Brain aging may be programmed by early-life stress. Aging affects males and females differently, but how perinatal stress (PRS) affects brain aging between sexes is unknown. We showed behavioral and neurobiological sex differences in non-stressed control rats that were strongly reduced or inverted in PRS rats. In particular, PRS decreased risk-taking behavior, spatial memory, exploratory behavior, and fine motor behavior in male aged rats. In contrast, female aged PRS rats displayed only increased risk-taking behavior and reduced exploratory behavior. PRS induced large reductions in the expression of glutamate receptors in the ventral and dorsal hippocampus and prefrontal cortex only in male rats. PRS also reduced the expression of synaptic vesicle-associated proteins, glucocorticoid receptors (GR), and mineralocorticoid receptors (MR) in the ventral hippocampus of aged male rats. In contrast, in female aged rats, PRS enhanced the expression of MRs and brain-derived neurotrophic factor (BDNF) in the ventral hippocampus and the expression of glial fibrillary acidic protein (GFAP) and BDNF in the prefrontal cortex. A common PRS effect in both sexes was a reduction in exploratory behavior and metabotropic glutamate (mGlu2/3) receptors in the ventral hippocampus and prefrontal cortex. A multidimensional analysis revealed that PRS induced a demasculinization profile in glutamate-related proteins in the ventral and dorsal hippocampus and prefrontal cortex, as well as a demasculinization profile of stress markers only in the dorsal hippocampus. In contrast, defeminization was observed only in the ventral hippocampus. Measurements of testosterone and 17-β-estradiol in the plasma and aromatase in the dorsal hippocampus were consistent with a demasculinizing action of PRS. These findings confirm that the brains of males and females differentially respond to PRS and aging suggesting that females might be more protected against early stress and age-related inflammation and neurodegeneration. Taken together, these results may contribute to understanding how early environmental factors shape vulnerability to brain aging in both sexes and may lay the groundwork for future studies aimed at identifying new treatment strategies to improve the quality of life of older individuals, which is of particular interest given that there is a high growth of aging in populations around the world.