The aim of this study was to explore cross-sectional and longitudinal aging differences in immediate and delayed visual and verbal memory abilities in individuals with Autism Spectrum Disorder (ASD) compared with neurotypicals (NTs). We measured hippocampal size, fornix fractional anisotropy (FA), and hippocampal and fornix freewater to understand how aging impacts memory structures. Longitudinal findings highlight vulnerabilities in immediate verbal memory and hippocampal volume, while cross-sectional findings indicate fornix freewater may increase at a faster rate in adults with ASD. Future research will examine cognitive and structural sex differences and will study how cognitive measures correlate with structural measures.