Amyloid related imaging abnormalities (ARIA) are the most significant risk associated with the use of anti-amyloid monoclonal antibodies (MAB) for Alzheimer’s disease (AD). Currently, the presence of the APOE ε4 allele is the best predictor for the development of ARIA. However, the degree of baseline memory impairment has not been fully explored as a risk factor for ARIA. Here, we examined MAB outcomes in a memory clinic population and compared patients with AD who developed ARIA to a case-matched group who did not develop ARIA. Participants who developed ARIA had greater numbers of recall intrusions and false positives, both markers for memory consolidation, at baseline than those who did not develop ARIA. We also observed greater baseline hippocampal and supplementary motor cortical atrophy with ARIA. These differences remained when controlling for the APOE ε4 allele and the presence of pretreatment microhemorrhages. Further investigation of memory impairment and associated brain atrophy is warranted to understand ARIA risk and MAB outcomes in AD.