Study Objectives: A critical role linking sleep with memory decay and β-amyloid (Aβ) accumulation, two markers of Alzheimer's disease (AD) pathology, may be played by hippocampal integrity. We tested the hypotheses that worse self-reported sleep relates to decline in memory and intra-hippocampal microstructure, including in the presence of Aβ. Methods: Two-hundred and forty-three cognitively healthy participants, aged 19-81 years, completed the Pittsburgh Sleep Quality Index once, and two diffusion tensor imaging sessions, on average 3 years apart, allowing measures of decline in intra-hippocampal microstructure as indexed by increased mean diffusivity. We measured memory decay at each imaging session using verbal delayed recall. One session of positron emission tomography, in 108 participants above 44 years of age, yielded 23 Aβ positive. Genotyping enabled control for APOE ϵ4 status, and polygenic scores for sleep and AD, respectively. Results: Worse global sleep quality and sleep efficiency related to more rapid reduction of hippocampal microstructure over time. Focusing on efficiency (the percentage of time in bed at night spent asleep), the relation was stronger in presence of Aβ accumulation, and hippocampal integrity decline mediated the relation with memory decay. The results were not explained by genetic risk for sleep efficiency or AD. Conclusions: Worse sleep efficiency related to decline in hippocampal microstructure, especially in the presence of Aβ accumulation, and Aβ might link poor sleep and memory decay. As genetic risk did not account for the associations, poor sleep efficiency might constitute a risk marker for AD, although the driving causal mechanisms remain unknown.