Older adults with epilepsy frequently report changes in memory function, with a concern about degenerative disease. However, the precise limit between memory changes related to the effects of longstanding chronic epilepsy and physiological aging on memory abilities is very difficult to specify. Knowledge about characteristics of physiological cognitive aging, its influence on memory test results, and its interactions with clinical variables related to epileptic disease are therefore essential to interpret memory scores. This paper aims at presenting an overview of the different factors that might modulate the developmental trajectory of episodic memory in chronic temporal lobe epilepsy. For this purpose, the effects of age on memory in healthy and epileptic persons will be first reviewed with a special focus on episodic memory functions in chronic temporal lobe epilepsy. Finally, scientific evidence supporting or not the presence of an accelerated memory decline in patients with longstanding epilepsy will be presented by discussing the interests and the limits of the existing literature. Identifying the best outcome measures to assess episodic memory and understanding the risk factors that might precipitate memory decline are key knowledge to improve the clinical care and quality of life of persons with chronic temporal lobe epilepsy.