Abstract Background and purpose The specific effects of antiseizure medications (ASMs) on cognition are a rich field of study, with many ongoing questions. The aim of this study was to evaluate these effects in a homogeneous group of patients with epilepsy to guide clinicians to choose the most appropriate medications. Methods We retrospectively identified 287 refractory patients with medial temporal lobe epilepsy associated with hippocampal sclerosis. Scores measuring general cognition (global, verbal and performance IQ), working memory, episodic memory, executive functions, and language abilities were correlated with ASM type, number, dosage and generation (old vs. new). We also assessed non‐modifiable factors affecting cognition, such as demographics and epilepsy‐related factors. Results Key parameters were total number of ASMs and specific medications, especially topiramate (TPM) and sodium valproate (VPA). Four cognitive profiles of the ASMs were identified: (i) drugs with an overall detrimental effect on cognition (TPM, VPA); (ii) drugs with negative effects on specific areas: verbal memory and language skills (carbamazepine), and language functions (zonisamide); (iii) drugs affecting a single function in a specific and limited area: visual denomination (oxcarbazepine, lacosamide); and (iv) drugs without documented cognitive side effects. Non‐modifiable factors such as age at testing, age at seizure onset, and history of febrile seizures also influenced cognition and were notably influenced by total number of ASMs. Conclusion We conclude that ASMs significantly impact cognition. Key parameters were total number of ASMs and specific medications, especially TPM and VPA. These results should lead to a reduction in the number of drugs received and the avoidance of medications with unfavorable cognitive profiles.