This study aimed at investigating the effects of acoustic distance and of speaker variability on the pre-attentive and attentive perception of French vowels by French adult speakers. The electroencephalogram (EEG) was recorded while participants watched a silent movie (Passive condition) and discriminated deviant vowels (Active condition). The auditory sequence included 4 French vowels, /u/ (standard) and /o/, /y/ and /ø/ as deviants, produced by 3 different speakers. As the vowel /o/ is closer to /u/ than the other deviants in acoustic distance, we predicted smaller mismatch negativity (MMN) and smaller N1 component, as well as higher error rate and longer reaction times. Results were in line with these predictions. Moreover, the MMN was elicited by all deviant vowels independently of speaker variability. By contrast, the Vowel by Speaker interaction was significant in the Active listening condition thereby showing that subtle within-category differences are processed at the attentive level. These results suggest that while vowels are categorized pre-attentively according to phonemic representations and independently of speaker variability, participants are sensitive to between-speaker differences when they focus attention on vowel processing.