When exposed to a spoken message, a listener takes into account several sources of linguistic and indexical information. Using the mismatch negativity (MMN) response, we examined whether the indexical information about the sex of the speaker influenced the processing of semantically gendered spoken words. Female participants listened two semantically gendered French words, one masculine and one feminine representing human beings, said either by five male or by five female speakers. The opposite sex voices produced an enhancement of MMN response. In line with interactive connections between indexical and linguistic information processing through activating lexical memory traces, the results showed more pronounced MMN response when the sex of the speaker matched with the gender of the word. Furthermore, there was a later detection of the incongruence between the sex information about the speaker and the gender of the word, shown by an enhancement of MMN response. Overall, these findings suggest that the listeners integrate the indexical information about the sex of the speakers both at the lexical selection level and at a higher-level processing such as the grammatical access.