Several studies have suggested that women are more sensitive than men to emotions in general. Event-related potential (ERP) studies have demonstrated N2 and P3b modulations, suggesting that women allocate more attentional resources to emotions than men do. However, the exact origin of this emotional modulation by sex is still a matter of debate. We wondered whether these sex differences might be due to some specific personality traits of women and men. Thirty participants (15 males and 15 females) were selected so that there were no sex differences on alexithymia, or depression and anxiety scales. The participants were asked to complete a “modified emotional” oddball task, in which they had to detect deviant stimuli among frequent neutral ones as quickly as possible. Behavioral performance, N2 and P3b ERP data were analyzed. When personality factors were controlled for, the sex differences on N2 and P3b components of the ERPs disappeared. Moreover, linear regression analyses showed that alexithymia was much better than sex at predicting the N2 latencies, while depression was the best factor for predicting the P3b latency. These results suggest that personality factors should be taken into account when sex differences on emotional processing are investigated.