Much research on color representation and categorization has assumed that relations among color terms can be proxies for relations among color percepts. We test this assumption by comparing the mapping of color words with color appearances among different observer groups performing cognitive tasks: (1) an invariance of naming task; and (2) triad similarity judgments of color term and color appearance stimuli within and across color categories. Observer subgroups were defined by perceptual phenotype and photopigment opsin genotype analyses. Results suggest that individuals rely on at least two different representational models of color experience: one lexical, conforming to the culture's normative linguistic representation, and another individual perceptual representation organizing each observer's color sensation experiences. Additional observer subgroup analyses suggest that perceptual phenotype variation within a language group may play a greater role in the shared color naming system than previously thought. A reexamination of color naming data in view of these findings may reveal influences on color naming important to current theories.