As the successful implantation of CAD (computer-aided-design) technology in garment industry, the 3D virtual garment technology has attracted a great attention of textile/garment companies. However, wearer’s perception on virtual garment in terms of fitting and comfort has never been systematically studied. In this paper, we propose an original customized 3D garment collaborative design process by integrating interactions between the designer and the specific consumer. In this context, a normalized sensory evaluation on garment fitting effects will be organized in a virtual environment, in order to enhance communications of the concerned actors on perception of products. Also, by learning from the measured distances between the garment surfaces and the mannequin (input) and sensory descriptors (output), we model the relationship between the garment design parameters and the human perception on fit of the finished virtual products. Using this model, we can estimate the fit perception for a specific garment size on a specific body shape without any real try-on experience. In practice, the proposed collaborative design process will permit to develop an online recommendation system for garment size selection and fit estimation. Furthermore, it will permit to recursively modify the initial garment patterns according to the consumer’s fit preferences so that they can obtain a real personalized garment. In this way, the consumer will be directly involved in the product design process by performing a series of sensory evaluations on virtual garment fit in order to obtain a desired finished product. This process has been validated by creating a collection of T-shirts meeting requirements of various customers.