In the context of fashion/textile innovations towards Industry 4.0, a variety of digital technologies, such as 3D garment CAD, have been proposed to automate, optimize design and manufacturing processes in the organizations of involved enterprises and supply chains as well as services such as marketing and sales. However, the current digital solutions rarely deal with key elements used in the fashion industry, including professional knowledge, as well as fashion and functional requirements of the customer and their relations with product technical parameters. Especially, product design plays an essential role in the whole fashion supply chain and should be paid more attention to in the process of digitalization and intelligentization of fashion companies. In this context, we originally developed an interactive fashion and garment design system by systematically integrating a number of data-driven services of garment design recommendation, 3D virtual garment fitting visualization, design knowledge base, and design parameters adjustment. This system enables close interactions between the designer, consumer, and manufacturer around the virtual product corresponding to each design solution. In this way, the complexity of the product design process can drastically be reduced by directly integrating the consumer’s perception and professional designer’s knowledge into the garment computer-aided design (CAD) environment. Furthermore, for a specific consumer profile, the related computations (design solution recommendation and design parameters adjustment) are performed by using a number of intelligent algorithms (BIRCH, adaptive Random Forest algorithms, and association mining) and matching with a formalized design knowledge base. The proposed interactive design system has been implemented and then exposed through the REST API, for designing garments meeting the consumer’s personalized fashion requirements by repeatedly running the cycle of design recommendation—virtual garment fitting—online evaluation of designer and consumer—design parameters adjustment—design knowledge base creation, and updating. The effectiveness of the proposed system has been validated through a business case of personalized men’s shirt design.