Introduction Very few studies to date have investigated cancer mortality in non-ferrous metal smelters. Existing studies mainly focus on lead exposure and have reported inconsistent results. The aim of this study was to investigate the risk of excess cancer mortality in the employees of a lead and zinc smelter located in the north of France by 1) comparing mortality in this cohort of employees with that of the regional population; 2) studying the associations between occupational exposure and cancer deaths. Methods The study cohort was composed of male workers, born in France, who had been employed by the company in question for at least 365 consecutive days. A company-specific job-exposure matrix was produced in order to calculate a cumulative exposure index for 15 toxic agents. Deaths of smelter employees which occurred between January 1, 1968 and December 31, 2015 were compared to those in the regional population (standardized mortality ratio, SMR). The relationships between the cumulative exposure indexes and mortality by cancer site were studied using Cox regression models with age and the 20-year lagged cumulative exposure index as time-dependent variables. Results Vital status was found for 2177 of the employees in the cohort (98%). Median follow-up was 34.8 years (interquartile interval = 24.3–44.8), totaling 74,437 person-years. Compared to the regional population, no excess risk of all-cause mortality (n = 913, SMR = 0.96, 95%CI:0.90–1.02), nor of cancer mortality (n = 338, SMR = 0.97, 95%CI:0.87–1.08) was found. An overall significant excess risk of cancer mortality was found for employees who worked in this non-ferrous metal smelter for a period of between 15 and 29 years (n = 139, SMR = 1.23, 95%CI:1.04–1.45). Asbestos exposure was found to be associated with an increased risk of mortality for all cancer sites (p = 0.0012), lip-oral cavity-pharynx malignant neoplasms (MN) (p = 0.0141) and trachea-bronchus-lung MN (p = 0.0018); lead exposure was associated with the same risk for lip-oral cavity-pharynx (p = 0.0378) and liver MN (p = 0.0155); aromatic amine exposure with bladder MN (p = 0.0002); chromium exposure with colon-rectum-anus MN (p = 0.0057) and colon MN (p = 0.0315); bismuth exposure with rectal MN (0.0011) and sodium hydroxide vapor exposure with laryngeal MN (0.0150). Conclusion Including occupational exposure to numerous toxic agents other than lead in this study of smelter mortality has made it possible to identify associations between different toxic agents and cancers, opening up new avenues for future research.