Numerous studies reported a significant decline in physical activity level in adolescents as a result of the COVID-19 lockdown. Physical fitness is recognized as a powerful marker of health in youth. The aim of this study was to evaluate the impact of the COVID-19 pandemic lockdown on health-related physical fitness in French adolescents. Two cross-sectional studies were performed comparing two different groups of French adolescents, before (sample 1) and after the first lockdown (sample 2). A total of 1231 adolescents (aged to 16.5 ± 1.5 years) participated in the two cross-sectional studies. Complete data for physical fitness and anthropometrics data were obtained. Adolescents from sample 2 showed lower physical fitness levels compared to adolescents from sample 1. Regarding physical fitness for boys and girls, physical fitness levels were significantly lower in both sex between adolescents from the sample 1 and adolescents from the sample 2, except for cardiorespiratory fitness and flexibility for boys and girls, respectively. The physical fitness global score was also significantly lower between adolescents from the sample 1 and 2 for boys (−9.8%, p < 0.01) and girls (−16.2%; p < 0.01), respectively. Overall, the higher difference was found for performance in the speed body displacement test (−30%). A difference of 12.8% and 25% was observed for boys and girls, respectively.