Moderate physical activity can be experienced by some as pleasurable and by others as discouraging. This may be why many people lack sufficient motivation to participate in the recommended 150 minutes of moderately intense exercise per week. In the present study, we assessed how pleasure and enjoyment were modulated differently by one’s tolerance to self-paced physical activity. Sixty-three healthy individuals were allocated to three independent experimental conditions: a resting condition (watching TV), a cycling in silence condition, and a cycling in music condition. The tolerance threshold was assessed using the PRETIE-Questionnaire. Physical activity consisted in cycling during 30 minutes, at an intensity perceived as “somewhat difficult” on the Ratings of Perceived Exertion Scale. While controlling for self-reported physical activity level, results revealed that for the same perception of exertion and a similar level of enjoyment, the High Tolerance group produced more power output than the Low Tolerance group. There was a positive effect of music for High Tolerant individuals only, with music inducing greater power output and more pleasure. There was an effect of music on heart rate frequency in the Low Tolerant individuals without benefits in power output or pleasure. Our results suggest that for Low Tolerant individuals, energizing environments can interfere with the promised (positive) distracting effects of music. Hence, tolerance to physical effort must be taken into account to conceive training sessions that seek to use distracting methods as means to sustain pleasurable exercising over time.