We study a family of optimal control problems in which one aims at minimizing a cost that mixes a quadratic control penalization and the variance of the system, both for finite sets of agents and for the limit problem as their number goes to infinity. While the solution for finitely many agents always exists in a unique and explicit form, the behavior of the corresponding limit problem is very sensitive to the magnitude of the time horizon and penalization parameter. For the variance minimization, there exist Lipschitz-in-space optimal controls for the infinite dimensional problem, which can be obtained as a suitable limit of the optimal controls for the finite-dimensional problems. The same holds true for the variance maximization with a sufficiently small final time. Instead, for large final times (or equivalently large penalizations of the variance), Lipschitz-regular optimal controls do not exist for the macroscopic problem.