MgV2O4 is a spinel based on magnetic V3+ ions, which host both spin (S=1) and orbital (leff=1) moments. Owing to the underlying pyrochlore coordination of the magnetic sites, the spins in MgV2O4 only antiferromagnetically order once the frustrating interactions imposed by the Fd¯3m lattice are broken through an orbitally-driven structural distortion at TS≃60 K. Consequently, a Néel transition occurs at TN≃40 K. Low-temperature spatial ordering of the electronic orbitals is fundamental to both the structural and magnetic properties; however, considerable discussion on whether it can be described by complex or real orbital ordering is ambiguous. We apply neutron spectroscopy to resolve the nature of the orbital ground state and characterize hysteretic spin-orbital correlations using x-ray and neutron diffraction. Neutron spectroscopy finds multiple excitation bands and we parametrize these in terms of a multilevel (or excitonic) theory based on the orbitally degenerate ground state. Meaningful for the orbital ground state, we report an “optical-like” mode at high energies that we attribute to a crystal-field-like excitation from the spin-orbital jeff=2 ground-state manifold to an excited jeff=1 energy level. We parametrize the magnetic excitations in terms of a Hamiltonian with spin-orbit coupling and local crystalline electric field distortions resulting from deviations from perfect octahedra surrounding the V3+ ions. We suggest that this provides compelling evidence for complex orbital order in MgV2O4. We then apply the consequences of this model to understand hysteretic effects in the magnetic diffuse scattering where we propose that MgV2O4 displays a high-temperature orbital memory of the low-temperature spin order.