Materials properties are determined by their compositions and structures. In ABO3 oxides different cation orderings lead to mainly perovskite- or corundum like derivatives with exciting physical properties. Sometimes, a material can be stabilized in more than one structural modification, providing a unique opportunity to explore structure-properties relationship. Here, CoVO3 obtained in both ilmenite-(CoVO3-I) and LiNbO3-type (CoVO3-II) polymorphs at moderate (8–12 GPa) and high pressures (22 GPa), respectively are presented. Their distinctive cation distributions affect drastically the magnetic properties as CoVO3-II shows a cluster-glass behavior while CoVO3-I hosts a honeycomb zigzag magnetic structure in the cobalt network. First principles calculations show that the influence of vanadium is crucial for CoVO3-I, although it is previously considered as non-magnetic in a dimerized spin-singlet state. Contrarily, CoVO3-II shows two independent interpenetrating antiferromagnetic Co- and ferromagnetic V-hcp sublattices, which intrinsically frustrate any possible magnetic order. CoVO3-II is also remarkable as the first oxide crystallizing with the LiNbO3-type structure where both metals contain free d electrons. CoVO3 polymorphs pinpoint therefore as well to a much broader phase field of high-pressure A-site Cobaltites.