The ability to switch between egocentric (subject-to-object) and allocentric (object-to-object) frames of reference is a pivotal requirement in many everyday life spatial activities. Behavioural and neurofunctional evidence has demonstrated that the structure of an environment can influence our ability to represent spatial information. Given the complexity of our spatial environment, this raises the question of how enduring (aligned/misaligned) environmental axes can influence a demanding but necessary process in everyday spatial tasks, such as switching from one frame of reference to another. To the best of our knowledge, no study so far has investigated this issue. In the present study, an egocentric-allocentric visuo-spatial memory task was administered using Immersive Virtual Reality technology, while the orientation of the environmental axes was either parallel (aligned condition) or rotated by 45° (misaligned condition) with respect to the egocentric perspective. Participants were required to memorize triads of objects and then to provide switching (from egocentric-to-allocentric or allocentric-to-egocentric) and non-switching (only-egocentric or only-allocentric) spatial judgments of relative distances in the two virtual environments. Results showed that overall spatial judgments were worse in the misaligned than aligned condition, and in the allocentric-based than egocentric-based judgments. Interestingly, a specific difficulty due to the switching from an allocentric to an egocentric condition in the misaligned environment emerged. Moreover, the allocentric based (Allo-Ego, Allo-Allo) spatial judgments were more penalized in the misaligned than aligned condition. These results suggest that the presence of an environmental frame misaligned with the egocentric learning perspective has a negative impact on allocentric representations, especially in the switching condition when the first anchor point is allocentric. Therefore, the processes of switching between unstable local spatial array and stable body position are significantly influenced by the representation of an enduring allocentric environmental frame.