Fabrication of fine grain structures in Al alloys at elevated temperature is of great difficulty, owing to the dramatic recrystallization and grain growth. In this study, particles with both (sub)micron and nano-scaled sizes are introduced in Al matrices (composites) to deeply refine and stabilize the grain structures at high temperature. Microstructural evolutions of the composites during deformation and annealing are characterized in details compared to the alloy counterparts without particles. Consequently, more rapid and obvious grain refinement are induced in the composites. Mechanisms of these grain refinement are discussed in terms of the influences of dual-size particles. Initial particle clusters can enforce grain fragmentations during unidirectional extrusion, and the subsequently dispersed micron particles accelerate the grain subdivision during orthogonal extrusion. Meanwhile, massive (sub)micron particles and dense nanoparticles promote dynamic recrystallizations in composites during deformation, thanks to the particle stimulated nucleation and lattice rotation. According to the driving and dragging forces calculation and textural results, grain growth is the major mechanism to determine the annealed structures of composites and alloys. As a result, uniformly fine grain structures are achieved in composites, approaching a stable critical size decided by the Zener pinning of nanoparticles.