When retrieving Aerosol Optical Depth (AOD) from passive satellite sensors, the vertical distribution of aerosols usually needs to be assumed, potentially causing uncertainties in the retrievals. In this study, we use the Moderate Resolution Spectroradiometer (MODIS) and Visible Infrared Imaging Radiometer Suite (VIIRS) sensors as examples to investigate the impact of aerosol vertical distribution on AOD retrievals. A series of sensitivity experiments was conducted using radiative transfer models with different aerosol profiles and surface conditions. Assuming a 0.2 AOD, we found that the AOD retrieval error is the most sensitive to the vertical distribution of absorbing aerosols; a −1 km error in aerosol scale height can lead to a ~30% AOD retrieval error. Moreover, for this aerosol type, ignoring the existence of the boundary layer can further result in a ~10% AOD retrieval error. The differences in the vertical distribution of scattering and absorbing aerosols within the same column may also cause −15% (scattering aerosols above absorbing aerosols) to 15% (scattering aerosols below absorbing aerosols) errors. Surface reflectance also plays an important role in affecting the AOD retrieval error, with higher errors over brighter surfaces in general. The physical mechanism associated with the AOD retrieval errors is also discussed. Finally, by replacing the default exponential profile with the observed aerosol vertical profile by a micro-pulse lidar at the Beijing-PKU site in the VIIRS retrieval algorithm, the retrieved AOD shows a much better agreement with surface observations, with the correlation coefficient increased from 0.63 to 0.83 and bias decreased from 0.15 to 0.03. Our study highlights the importance of aerosol vertical profile assumption in satellite AOD retrievals, and indicates that considering more realistic profiles can help reduce the uncertainties.