We have previously shown that when a stimulus consisting of two concentric rings moves, saccade latencies are much longer (by 150 ms) when attention is directed to the larger ring than to the smaller ring. Here, we investigated whether this effect can be explained by a deferral of the “cost” of making a saccade while the target remains inside the attentional field, or by purely visual factors (eccentricity or contrast). We found 1) latencies were shorter when attention was directed to small features irrespective of retinal eccentricity; 2) saccade latency distributions were systematically determined by the ratio between the amplitude of the stimulus step and the diameter of the attended ring: stimulus steps that were larger than the attended ring resulted in short latencies, whereas steps smaller than the attended ring resulted in proportionally longer and more variable latencies; 3) this effect was not seen in manual reaction times to the same target movement; and 4) suprathreshold changes in the contrast of targets, mimicking possible attentional effects on perceived contrast and saliency, had little effect on latency. We argue that the spatial scale of attention determines the urgency of saccade motor preparation processes by changing the rate and rate variability of the underlying decision signal, to defer the cost of saccades that result in little visual benefit.