When the size of visual objects matches the size of the response required to perform the task, a potentiation effect has been reported, with faster responses in compatible than incompatible situations. Size compatibility effects have been taken as evidence of close perception-action interrelations. However, it is still unclear whether the effect arises from abstract coding of the size of stimulus and response or from the evocation of grasp affordances from visual objects. We aimed to disentangle the two interpretations. Two groups of 40 young adults categorised small and large objects presented in standardised size as natural or artefacts objects. One group categorised manipulable objects that may be associated with small or large size properties and evoke power and precision grasp affordances. The other groups categorised non-manipulable objects that may only be associated with small or large size properties. Categorisation responses were made by reaching and grasping a monotonic cylindric device with a power and precision grip in a grasping condition and with large or small touch responses in a control condition. Compatibility effects were found in both grasping and control conditions, independently of the manipulability or category of objects. Participants were faster when the size of the expected response matched the size of the object than when they mismatched, especially for power grasp or whole-hand touch responses. Overall findings support the abstract coding hypothesis and suggest that compatibility between the conceptual size of the object and the size of the hand response is sufficient to facilitate semantic categorization judgements.