Nope...as long as it's in the same vertical plain as the original orifice, you just move whichever accessory you're using back the same distance & install the pin...which now clears...
But isn't that moving the weight away from the rear cross-member by another inch or more? Wouldn't that increase the lean effect / decrease the maximum weight capacity of a hitch-mounted cargo box?