No harm. All you'll do is cause the engine to run at a higher RPM (Yes, higher the RPM the more wear over the very long run). The suffering will come from your wallet, higher RPM = more fuel consumed.