I actually had to go outside and check to see if I had a grab bar on the driver's side at all...
I say remove it -I didn't even know it was there!

You could replace it with a small strip of foam padding.