Mine 'clicked' during decel. I didn't notice any vibration at highway speed. If it was there, it was probably hidden by the road noise of my Wrangler MTs.

If you've been driving with a torn boot, the CV is the most likely culpret. Stick your finger inside the boot & see how gritty the grease is.