I've had this happen before, but I always attributed it to the varying road conditions and the wide tires on the VX (Nitto Terra Grapplers). If the road crowns to the right then the VX usually pulls right, or if there are some wallows in the road from heavy truck traffic the VX will get a little squirrely as the wide tires bite on different parts of the wallows. I've had other vehicles with wide tires and they tend to squirrel all over the road. I know it sounds goofy, but does the VX pull differently on the same patch of road?

It could be a caliper hanging up, but I would think that there is enough pressure in the brake lines to overcome that unless it's something drastic (like a siezed caliper).