As far as I can tell the swivel on allows you access to remove and mount the base and the mirror.
Presumably if you broke the top part of the mirror you wouldn't need to remove the base from the fender, just the broken top portion. You would do this by swiveling the mirror to reveal the mounting pieces.
This is also where the screws go through the base and you bolt the metal mount under the fender. You can easily do this differently to make mounting the one piece mirror to the fender simple and easy. If you're not trying to remove or install the mirror there is really no need for the swivel. It's not something you use when trying to use the mirror anyway.
Really wouldn't be a big deal for me either way.