I took mine off two years ago. Your fears are unfounded. No problem, better ride and it still handles the same. What's the point of IRS if you tie them together, lol. Ok, if your doing slolam courses I can see it, otherwise take it off, but don't throw it away. For resale you might want to put it back on. HOWEVER, don't remove the rear! I did that and it is noticeably a different cat. Still driveable but with a degree of caution. Finally, take off the front but not the rear.
What prompted me to take the front sway bar off was I remembered the first thing to go on off-road VW's is the front sway. Hold it! I know you're going to say the VW has trailing arm IRS and the VX has A-Arms but with stiff torsion bars and shocks up front as well as in back the front sway bar is just an unwelcome passenger IMO.