if the a/c works when you fiddle around with it its a good thing that means you have adequate refridgerant and dont have a leak and the compressor works! and is more than likely a bad electrical connection somewhere.
I havent looked at the vx a/c system but many vehicles also use vacuum to switch the blend doors and temp control in the HVAC plenum so could be a vacuum line leaking. I'd start by getting to the back of the heater/ac controls and work back through there.

If the a/c is poor even when it does work, try and borrow a set of R134 manifold gauges hook them up, (the high and low pressure sides have different size fittings so you cant screw em up) then if the system has been at rest or not used for some time, both guages should read approximately the same and with R134a will read in psi approximately what the ambient air temp is.

If its 70 deg out and you read 20psi on both guages you have lost refridgerant, start looking for signs of oil around the piping, connections etc.

Does the compressor kick in? again with low refridgerant the low press cut out will prevent the compressor from starting.

As for the a/c on defrost ....here's the techpoop... your windows fog up because of high humidity in the air and generally cooler temperatures, outside. The dewpoint temp is the temp at which condensation will be drawn out of the air and will settle on a cool surface, ie when the air cannot hold any more moisture. If you bring a cold beer out of the fridge on a humid day, it will instantly be covered in condensation because its surface temp is below the dew point temp of the air.

Soooo, the air passing over the evaporator coil which is very cold and is always going to be below the dew point temp will condense the moisture from the air inside the vehicle onto the coil and therefore clear the fog from the windows!!

Brilliant innit?