I agree with a lot of what you're saying. You lose MPG as you go up in altitude. An intercooler does make the air denser, so I guess it would be comparable to driving at a lower elevation.
But I do agree, for me, it's not about driving conservatively. The intercooler provides more oxygen, so that I can use more skinny pedal for the turbo or supercharger. I think in and of itself, the intercooler provides more oxygen, and if used conservatively, would aid in MPG. I know some people that own Turbos on the 92x site, but drive it below boost, so they can run regular gas, and they do report very good gas mileage. To me, it sounds a little ridiculous. Kind of like buying full skid plates, but not wanting to take it off-road.