Jon, you might be correct. I remember removing the lower ball joint from the passenger side when I did my half shaft swap, but I don't remember removing it from the driver side. I could swear that there was enough room with the lower ball joint still attached to get the inner CV joint taken apart. They say that the memory is the first to go so maybe I did remove the lower ball joint on the driver side. 
Well, if you can't get the inner CV joint disassembled with the lower ball joint attached we'll have our answer and you'll have to remove the lower ball joint to get everything to come out. Sorry for any confusion.
"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong, it usually turns out to be impossible to get at or repair."
-Douglas Adams, Mostly Harmless