I didn't remove the rubber piece. My pads were still fairly new so the piston wasnt all that far out (I know that doesnt mean its not dirty) the rotors were garbage so I replaced it all at the same time. The pads and rotors were pretty think so maybe they are just dragging. I did notice they were a lot more on the warm side in the first few days and they dont seem as hot now, but is has also been pretty cold here so I didnt know if that had anything to do with it.
I am gonna be driving about 1000 miles soon and dont want any grenading happening.