Turns out the rubber thing actually hooks up under the plastic so you're all kinda right. The easiest way to do it is to take a small screwdriver and push under the rubber and OUT. Then lift the rubber and slide the whole thing IN to expose the screw.