I also think you might be able to lightly "pick" some of the specs off with a fingernail w/o damaging the cladding. Or even use a scotch-brite kind of pad to lightly chip it off w/o marring the cladding surface.

Also, would one of those clay bars pull some of it off? I haven't used my clay bar yet, so I don't have any idea how well it work on this. Kind of like when we were kids we used Silly Putty to pick up stuff.