I think I may have found the problem. Today as I touched up my yellow front door speaker covers, I noticed a mistake. I was out of Silicon Spray so I used White Lithium Grease instead to lubricate the seal/track for the windows....hmmmm.... it was crusty and all gunked up. I cleaned them very well with silicon spray and a towel and lubricated (this time with only silicon spray) and so far the problem has vanished.

Of course, I did this a couple of days ago and it hasn't been as cold, so the jury's still out but once I cleaned everything even in the warm weather it was a very noticeable difference of how fast the windows go up and down.