I notice a huge difference in the loose steering feeling between my summer and winter tires.

Summer: 285/55-18 Pirelli Scorpion Zero
Winter: 245/75-16 Firestone Destination M/T

Different rims and all but the tread patch is over 10" across on the summer tires, versus 7" with the winter. Plus my summer tires are well worn.

On well traveled paved roads, especially at traffic lights where the pavement has deep grooves, my summer rubber really throws me around. The winter tires don't follow the groves at all and make the steering feel much tighter.

I did notice an improvement when I had the front berings repacked, but the difference in tire width is like night and day.

Since I have the same tires I would say (given the groves of the road your on) that the majority of what you are feeling can be fixed with narrower, less worn tires. YMMV