Aftermarket speakers distort earlier than stock speakers do?

I had a completely stock non hk sound setup, then wanted to upgrade due to broken speakers. I replaced the front midbasses with Alpine SPS-510's and rear midbasses with Alpine SPS 410's, as well as replacing the stock headunit with a Pioneer MVH-X560BT. At lower volumes the system sounds great, but lacking a little on the low end. But if I put the top down or even roll a window down, the speakers distort before I can turn them up enough to hear them. My aftermarket speakers distort at an insanely low volume, and it's very frustrating. The speakers are 4 ohm and the factory amp is 2 ohm (I believe), so would a new 4 ohm amp help? Or could it be the polarity somewhere (but I doubt that). Or are all 4 ohm speakers just that quiet?