I think it is more people who just stay in the over taking lane for no reason at all and holds up people who go a little faster than the speed limit (lets be realistic here, I'm sure everyone travels a little faster than the speed limit from time to time).
But it is quite annoying when you want to over take but the people in the right lane just don't budge at all. They drive side by side the car in the left lane (and I am sure some people do this just to piss off other drivers).
Nothing wrong with trying to over take doing the speed limit, but if you are not over taking then just pop back into the left lane. Just let people who speed go by you and pray there is a HWP up ahead and giggle when it happens.
Edit: Just to add: it is also illegal to "disrupt" the flow of traffic.