I agree and I disagree. I agree because like you I don't normally drive my car daily for a number of reasons (security, cost, its general habit of breaking, and the hassle of every dick annoying you) but when I do it give me a real kick and I don't mind paying for that. But that is a convenience I enjoy because I have a reasonable well off family with excess cars, inner city housing, and enough disposable income to waste it on toys like GT-Rs.
On the other hand, its the families who live out woop woop with poor access to public transport (as opposed to access to the poor public transport like us inner city folks ) why rely on their cars and don't have much disposable income at the best of times.
Regardless, I don't agree with the whole "cut the tax" argument that seems so popular for various reasons I could bang on about if so provoked