My mate has an E46 M3, and believe me it is a crazy machine. I would not say it is a lesser car than a GTR, simply a different car. What you say about it being luxo may be true, but underneath it there is still a very hard sports car. Consider that both the E36 and E46 M3s were PCOTY at Motor.
In terms of driving, the M3 is exactly as people describe - instant throttle response, feelsome and talkative steering etc etc, and boy do they drift well in standard form! There may be nothing like the sound of a turbo spooling, wastegate opening and bellow of an RB26, but once you hear a 3.2 BMW M52 engine at 7000RPM the hairs on your back will stand up. Remember it is one of the most celebrated straight six engines in the world.
The E36 is a bit of the unloved child of the M3 family; it didn't have the fat guards of the E30 or E46, and was a bit of a sleeper in comparison. But the underlying mechanicals are basically the same as the E46 less the tricky diff and SMG update. OK, the motor is supposedly all new, but the basic architecture and design remain. I like the more Q-car look of the E36. And yes i would give my right leg to have either a GTR or M3 in my driveway.
Hope that dribble helps.