He's just jealous, and is most probably trying to make himself feel better.
Do you really care what someone who obviously craves the attention of others thinks?
A true car enthusiast doesn't want to get looks from others, they drive the car because they love it.
You made the right choice, and it's not like your car is common. In fact, your car will almost certainly become a classic, given that it's a GTR V-SPEC and Series 3.
If I were you though, i'd keep the original wheels and try to keep as much of the original parts as possible because in the future it will help it to retain it's value.