Not only is it YOUR hobby mate, it's YOUR choice, and YOUR family... Not your Father-in-Law's... I wouldn't give a moments thought to his "advice"...
You're a grown adult raising a family and it's you're perogative to do that anyway you please within the boundarys of the law...
I feel your pain though... I've got 3 young boys, a father-in-law who couldn't care less about cars / bikes / engines / motorsport, and a severe addiction to motorbikes, fast cars and track days... I'm yet to hear anything about it but I know it's frowned upon...
Keep on doing what you're doing... They will learn to accept it...