I have installed a switch that runs inline with the 4WD fuse. I use it all the time and have seen no problem for the last year of using it.
Of course, the ignition has to be off before the use of the switch will take effect. (ie: no alternating between 4WD and RWD on 'the fly')
If you don't wish to install a switch, simply remove the 4WD fuse from in the engine bay. And no, ABS is not affected by this.