Its a couple of hours drive. Have some cash handy as the tolls are killer.
Fuji Speedway is near the mountain, sometimes there are private drift and club days on.
Daikoku is hit and miss. Friday night and weekends are busier but usually closed by police if it gets packed.
If you go to Daikoku DONT miss the right hand lane that goes to the parking area!