When it cranks, does it crank slowly?
My guess is that something is draining the battery when you leave the car, and making it hard to start later on. But, when you leave it off for just a few seconds, it'll restart no problems because it hasn't had time to drain the battery.
Use a multimeter and check if there is significant current flow from the battery when the car is off.