If that was the case Richard it would be making the noise all the time. I definitely believe it's the booster as I have replaced these a couple of times, especially on 32R's. It's a full replacement job as I'm not sure if they can be rebuilt, $200+ I think for the part. Not the easiest job, you have to unbolt the brake booster and push that forward as far as it will go, so that you can wiggle the clutch booster out. And yes it should be safe to drive, have you noticed a slightly higher idle than usual? Or a heavier clutch pedal?