Its not really going to damage anything. However, if you think it sounds quite nasty, get it changed. If you have changed your clutch recently then don't worry about the clutch, just get a bearing. If you've had the car for a while and the clutch is of unknow origin and age, change the lot.
From what I have seen, with general road use, the bearings normally either outlast, or die around the same time, as a clutch. And they are usually changed with the clutch, so if the bearing is dead, clutch is usually getting close too. Just for your thoughts.