Yep. Agree with the comments above........... Sounds like your spigot bearing is too small.
Some advice from an old fart... Don't try to grind the bastard out with a file or drill etc. Wait and get the right sized spigot bearing/bush. They are an important part of the drivetrain, and the time you lose sourcing the right bearing will pay for itself many times over with many kilomoteres of stress free driving in the future.
Good luck, Steve.