A - Technically not assault. Permission was granted by both him and his mother.
B - Call and report it to the coppers. They won't do anything until it really escalates however it will be on record. so, if he does try and prevent you from doing anything that is relation to the child (change schools, act as a responsible guardian etc) his irresponsibility is on file.
This is all based on my legal knowledge from a few years ago. Contact a lawyer for more thorough info or next time he calls, tell him to stop being a c6ck and suck one instead.
Know what you mean though, the girlfriend has moved in with me and she gets strife from her ex regarding their son. She mvoed up here from Melbounre though.