Child Custody

My child’s father is threatening to take me to court for custody of my 14 year old son.
We both are responsible parents for him as I’m his mother and his fathers name is on the birth certificate
I currently live in a 2 bed council House with my partner and there are 4 children. 2 of the children are his, he also has an 11 year old girl but he only wants my son not my daughter. He says my son needs his own room and space. We have solved this problem because my partner has just purchased a 3 bedroom property which we move into in 3 weeks time so my son will have his bedroom but the father is still continuing with threats of court.
He pays child maintenance for 2-3 times a week and this is when he has my son overnight. The original arrangement was only 2 overnights a week which is something he wanted and we agreed to up until recently when he started paying child maintenance through the CMO.
He wanted mediation which I agreed to and we both attended the first session and I wanted another sessions to try and resolve the issue but mediation broke down due to him not paying his fee and so they cancelled.
My son has expressed he would like to stay with his dad full time until move. I’ve not agreed to this, due to his dad working away each month for days at a time with little notice. Therefore there would be no structure and routine.
The children are very well looked after and I have given everything to raise my children for majority on my own and now all.of a sudden he wants custody of my son and I don’t know what ground he has because we now have a new build 3 bedroom house where he has his own room. Please can you advice, can he do this?