Yep. It is wrong and it is weird.
Go to the ex first and try to diplomatically reason (and then maybe work in a little aren't you humiliated?) with him/her. He/she needs to understand (and you can calmly lay out why) how it is inappropriate and how it interferes with your life with your blood family. You do not have any problem with visiting - because they are have a relationship and you are ok with that, but living with them. Please move on.
But it is not a matter of ';letting'; the person do it. You do not actually have any say.
That said - if he ignores your plea, then go to your parents and tell them that it puts you, your kids and husband in an awkward situation and makes it hard to visit. You can ask respectfully that they talk to the ex and say ';hey, we like you and want to still be a part of your life, but we think it would be best for everyone if you find a place of your own';.
Him first, then the parents. It is the adult way.Would u let your ex live with ur family even if?
If he/she is still friends with your family and you no longer live there, it's none of your business. That's your families problem then. If they don't want him/her there, they are fully capable of kicking him/her out. If they do, that's their business. Frankly, it's something you should stay out of. Just because you don't like him/her anymore doesn't mean they can't.