Family visa application

Hi, I am Swedish citizen and married to a Chinese national since 3 years and I have been staying there on one-year L which has been renewed in Sweden every year. We live in a very small town in Hubei. We had a son born in China 18 months ago and when asking for a exit permit we had a really big fight with the local PSB to get it and it ended up with Swedish Embassy contact the Foreign Affairs to step in and correct the PSB before it was issued. My son holds a Swedish passport now as well and when we tried to get a family L with unlimited stay for him and me the local PSB of course refused to assist and help. Is there any other way to get it or is the only way to got through PSB where Hokou is registered of Chinese inviter? We will not get this as long as it needs to be done there. Please advise!