How to deal with not wanting kid going to grandparents?

So my 6yo father passed when he was under a year old. His parents/my kids grandparents kept in the picture very, very much. We lived together some time after his passing and would still see them frequently after moving out. When I moved into another city they would still get him from time to time for a weekend or so as he wasn’t in school yet.

I’m really grateful for the love they have for him. The time, money and effort they choose to give him consistently. I appreciate they tell me often to let them know if I ever need their help with anything for him such as watching him, clothes, school supplies etc. Again I’m very grateful.

I’ve moved a bit further out now and he’s now in school. They do not see him often anymore. Since school starting I’ve decided it was best for them to have him a week during summer break, and a week over winter break. Because of how much my kid and them love each other, they miss each other a lot!!

My problem is his how bad it is when he comes back from being over there a while. He cries, sobs!! He screams he wants to go back with. This last time he threw himself on the ground, spit on the floor, banged his head on the wall, claimed he wanted to die, that he should be with his dad in heaven and it was SO overwhelming for me to see him like that.

I thought maybe if he went a weekend instead of a full week it wouldn’t be as bad, but yes it was. To have him scream and cry outside my house because he has to stay scares me that the neighbors will think he is being abused or something with me. Obviously it isn’t the case I just think like any other kid he’d prefer to be where he is spoiled and has no rules. Shit, I would too lol.

I don’t want this to happen again. To hear my son scream he wants to die because he has to be here after x amount of time with them really bothers me. I feel bad because I know he’s pretty much a part of their son they get to still hold and love on. It breaks my heart, truly. But it breaks my heart more for my kid.

I don’t know how to just tell them no, because I know they just love him and want to be a part of his life, but it feels like too much. Other little things I dislike, for example they let their kids watch Family Guy and I dont allow that here, but I don’t think I can really make rules for their house & kids.

What do I do? I obviously want them to have a relationship and everything but I feel at this time, during this age it seems too much for my son to just enjoy his time there and come back without being so hurt he has to stay.

We do things here. I buy him wants and needs myself. I take him out places. His stepdad who he chooses to just call ‘dad’ also loves him and spends times with him as much as I do. He enjoys their night when stepdad watches him and I have to work, cus they have a “party” which is just my kid eating chips or some cookies lol. The downside to us I suppose is he goes to school here, we have rules, he has his little chores to do, you know the not fun stuff.

They talk on the phone here and there. Once in a while a video call. But it’s the same, he ends up crying for a long time. I know he must miss them a bunch, his uncle is around his age and they’re the best of friends but I dread his behavior when he comes back and it will last for a few days after. It must be emotionally hard for him, and it is for me too to see him like that.

Helpppppp!!!!!