If the coach wants to try this, and you trust the coach, I do not think it unreasonable to consider this. I think either of my sons would have enjoyed practicing with older boys at that age. And having a "little brother" at practice to be considerate of might actually help some of the older kids focus/behave better.
I do think a talented child being pushed 'faster' than his maturity allows is not a good idea. But this might happen with any age child, and it does not sound like that is what is happening here? It sounds more like the coach is trying to find a practice group that would be best for your son, not pushing your son to advance faster than normal?
But there are certainly many potential downsides. Here are a few things off the top of my head I would clarify/think about first:
How many hours/how many days would your son practice? A 4 year old beginner who is a couple years from competing should probably not be practicing nearly as long or as often as more advanced boys typically would. I would also clarify conditioning expectations etc. as I imagine they should be different for your son than for older boys.
What would coach do to insure safety? In my experience boys roughhouse, push, and pick on each other to some degree, and that happens at gym practice like anywhere else. If everyone is much bigger than your son, or they decide he is a good target for teasing, that may make things hard for him. Standing on line waiting to 'go' on something is where this has been the worst at our gym. Especially around the pit which seems to scream to some boys "push someone into me!"
How are you going to feel about your 4 year old being exposed to the language used by typical 8-11 year old boys? I have had to have many talks with my sons over the years about 'gym talk' staying at the gym.
Are the older boys allowed on phones/other devices at practice or on breaks? This has been a major concern of mine over the years. Kids can and do bring up anything that is on the internet with these devices. Anything. And they often cannot wait to show their friends.
Any problem with him moving back to age level group if you decide this is not working?
I personally would not care about other parents and their concerns. If they have a problem with your kid practicing with their kids, they can take it up with the coach. I personally would much rather my 10 year old practice with a well behaved 4 year old than with a poorly behaved 12 year old.