Yeah, even though we are a big country in land and population, quality boys programs are few and far between (compared to girls), so can really only pull from a fraction of the country.
The college thing is huge. In the USA a 4-year college education is expected for just about every kid doing even close to average of better in school. And gymnasts tend to be high achievers. The vast majority of gymnasts will attend university right after high school. With so few mens college gym programs, motivation to stay in the sport is difficult except for the few at the top of the podium at age 16 or so who will get into these few remaining college programs, and we all known it take years more to really develop in MAG post puberty. The top 16 year old level 10 may not be the top 23 year old in prime olympic shape, but these are the ones going into NCAA. That video of Kohei as a junior on Youtube shows a gymnast who is decent but not anything spectacular. In the US, Kohei might have been overlooked as a junior and not made NCAA, and never continued in gymnastics. Imagine that!
The pathway to World/Olympic level gym in the US has a huge roadblock at age 18, even for the relatively few boys who do happen to live near a quality gym, do overcome the stigma, and do develop the passion.
I'll add that I think the men we do have are amazing, and performed well in Rio. 5th place team, 2 men in the top 15 AA in the world is super, and all 5 making a final as individuals. Hitting 6 for 6 is definitely different than 4 for 4. And the US ladies are not going up against a Kohei, Deng, Whitlock, and Vernaiev from other countries. Mustafina and Chunsong are lovely, but have gaps in their repertoire. The US women simply aren't facing the high caliber of world-level competition right now as the US men.