I'm definitely not an expert and my dd has only competed compulsories so far (she will be competing level 5 starting next month), but I do think compulsories are important for teaching the basics, teaching good form and really providing a strong base for the optional level skills. I guess it's just a different philosophy than some gyms have though. Dd's gym believes in really focusing on perfecting the building block skills such as the back handspring, handstand, kip, etc before they begin training the higher level skills. For example, until you have a very strong kip with no bent arms and a straight tight cast, they won't let you move on to other, more advanced, bar skills. And it's not enough to have a pretty good back handspring in order to move on to the back tuck or layout, it has to be near perfect.
I understand that many gymnasts that skip levels and score out may score lower because they haven't spent the time really learning the routines and focusing on the details. But I have seen other gymnasts that score out with lower scores because they really don't have the skills that strong, but they (coaches? parents?) want them to move up anyway. They say that in optionals they can focus on their strengths and avoid the skills they have trouble with. At dd's gym, by doing compulsories, and being expected to score well there or not move up, they are really forcing the gymnasts to have all their building block skills solid and I think it sets them up nicely for the more advanced skills.
I also don't think there is a 100% correlation between scores in compulsories and scores in the upper level optionals, but do think that most girls that do very well in the lower levels and that stick with it, do tend to do well in the upper levels too. Of course there are some girls that struggled more in the lower levels and then really started excelling in the upper levels. And there are some that did well in the lower levels and not in optionals. But I have noticed that most girls that are really doing well in levels 9&10 in my state/area, also really did well when they were younger. I've clicked around on mymeetscores sometimes just out of curiosity and the ones that are placing high in level 10 generally were winning at level 4 too.