It is definitely possible--our adult team has several members who left the sport for periods of years and came back to compete levels 7-9. For most, skills came back fairly quickly, strength was harder
The hardest part is often finding places that do adult classes at all, much less allow adults to compete (with the extra hours and actual coaching). You might have to do rec classes and be very self-directed. If you're in college or near one, look into if there's an NAIGC club team. Rules vary by school, but many don't require you to be a student to be on the team and will have a lot of people with similar goals.