I agree with Coach Todd about your shoulders and wish to add my two cents to this thread...
This is very common with gymnasts who are at the very phase of learning a back walkover as you described. Without the strength and ability to draw your body up off the floor using your shoulders, many gymnasts drift off to the side and even twist away or flip to the side.
Here is a great drill that teaches a gymnast what to do...get in a handstand against a wall with only your feet touching the wall and with your hands further away from the wall than you are used to. Without using your feet, (do not push the wall with your feet) lean and use your shoulders to pull yourself off the wall and come down. As you get used to this then keep starting further and further away from the wall. You can split down and step out if you want after your feet come off the wall.
Your goal is to be able to pull off the wall from a level position to the floor. That's right, bent way over in a handstand with your feet touching the wall and your legs level with the floor. If you can pull off the wall from there and come down then you have more than enough ability to do a back walkover starting with your leg up.
To improve the technique look in while you do this. Look at your belly as you pull away from the wall. If you are splitting and stepping out then you can look in hard at your knee.
Before you actually try the back walkover after mastering this drill it can help to do this same drill in a handstand split starting with your back leg foot touching the wall. To increase the difficulty, use a folded mat instead of the wall so you can start from a lower and more difficult position than the wall in a handstand split.
Remember not to push off your foot. Your back foot should lift and float off the folded mat. Same goes for the real thing when you get to it. The more you need to push off the supporting foot the worse your back walkover is even if you can do it by pushing off your foot. The point is that you should be able to draw yourself up off the beam without pushing off using the strength and lean of your shoulders. Yes, this is a hyper extension of the shoulders. Or a plange action, if you will.