To fix the problem try rotating your torso with each stroke. Visualize pointing your belly button to each side of the pool with each stroke, like rocking a boat from side to side. Here are a few drills to work on rotation. I suggest using fins as well so you don't have to worry about sinking while getting these right.
Start with kicking 6-12 beats on each side, then switching. Place your ear relaxed on your arm...
...once that is mastered you can add 3 strokes in between and a slow relaxed arm recovery
Another thing to try is relaxing your shoulders and forearms during the recovery phase (when your arms are out of the water) of the stroke. This will prevent your body being thrown out of alignment by a tense arm swinging across your body. These two drills lift your elbow and even shaking your hands at the wrist during recovery can help you stay loose.