Shall I assume they walk in to the wind, or was I just unlucky again?
There is no set rule on how they walk in relation to the wind, but they do USE the wind 100% of the time. Their nose is their best defense.
They will walk into the wind if they feel that it is safe. They'll walk across the wind too. What I see them doing a lot and have read guy who study deer movements report is they often like to walk diagonally with the wind, kind of a quartering away angle. This lets them see in front of them but also smell what might be upwind of them as they move across the terrain.
But honestly, that is deer hunting. You see them in one spot, move your set up, and next time they are in your old spot. Happens to all of us at some point and time.