Had that happen to me a couple of times as well. the most interesting one when we were racing with smaller dinghies and we found ourselves always waiting at the finish line waiting for the other boats to finish, so me and my crew had a few beers in between races (we had a cooler). On the third race we were wondering why the smaller boats were leaving us behind, so we assumed it was the beer, we actually managed to get to the windward mark, and when I was about to raise the board for the down wind leg I noticed that it was already up, so apparently it was the beer!
Winds were light to moderate and we actually didn't have any trouble tacking without the board, we were just getting a whole lot of leeway.