There has been quite a bit of extensive testing on aircraft wings using vortex generators. Some of the articles I've read about them look quite interesting showing positive results. I'm not sure it would be of any real benifit on small turbines though. Most of the testing that has been done with them is on aircraft that fly in the range of 100+ mph, a bit out of the range were working in. It's possible there would be some benifit to them even in low winds... I've never experimented with them so I couldn't say one way or the other. They may work on HAWT's much better since the blades are running at higher speeds.
Rotornuts played with the idea some time back, the tubercles, I don't remember what the outcome was...
.