I think you're about to teach yourself a LOT of engineering the hard way. . .
improper aerodynamic balance, possibly caused by mounting position on the profile, or centrifugal or lift forces, is a avenue to explore. based on your comment about the blade tips changing where they track, this is a strong possibility. solutions to this will probably involve new or substantially modified blades with different bending and mounting characteristics.
too much flymass or too little spring might also cause your problems.
run the thing on the ground without blades and watch what it does. you might go as far as to put weight equivalent to the blades on the thing, such that the 'blade' mass runs at the blade CG station. . . in fact, that's what I would do in your position.
if it's really, really, REALLY well behaved, look at the blades.
if it's still violent or marginal, look to the weights, the spring, and all the things Ron mentions.
too, if your flymass is a significant fraction of the total rotating mass, you'll have major hunting problems as the moment of inertia changes due to the flyweights changing station. . . .
the trick with troubleshooting something this complex is to make gradual changes in ONE VARIABLE ONLY until you satisfy yourself that that variable is not part of the problem.
good luck!
-Dan