Cool CAT,
I'm not ignoring you, I just don't have time to spend on the forum any more it seems. Springtime...
I tried to find time for an answer at lunchbreak, even started a spreadsheet with your numbers, had to quit before typing a reply. Damn job...
First thing to do is sort out the AC and DC. The load is DC, so figuring out the V and A at the load would make more sense than trying to figure out the power in AC phases and then working through to the load.
So the measured 177VAC was read across 2 phases in Wye at 654 RPM. Let's stick with that for a moment and get to the rest of the speeds later. That VAC is rectified into DC, and 3-phase rectifier gives you 6 peaks per cycle, and the peaks equal the peak of the AC waveform. Assuming the AC is a sine (close in your case, not so for battery charging) then the VDC is equal to VAC*1.41 = 250 V.
The current is a bit less obvious, but I have tried measuring simultaneously to check, and I get about 1:1 ratio when converting the current in AC to DC, but only in Wye. So let's say the current through the resistor is 15.7 amps.
The power dissipated by the resistor is (250V)*(15.7)=3925 Watts. I'd like to see a photo of the resistor! Either it's huge, or you have rigged up a water-heating element.
At the same time, the windings are shedding heat. That's all AC so the formulas are something you can look up: P=(3 ohm)*(15.7A)^2 *1.73 = 1281 Watts
Ignoring all the other little parasitic losses (small %) the total power driving the shaft must be 3925 W + 1281 W = 5206 W, or 6.98 HP. Yeah no kidding the lathe was hot!
The efficiency of the generator is 3925/5206 = 75% These motor conversions sure like to run at high voltages.
Comparing the open-circuit Volts/RPM, which was about 0.4, to this level, 177v/654RPM = 0.27 v/RPM. Looks good, because there is always a voltage drop when current flows. If you can get back to the lathe, try it with two of those 10.8 Ohm resistors in parallel, at about 2.7 Ohms. Somewhere I read that the source-load efficiency is highest when they have the same impedance. That may just be audio-amplifier lore, but maybe a similar principle applies here. I would expect to find a much faster rise in power, too, resulting in a lathe that won't go past 400 RPM.
16 Amps is probably close to the rated current of the original motor. Now that you have it re-wound who can really say what the peak is now, but a reasonable guess would be to compare the original circ-mils of the original wire to the circ-mils of the new wire you've used and that will scale it up into the ball park.
Since you are looking for way way more power and I guess you still want to gear up for speed, then I think you'll have to come up with some other driving power if you want to test for higher output. Do you have a tractor with a PTO?
Okay enough numbers. (somebody check my math please?) It's all just calculations but they do tell a story that makes sense.
One more thing: I saw the base of the motor on a board to keep it steady while you were driving it. Good for getting electrical numbers. How about mechanical power numbers. Go back and look at how I set up my tests - wanna try a beam+scale like that? Then you can see how close or how far my guesstimate about input shaft power was.