All this seems a long time ago, I have forgotten what this was all about.
I assume you are using round magnets with the stator in the picture. If so I think the big improvement you could make is to use all the space and wind with thicker wire or more starands so that the coils touch. if your turns were correct before keep them the same.
Coil shape is not so important. If the magnet spacing is ideal then round coils would be better but I suspect you are using fairly closely spaced round magnets and the elliptical coils that you used are a good compromise. making the things trapezoidal may link you a bit more flux but the turn length will be greater and you will have more resistance. It's all a trade off, the ideal coil shape only applies to a single turn. With real coils there is no room for all the turns to be ideal and it becomes a compromise.
You look to have wasted a lot of space that could have used more copper, that really is the only way to get more out . The heat is due to more current than the wire can handle, it has nothing to do with coil shape, all losses are resistive, nothing to do with coil shape, cancellation or anything else as long as your wire is not thick enough for eddy loss in the wire itself.
Flux