Things are rather easier with wind power as the duty cycle of wind on most sites is low, the average power is way below the peak. I promise that all these alternators will fry at claimed rating on a full load bench test. There is also the unknown effect of wind cooling, it is significant but probably not as high as suspected.

How this affects things in real life depends on the circumstances. For grid tie or heating there is no justification for low efficiency and the present designs can manage rather more than indicated by running stalled as in common battery charging.

For battery charging loads it may be a reasonable assumption that you need a big turbine to give worthwhile power on low wind days so on high wind days the batteries will be dumping. Increasing high wind output may be of little benefit. It is only if you can use the excess power there is any point in increasing output. This may be the case for Dan and running stalled with low speed and stress and easy control may out weigh the other considerations.

For smaller machines there is more need to extract more power in times of high wind and load matching that keeps the prop speed up and alternator efficiency up will double the power out in the 20 mph region. This inherently needs a higher cost alternator with more material and the temperature problem stays within reason and would do so up to 3 times the output of conventional loading.

If you want even more or want these larger powers from a stalled machine then stator temperature does start to become an issue.  if you want high efficiency it has to come at a price but with the direct connection the overall efficiency needs to be low. You have a choice of a high efficiency alternator and resistive heating in series resistors or you can dissipate the heat in a lower efficiency alternator.

If you can find effective ways of removing the heat you get a cheaper machine for the same output. This is exactly what motor conversions do, in real terms they are generally less efficient but are extremely well cooled, you get less power for a given wind speed but a higher overall current for a given thermal limit.

What your chances of great improvement are using different materials I really don't know, I can't see anything even approaching the same thermal capabilities as a coil wound in iron slots. Coils not potted and exposed to the air will be better than potted, but the cooling surface area is small. There is no chance of any metal type heat sink to extend the cooling area. It will need careful experiment to see if various fillers are significantly better than unpotted coils.

I think you will have to go for very expensive epoxies to even beat polyester, they are thermally quite poor and you will struggle to get to class H temperatures. How much you will gain from fillers again will need lots of experiment, at best I don't think the gain will be great. Benefit from cooling fins is dependent on a good temperature flow through the material, if you can't get this nearly all the cooling will be near the coils, unless your mixture will give you temperatures of 60deg or over several inches from the coil you are on a looser.

I really don't think alumina is going to be revolutionary. Sintered alumina as used in integrated circuits is excellent but once you use particles in a resin binder I can't see it being very good.

I still think metallic powder may be better but that is not going to be spectacular either.

Much room for experiment and I am sure there is quite a bit to gain, but I doubt that it is going to revolutionise performance to solve all the issues of this type of machine. There are better constructions that retain 90% of the advantage and are far better cooled but are more suited to radial construction.

I think this issue will keep cropping up, it does so regularly, about time someone actually tried something and done some tests instead of dreaming of the perfect material. I think methods are more important than the filler material.