After examining the supplied basic schematic of the converter,
I have found the problems why the converter is so in-efficient and why the Power Mosfet is over heating, as well as, the lack of output voltage accuracy
The design is quite basic and the designers did not know what type of voltage isolation is needed to attain high efficiency, accurate output voltage and good high voltage dynamic operation.
-- 1) the power supply for the TL598 is formed by a Zener diode and a 500 Ohms 50 Watts resistor, same thing goes for the voltage for the MosFet driver.
-- 2 ) The voltage range of the input may go to 85 volts or even higher. and the TL598 is capable of 40 volts maximum
-- 3 ) The circuit bias supply voltage for the Mosfet driver is set to 12 volts, so it cannot swing the input voltage range to maintain proper gate voltage.
-- 4) the Feed back path goes into the error amplifier which was set as a pure DC gain amplifier and that is a NO, NO and NOT !. The amplifier requires limited AC gain and open DC gain for the loop to fully close to set the output voltage accurately.
Solutions: The Mosfet driver needs to have a high voltage offset type, IRF and many manufacturers do have many types, some from 100 to 600 volts capability and some with 2 or more Amps driving capabilities.
-- The power supply for the TL598 needs to be started with a low current source and once it starts to charge one can add a two chokes in series and a diode to charge the device supply with enough current capabilities, or the main choke can have an additional terminal that sets the voltage to a 1/4 of the battery voltage. or another additional circuit to connect the 12 volts lower battery to the device supply, lastly a small IC to produce the necessary voltages ( this is my preferred way).
Theses changes alone will bring the efficiency to the upper 80's or lower 90's and the output voltage accuracy is improved to a few millivolts range.
Also the high voltage detector is not needed allowing the converter to continue working with the higher volts, limited in this case by the breakdown of the Power MosFet.
Lastly, the linear charge regulators for the small amp-hour batteries are not the best solution, both should be changed to a better circuit using the same circuit for the high power converter to get better efficiency, as a matter of fact that should be the optimum solution because these circuits can be used to replace the main one in case the main is damaged -- I have done these with several systems.
Nando