Sounds like your are not sure what you have created. Really be careful that you have not created a fire hazard. For a charger, my criteria, if you can't touch it without getting blister burned then it is too hot.
First step is knowing what transformer you are starting with and if it appropriate for what you are doing with it. Just because it came from a UPS does not mean it is right for your charger application. Did it come from a 12v, 24v, or greater UPS system.
Measure the open circuit AC voltage from the transformer. If you are more then 15 to 16 vac it can be problem using it in the appearent charger config you are attempting. This is assuming you are going after a single 12 v battery charger.
If you can, load the transformer with power resistors or water heater elements in attempt to get at least a 10 to 20 amp AC load on it if you think that is what the transformer is capable of. Measure the vac with load. Should not drop more then 3 or 4 vac from unloaded voltage.
Second, If you claim you are delivering 40 amps then the alternator rectifiers are dissipating about 20 to 30 watts. This is alot of heat to get rid off and requires a large heat sink. In a car alternator the rectifiers are mounted to case of with the pulley fan blades running air through the alternator to help cool the rectifiers.
At the end, you have a very uncontrolled charger that without alot of careful supervision while charging a battery will likely damage the battery.
Anyway, to answer your specific question. The dominate problem excessive heat causes in a power transformer is wire insulation degradation. The enamel or Formvar insulation will get brittle and cracked when subjected to >100 deg C. Eventually you will get a turn to turn short between the windings. Assuming you have a fuse in the AC plug side, it will blow when you get a shorted transformer.