I tried to use power from one of a string of 2 batteries when I converted my old 6v truck to 12v with a new alternator. Changed the bulbs. Short cranking cycles won't hurt the starter. But the concern was the heater motor. So I ran it off one of two 6v batteries in string series charged by alternator.
When one battery got discharged, its resistance rose, preventing recharging of whole string. So I ended up finding a heavy resistor to drop voltage to heater motr, everything is on 12v now.
I do agree that fence charger is smll load, may cause no problem , particularly if you shift that load from one bat to other periodically.
Another problem to watch for, however is that fence chargers, or at least the one I had, send a hi-V pulse back on the supply line. I tried running a fence off the same battery that also ran radios in the house, and fried the radios. I advise a separate system to run fence charger.