Derek, I assume both 12v banks are and remain connected while you are discharging one set.
Think of it this way, you have two tanks of water cross connected, and therefore at the same level.
Now you start drawing water from one tank, so water will flow from the other to try to maintain equal levels.
If I understand you correctly what you are experincing is normal. Have you done a discharge test on your batteries? This will establish the remaining capacity of your batteries.
To test;
Fully charge a bank, including a 3 hour equalising charge (assuming flouded lead acid), isolated that bank and allow to sit for say 12 hours(this lets the surface charge disipate).
Place a load on the batteries equal to 10% of amp hour rating, monitor each cell if possible, and cease the discharge test at 1.85V for any cell, under load.
Multiply the amps x time, the closer this figure is to 20% of your batteries amphour capacity the better your batteries are.
Now don't re-connect both banks until you have charged the test bank to equal volts with the other, or large currents will flow.
Hope this helps
allan down under