It's a simple matter of elimination.
You've started the elimination process by measuring short circuit current.
Now, disconnect the battery and connect a 12V/4A (55W) car headlight. How much current does flow? 3.5A, or close?
Next, connect the battery to a normal battery charger. How much current does flow?
Finally, if you're worried about the resistance being an issue: connect the panel to the battery/charge circuit with short leads. Does this change things?
The battery is my prime suspect. But, sequential elimination of problems will lead you to the answer.