Hanski
I am a bit confused so if i go off track please excuse me.
First the theory as I understand it.
Your battery is connected to the shunt and also to the charging supply be it wind solar or battery charger. When the charge reaches your preset voltage you set with the pot, in your case 13.7v the relay should switch and the load which you put on it will take the voltage down and the battery will stop charging.
Points to consider
1. the load must be more than the supply from the charging supply.
if it is to low the voltage to the battery will keep going up
2. if it is to high the supply will short out. if it is a wind the mill might stop or battery charger blow fuses but not solar they don't care.
The battery wont lose charge until it is drained by an appliance attached to it and when it reaches 11V? the dump will switch off and the battery will charge to 13.7v again and the process continues.
Now to your problem
1. connect the relay to your battery directly and make sure it switches correctly
if not the relay is wrong
if ok then connect the positive lead of the relay to the circuit board and the negative directly to your battery negitive but don't put any load on the relay output. Charge the battery and see if it switches ok. use a multimeter directly to the battery to check the voltage as this is what powers the relay, if ok then attach the dump you have to the relay. The dump must NOT be connected to the battery in any way.
Try this and let me know how you go
Trivo