Maybe add a hunk of wire through the big hole? Maybe a nylon lock-nut on the inside? Maybe just a giant squirt of silicone calking in the big hole part?
'big hole' = key way.
The brackets are strong enough. That idea about gravity holding them down idea is not. G-