If you had some way to determine how far down it was to clay, you'd have a better idea of how well your structure will hold. It may be shallower than you think, but building on the aggregate is a recipe for failure. If the large rocks are, say, in the ton range, then it should hold. Anything smaller than a cubic metre is really just gravel to a storm surge.
Not to sound snooty, but you're building on sand, just that the particles are much bigger. I'd look into some sort of mechanical driver. Maybe heat-treat the driving ends of your rebar and send them right thru the rocks. Put enough of 'em in (3 per sq ft) and it should hold.
Better yet, call a soils lab and do a core sample. If you're gonna invest close to five figures in the project (don't grimace, you'll be there sooner than you think), a few hunnerd turtlebuks to find out what's down there will give you peace of mind when the hi flow comes, provided you built following the data of the test.
I just bit the bullet and rented a small track hoe and cleared out everything down to clay, on the good side. Cheaper in the long run, for me.
Just sharing my (few) successes and many failures, skid...
Turtle