With enough organic matter, clay eventually makes the best soil. You’ll get there! 

Whatever organic matter you can get, just get it on the surface and let the worms do their thing. You can pile it up and plant into it, let the roots help the worm out.