Have you heard of SeaweedFS? Pretty different project, but there might be some good ideas in there for scaling when dealing with lots of small files and a designing distributed hash based lookup.

https://github.com/seaweedfs/seaweedfs?tab=readme-ov-file#master-server-and-volume-server