Hi all! I have just pushed a major update to the negentropy project. It implements protocol version 1 (previous was version 0).

The protocol has been simplified: ID size is no longer configurable, there are no more continuation ranges, and the output can be constructed in the same input-scan pass.

There is a new fingerprint function, based on an incremental hash. This allows fingerprints to be pre-computed and stored in a tree structure. The C++ implementation includes a B+Tree implementation that allows fingerprints to be computed without collecting IDs from the entire DB.

I have written a comprehensive article that goes over the theory of RBSR and the negentropy implementation:

https://logperiodic.com/rbsr.html

Comments are appreciated!

Finally, I have integrated the new version of negentropy and the B+Tree with strfry. It's in a development branch and not quite ready for production, but my testing indicates this will be a massive improvement for full-DB syncs, especially on relays with very large DBs. In-sync or nearly in-sync relays should sync almost instantly with negligible resource usage. Relays will also use the B+Tree for filters that contain only until and/or since, meaning that date-range full-DB syncs are also efficient. Syncing arbitrary filters works as before (but I have begun work to make these more efficient as well).

Unfortunately, this is a breaking change for the negentropy protocol (this really should be the last one!) and will also require a new DB version.

I'm going to take this opportunity to make a few more breaking DB changes, and plan to release strfry 1.0 after a beta testing period.


Yes but how long is your OP_CODE field?


i wish i could zap this note


👀

nostr:nevent1qqsyazdcuyjfy3pya25ha8750uxpf4d3cap252kv6vs4w8dtatvwv9cpz3mhxue69uhhyetvv9ujuerpd46hxtnfdupzqgvz8pp38yu4n4kgv9arhkyexqafvcymgjnyf6tn3ygr3f77sc3dqvzqqqqqqyvfrcww


Thank you for doing computer science so I don’t have to!


Thank you Doug for your fine engineering 👌


Beautiful, more efficient sync.

I am personally invested in competing with all of this stack from first principles, but I love to see this.

nostr:nevent1qqsyazdcuyjfy3pya25ha8750uxpf4d3cap252kv6vs4w8dtatvwv9cpz3mhxue69uhkummnw3ezummcw3ezuer9wcpzqgvz8pp38yu4n4kgv9arhkyexqafvcymgjnyf6tn3ygr3f77sc3dqvzqqqqqqy3dxwps


Oooh. Distributed systems. ❤❤❤

Will read the paper.