Thank you. I'm still improving and studying better the negentropy protocol.
I converted the code from C++ to Rust in basically a day and half (I'm not a C++ dev so some thing could be converted badly) and are there some things to fix/improve.
Like, I noticed an unstable performance with reconcilie_with_ids method with a set of 50000 items: it usually took 600 ms the final reconciliation but often it took 2 or 4 secs (executing exactly the same code).
Thanks, I'll replace the String with a vector.
BTW, I'm planning to release also bindings for some languages (Kotlin, Swift and Python) using UniFFI, like I have done for rust-nostr libraries.
600ms seems too long for that set size. I will take a look at the performance when I have my Rust harness working. Let's make sure it's working the same way as the C++/JS impls first, and then we can see if there any speed-ups.
One optimisation I did in the C++ was to ensure that the XORing is always done word-wise, instead of byte-for-byte. In fact, with attention to the alignment, I got the compiler to emit PXOR SIMD instructions to do multiple words in a single instruction.
That's very cool about the UniFFI bindings, I'm looking forward to it!
Performance issues should be solved now: reconciliation of 1 million ids took ~400 ms
Found another optimization: 1 million ids reconciled in ~100 ms
Nice, will look into amd optimizations in the damus+nostrdb version. Not sure if you’ve seen nostrdb yet but its a pretty close copy of the strfry design, but meant to be an embeddable db like sqlite. Really impressed by the share nothing architecture, clustered indices and flatbuffers, so i copied most of that 😅. Keep up the good work!
https://github.com/damus-io/nostrdb
Yes, I have seen nostrdb. I think it's a great idea, and that it will be a big benefit for native clients. Let me know if I can help in any way!
I was thinking about it a bit and maybe flatbuffers is a bit overkill for the indexed nostr events in strfry, and I'm working on a branch that uses a custom event representation similar to nostrdb. I usually use flatbuffers by default because the generated code is convenient and it's easy to evolve the schema in a backwards compatible way. Also it comes with some nice utilities for debugging and securely parsing untrusted data (not needed here).
BUT the core nostr event layout is pretty much fixed and saving some space here is important not least because it will let us pack more records into the page cache. I'll keep you posted on how that goes. Cheers!
Awesome! being able to generate flatbuffer readers for other langs is probably the main advantage, I was going to use it for nostrdb notes but I couldn’t make the tag arrays work with the swift generator :(
Oddbean new post about login | logout