Oddbean new post about | logout
▲ ▼
jb55 | 1 months ago (raw) | root | parent | reply | flag +2 
 messagepack would work fine I think. biggest gains would be parsing efficiency and battery life gains. decoding json sucks and is slow.

nostrdb has optimizations to skip parsing altogether when it doesn’t need to (stops parsing at the id field if we already have the note). The performance boost there is nice. messagepack or some other format would be another boost.

The *ideal* way would be something like flatbuffers, where you could just memcpy the note into your db… but is more complex.
▲ ▼
frphank | 1 months ago (raw) | root | parent | reply | flag 
 CBOR is basically the RFC standardized version of messagepack.

I recommend CBOR.
▲ ▼
semisol | 1 months ago (raw) | root | parent | reply | flag 
 I proposed a format for that
▲ ▼
mleku | 1 months ago (raw) | root | parent | reply | flag 
 my binary codec already does a lot of that memory remapping of fields, as the runtime and database versions of the data are basically the same - it keeps the id/pubkey/signature fields (including in common tags) in raw binary and unpacking it into the runtime format is just a matter of creating pointers

the hex encoding is also done with a SIMD hex encoder, and the sha256 hash is done also with a threaded worker based single instruction multiple data so, on avx512 and avx2 that means it runs 2 hashes per CPU thread

switching the binary fields to be kept as binary except up to the wire has a massive benefit

it is so nearly close to being a perfectly servicable wire codec as well, i just didn't design an envelope encoding or binary serialisation for the filters

but other languages probably won't support this kind of optimization very well certainly not javascript

i don't get how javascript parsing is really much slower working in native json (which should be optimized to the max) versus making javascript work with foreign binary data formats for these binary fields

but totally understand why it's hard to make clients that aren't JS/DOM/HTML/CSS based... the whole tech industry has focused on this universal platform and its abomination of a scripting language to the expense of serious languages supporting native apps, and the total lack of adequate multi-platform targeting and adequately broad language support for cocoa, win32, gtk and qt - ironically most of the time it's either electron, putting the front end in a browser engine, or some kind of simple and fairly inadequate immediate mode or similar direct opengl/vulkan based thing (eg imgui, egui, gio, fyne, nucular)