Oddbean

i just learned about this tho https://github.com/deneonet/benc from this: https://alecthomas.github.io/go_serialization_benchmarks/ and i can't help myself but implement it lol faster than flatbuffers... faster than capn proto the benefits of those flat/decode on demand style of codecs are zero when you actually need the whole data and taking advantage of it, requires refactoring an entire search matching library (filters in nostr) so, i'm just gonna go with this its syntax is like Go except also like Protobuf it has some scheme about versioning and shit but honestly for this use case, like, such as replacing the entire nostr encoding, with the envelopes, filters and events... this would be the hands down best option because the data usually needs to be matched on several criteria in the "extra filter" used in the database indexing scheme that fiatjaf devised, so the extra logic to decode them before running the match when half the time half the fields need to be decoded, and most of the fields except for two are already just raw bytes in my data structure format... this is it, no more fucking around, the end, faster than this is impossiburu

i can reserve the option of later changing it again but just following the same pattern i used to do protobuf should be easy to make the same thing... the only extra overhead it does is copying out the slice headers of the benc formatted structure and allocating/copying into the "native" realy style

well, on the basis of its raw throughput in my bulk decode encode decode test it's all about the same probably the reason is that my data structure is more suited to protobuf than benc, so even though protobuf does reflection the overhead does not make it slower they are basically the same for this use case the throughput is about 58mb/s for a single thread unmarshal from json, encode to binary, decode from binary, about 30mb/s to do that plus check the ID hash and encode/decode through json my JSON encoder is the bomb for this shit... if it wasn't for it being a bigger data size i'd say this is the way to go in any case, i'm leaving the code in there for the different binary codecs in case for some reason it seems like a good idea to work with them later the big problem i foresee is that to make it go any faster i need to adapt the runtime data structure of my events to BE the benc encoded version, which uses slices wrapped in structs, both the protobuf and benc versions have this problem of needing a shim to change the data structure, and probably that overhead is the last bit... so, i'm just leaving it with protobuf for simplicity