JSON or CBOR 

 not CBOR because bluesky and IPFS and fuck that shit

protobuf is the most broadly supported and best maintained binary codec 

 Something like protobuf? 

 Easier and better option - relays should support #Reticulum as well as HTTP(S).

Reticulum packets are always compressed at the transport layer.

It's an established standard that "just works", no need to herd cats into a completely new binary protocol. 

 nostr websockets are generally flate compressed by default also... for the most part this reduces the size of the hex encoded fields almost to binary, and greatly compresses long text segments and repeating fields like in req envelopes 

 Don't forget LXMF 

 https://github.com/markqvist/lxmf

#Reticulum 

 Yes, 💯. I keep fighting json crapola on HAMSTR. Ended up compressing and serializing for now. 

 write a codec. simple

in my nostr codec it decodes all the nasty hexadecimal fields (p tags, e tags, a tags) into raw binary for internal use to save memory and speed up matching, and this is also then transformed to the binary format used by the database

all you have to do is write one good codec and done, modularity solves the problem 

 You lost me at write a code. 😂 

 structure is already there, i just wrote a thing that converts between one format and another, in a way that is far more efficient than what fiatjaf wrote with go-nostr 

 Ah. Gotcha 

 JSON is the reason 90% of nostr devs where able to easily learn nostr 

 don’t need to change your mind. 
think what you want. 

 you need to go study the principles of Unix if you don't understand why having the messages in JSON is better..

the default websockets use Flate compression and this is pretty decent at deduplicating the high entropy pubkeys

if you want to make a relay and client that works with a binary format, go make it, but you will have a bad time if its primary support of JSON is not kept fully up to date, interoperability > all

by obscuring data in a binary format you create all kinds of problems with interoperability, debugging, and so on, i'm not saying don't do it, but it's a low priority compared to fully supporting the easy to debug human readable format, even if json is shitty for certain parts of it syntax it's still readable

here's the principles of unix, summarised by Brave's AI:

-----

Write programs that do one thing and do it well: Focus on simplicity and single-purpose tools.
Write programs to work together: Design tools to be modular and interoperable.
*Write programs to handle text streams: Use text as a universal interface for data exchange and processing.*

Additional Principles

From “The Art of Unix Programming” by Eric S. Raymond:

Rule of Clarity: Clarity is better than cleverness.
Rule of Composition: Design programs to be connected with other programs.
Rule of Separation: Separate policy from mechanism; separate interfaces from engines.
Key Concepts

*Plain text: Store data in plain text files.*
Hierarchical file system: Organize files and directories in a hierarchical structure.
Treat devices and IPC as files: Use a file-like interface for device management and inter-process communication.
Use software tools: Prefer small, specialized programs (tools) over monolithic applications.
Command-line interface: Use a command-line interpreter to string together tools and execute tasks.
Notational Conventions

 

 t-y 4 input! 

 this is the right answer

i don't see how anyone could disagree 

 Out of curiosity: what do you think of systemd?

🫂 

 i don't really like it, i prefer the old days of rc.d

systemd is too complex 

 Good text. I'm almost there with our new relay. Last step missing is group multiple tools on the command line.  

 Yeah, this approach allows for "small specialized programs". 

Such as: a binary protocol to optimize send and receive speed and bandwidth, primarily over mobile networks. 

Protocols have different design goals.
 

 yes, but don't turn it into a waste of dev resources like Wayland or most of Linux 

 so sure in your conviction or something else? 

 I wrote a NIP for this. Got almost zero engagement.  People want baubles. 

 Lol, it took me several seconds to realize you weren't talking about security options, and was wondering wtf a JSON option was 😆  

 imo too early for that kind of optimization, nostr having a trivial-to-learn protocol is one of its charms, and a simple text protocol allows for easier prototyping and experimentation in the early days

mind that this is only about text messages, which are very small compared to other media, would agree encoding images and video in JSON would be absurd 😀 

adding another protocol and the logic for supporting both of them adds a lot of hassle--like sure HTTP eventually got binary protocols in the form of HTTP2, HTTP3, when giants like google with massive web servers tried to squash out every byte and sponsored this kind of development, but i'm not sure massive relays like that what we want in the first place 

 I think the later we move, the harder it will be to ditch JSON, isn't it? 

 that's a fantasy imo, there will be no ditching, the old protocol will always be around, and there will always be need to support it too 

 It's over, billions must simdjson... 

 billions? hah, that presumes adoption already 😀 
to get there, there's probably more worthwhile things for both relay and client authors to focus on than duplicating protocol work 

 Can nginx and browsers automagically gzip the json? 

 yea, the websockets protocol has built-in support for transparent per-message compression, using deflate (gzip) 

 And one message could be an array containing the result of some query, i.e. all posts by people I follow in the last day.

That's probably not much slower than going from json to binary and back. But haven't benchmarked.

Would be good is there was a relay health tool that checks if this is configurered. 

 i'm not sure-AFAIK, the protocol sends only one event per message

deflate is very good in catching repeated text within a message, compressed JSON will get pretty close to a naively implemented binary protocol, for large data
it's also very fast

with lots of small messages it will be less effective because the same text repeated over and over won't be compressed, this would ideally need some kind of shared dictionary like HTTP2's header compression 

 This would be a good nip, result batches to allow better compression ratio 

 that made sense to me, but apparently, websocket can already do this on the transport level:

nostr:nevent1qvzqqqqqqypzqlxr9zsgmke2lhuln0nhhml5eq6gnluhjuscyltz3f2z7v4zglqwqqsqz35fqcqu7qmplf6z7wmyxs7fv84rezvj24kevz5h54ntdhwsj6cyz2c99
so now it's less clear what the win would be 

 Considering the messages are coming from different sources, doubt that there's much common text between them. It should be easy to do a quick analysis to figure this out. 

 yes, that would be something to figure out first

HTTP2 benefited from decades of HTTP use, and a virtually ossified protocol, making it possible to design a hyper-optimized protocol for specifically that use-case

just throwing something together with protobuf is not quite that, and has the risk that another binary optimized protocol will be needed later, and of course people will want support for all three... 😀 

(HTTP3 didn't change much compared to HTTP2, the main change is switching the transport to QUIC and pretty much all other changes are related to that, but still-) 

 strfry already does stream / rolling window compression for clients that support it.. 

 TIL this is a thing at all, it's even defined in the same RFC ( https://datatracker.ietf.org/doc/html/rfc7692 ) as per-message websocket compression, a matter of reusing the LZ77 context between messages 

 Json was a mistake. 

 good luck having fun getting agreement on binary codecs

you have all the CBOR fans, and the Protobuf fans and the MsgPack fans and then there is retarded custom codecs like the one i made for database and the one fiatjaf made for database but at this point these only encode events

in principle a good idea but the only way you are gonna get this to work is getting several clients to adopt it and at least one relay that makes this available, and probably you still have to stick with websockets 

 CBOR 

 Gray beard approved.  

 you see, there is a problem, i don't like cbor after dealing with it in my work with bluesky

so why do you love it so much? why do you prefer it compared to msgpack or protobuf or flatbuffers or capnproto?

keep in mind this needs to connect together apps written in go, rust, javascript, java, c, c++ and maybe at least reasonable support for python and c#

from what i've seen, and it's been a while since i've looked that close, protobuf was the most well and fully-supported across all of these languages

personally, i would want to use flatbuffers, and currently the Go version of this protocol encoding is shitty, but i could imagine myself reworking it in a month or two into something that is really natural to use with Go, because i have already in fact written an on-demand protocol buffer encoder, twice, in fact, simple TLV thing that was based on ad-hoc dynamic interface array message descriptors 

 m 

 CBOR is RFC standardized, protobuf is Google's personal pet project and with their schemas usually also too complicated. 

 🎯 

 as someone who works with interfaces and not variants i don't like any protocol that doesn't have static typing and i honestly don't see the point of complex rpc specification compilers when it's just a friggin function call

about 10% of cases, partial failures can happen that should really return an error and a partial value (eg, a result being an array of fields from an array of inputs) but nope, can't do that with typical C++/Rust error handling idiom

this forces you to design APIs as single shot rather than batched which is a pretty rigid and performance limiting way to do things, especially when it is easy to aggregate queries to be handled by for example databases with concurrent queuing (especially distributed ones where the processing can be fanned out to multiple servers)

i say, if you make the client and build out a relay interface go ahead, make your hipster CBOR encoder API

but for the love of God do not make up your own fancy new RPC API if you do not intend to support a sufficient range of common language bindings, NATIVELY... just take the json and binary encode it, don't fuck around 

 Yeah, protobuf is gRPC standard, like @Michael J suggested. 

 i think in all this debate people just need to make a relay/client pair that understand the new protocol and done 

 I don't understand why they want to standardize this. Different devs will want different protocols and it's irrelevant to core interoperability because you can always communicate over json. 

 All I really care about is signed events that I can parse and verify with a npub. Plus a few conventions around kinds. If signed events can be serialized for more efficient transmission, great, but that’s icing on the cake. 

 messagepack would work fine I think. biggest gains would be parsing efficiency and battery life gains. decoding json sucks and is slow.

nostrdb has optimizations to skip parsing altogether when it doesn’t need to (stops parsing at the id field if we already have the note). The performance boost there is nice. messagepack or some other format would be another boost.

The *ideal* way would be something like flatbuffers, where you could just memcpy the note into your db… but is more complex. 

 CBOR is basically the RFC standardized version of messagepack.

I recommend CBOR. 

 I proposed a format for that 

 my binary codec already does a lot of that memory remapping of fields, as the runtime and database versions of the data are basically the same - it keeps the id/pubkey/signature fields (including in common tags) in raw binary and unpacking it into the runtime format is just a matter of creating pointers

the hex encoding is also done with a SIMD hex encoder, and the sha256 hash is done also with a threaded worker based single instruction multiple data so, on avx512 and avx2 that means it runs 2 hashes per CPU thread

switching the binary fields to be kept as binary except up to the wire has a massive benefit

it is so nearly close to being a perfectly servicable wire codec as well, i just didn't design an envelope encoding or binary serialisation for the filters

but other languages probably won't support this kind of optimization very well certainly not javascript

i don't get how javascript parsing is really much slower working in native json (which should be optimized to the max) versus making javascript work with foreign binary data formats for these binary fields

but totally understand why it's hard to make clients that aren't JS/DOM/HTML/CSS based... the whole tech industry has focused on this universal platform and its abomination of a scripting language to the expense of serious languages supporting native apps, and the total lack of adequate multi-platform targeting and adequately broad language support for cocoa, win32, gtk and qt - ironically most of the time it's either electron, putting the front end in a browser engine, or some kind of simple and fairly inadequate immediate mode or similar direct opengl/vulkan based thing (eg imgui, egui, gio, fyne, nucular) 

 Maxim already implemented it in a weekend.  

https://github.com/renostr/ 

 Don’t tempt me!

If you want it, we’ll do it. CBOR. 

 Not sure about this one. JSON has a lot of advantages, size is negligible compared to media files people transfer all the time.

What relays should provide, next to websockets is a plain HTTP API.

This lowers the bar even more for new nostr  developers and makes things easier for prototype and lots of no-realtime apps (not having to handle closing sockets, reconnection logic, etc).

You can have both served in the same path on the same process. Imo this should've been part of the spec, it might be too late

nostr:note142wvcpyyla0zv4lpnfnaj00yvx5xyh3edj0cxsuwgme83ldeky3sfvva7r 

 constrained 

 Fields like event id, sig, pubkey, p & e tags would be 50% smaller in binary vs utf-8 hex. Follow lists are currently quite large and their size would be basically cut in half.

But if we implemented NIP-114, we could save even 90% or more in bandwidth and some processing as well, depending on how many relays you connect to. Hoping to find some time for it again. https://github.com/nostr-protocol/nips/pull/1027 

 CBOR + NIP-114 + Negentropy https://i.nostr.build/9Ib46NS1T1To4mlN.jpg  

 Please don’t merge it. 

Plain simple json has a big advantage for specifications.

Bitcoin is still near to not understandable because it’s all binary protocol.

You could simple **gzip** the request using standard http request and it’s all fine. 

 Oh yes please. 

#nostr is a data hog (which I just found out). I normally use unlimited data plans with extremely generous data allowances.

Will have to adjust if I do travel to Australia with shitty internet.

nostr:note142wvcpyyla0zv4lpnfnaj00yvx5xyh3edj0cxsuwgme83ldeky3sfvva7r  

 JSON should be required, but there's nothing wrong alternative encodings 

 Could just use protobuf as a drop in alternative? 

 CBOR, protobuf 🤮  

 Is running both the default setting in strfry?  

 Why make it a core part of a relay instead of an optional service?

I could spin up a relay with, say, a gRPC service attached to it.  JSON is free, gRPC (which is binary over HTTP/2) is for paying clients.  Now I've financed my relay and given a high speed, low-overhead option to the people who want it. 

 and also, nostr 2026 clients that would use it and a pony. 

 Wtf is gRPC? Binary is binary 

 https://grpc.io/

It's a well-reputed RPC framework designed for fast inter-service communication. 

 I don't believe you or know what RPC is to understand you in the first place 

 Remote Procedure Call (RPC) is a protocol by which one bit of code can invoke functions on a different program, even if that program is running on a different machine, as if it was just another locally defined function.  There are a few flavors of RPC, but gRPC is the most common.  It uses a language called protobuf to define the inputs and outputs of each function, then uses that definition to automatically generate client code by which a program can invoke that remote procedure.

TL;DR—I could write code on my laptop that seamlessly uses other code living on a server in a data center hundreds of miles away.

In this scenario, gRPC encodes the messages between my computer and the remote server in binary, and transmits up to several at a time to maintain fast processing speeds. 

 I don't care. You said something about it being for paying clients. I'm not paying anyone to use binary. When there's a good version of nostr, it won't use json for shit that should be encoded in binary, and I still won't be listening to anyone who pretends I'm supposed to pay them for using binary. 

 Have you ever heard of noSQL databases? 

 No and I don't care since SQL is also not binary 

 All nostr needs at its core is binary + unicode + udp or another method of transfer

Anyone who doesn't understand this is a hecker 

 I'd love to see your implementation of this concept! 

 You're looking for someone else then. Criticizing the work of others isn't the same as actually joining in the work oneself. I don't know if I'm on your side since you're a human and humans are the reason I don't know if Digit is safe. 

 I brought up UDP, once, and everyone thought I was just being retarded. Which I probably was. 😂 

 You weren't imo. UDP is cool

TCP and other stuff works fine tho. Anything that works, works 

 One more note - web browsers and existing web protocol are bad, hence the need for nostr, hence how a big flaw in nostr's design is trying to have compatibility with outdated shit at the core of the protocol instead of building the core of the protocol to be future-ready while leaving everything web-browser-compatible to be additions or extensions built atop the core protocol 

 You don't have to pay anyone for anything, on Nostr, as you can run your own or find someone who will let you use theirs for free.

Irrelevant to the point about gRPC. 

 Scroll up. gRPC is the irrelevant thing here, that's what I've been trying to tell the guy above the whole time. 1s and 0s are called binary, I don't need to hear the term "gRPC" to discuss how nostr shouldn't use javascript for things that should be binary.

Again though, always nice to see you reply 

 It's Comp Sci 301 type stuff. Very common. 

 I am too lazy to understand. Nice to see you replying here though 

 Relays should stop spam and child porn first  

 Bluesky user type reply 

 The people who don't understand you are suffering from laziness, specifically a phenomenon I call "inappropriately human-optimized code." They're fixated on treating machine language like human language and trying to make the digital ecosystem do the same, like they don't know what a computer is.

https://wikifreedia.xyz/inappropriately-human-optimized-code/npub1wamvxt2tr50ghu4fdw47ksadnt0p277nv0vfhplmv0n0z3243zyq26u3l2