Oddbean

Nostr events are transferred between clients and relays over WebSocket, but that requires servers to be online. A universal file format is needed to store events for backup and transfer between people and systems. [JSONL](https://jsonlines.org/) is a simple format that stores one JSON object per line. This is ideal for Nostr events, and is already compatible with a bunch of existing tools including [nak](https://github.com/fiatjaf/nak), [strfry](https://github.com/hoytech/strfry), and [Ditto](https://gitlab.com/soapbox-pub/ditto). It is memory-effecient because only one line needs to be processed at a time, and it is universally compatible because it's just Nostr events. ## Storing Events JSONL files should end in a `.jsonl` file extension. Each line should have a complete, valid Nostr event. ```sh # Example storing jsonl events with nak nak req wss://ditto.pub/relay > events.jsonl ``` ## Importing Events You can `cat` a JSONL file to stdin of various programs. ```sh # Example importing jsonl events with nak cat events.jsonl | nak event wss://ditto.pub/relay ``` ## Filtering Events Use the `jq` command to filter down an JSONL file. ```sh # Get only kind 0 events jq -c 'select(.kind == 0)' events.jsonl ``` You can filter before importing: ```sh # Get only kind 3 events then import them with nak jq -c 'select(.kind == 3)' events.jsonl | nak event wss://ditto.pub/relay ``` Or filter while exporting: ```sh # Get all events and then post-filter them nak req wss://ditto.pub/relay | jq -c 'select(.kind == 3)' > events.jsonl ``` Of course you can chain many shell commands together to do all sorts of useful things! ## Sharing Events JSONL events can be shared on cloud services, compressed, emailed, transferred with rsync, or any other method. Another powerful method is to host the file on an HTTP server, allowing users to `curl` it and pipe it directly into other programs. ## Suggestions to Relay and Client Developers Relay and client developers should allow users to export/import events from their software in JSONL format. For relays, since the public interface is often restricted, relay software should add direct import/export of jsonl. That way relay operators would be able to shell in and run a command to get events in jsonl, and to import from jsonl. Ideally these commands would use stdin and stdout so they can be piped to other commands for filtering or direct transfer. Clients should also offer an import/export screen inside the app in jsonl format. I would suggest a separate export of all "your" data, and another one for all cached data. ## Usage With Existing Tools Here are examples of how to use JSONL events with some existing tools. ### nak Change the relay to your desired relay. Try `nak help req` to see all the available filtering options. ```sh # Export nak req wss://ditto.pub/relay > events.jsonl # Import cat events.jsonl | nak event wss://ditto.pub/relay ``` ### strfry See strfry's [README](https://github.com/hoytech/strfry#importing-data) for more information. `--since` and `--until` flags are supported during export. ```sh # Export ./strfry export > events.jsonl # Import cat events.jsonl | ./strfry import ``` ### Ditto ```sh # Export deno task db:export > events.jsonl # Import cat events.jsonl | deno task db:import ```