a Relay Test Suite is in the works...

I've already got ideas for 94 tests and as I write these tests they expand out into even more tests.

I hoping that exposing relay behavior will drive relay software towards consistently supporting the same way of doing certain things, and that this will help nostr compatibility.

Tests include things like:
* Can the public write events?
* Can an authenticated person write somebody else's events?
* Does the relay respond to minimally constrained filters?
* Does limit 0 give an EOSE?
* When you write an event and get OK-true, can you read it directly back?
* Does limit behave properly when a filter has multiple pubkeys and multiple kinds?
* Does it require AUTH to read DMs and giftwraps?
* When submitting a replaceable event, are the prior events replaced?
* When submitting a replaceable event with newer ones already present, is it rejected or ignored?
* Does deletion of a replaceable event preserve the events that are newer than the deletion event?
* Does it prompt for auth initially?
* How large of an event can it handle?
* Does it verify signatures?
* How does it react to invalid UTF8?
* Does it handle all JSON escape codes? What about surrogate pairs?
* Does it preserve field order in events?
* Does it preserve fields in events not defined by the nostr spec?
* How many subscriptions does it allow simultaneously?
* Do events with negative created_at values work?  What about exponential notation? What about numbers that won't fit in a u64?

... that kind of thing.

If you want to help contribute, even if you just have an idea of what to test, come on over to https://github.com/mikedilger/relay-tester and give feedback, open an issue, or better yet submit a PR.

What about the other things I'm working on?

* Gossip is on pause while we wait for the UI developer to handle some personal affairs, then we will have annotations and NIP-17 DMs.
* Chorus is stable and upgraded to use Pocket
* Pocket has been completed and is stable too, but I haven't tried to make it useful outside of the scope of a relay yet.


Are you testing the relay tester right now? Because my relay keeps crashing. 😂


very interested to see this


Great work Deelguh


More than this I think we just need active relay development lol


Haha!  no


I'm available if you need relays to test. I've strfry, chorus, rnostr, replicatr, nostr-rs-relay... pick one! :) ⚡


This is amazing.


Very cool. I might use you then. I haven't tried setting up the other relays besides nostr-rs-relay and chorus.

FYI  The tester is designed to test a relay software implementation, not a live running relay, because:

 1. It might crash a live relay,
 2. It will put a lot of junk events into the relay, and
 3. it might get false results if it ends up finding events it didn't inject

But I'm sure someone will ignore that recommendation.


This "will be" amazing.

Those tests aren't written yet.  It only does 18 tests so far, the rest are placeholders, and I'm still doing major refactors which are much easier with less tests.


Yes, I'm doing a docker compose with all popular relays available to be tested (so, not used).
Also trying to compile relay-tester from your repo, I started and it seems huge! :D


I need a few weeks to write the bulk of the tests. You can play with it now if you want but it's only 18 pretty stupid tests so far.


Don't worry, love to play and faceplan-me a lots of Rust error message 😂


I want this asap, please :D


Que bien ✨🫡👍👍


:omgclap:


The fact that you have a list of things is already amazing!


I waited about a year for somebody else to do it and eventually got tired of waiting.  Then when I started thinking about tests I couldn't stop.  I couldn't sleep I kept getting up and writing down more test ideas.


I can say this is Nostr Implementation Possibilities for Relay Design.🫢


Binary build is ✅
First tests are ✅
Need now to build docker compose to allow ssl Relay to use wss and start first tests ✅


Building image is done ✅ I'm now able to rebuild the binary each time you'll update the cod and make new tests. Each relay will have SSL cert and tested with wss//.
Current tests on a strfry default config relay:

RESULTS:
nip11_provided: PASS
claimed_support_for_nip4: PASS
claimed_support_for_nip9: PASS
claimed_support_for_nip11: PASS
claimed_support_for_nip26: INFO (not supported)
claimed_support_for_nip29: INFO (not supported)
claimed_support_for_nip40: PASS
claimed_support_for_nip42: INFO (not supported)
claimed_support_for_nip45: INFO (not supported)
claimed_support_for_nip50: INFO (not supported)
claimed_support_for_nip59: INFO (not supported)
claimed_support_for_nip65: INFO (not supported)
claimed_support_for_nip70: UNTESTED
claimed_support_for_nip94: INFO (not supported)
claimed_support_for_nip96: INFO (not supported)
public_can_write: PASS
public_can_read_back: PASS
supports_eose: PASS
find_by_id: UNTESTED
find_by_pubkey_and_kind: UNTESTED
find_by_pubkey_and_tags: UNTESTED
find_by_kind_and_tags: UNTESTED
find_by_tags: UNTESTED
find_by_pubkey: UNTESTED
find_by_scrape: UNTESTED
find_replaceable_event: UNTESTED
find_parameterized_replaceable_event: UNTESTED
replaceable_event_removes_previous: UNTESTED
replaceable_event_doesnt_remove_future: UNTESTED
parameterized_replaceable_event_removes_previous: UNTESTED
parameterized_replaceable_event_doesnt_remove_future: UNTESTED
since_until_include_equals: UNTESTED
limit_zero: UNTESTED
event_always_gets_ok_reply: UNTESTED
auth_always_gets_ok_reply: UNTESTED
limit_works_across_multiple_filter_groups: UNTESTED
serves_post_eose_events: UNTESTED
no_timeout_while_subscribed: UNTESTED
nip4_dms_require_auth: UNTESTED
delete_by_id: UNTESTED
delete_by_id_of_others: UNTESTED
resubmission_of_deleted_by id: UNTESTED
delete_by_npnaddr: UNTESTED
delete_by_npnaddr_of_others: UNTESTED
delete_by_npnaddr_preserves_newer: UNTESTED
resubmission_of_deleted_by_npnaddr: UNTESTED
resubmission_of_olderthan_deleted_by_npnaddr: UNTESTED
resubmission_of_newerthan_deleted_by_npnaddr: UNTESTED
delete_by_pnaddr: UNTESTED
delete_by_pnaddr_of_others: UNTESTED
delete_by_pnaddr_preserves_newer: UNTESTED
delete_by_pnaddr_bound_by_dtag: UNTESTED
resubmission_of_deleted_by_pnaddr: UNTESTED
resubmission_of_olderthan_deleted_by_pnaddr: UNTESTED
resubmission_of_newerthan_deleted_by_pnaddr: UNTESTED
deleted_returns_ok_false: UNTESTED
prompts_for_auth_initially: FAIL
can_auth_as_unknown: UNTESTED
unknown_can_write_own: UNTESTED
unknown_can_readback_own: UNTESTED
unknown_can_write_other: UNTESTED
unknown_can_readback_other: UNTESTED
can_auth_as_known: UNTESTED
known_can_write_own: UNTESTED
known_can_readback_own: UNTESTED
known_can_write_other: UNTESTED
known_can_readback_other: UNTESTED
giftwraps_require_auth: UNTESTED
large_contact_lists: UNTESTED
preserves_json_field_order: UNTESTED
preserves_nonstandard_json_fields: UNTESTED
handles_event_kind_larger_than_16bit: UNTESTED
handles_filter_kind_larger_than_16bit: UNTESTED
handles_event_created_at_larger_than_64bit: UNTESTED
handles_filter_created_at_larger_than_64bit: UNTESTED
accepts_events_before_nostr_was_invented: UNTESTED
accepts_negative_event_created_at: UNTESTED
accepts_events_in_the_near_future: UNTESTED
accepts_events_in_the_distant_future: UNTESTED
accepts_negative_filter_created_at: UNTESTED
accepts_filter_created_at_in_near_future: UNTESTED
accepts_filter_created_at_in_distant_future: UNTESTED
handles_all_json_escape_codes: UNTESTED
handles_surrogate_pairs: UNTESTED
verifies_signatures: UNTESTED
accepts_invalid_utf8: UNTESTED
accepts_null_characters: UNTESTED
handles_filter_prefixes: UNTESTED
keeps_ephemeral_events: UNTESTED
max_subscriptions: UNTESTED
allows_immediate_reconnect: UNTESTED
idle_timeout_if_unsubscribed: UNTESTED
handles_empty_tags: UNTESTED


This is fucking brilliant.


Onward 🫡