I asked claude about it, is it hallucinating?  ->

There seems to be a misunderstanding in that statement. Let me clarify how ECDH (Elliptic Curve Diffie-Hellman) key exchange works, particularly with respect to public key encoding:

In ECDH, the shared secret is computed based on one party's private key and the other party's public key. The result should be the same regardless of which party computes it.
The "02" or "03" prefix in a 33-byte compressed public key encoding doesn't affect the shared secret computation. Here's why:

The prefix (02 or 03) only indicates whether the y-coordinate of the public key point is even or odd.
When decompressing the public key, this prefix is used to reconstruct the full (x, y) coordinates of the point.
The actual ECDH computation uses the full (x, y) coordinates, not the compressed form.


The shared secret in ECDH is typically the x-coordinate of the resulting point from the scalar multiplication of one party's private key with the other party's public key point.
Whether you start with a "02" or "03" prefix, you'll end up with the same full (x, y) coordinates after decompression, and thus the same shared secret.

If you're experiencing different shared secrets based on the prefix, it could indicate an implementation error, possibly in the key decompression step or in the ECDH computation itself.
To ensure correct ECDH implementation:

Properly decompress the public key if it's in compressed format.
Use the full (x, y) coordinates for the ECDH computation.
Use only the x-coordinate of the resulting point as the shared secret.

If you're still encountering issues, it would be helpful to review the specific implementation or library you're using for ECDH computations. 

 it's 1 bit missing, claude is wrong

i literally was getting ~50% error on an ECDH test that used secrets that weren't checked for being even pubkeys

as soon as i made it check for 2 only, fixed 

 now i need to check if go-nostr keygen does it right.. thats typically what ive been using.. 

 spoiler alert: it doesn't check

i just never encountered this before, because it looks like most generators check for a 2 key and either invert it or reject it

it's not specified in either nip-04 or bip-340 except obliquely and not explicitly

you can't get around it, claude is wrong, the ECDH computation makes two different keys depending on whether it's an odd or even Y 

 huh, but it has the pubkey, why wouldnt it be able to know its even or odd when computing shared secret?  im missing something here.. 

 it's not possible for the sender to know the extra bit, they pick the wrong one, the receiver at best could send back a reply "wrong bit" 

 damn 🌋🌊 

 I think it's time to hopefully ask @fiatjaf and maybe @JeffG 

Something seems really wrong. This is nothing that's come up in my tests before. Were all missing something?? This is a huge deal! I beleive this would be part of an EC twist vulnerability... 

 CC @@Little Mikey D @Vitor Pamplona  

 i wasn't aware of what this exactly means, it's not a vulnerability, it's a bug that surely is killing nostr adoption because everyon expects DMs to work, but they obviously can't work without that extra bit

if my dm partners are mostly different sign keys to me, we can't use it 

 BIP-340 specifies choosing the 02.  Look at the section "Implicit Y coordinates" 

 so using code that doesn't check for it is wrong

go read some source code of nostr libraries again with regard to key generation

i'm not gonna pretend i checked the javascript or rust libraries, as i hate both of these languages but the Go libraries DO NOT CHECK

just write a short piece of code generating new keys and then use two of them to make an ECDH key both ways

then you'll know if the library is doing it right 

 Good luck improving the NIPs.  In another project, I've bene working on a JSON-LD schema for nostr profiles.  It might be possible to add the sign to list of fields, if you dont get your issue resolved quickly.

https://w3id.org/nostr 

 You'll never prevent 3's because of hardware wallets.  I really wish nostr had been the full 33 byte pubkeys. 

 that's why i'm betting on a sign flag in the kind 0 as the solution, nothing else will work 

 Got it.  I wanted this for years.   

 i love the x-only pubkey thing but it only dawned on me this last day that it breaks ecdh 

 I hate actually writing anything in the NIPs area any more.  Because everything they touch there's a good chance they will make it worse.  I am scared even mention the few decent NIPs in there in case they get ruined. 

 yeah, i feel that way too, but this ECDH and key generator thing needs to be made really loud because it is killing the most important thing after shitposting 

 You are right but the staggering incompetence is off the charts. 

 the documentation is the root of the weed tho 

 Yes, dont underestimate their ability to make it worse! 😉   

 Guess and check is not an option IMO, the nip needs to be updated. I know you guys have your issues with the repo stuff, but has an issue or a PR been submitted on the nips repo? I wan't to get moving on this. I don't think every man for himself is a good or safe option.  

 yeah, it really needs to be stipulated... in my library now the generator functions derive the compressed pubkey and retry if they get a 3

but at the same time we gotta do something about users that have them and don't want to make new ones 

 Very true. Although better now than later 

 yeah, exactly, the sooner all generators respect this the less strange bumps we get from users trying to use nostr functionality that requires ECDH to work 

 How are you detecting if decryption fails? Are you checking if the nip44 mac fails? Since nip04 has no method of verifying plaintext, how do you check on that? 

 i'm testing the ECDH on two keys, and as expected getting a 50% failure rate if i just use randomly generated secret keys and don't screen out 3s (could be 2s for all it matters)

just do a simple test - generate two keys, derive the secret in both directions while overwriting teh first byte of the pubkey with a 2 no matter what, then compare the shared secret result, and count how many times they don't generate the secret

spoiler alert: ~50% +/- 0.1% for 10,000 samples

it's a very simple test to write, i'm just busy, i bumped into it, oop bug, fix, onto other things, let the people know about it

i don't see any way around it

3 pubkeys don't have the same secret as otherwise identical 2 pubkeys 

 yes but how do you know the shared secret is correct?  

 secret A + pubkey B = secret B + pubkey A

this is the entire premise of Diffie Hellman Key Exchange

if that equation doesn't hold, then one side has something wrong with it, and is not as labeled 

 Yeah but I mean in the wild where you don't have secret B. 

 you get a different secret, and the equation is false 

 But there is no way of knowing if the shared secret is invalid unless you can have both sides of the equation, which we can't have.  

 It would be nice to have "3" keys in your kind=0 profile.  I have wanted that for years.  But years later we still dont have it.  And I dont think we will have it in future.  It would have been nicer if pubkeys were 33 bytes in the first place.  At this point I dont want anything to change, becasue the chance of breaking a working system is too high. 

 adding a flag to the kind 0 profile metadata is the only practical solution, as well as a strong stipulation about it both in the nip-01 and the - one that describes the kind 0 event

it has to be addressed, and once it's obvious to any implementer and all implementations used in the majority of cases are compliant the problem will disappear and nobody will have to actually put the optional flag in the kind 0

i have had problems with DMs myself, and this is a big obstacle for use of nostr DMs as a control mechanism as a way to communicate with customers especially, just ask nostr:nprofile1qy2hwumn8ghj7un9d3shjtnyv9kh2uewd9hj7qgwwaehxw309ahx7uewd3hkctcpz9mhxue69uhkummnw3ezuamfdejj7qg6waehxw309ahx7um5wgh8g6r9wdsk6etrv96zu6t09uq3kamnwvaz7tm5dpjkvmmjv4ehgtnwdaehgu339e3k7mf0qyf8wumn8ghj7mn0wd68yv339e3k7mf0qythwumn8ghj7un9d3shjtnwdaehgu3wvfskuep0qqs8eseg5zxak2hal8umuaa7laxgxjyll9uhyxp86c522shn9gj8crsy7rryg  the thing i'm saying is that i believe this problem has silently retarded nostr adoption 

 if you are able to extend kind=0 ill be impressed and happy, but im not going anywhere near it, because the chance of breaking the whole protocol is too high.  not alot of people have complained on this issue, so we can just use work arounds I think. 

 well, at least make a loud pronouncement about not making odd npubs lol 

 I think this might be an acceptable solution. I suppose it cuts the key-space in half though? 

 that was implicit and is mentioned in BIP-340

just keep in mind that as it was, the bit-strength of secp256k1 is about 126 effective bits, and this only shaves off half a bit

the permutations of those bits is an extra factor of two for every bit so it's not a linear relation, losing one bit is really losing 1/126th of the security

the chances of collision are doubled and the time to average brute force is half, yes, but we are talking about a process that would take hundreds of years for one key with all the computers in the world 

 no, i think that's a bad idea.  its doubling down on a bad design choice.  lots of things will and should produce odd npubs, like hardware wallets etc.  and there already are many users with '3', its not fair to invalidate, deprecate or cancel those accounts.  just let 2s be 2s and 3s be 3s.  and find work arounds such as kind=0.  in any case im not touching it, because the protocol might get even more broken by well-intended efforts to fix a bad original choice.  and we cant cancel half the user base, the system is already running... use workarounds! 

 i understand your point but the change i propose only affects two points of the system, where your proposal means a whole shitload of code has to be rewritten to accept 64 or 66 character pubkeys in the event, this is a larger compliance burden 

 To be clear.  I propose no change.  The protocol is working.  I propose dont change it, and dont touch it.  Find a NIP work around for one of the other NIPs that's fine.  Let 2s be 2s and 3s be 3s, because we already have a user base of 2s and 3s and cant change that now.  Extending profiles is a good way.  I actually think profiles should be extensible by design, in a permissionless way. 

 you are saying allow 66 character pubkey fields tho, or what? 

 No.  Let me be very clear now.  Dont change the protocol.  Let 3s exist on the network.  Dont touch it, dont break what we have.

I have no proposal.  Whatever workaround you want, is find by me.  Just dont change the protocol.  Im specifically not getting involved, because I dont want the protocol to change.

I'll just cross my fingers on that, tho! 

 ah, yes, so you are gonna do the same as what the rest of them do because of the survivorship bias of not seeing the users who left for reasons of not being able to use DMs, helpful. 

 it is the fault of the NIP maintainers also, btw, because BIP-340 clearly states this

but then again BIP-340 is horribly worded as well, it took me some time to figure a lot of it out... my fork of btcec has the extra 4 test vectors in the test suite that roasbeef refused to bother to complete, i even handed it to him on a platter 

 It's not that.  There's tons of ways to do a DM already.  They all work.  Cant really invalidate user accounts on an existing system, that prides itself on users not being able to be cancelled.  That would go against the marketing.  Devs will just have to find another way, of which there are multiple, and they'll figure something out. 

 yeah, i've been thinking about it, because i have part way done the tooling to manage a paid relay service using DMs and for the customer side we need to be able to tolerate anything, so there's things that can be done

one, the relay will never generate or use 3 keys, that's now fully fixed in my codebase

two, the initial messages with a user will send out assuming both 2 and 3, and each one will have a different string that the user is asked to paste and return, and this will confirm, and if 3 keys are discovered they will be stored in the database - probably just make a public event with a note saying "this key is an odd key" so others can discover this after that

i think that is enough measures

other than simply pointing it out to the nip guardians that their protocol uses ecdh without strict odd prohibition (as you say, you can either roll again for a 2 or you can modulo subtract the curve order G from the key and voila - considering this avoids needing to derive the key again i might need to revisit this option 

 user A gets secret A and user B gets secret B and they can't communicate

it's a 2x2 grid, two cases where A and B pubkeys are 3s you can't get the same shared secret

most generators already either reject 3s or subtract the curve order from the secret to get the opposite sign, but even with this in mind you still have teh problem that you have to try a second one to test if it's just the wrong one and that only gives you 25% of cases where they are wrong, in the other direction, where you guess the pubkey prefix/sign, it works

i have seen this in DMs before and it was infuriating, but i didn't test it enough, i'd generate a key, have problems, and without knowing what was going on, that key would be discarded and another one generated, and 50% of the time it would work and seem like the stars were out of alignment, but it's a bug in the protocol that needs to be strongly stipulated about the keys being ONLY 2 prefix, even pubkeys because of ECDH

the ways around it are difficult, the only solution i see is signalling optionally in kind 0 if you have an odd pubkey

that solves the problem

if two people have 3 keys and their software guesses 2s both sides can't communicate, otherwise, you have a case where one way works the other doesn't 

 I guess what I'm trying to say is, from a libraries' perspective, there is no way (unless the caller tells me) for me to know what the proper sign is. It will just happily decrypt garbage and return it to the user. I can't know if the decrypted data is correct. In nip44 we can check the MAC, but for nip04 you just have to hope for the best?? 

 all events have a MAC in fact, the presence of it in NIP-44 is redundant, that's what the event signature is

it doesn't have to authenticate on the plaintext after you decrypt it, if you already certified the ciphertext

this is another retarded element of nip-44 

 But I still can't know if the decrypted plaintext is correct. That's what I'm saying. The signature tells me nothing about the plaintext 

 yes, this is correct, you would have to have a sentinel to enable this, the first byte even it could be, or maybe better first 4 bytes to eliminate the chances of decrypting the same by both 

 also, yes, you don't need that bit for signature verification, that's one of the neat things about Schnorr signatures

but it does not apply to ECDH 

 two points though

one, having to decrypt the whole message and then discover you need to flip the bit is wasteful of computation and time

two, it still doesn't fix the problem of two 3 key users with software imputing 2 keys