Oddbean new post about | logout

Notes by mleku | export

 #devstr #progressreport w00t, i've now got the queries bumping their last access time counters in #realy

now i can add back the garbage collector... first the counter, which collects all of the events, their data size (including indexes), and last access time, this list then has a sorter, which sorts them by their access time

then the mark function, which then goes in ascending order (oldest) and selects the list of events that exceed the low water mark target (the GC triggers when the count exceeds the high water mark, and we mark the ones that would bring it to the first event under the low water mark)

then finally with all of the events selected that we are going to delete, we run the "sweep" function which deletes the events that are the least recently accessed that give us our low water mark target, and voila

this is IMO an essential feature to put on a database that can potentially grow very large, because there really is no other way to deal with containing the event database size to the available storage on your server without this, IMHO this is a mandatory feature for a relay, but afaik no other relay (except my buggy previous version, replicatr) actually has this... 

i could be wrong... 

probably many other database drivers have access time marking but idk if anyone has bothered to do this because the most popular relay, strfry, is used by hipster damus boi who resolves this issue by his periodic nukenings

realy will have its own periodic nukening automatically so you don't have this issue of hot data going dark like a nukening causes 
 yay, count function done and dusted... next! 
 so Coracle stopped loading on Chrome for the last couple days. 

i finally tried deleting from hi... 
 must have been a bug in there somewhere 
 reminds me of the ones with the flies and the birds disrupting speeches also

God Has Spoken Their Doom 
 #realy #progressreport

it appears that i have finally squashed all the most salient causes of forever loops in my event query code

i have also implemented a separate count function that avoids decoding events if it can, so it performs better, it returns an approximate flag if it finds replaceable events because it is planned to not actually delete them, but i haven't written the post-processing in the query to enable this or removed the delete from replacement on event save (coming soon, on request from my shadowy super-nostr-dev friend who needs this for his work)

amethyst's endless stream of replaceable events helped expose more of the issues in the query and save for replaceable events, which was an edge case that neither nostrudel nor coracle triggered, this also now does use a bit of memory when it happens but it's quickly freed within the following minute (on more constrained hardware with less memory - eg 1gb - this might cause a brief spike of CPU usage to manage the more aggressive garbage collection - nothing i can really do about that...)

realy now uses 1-2gb of memory most of the time now, closer to 1gb

i think it's getting very close to release candidate, which will be a minor version bump to v1.1.x

there has been quite a few breaking changes, but i don't think anyone is importing it yet so 😅 whew

when it's bumped to v1.1.x i'm going to probably deploy it remotely on my VPS server

ah yes, i have other work to do today but the other major features that will be part of the v1.1.x will be a working garbage collector, a framework for shared/remote second layer event stores (even algaefs or blossom potentially could be implemented) and i also want to get around to making a badger database driver for btcd... too many things to do

better check my calendar and make sure i'm not slacking on someone haha 
 yeah it has the vibe of a prison entry hall... the one in Sofia has a similar vaulted ceiling in the center 
 is it the one in southwest Amsterdam? i forget the name of it... i stayed there when they used it as the winteropvang... 

yeah that's the thing too... prison populations in most of the west are mostly pot smokers, and a lot of them get worse because of that... in the netherlands they de facto don't put people in prison for that, and that meant a lot of prisons were unnecessary 
 this is what an OMAD meal looks like

it's a bit weird to put that much in one go but you seriously barely want for anything for the rest of the day

works best if it's really nicely fatty beef like Angus or Wagyu properly finished 18-24 months, the fat contains some 30% stearic acid which is a key precursor to a hormone that tells you you are not hungry 
 yeah, i think that i probably want to try OMAD again in the future... just gotta get a good source of beef lol

nothing makes you feel satisfied or gives you the energy to go all day long like around 100-120g of animal fat 
 that's not a given considering those shitcoins don't have instant payments 
 This is probably not a new idea but why not allow a fee to start a DM via Nostr, or allow users t... 
 IMO just block messages at the relay level (paid relay with auth) unless the user follows them... IMO, out of the blue DMs are never honest, it's always fine if you have already followed them

possibly web of trust could open that a little wider by possibly allowing follows of your follows to message you as well

i'm thinking about how to do this, for now it's not implemented

IMO it's better to control this traffic at the relay level than the client level

i also intend to make it so automatically queries have muted npubs removed from results since client devs seem to be loath to allow users to completely hide muted events 
 haha, there's no way the numbers at each point are correct, you can see in the very last frame on the bottom left what looks to me like Madeira and i seriously doubt i'm the only one running a node on this island haha

i would guess there's more like 20 bitcoin nodes on this island at least 
 I had a cat die from paralysis tick... Horrible creatures... You have to be in dry and high places to avoid most of these pesty bugs. 
 most of it is desert. only the right edge and a little of the right bottom are green and the bottom left tip are green, and some of that bit on the top left

the land has been above the water for longer than most of the rest of the world... the only volcanic parts are what i just described, or if they aren't volcanic, they have a lot of humidity blowing in from the sea

i think that there is good reason to suspect that not just australia but also antarctica were face forward at the time a massive X/gamma/cosmic/etc blast from a nearby supernova went off, and the whole place was sterilised and just turned into dust and eventually blew away

idk where the idea came from exactly but something like 40k years ago this likely happened

the main point is that there is very little carbon on the land, at all... 

and maybe after it got sterilised maybe it moved way south so it was out of the way of the big flows of water which might have helped plants colonise the land, but that never happened in the last 10k years or so 
 so, bitcoin is now gonna go through some serious bug-thrashing

i hope we are ready

i think we are

maybe the feature creeps will pull their heads in after a thousand asshats try to build L2s and tokens on bitcoin 
 yeah, blockstream was the first

i can't even

people really don't seem to understand the difference between money and a security

oh well, in 4 years time they might start to have more idea
 
 or everything 
 luddites are commies

this shit is just doomers making bullshit stories about how the world is worse because the perfect golden age is past and everything is falling apart

what's hilarious about these numbskulls is they are basically nihilists. let's say 'nihilist lite'

this is what all people that think that everything is going downhill and nothing is going up 
 i was wondering at first because of the small size of the gif if that was actually hillary 
 Well well well, looks like the pet rock is turning into a pet boulder 
 pet snowball 
 and then pet apocalypse 
 Being an iOS user feels like being a second-class citizen.

Time to replace my lost Android... 
 i'm getting one android to just do my banking and occasionally do navigation with all radio off in both cases

my "degoogled" phone i already have it, it has a screen with like 2 rows of 12 characters, old school LED array, i forget what they are called, too long ago i first had one (before AMOLED). it's about the size of your thumb and can only do voice and sms

i just have been trying to figure out how to arrange it... the main problem actually has been the way these batteries are built in and when you leave them plugged in all the time, they literally inflate with hydrogen and are at risk of catching fire

so i'm rooting the bank terminal device, and installing a battery charge regulation app that keeps it from ever going over 80% and that should prevent the thing swelling... and the dumb bank gets their small task of maintaining their one terminal system, so i am relegating it to not being my phone at all, just a dumb brick in the corner sitting on top of my computer case 
 just hoping the bank app won't complain the system is rooted, i had some other stuff complain about that

may have to get a dongle that offloads that from the device by reading the device charge level and actually, you know, doing what i want it to,ie, not blowing up with hydrogen lol 
 also, yeah, i'm gonna go through my gmail, what few things i have tied to it, and move them all to proton, and yeah, i'll be off google completely then, except for the unavoidable stuff i can't remove from the bank terminal brick

will be nice, just thinking about it, how much i would like to not feed google anything

gonna have to wean myself off translate (use about every other day) and maps (about once a week)

only google related thing i will use is Go which is fine because it's actually good enough they aren't running a persuasion bullshit campaign to get hipsters to fawn over it like Rust and Mozilla 
 Caveman here reporting for duty.

I will likely have to keep play services on it after installing the xapo app but I will have root so I should be able to remove everything else. And yeah just osmand, for the rare navigation requirement...

I can't see myself ditching it completely from desktop but I can stop using mail and calendar, maps on desktop I use maybe 2x a month, translate maybe 5

Yeah, but anyway fuck google. 
 Theres a limit to how many nostr pubkeys you can follow.  and its 928. 👀 
 most likely because that's over 64kb... strfry... i'm lovin it

khatru and relayer crap out over 500kb 
 they also sing very pretty songs 
 especially the ones with the red/orange chest 
 avocado is proof that #carnivore is wrong

#paleo, not carnivore or #keto 
 aaaavvoooo caaaadddooo 
 yeah, i think the "definitely bad" list is valid but the "eat only this" list is stupid

i can say that seed oils and artificial sweeteners definitely seem to cause magnesium, boron and potassium deficiency though

and this peripheral neuropathy... i think Jack Kruse is pretty much on point about all this stuff and he's not rabid carnivore or keto, his recommendations are mostly paleo and related to whatever the modern poisons have upset 
 yeah, here in europe maybe 75% of packaged/bottled drinks are contaminated... a very small number of them use stevia which is mostly ok but i think it's also a problem similar to liqorice 
 @satsmcd is here for pickle ball, shitposting, and engagement 

Unfortunately he hates 🥒 🥎 ... 
 devs are kooky, no deletes they say, it will be fun they say 
 nostr1.com is experiencing a partial unplanned outage. (and nostr21.com relay) most relays still ... 
 it is large because it's rather clever, but maybe too clever 
 probably not much because it basically offloads a heap of work from the app itself, one is the signer, one is the relay, one is notifications... they all sit there doing nothing until teh user pokes at them

oh yeah, pokey might be a bit of a power drain, the other to no 
 nostr:nprofile1qydhwumn8ghj7mn0wd68ytnnv96xsmmpwfjx2u3wvdhk6tcprfmhxue69uhkummnw3ezuurvv43xx6rpd9hzummjvuhszxmhwden5te0dehhxarj9ehhyctwvajhq6tvdshxgetk9uq3kamnwvaz7tmwdaehgu3wvdhkjmnxw4hxg6t59e3k7mf0qqszrhkdhjxqa2urdvnj8uu472amh96d7pp445wer7jnnnfn2q2mf4qz4m04d  they said glory!

nostr:nevent1qvzqqqqqqypzqtr9jsrjtwl3p0am754hd3qkqe65gsfxfac860vuc882hpkh8ltlqy2hwumn8ghj7un9d3shjtnyv9kh2uewd9hj7qghwaehxw309aex2mrp0yh8qunfd4skctnwv46z7qpq9ljach59dhw5z9n6udxgkf8z7js6wzjs9ulwzpa0ekhuv70vrfqqd6v7k7 
 https://cdn.satellite.earth/7da12f5e84f9d7b7b33eb124fe460f3ca3290794b4d9514fdbe925922f5b54d8.png

inspired by the most prolific cause of my relay to crash, nostr:nprofile1qy88wumn8ghj7mn0wvhxcmmv9uq3wamnwvaz7tmkd96x7u3wdehhxarjxyhxxmmd9uq3zamnwvaz7tmwdaehgu3wwa5kuef0qyt8wumn8ghj76rfwd6zumn0wd68ytnvv9hxgtcqyprqcf0xst760qet2tglytfay2e3wmvh9asdehpjztkceyh0s5r9c9xkzen   and his amethyst 
 i am not working on my relay any more tonight, that's it... that tears it

so i'll try putting my relay back on but i expect it to blow up memory usage again... maybe this time i will catch it at the right time to get an idea where in my code i have erred 
 no, amethyst crash realy

it's ok, it is my bug, so i will fix it, i'm glad to have another thing breaking my shit because probably otherwise it doesn't get tested

most likely it's just doing something reasonable that i didn't think of in my algorithm 
 i should write an orealy book with teh title

Bug Driven Development

subtitle:

How To Write Nostr Stuff, and turn all your hair gray

----

so, why am i posting this?

because i literally just open up amethyst, and it exposes bugs in my code for replaceable event handling, relay goes completely nuts and burns all the memory until it gets killed 
 #Bounty Alert:

100K sats for a great Coffee Shop name that is bitcoin/nostr related.

Name must ... 
 Ristretto Nakamoto 
 not sure if you considered this but running COUNT queries over windows of time that are short enough should let you estimate good limit boundaries in those time periods

it reminds me that i currently still just run a query which decodes the events in response COUNT... when in fact, the search first makes a set of indexes and then searches for them and returns the list of derived keys for the actual events. this number would be the answer.

you could even do bisection searches so at first you only query half each side of your window and then break it down from there to find the smallest window size that works and then you can paginate based on that (ie, it will never exceed a limit once you have that, unless somehow a lot of backdated events get injected)

pagination is always a two step even with more "sophisticated" database query systems anyway 
 one filter per sub... do you really need more than 80? khatru and relayer don't even have a limit for this (probably because Go can handle that shit anyway, supposedly) 
 yeah, amethyst is special

it's my new way to discover more bugs in my code

https://cdn.satellite.earth/7da12f5e84f9d7b7b33eb124fe460f3ca3290794b4d9514fdbe925922f5b54d8.png 
 nah, it's enough to just add to my relay list and open it up and then when everything explodes, hit the task manager and kill 
 diet cherry of course, to maintain his demon brain, since apparently all the presidents and whatnot seem to thrive on this shit 
 meh, client devs are the most prominent and least qualified

relay devs are like, can we count them on more than one hand at this point?

the influencoor/follower mindset has poisoned the dev process by putting too much energy where it has least effect for the business, i think you can say 
 yeah, maybe i'll get some funding to focus my work on this, with a long term of funding... it's a hard problem and you don't know whis until you actually try to build a relay and then watch your logs as the clients sodomize it 
 apparently my bitcoind leveldb corrupted *yet again* so I have to redownload the entire blockchai... 
 have fun when the filesystem itself needs to be accessed outside of the boot environment because 

for some reason, these BSD (and thus solaris) unixes aren't up to speed on making disk mounting and fixing easy, you know, mount... umount... fsck...

these tools won't help you with zfs and ufs

its probably the biggest reason why nobody uses BSD or solaris anymore

it's not intuitive at all 
 zfs is an abomination from Sun Microsystems Solaris OS which is based on BSD

BSD also has a shitty, similar thing that ZFS is an quote "improvement" on

you can't use mount, umount, or fsck... it's all some bullshit zsomething this zsomeotherthing that 
 no, what you need to do is stop the daemon, duplicate the state of it, and restart it, and if it fucks up you just zap the old version, copy back the new version, and wait an hour or two and it's back 
 i use badgerdb to store the actual events as values in keys in my database, but they are rarely over 500k... bitcoin blocks are 2-4mb typically, honestly, they should not be stored in the DB, the differential is the age old stacking problem of slivers and chunks that used to be a big problem with networks until about 10 years ago

if i was going to write a database driver for btcd i'd use badger for the indexes, and store the blocks in a flat filesystem named by the block hash... they are too big

or, it might work with badger, because badger actually stacks the values in one file and the keys in another...

anyway, my point here is that the entire filesystem of bitcoin's leveldb mutates so much you can't really snapshot it properly, they randomly change half the dataset for whatever reason and when you use `cp` with `-rfvpu` which retains the perms and only copies files that have been changed... it still copies all of the files because all of them have been changed so, yeah

it's dumb because bitcoin values are mostly just the blocks, and the indexes are mostly just keys, so having them separated would actually make backup time-effective instead of a collossal pain in the ass 
 i use badgerdb to store the actual events as values in keys in my database, but they are rarely over 500k... bitcoin blocks are 2-4mb typically, honestly, they should not be stored in the DB, the differential is the age old stacking problem of slivers and chunks that used to be a big problem with networks until about 10 years ago

if i was going to write a database driver for btcd i'd use badger for the indexes, and store the blocks in a flat filesystem named by the block hash... they are too big

or, it might work with badger, because badger actually stacks the values in one file and the keys in another...

anyway, my point here is that the entire filesystem of bitcoin's leveldb mutates so much you can't really snapshot it properly, they randomly change half the dataset for whatever reason and when you use `cp` with `-rfvpu` which retains the perms and only copies files that have been changed... it still copies all of the files because all of them have been changed so, yeah

it's dumb because bitcoin values are mostly just the blocks, and the indexes are mostly just keys, so having them separated would actually make backup time-effective instead of a collossal pain in the ass 
 lol, the events are stored in values, the indexes in keys

and yes, this would instantly solve the problem of bitcoin

if i had any time spare i'd make a PR to create a btcd database driver with badger because it would probably fix most of its slow IBD 
 so, idk how it was working yesterday, but now mysteriously i have a bug in tags

tags are not either being saved or loaded

since they appear to print correctly at the point of saving in the logs, marshaling what i expect, it must be in the binary codec either encoding the tags to store or decoding them from the store

and since the problem just appeared, i'm guessing it's decoding but i gotta check this. it's breaking my access control system, although never mind it's not retrieving the events correctly either

#devstr 
 yes, i forked all the things... and i got sick of khatru so i forked relayer, realyer was much easier to deal with, i've made more progress in 3 months than i made in 6 on khatru 
 OK so, first try with nostr:
I'm using "gossip" (github.com/mikedilger/gossip) because it's open ... 
 you just need to figure out the relay configuration

it's too obfuscated for me... gossip is written in Rust so i'm not touching it, and i don't understand how other people figure out how to get it working, it seriously is a several hour episode the one time i got it sorta working, i personally feel that people who use languages like C++ and Rust have their models of reality warped in a way that is difficult for primitive natural minds to decode the output from 
 oh yeah, you could also use wireguard and host it on a VPS only accessible over wireguard 
 well, even you could run it on your own home server, if it has a wireguard tunnel to a VPS and you would then be able to access it away from home too 
 i seriously doubt mike's client can cope with two instances writing to the app DB... that stuff will just be in the browser cache if you run a custom instance of coracle or nostrudel 
 have fun causing race conditions

welcome to the real world of concurrenc networks 
 yeah, this is also one of the big hassles with web apps is the web browser doesn't easily let you connect the app to a local cache relay... you could then also connect that cache to wireguard and use the same cache on your other devices as well

to do shared state like this you have to build a separate configuration/cache service 
 nostr:npub1ye5ptcxfyyxl5vjvdjar2ua3f0hynkjzpx552mu5snj3qmx5pzjscpknpr does Nostrudel ever use the... 
 Yeah I have seen this in next also, puts the wrong display name and avatar on posts sometimes 
 yeah it first appeared in yesterday's update i think 
 you might like Suspicious Observers... there is growing amounts of science papers about the influence and correlations between solar activity and weather, and he is constantly bumping new found papers talking about new data relating to this... i'm a bit addicted to the excitement of "wen next big CME and aurora" these days because they are clearly going to be fairly frequent for the next year or two, we have already had three major low latitude auroras 
 I recently to learned that you can wire the DC output of a solar charge controller directly to th... 
 So as long as your batteries are sufficient voltage I guess? 
 probably just need a voltage triggered off switch that just turns all that side of the circuit off once it drops below the voltage the devices need 
 👀

nostr:nevent1qqs2ah52eu26yeh8a2dpz2lahhl8t7gk56vxnfv72rqr9jxnvcnu46spzamhxue69uhhyetvv9ujuu... 
 scammers must be afraid of me, i get like maybe one a month 
 So, after reading the last few chapters of the Book  of Judges, It starts going off the rails aft... 
 i think it was the catholic orthodoxy who invented the "word of god" meme and it was always before that understood that humans wrote it, from their own experiences or what they heard from someone

this meme of authority totally poisons the whole thing, it makes nonbelievers justifiably incredulous