#devstr #progressreport w00t, i've now got the queries bumping their last access time counters in #realy
now i can add back the garbage collector... first the counter, which collects all of the events, their data size (including indexes), and last access time, this list then has a sorter, which sorts them by their access time
then the mark function, which then goes in ascending order (oldest) and selects the list of events that exceed the low water mark target (the GC triggers when the count exceeds the high water mark, and we mark the ones that would bring it to the first event under the low water mark)
then finally with all of the events selected that we are going to delete, we run the "sweep" function which deletes the events that are the least recently accessed that give us our low water mark target, and voila
this is IMO an essential feature to put on a database that can potentially grow very large, because there really is no other way to deal with containing the event database size to the available storage on your server without this, IMHO this is a mandatory feature for a relay, but afaik no other relay (except my buggy previous version, replicatr) actually has this...
i could be wrong...
probably many other database drivers have access time marking but idk if anyone has bothered to do this because the most popular relay, strfry, is used by hipster damus boi who resolves this issue by his periodic nukenings
realy will have its own periodic nukening automatically so you don't have this issue of hot data going dark like a nukening causes
#realy #progressreport
it appears that i have finally squashed all the most salient causes of forever loops in my event query code
i have also implemented a separate count function that avoids decoding events if it can, so it performs better, it returns an approximate flag if it finds replaceable events because it is planned to not actually delete them, but i haven't written the post-processing in the query to enable this or removed the delete from replacement on event save (coming soon, on request from my shadowy super-nostr-dev friend who needs this for his work)
amethyst's endless stream of replaceable events helped expose more of the issues in the query and save for replaceable events, which was an edge case that neither nostrudel nor coracle triggered, this also now does use a bit of memory when it happens but it's quickly freed within the following minute (on more constrained hardware with less memory - eg 1gb - this might cause a brief spike of CPU usage to manage the more aggressive garbage collection - nothing i can really do about that...)
realy now uses 1-2gb of memory most of the time now, closer to 1gb
i think it's getting very close to release candidate, which will be a minor version bump to v1.1.x
there has been quite a few breaking changes, but i don't think anyone is importing it yet so 😅 whew
when it's bumped to v1.1.x i'm going to probably deploy it remotely on my VPS server
ah yes, i have other work to do today but the other major features that will be part of the v1.1.x will be a working garbage collector, a framework for shared/remote second layer event stores (even algaefs or blossom potentially could be implemented) and i also want to get around to making a badger database driver for btcd... too many things to do
better check my calendar and make sure i'm not slacking on someone haha
is it the one in southwest Amsterdam? i forget the name of it... i stayed there when they used it as the winteropvang...
yeah that's the thing too... prison populations in most of the west are mostly pot smokers, and a lot of them get worse because of that... in the netherlands they de facto don't put people in prison for that, and that meant a lot of prisons were unnecessary
this is what an OMAD meal looks like
it's a bit weird to put that much in one go but you seriously barely want for anything for the rest of the day
works best if it's really nicely fatty beef like Angus or Wagyu properly finished 18-24 months, the fat contains some 30% stearic acid which is a key precursor to a hormone that tells you you are not hungry
yeah, i think that i probably want to try OMAD again in the future... just gotta get a good source of beef lol
nothing makes you feel satisfied or gives you the energy to go all day long like around 100-120g of animal fat
IMO just block messages at the relay level (paid relay with auth) unless the user follows them... IMO, out of the blue DMs are never honest, it's always fine if you have already followed them
possibly web of trust could open that a little wider by possibly allowing follows of your follows to message you as well
i'm thinking about how to do this, for now it's not implemented
IMO it's better to control this traffic at the relay level than the client level
i also intend to make it so automatically queries have muted npubs removed from results since client devs seem to be loath to allow users to completely hide muted events
haha, there's no way the numbers at each point are correct, you can see in the very last frame on the bottom left what looks to me like Madeira and i seriously doubt i'm the only one running a node on this island haha
i would guess there's more like 20 bitcoin nodes on this island at least
most of it is desert. only the right edge and a little of the right bottom are green and the bottom left tip are green, and some of that bit on the top left
the land has been above the water for longer than most of the rest of the world... the only volcanic parts are what i just described, or if they aren't volcanic, they have a lot of humidity blowing in from the sea
i think that there is good reason to suspect that not just australia but also antarctica were face forward at the time a massive X/gamma/cosmic/etc blast from a nearby supernova went off, and the whole place was sterilised and just turned into dust and eventually blew away
idk where the idea came from exactly but something like 40k years ago this likely happened
the main point is that there is very little carbon on the land, at all...
and maybe after it got sterilised maybe it moved way south so it was out of the way of the big flows of water which might have helped plants colonise the land, but that never happened in the last 10k years or so
so, bitcoin is now gonna go through some serious bug-thrashing
i hope we are ready
i think we are
maybe the feature creeps will pull their heads in after a thousand asshats try to build L2s and tokens on bitcoin
yeah, blockstream was the first
i can't even
people really don't seem to understand the difference between money and a security
oh well, in 4 years time they might start to have more idea
luddites are commies
this shit is just doomers making bullshit stories about how the world is worse because the perfect golden age is past and everything is falling apart
what's hilarious about these numbskulls is they are basically nihilists. let's say 'nihilist lite'
this is what all people that think that everything is going downhill and nothing is going up
i'm getting one android to just do my banking and occasionally do navigation with all radio off in both cases
my "degoogled" phone i already have it, it has a screen with like 2 rows of 12 characters, old school LED array, i forget what they are called, too long ago i first had one (before AMOLED). it's about the size of your thumb and can only do voice and sms
i just have been trying to figure out how to arrange it... the main problem actually has been the way these batteries are built in and when you leave them plugged in all the time, they literally inflate with hydrogen and are at risk of catching fire
so i'm rooting the bank terminal device, and installing a battery charge regulation app that keeps it from ever going over 80% and that should prevent the thing swelling... and the dumb bank gets their small task of maintaining their one terminal system, so i am relegating it to not being my phone at all, just a dumb brick in the corner sitting on top of my computer case
just hoping the bank app won't complain the system is rooted, i had some other stuff complain about that
may have to get a dongle that offloads that from the device by reading the device charge level and actually, you know, doing what i want it to,ie, not blowing up with hydrogen lol
also, yeah, i'm gonna go through my gmail, what few things i have tied to it, and move them all to proton, and yeah, i'll be off google completely then, except for the unavoidable stuff i can't remove from the bank terminal brick
will be nice, just thinking about it, how much i would like to not feed google anything
gonna have to wean myself off translate (use about every other day) and maps (about once a week)
only google related thing i will use is Go which is fine because it's actually good enough they aren't running a persuasion bullshit campaign to get hipsters to fawn over it like Rust and Mozilla
Caveman here reporting for duty.
I will likely have to keep play services on it after installing the xapo app but I will have root so I should be able to remove everything else. And yeah just osmand, for the rare navigation requirement...
I can't see myself ditching it completely from desktop but I can stop using mail and calendar, maps on desktop I use maybe 2x a month, translate maybe 5
Yeah, but anyway fuck google.
yeah, i think the "definitely bad" list is valid but the "eat only this" list is stupid
i can say that seed oils and artificial sweeteners definitely seem to cause magnesium, boron and potassium deficiency though
and this peripheral neuropathy... i think Jack Kruse is pretty much on point about all this stuff and he's not rabid carnivore or keto, his recommendations are mostly paleo and related to whatever the modern poisons have upset
yeah, here in europe maybe 75% of packaged/bottled drinks are contaminated... a very small number of them use stevia which is mostly ok but i think it's also a problem similar to liqorice
probably not much because it basically offloads a heap of work from the app itself, one is the signer, one is the relay, one is notifications... they all sit there doing nothing until teh user pokes at them
oh yeah, pokey might be a bit of a power drain, the other to no
nostr:nprofile1qydhwumn8ghj7mn0wd68ytnnv96xsmmpwfjx2u3wvdhk6tcprfmhxue69uhkummnw3ezuurvv43xx6rpd9hzummjvuhszxmhwden5te0dehhxarj9ehhyctwvajhq6tvdshxgetk9uq3kamnwvaz7tmwdaehgu3wvdhkjmnxw4hxg6t59e3k7mf0qqszrhkdhjxqa2urdvnj8uu472amh96d7pp445wer7jnnnfn2q2mf4qz4m04d they said glory!
nostr:nevent1qvzqqqqqqypzqtr9jsrjtwl3p0am754hd3qkqe65gsfxfac860vuc882hpkh8ltlqy2hwumn8ghj7un9d3shjtnyv9kh2uewd9hj7qghwaehxw309aex2mrp0yh8qunfd4skctnwv46z7qpq9ljach59dhw5z9n6udxgkf8z7js6wzjs9ulwzpa0ekhuv70vrfqqd6v7k7
i am not working on my relay any more tonight, that's it... that tears it
so i'll try putting my relay back on but i expect it to blow up memory usage again... maybe this time i will catch it at the right time to get an idea where in my code i have erred
no, amethyst crash realy
it's ok, it is my bug, so i will fix it, i'm glad to have another thing breaking my shit because probably otherwise it doesn't get tested
most likely it's just doing something reasonable that i didn't think of in my algorithm
i should write an orealy book with teh title
Bug Driven Development
subtitle:
How To Write Nostr Stuff, and turn all your hair gray
----
so, why am i posting this?
because i literally just open up amethyst, and it exposes bugs in my code for replaceable event handling, relay goes completely nuts and burns all the memory until it gets killed
not sure if you considered this but running COUNT queries over windows of time that are short enough should let you estimate good limit boundaries in those time periods
it reminds me that i currently still just run a query which decodes the events in response COUNT... when in fact, the search first makes a set of indexes and then searches for them and returns the list of derived keys for the actual events. this number would be the answer.
you could even do bisection searches so at first you only query half each side of your window and then break it down from there to find the smallest window size that works and then you can paginate based on that (ie, it will never exceed a limit once you have that, unless somehow a lot of backdated events get injected)
pagination is always a two step even with more "sophisticated" database query systems anyway
one filter per sub... do you really need more than 80? khatru and relayer don't even have a limit for this (probably because Go can handle that shit anyway, supposedly)
meh, client devs are the most prominent and least qualified
relay devs are like, can we count them on more than one hand at this point?
the influencoor/follower mindset has poisoned the dev process by putting too much energy where it has least effect for the business, i think you can say
yeah, maybe i'll get some funding to focus my work on this, with a long term of funding... it's a hard problem and you don't know whis until you actually try to build a relay and then watch your logs as the clients sodomize it
have fun when the filesystem itself needs to be accessed outside of the boot environment because
for some reason, these BSD (and thus solaris) unixes aren't up to speed on making disk mounting and fixing easy, you know, mount... umount... fsck...
these tools won't help you with zfs and ufs
its probably the biggest reason why nobody uses BSD or solaris anymore
it's not intuitive at all
zfs is an abomination from Sun Microsystems Solaris OS which is based on BSD
BSD also has a shitty, similar thing that ZFS is an quote "improvement" on
you can't use mount, umount, or fsck... it's all some bullshit zsomething this zsomeotherthing that
no, what you need to do is stop the daemon, duplicate the state of it, and restart it, and if it fucks up you just zap the old version, copy back the new version, and wait an hour or two and it's back
i use badgerdb to store the actual events as values in keys in my database, but they are rarely over 500k... bitcoin blocks are 2-4mb typically, honestly, they should not be stored in the DB, the differential is the age old stacking problem of slivers and chunks that used to be a big problem with networks until about 10 years ago
if i was going to write a database driver for btcd i'd use badger for the indexes, and store the blocks in a flat filesystem named by the block hash... they are too big
or, it might work with badger, because badger actually stacks the values in one file and the keys in another...
anyway, my point here is that the entire filesystem of bitcoin's leveldb mutates so much you can't really snapshot it properly, they randomly change half the dataset for whatever reason and when you use `cp` with `-rfvpu` which retains the perms and only copies files that have been changed... it still copies all of the files because all of them have been changed so, yeah
it's dumb because bitcoin values are mostly just the blocks, and the indexes are mostly just keys, so having them separated would actually make backup time-effective instead of a collossal pain in the ass
i use badgerdb to store the actual events as values in keys in my database, but they are rarely over 500k... bitcoin blocks are 2-4mb typically, honestly, they should not be stored in the DB, the differential is the age old stacking problem of slivers and chunks that used to be a big problem with networks until about 10 years ago
if i was going to write a database driver for btcd i'd use badger for the indexes, and store the blocks in a flat filesystem named by the block hash... they are too big
or, it might work with badger, because badger actually stacks the values in one file and the keys in another...
anyway, my point here is that the entire filesystem of bitcoin's leveldb mutates so much you can't really snapshot it properly, they randomly change half the dataset for whatever reason and when you use `cp` with `-rfvpu` which retains the perms and only copies files that have been changed... it still copies all of the files because all of them have been changed so, yeah
it's dumb because bitcoin values are mostly just the blocks, and the indexes are mostly just keys, so having them separated would actually make backup time-effective instead of a collossal pain in the ass
lol, the events are stored in values, the indexes in keys
and yes, this would instantly solve the problem of bitcoin
if i had any time spare i'd make a PR to create a btcd database driver with badger because it would probably fix most of its slow IBD
so, idk how it was working yesterday, but now mysteriously i have a bug in tags
tags are not either being saved or loaded
since they appear to print correctly at the point of saving in the logs, marshaling what i expect, it must be in the binary codec either encoding the tags to store or decoding them from the store
and since the problem just appeared, i'm guessing it's decoding but i gotta check this. it's breaking my access control system, although never mind it's not retrieving the events correctly either
#devstr
yes, i forked all the things... and i got sick of khatru so i forked relayer, realyer was much easier to deal with, i've made more progress in 3 months than i made in 6 on khatru
you just need to figure out the relay configuration
it's too obfuscated for me... gossip is written in Rust so i'm not touching it, and i don't understand how other people figure out how to get it working, it seriously is a several hour episode the one time i got it sorta working, i personally feel that people who use languages like C++ and Rust have their models of reality warped in a way that is difficult for primitive natural minds to decode the output from
i seriously doubt mike's client can cope with two instances writing to the app DB... that stuff will just be in the browser cache if you run a custom instance of coracle or nostrudel
yeah, this is also one of the big hassles with web apps is the web browser doesn't easily let you connect the app to a local cache relay... you could then also connect that cache to wireguard and use the same cache on your other devices as well
to do shared state like this you have to build a separate configuration/cache service
you might like Suspicious Observers... there is growing amounts of science papers about the influence and correlations between solar activity and weather, and he is constantly bumping new found papers talking about new data relating to this... i'm a bit addicted to the excitement of "wen next big CME and aurora" these days because they are clearly going to be fairly frequent for the next year or two, we have already had three major low latitude auroras
i think it was the catholic orthodoxy who invented the "word of god" meme and it was always before that understood that humans wrote it, from their own experiences or what they heard from someone
this meme of authority totally poisons the whole thing, it makes nonbelievers justifiably incredulous
Notes by mleku | export