Oddbean new post about | logout
▲ ▼
sommerfeld | 6 months ago (raw) | root | parent | reply | flag 
 I'm curious if you store all the notes/events with their content or you just keep track of their ID for stat purposes. If the former, I'm curious of your storage needs since it would mean storing every note ever published.
Do you crawl a fixed set of relays? Does it update constantly in real time automatically? Or do you run it once every X amount of time?
▲ ▼
brugeman | 6 months ago (raw) | root | parent | reply | flag +1 
 We collect all events from all relays for search (only standard kinds), so yes we store everything ever published, that's at an order of 1Tb of unique events atm. Stats only show the last 2/6 months, but they have to process everything to be correct. Stats are real time, it's a process that keeps all numbers in RAM and handles incoming events as they come.
▲ ▼
sommerfeld | 6 months ago (raw) | root | parent | reply | flag 
 Very impressive. Why store the note content though?
▲ ▼
brugeman | 6 months ago (raw) | root | parent | reply | flag 
 I need to serve raw data from our relay. Content isn't needed for stats atm, but it's stored.