Oddbean new post about login | logout because customers are charged so much for crawler traffic that they're leaving the platform. And also that if anyone wants to change it, it's trivial to change the defaults and Discourse has a lot of weight to throw around because people usually don't change defaults. Bing ended up fixing it because they were the adults in the room. When people pointed out that Discourse's change would cause problems for people, leadership responded with nonsensical angry comments: https://meta.discourse.org/t/handling-bingbot/84659/22.
Why not just not charge people for a trivial amount of crawler traffic? If the backend is decently fast, it should basically be free to eat the load and not charge customers for it (and yes, I know the 0.5 QPS was for one site and it will add up; even so). My CDN doesn't charge me for getting DDoS'd because that traffic isn't really "my fault" and that could easily be orders of magnitude more traffic than all crawler traffic for all crawlers across every discourse instance.
Anyway, the same thing that fundamentally concerns me about the Bing issue is what concerns me about the responses from the founder about Android performance where, in response to comments about performance being bad on a lot of Android phones, the founder has ranted, multiple times, about how the real problem is that Qualcomm engineers are terrible and deserve to lose their jobs. I did a quick Bing search to find some examples and he's still doing this in 2023. https://files.mastodon.social/media_attachments/files/111/064/900/759/469/311/original/9ef698ab7ab69bee.png
The Apple CPU org has, along with DEC in the StrongARM & Alpha era, one of the best executing CPU design teams ever assembled (and not coincidentally, they share a lineage via DEC->SiByte->PA Semi->Apple) and Qualcomm is at, what, maybe 75%-95% of the single-core performance? If only managing to get to 75% to 95% of the performance of an all-time-great team makes you a clown you deserves to lose their job, man, I have news for you. Of course I don't agree with this —
I've repeatedly pushed back against Casey Muratori's rants about how the developers of X are morons because they don't produce code that's as fast his Casey's code. There are a lot of reasonable reasons that people produce slow code. But if you're ranting about how people who aren't quite capable of keeping up with not just a world class team, but an all time great team, should lose their jobs while you're producing slowest-in-class software, like, come on dude.
Yeah, the Bing issue is fixed because Bing came from the Microsoft lineage/culture of fixing issues that impact end users even if the problem is obviously a technical or product issue that comes from outside of Microsoft. But in other cases, the "vendor" being ranted at either can't or won't fix the issue and end user gets stuck with the problem, although they do get some nice rants about how Qualcomm engineers deserve to lose their jobs, so at least there's that.
BTW, the rants from Discourse leadership about how Qualcomm engineers are clowns who deserve to lose their jobs for not keeping up with Apple are a red herring. I find Discourse unusably slow on fast machines, like the massive workstation I used to do log analysis across all Twitter logs. Discourse hijacks ctrl+F and its custom search is too slow to use on a super fast dev box. Even if Qualcomm produced world-class workstation performance on mobile, that wouldn't be fast enough for Discourse.
I got a response from a co-founder and CTO of Discourse. Cool Sure, everyone has a threshold for how much traffic is an attack. For easily cached traffic like this, maybe I'd start wondering about it at 10k QPS? 100k? If you think the threshold should be 0-1 QPS, fine But how do you reconcile this with the other founder's opinion that engineers who produce 75% of the performance of an all-time-great performance team are "brutally bad", "terrible at their jobs", and deserve to lose their jobs? https://files.mastodon.social/media_attachments/files/111/065/633/224/370/913/original/ead2b9cd9eea03a9.png
@ed709062 Adding caching should have a very high threshold. Almost none of the caches at Stack Overflow ended up being worth it (I did the analysis and we yanked them out). Adding caching is behind like all the other answers if the problem is one badly behaving bot.
@25b1da2f As I said, if someone wants to have a 100k times lower threshold for what's ok, that's fine by me. My core question is how does one reconcile that with the opinion that engineers who produce 75% of humanly achievable performance "brutally bad", "terrible at their jobs", and deserve to lose their jobs? It's a bit incongruous for leadership to trash people so harshly for getting moderately close to cutting edge perf when their own product is many orders of magnitude away from that.
@ed709062 I also wouldn't add the caching if bing were hitting me incorrectly at a rate 100,000X higher than stated. And the core answer is I don't get hung up on obvious hyperbole. Jeff's run his mouth about things on the internet for decades. He pokes the bears he wants. Sometimes (somehow) it makes things better for people on the internet. Many other times it misses (and it has missed me).
@25b1da2f I think the absolute rate matters here. The two numbers that have been stated as too much / attack-level numbers are 0.006 QPS and 0.5 QPS. I wouldn't add a cache for that level of traffic either. I'm just saying it's trivial to serve that much traffic and I'd expect a site that could be static, like a forum, would serve crawlers out of a CDN and not even notice the traffic even at 10k x that level. For the other part, I have multiple Discourse folks and the CTO in my mentions,
@25b1da2f complaining this is "unfair", "nonsense", etc., when Jeff and Sam both put out stuff that's more hostile and, IMO, much more unfair as well. In the original threads on the Bing they, they're just having tantrums. IMO, if they don't want people pointing out this out, maybe consider they shouldn't have their co-founders go around the internet, trashing things they don't like a super hostile fashion, especially when they bill themselves as a company that creates "civilized discourse"?
@25b1da2f In general, I'm not a fan of the missing stair / "that's just Bob" defense. I don't think Jeff making a lot of statements that are equivalent to his statements that Qualcomm engineers are "terrible at their jobs" makes this better or should make Discourse more immune to criticism. It's Jeff's prerogative to use his huge platform to shit on people doing work he doesn't like, but it shouldn't surprise employees that his company is going to get splashed with some shit as a result.
@25b1da2f And if a founder goes around loudly calling engineers terrible an publicly hoping they lose their jobs for not hitting absolute cutting edge perf while their company produces a product that famously has worst-in-class performance, I don't think it should come as a surprise when people pick on the performance. The same thing would happen if Casey Muratori or Jonathan Blow made a forum with searching-hijacking javascript that's too slow to use even on a massive dev workstation.
@25b1da2f Jeff seems unusually good at handling criticism in a lot of circumstances and I have a lot of respect for that, but if someone is going to spend years (coming up on a decade now) hostilely trashing engineers at a company for not quite achieving the best performance in the world, it seems quite funny to me when they then produce world leadingly slow software.
@ed709062 I agree with you that the irony is odious and unbecoming. And I hope my friend hears you. And, on the Qualcomm issue, they did eventually back up their trash and rewrote their javascript to please that processor. https://eviltrout.com/2016/02/25/fixing-android-performance.html The server side performance was a tragedy we all saw coming the day we heard what the stack was. At SO we turned a well performing language into greased lightning. Couldn’t be done on ruby.
@25b1da2f Fair enough. With respect to the Qualcomm trash talk, given that he's trashing engineers for being terrible for not hitting Apple-like perf, I would consider the trash talk to be backed up if Discourse had an implementation that was within a factor of two of what an all-time-great performance team could do, which isn't even close to the case. I've tried using forums that use Discourse after that 2016 blog post and still find search to be unusably slow with annoyingly slow loading.
@ed709062 5X improvement near the top of the stack doesn’t feel like something to sneeze at outright, and your line is fair given the gauntlet thrown, and I don’t envy any of the engineers who had to actually cash the checks.
@25b1da2f As someone who's worked for Twitter, I'm familiar with having a leader who writes checks the company can't cash. I'm not exactly a fan of that, but one thing that I liked about Twitter was that it had a culture where people felt comfortable publicly calling Jack out on bad comments, e.g., if Jack made a comment like the ones from Jeff or Sam that I've quoted, I'd expect @85f21341 and others to publicly call him out on that and for Jack to take it well (at least while he was still CEO).
@ed709062 sorry but this is absolute nonsense. When someone is running a DDoS attack on you, you block or throttle them, this is Internet basics. Crawling the same page 500 times a day is an attack, even you have a threshold, maybe its not 500, maybe its 5 million, I don't know.
@4958ec32 Sure, if I'm running on-prem, for things that are easily cached like crawlers, I might start worrying about at around 10k-100k QPS if I wasn't caching and much more if I was caching and you apparently worry about it in the single digit QPS range. How do you reconcile this with your co-founder's opinion that engineers who can only doing 75% of the performance of an all-time-great team are "brutally bad", "terrible at their jobs", and deserve to lose their jobs?
@ed709062 that post is from 2018, though, not 2023
@ed709062 also, remember Exynos? The bane of EU Samsung flagships, for the last few years always lagging behind Qualcomm even when both were using same core designs from ARM and same manufacturing process from... Samsung? To the point they pulled them back from their home market and ultimately from their flagships entirely? And it's not like Samsung is incompetent - Exynos is the direct descendant of (actually retroactively renamed to Exynos) Hummingbird SoCs which powered early Galaxy S phones and... iPhone 4 and everything else apple made with A4 chips. And the version for Galaxy phones was better than what Apple put in their phones! They just didn't manage to keep up that momentum. Qualcomm has for a long time managed to stay second to Apple when it came to mobile SoC designs, despite making many more designs (Apple was only really concerned with high-end mobile chips until they also started making high-end desktop chips) and fundamentally being much closer to the rest of the competition than Apple (using the same ARM core designs as them). If even being second-best makes you a clown I think we'll need a bigger circus considering the increasing competition in that market. Unless somehow Discourse works better on MediaTek Dimensity, Samsung Exynos, HiSilicon Kirin, Unisoc or JLQ SoCs I don't think it's fair to blame Qualcomm for anything here.
