Oddbean new post about login | logout How to not break 1/3 of the world's computer systems: 1) Do not apply updates to production without proper testing in a quality assurance environment first. 2) Never. And I mean never. Never install updates on a Friday. 3) Avoid Microsoft if applicable.
Perhaps they outsourced the QA dept...
Perhaps.
Not your QA department, not your QA.
https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/comment/ldw1yxe/ Sysadmins could not have prevented it.
Disagree.
By not deploying Crowdstrike?
Disable automatic updates policy and turn it back on after updates have proven valid and working in other environments. Never rely on vendors automatically updating your machines.
Seems like it affected several versions of the sensor (see the link I posted). There was no software update involved. It was in the channel file updates. I also don't admin Crowdstrike myself, so I don't know if that could have been prevented.
I'm not familiar with it either, just from what I read earlier. I now admin all RedHat Linux systems, with Oracle middleware running ERP systems and all of the infrastructure with those.
It’s looking like this was a failure of a rollout from crowdstrike, controlled by them and not administrators. Time to switch vendors for those who had it then. There are other great options out there to replace Crowdstrike but honestly, they could probably do the same level of bricking. It’s a controls issue. With great power comes great responsibility.
I am not familiar with this product, but from my reading this morning it sounds like a policy could have disabled automatic updates which could then hBe been turned on after tested in another environment. This mimics every other product I've used as well.
It auto updates so they can roll out things in near real time. Crowdstrike’s value prop is that they have telemetry from thousands of environments globally, they see shit before anyone else so they can remediate before customers even know there might be a problem. When you come from something like carbon black (which bluescreens shit everyday of the week) this is a win for security at scale. Obviously there is a tradeoff but people didn’t seem to consider it, until it materialised..
Word on the street is people were getting hit even with that turned off. It’s the pervasive issue with cloud products of trusting someone else with persistent admin privileges. https://image.nostr.build/016896d0b9a48988b8f3c83fe212bf0b7bdbda4942f96308ad75237108b519ca.jpg
Wow. That's fucking wild.
The fact that they can push kernel drivers at will without permission. insanity. 🤦♂️
The service is installed as admin. It can do whatever the hell it wants. This is standard for EDR agents.
It is installed as a kernel mode driver which is even higher privileges than a user mode admin. It isn't quite standard for EDR agents as there are a lot of solutions whose agents only use usermode hooking for their detections.
When I was surveying the landscape it was certainly true, but that was a while ago. Which are usermode only?
A lot are still usermode hooks. Some are mixed. For example, CarbonBlack has some kernel level drivers for file and network detections, but its prevention policies all operate as usermode hooks. https://github.com/Mr-Un1k0d3r/EDRs
Ya but we’re not just talking about detections. The agent for each of these still needs kernel admin to hook? So you’d have to abuse the deployed agent in most cases, which is too sophisticated for 99.999% of attacks, but the risk is still present, right?
No, you don't need kernel mode access to hook into API calls like NtReadVirtualMemory, NtOpenProcess, etc which are all API calls that exist in usermode space. Having usermode hooks certainly makes it easier for malware to thus unhook the security agent and avoid detection. So it is a trade off.
https://i.nostr.build/2QUsdprmYbl9S5GC.jpg
One of these days I'll try nixos. I just like elementary too much 😁
Why no test phases for updates and patching? So many heads should roll for this.
dont have Microsoft in my house, Apple OS, iOS, Linux Mint, Android.
