How to not break 1/3 of the world's computer systems:

1) Do not apply updates to production without proper testing in a quality assurance environment first.
2) Never. And I mean never. Never install updates on a Friday.
3) Avoid Microsoft if applicable.


:balenod:


Perhaps they outsourced the QA dept...


Perhaps.


Not your QA department, not your QA.


https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/comment/ldw1yxe/
Sysadmins could not have prevented it.


bingo


next time


Disagree.


It’s looking like this was a failure of a rollout from crowdstrike, controlled by them and not administrators. Time to switch vendors for those who had it then.

There are other great options out there to replace Crowdstrike but honestly, they could probably do the same level of bricking. It’s a controls issue. With great power comes great responsibility.


By not deploying Crowdstrike?


I am not familiar with this product, but from my reading this morning it sounds like a policy could have disabled automatic updates which could then hBe been turned on after tested in another environment. This mimics every other product I've used as well.


# 3 fixes 100% of the problem to begin with 😅😅😅


Disable automatic updates policy and turn it back on after updates have proven valid and working in other environments. Never rely on vendors automatically updating your machines.


I don’t care if it’s YOUR Friday, just never install updates on MY Friday. Stupid timezones.


It auto updates so they can roll out things in near real time. Crowdstrike’s value prop is that they have telemetry from thousands of environments globally, they see shit before anyone else so they can remediate before customers even know there might be a problem. When you come from something like carbon black (which bluescreens shit everyday of the week) this is a win for security at scale.

Obviously there is a tradeoff but people didn’t seem to consider it, until it materialised..


Seems like it affected several versions of the sensor (see the link I posted). There was no software update involved. It was in the channel file updates. I also don't admin Crowdstrike myself, so I don't know if that could have been prevented.


I'm not familiar with it either, just from what I read earlier. I now admin all RedHat Linux systems, with Oracle middleware running ERP systems and all of the infrastructure with those.


One of these days I'll try nixos. I just like elementary too much 😁


Word on the street is people were getting hit even with that turned off.

It’s the pervasive issue with cloud products of trusting someone else with persistent admin privileges. https://image.nostr.build/016896d0b9a48988b8f3c83fe212bf0b7bdbda4942f96308ad75237108b519ca.jpg


It's very nice to have everything in a git repo to rebuild it all if you need to.


Why no test phases for updates and patching? So many heads should roll for this.


Wow. That's fucking wild.


The fact that they can push kernel drivers at will without permission. insanity. 🤦♂


The service is installed as admin. It can do whatever the hell it wants. This is standard for EDR agents.


That’s the point. Two sides of the coin.


Outsourcing to defender == USG red/white/blue team. Or private security via these alternatives.

It’s compelling & this is the trade off.


It is installed as a kernel mode driver which is even higher privileges than a user mode admin. It isn't quite standard for EDR agents as there are a lot of solutions whose agents only use usermode hooking for their detections.


When I was surveying the landscape it was certainly true, but that was a while ago. Which are usermode only?


A lot are still usermode hooks. Some are mixed. For example, CarbonBlack has some kernel level drivers for file and network detections, but its prevention policies all operate as usermode hooks.

https://github.com/Mr-Un1k0d3r/EDRs


How to not break 1/3 of the world's computer systems:

1) Do not apply updates to production without proper testing in a quality assurance environment first.
2) Never. And I mean never. Never install updates on a Friday.
3) Avoid Microsoft if applicable.


Ya but we’re not just talking about detections. The agent for each of these still needs kernel admin to hook? So you’d have to abuse the deployed agent in most cases, which is too sophisticated for 99.999% of attacks, but the risk is still present, right?


dont have Microsoft in my house, Apple OS, iOS, Linux Mint, Android.


No, you don't need kernel mode access to hook into API calls like NtReadVirtualMemory, NtOpenProcess, etc which are all API calls that exist in usermode space. Having usermode hooks certainly makes it easier for malware to thus unhook the security agent and avoid detection. So it is a trade off.


Ah, ok, now I follow. Ya. I don’t recall any usermode only agents on my radar in 2017, probably because of the lesser features. Was leaning heavily on the response part.