Anyone else in IT awake tonight thanks to CrowdStrike BSOD meltdown? My company is in chaos. I see flights being grounded on CNN front page article, but they're not pointing at CS yet.

Yeah, started with reports of the call center being down and escalated quickly from there.

the Aussies pointed to them quite early.

https://www.abc.net.au/news/2024-07-19/global-it-outage-crowdstrike-microsoft-banks-airlines-australia/104119960

hey fam. i got your back:

CrowdStrikeがWindowsブルースクリーンの対処法を発表

1.セーフモードかWindows回復環境で起動
2.C:\Windows\System32\drivers\CrowdStrikeに移動
3.C-00000291*.sysにマッチするファイルを削除 ※アスタリスク部分は00000291abc.sysや00000291999.sysなどどんな文字にもマッチするという意味
4.起動 pic.twitter.com/FqhRb5nncW

— ロアネア@最多情報源バズニュース (@roaneatan) July 19, 2024

Well here's the culprit. Boot to safe mode and rename that file is the published fix.

Looks like my company is unaffected, at least the major systems.

Time for SentinelOne?

Is this a Microsoft pushed update?

Or do you have to have some connection to crowdstrike?

TexAg1987 said:

---

Is this a Microsoft pushed update?

Or do you have to have some connection to crowdstrike?

---

Not Microsoft. Only will affect you if you have the Crowdstrike Faclcon sensor installed. Unfortunately millions of people do...

We had a tech from CrowdStrike on a call with us and that was the fix they gave us, too.

Boot into safe mode
Navigate to %WINDIR%\System32\drivers\CrowdStrike\
Delete the file matching C-00000291.sys
Reboot

Lathspell said:

---

Time for SentinelOne?

---

We POC'd Crowdstrike several years ago but decided to go with SentinelOne. I recommend them.

We use Sentinel One, so we aren't affected by this issue, but we've had tons of issues with it over the years, too. Mostly performance related, and nothing to this extent, but there isn't a perfect solution.

Bitlocker + Crowdstrike = nightmare

We had Crowdstrike implementation in our next budget. Guess we're sticking with Trellix a little longer.

We went to CrowdStrike from Trellix (the former FireEye product, not the former McAfee). That solution really sucked though in regards to actual protection and resource use, became very evident once we rolled CrowdStrike out.

But yeah....bitlocker - nightmare.

Various custom supplied hardware with no safe-boot options - bigger nightmare.

Trying to troubleshoot it remotely over the phone with personnel with no IT experience - just shoot me.

Probably won't happen, but I'd like to know what exactly they did to bork things up this badly. Luckily not my silo, so I just get to sit back and watch the train wreck. Hardest part is not being able to script the fix. Manually doing a web console for VMs takes about 5-10 minutes per server to fix depending on reload/boot times. A few thousand servers can take a while.

Hopefully this helps one of y'all.

https://www.reddit.com/r/sysadmin/comments/1e708o0/fix_the_crowdstrike_boot_loopbsod_automatically/

Average Joe said:

---

Bitlocker + Crowdstrike = nightmare

We had Crowdstrike implementation in our next budget. Guess we're sticking with Trellix a little longer.

---

Yeah I've run into that on three machines so far. Not a showstopper you just have to log into the Microsoft site to get the recovery key. But it's time-consuming.

Tailgate88 said:

---

Average Joe said:

---

Bitlocker + Crowdstrike = nightmare

We had Crowdstrike implementation in our next budget. Guess we're sticking with Trellix a little longer.

---

Yeah I've run into that on three machines so far. Not a showstopper you just have to log into the Microsoft site to get the recovery key. But it's time-consuming.

---

The killer will be if you run bitlocker on-prem, and save your keys on your domain controller. Which is also protected by bitlocker and Crowdstrike.

sixiron said:

---

Lathspell said:

---

Time for SentinelOne?

---

We POC'd Crowdstrike several years ago but decided to go with SentinelOne. I recommend them.

---

Good choice! It's a fantastic product.

Tailgate88 said:

---

Average Joe said:

---

Bitlocker + Crowdstrike = nightmare

We had Crowdstrike implementation in our next budget. Guess we're sticking with Trellix a little longer.

---

Yeah I've run into that on three machines so far. Not a showstopper you just have to log into the Microsoft site to get the recovery key. But it's time-consuming.

---

PIA to walk users through the recovery process over the phone. Some get it real quick, others it's like trying to get them to land on the moon.

Wow. I'm in the IT field (networking side) and that was fairly close to gibberish to me. But, I appreciate your post and the link.

https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/

Crowdstrike has an automated way to deploy a fix, but you have to opt in for it. Once you opt in it takes about an hour, then you just have to reboot any system still affected.