Updated July 20, 2024 at 5:39 AM
CrowdStrike is actively working with customers affected by the flaw found in a single content update for Windows hosts. Mac and Linux hosts are not affected. This was not a cyber attack.
The issue has been identified, isolated and a fix has been deployed, we are directing customers to the support portal for the latest updates and will continue to provide complete and ongoing public updates on our blog.
Additionally, organizations are advised to ensure they communicate with CrowdStrike representatives through official channels.
Our team is dedicated to ensuring security and stability for CrowdStrike customers.
We understand the seriousness of this situation and deeply apologize for any inconvenience or trouble caused. We are working with all affected customers to get our systems back up and running and providing the service you expect.
CrowdStrike is operating normally and this issue does not impact Falcon platform systems. If your system is operating normally, protection will not be affected even if Falcon sensors are installed.
Below is the latest CrowdStrike technical alert with more information on this issue and workarounds that organizations can take. We will continue to provide updates to our community and the industry as they become available.
summary
- CrowdStrike is aware of reports of crashes on Windows hosts related to Falcon sensors.
detail
- Symptoms include hosts experiencing bugcheck\blue screen errors related to the Falcon sensor.
- On unaffected Windows hosts, the problematic channel file has been reverted and no action is required.
- Windows hosts that came online after 0527 UTC are also not affected.
- This issue does not affect Mac or Linux-based hosts.
- The channel file “C-00000291*.sys” with a timestamp later than 0527 UTC is the reverted (good) version.
- The channel file “C-00000291*.sys” with timestamp 0409 UTC is the problematic version.
- Note: It is normal to have multiple “C-00000291*.sys” files in your CrowdStrike directory. One Any file in the folder with a timestamp later than 0527 UTC becomes active content.
Current Action
- CrowdStrike engineering identified content deployments related to this issue and reverted those changes.
- If your host continues to crash and is unable to stay online to receive channel file changes, you can use the workaround steps below.
- We guarantee you CrowdStrike is operating normally and this issue does not impact Falcon Platform systems.If your system is operating normally, your protection will not be affected even if you have Falcon sensors installed. Falcon Complete and OverWatch services will not be interrupted by this incident.
Query to identify affected hosts using Advanced Event Search
Please refer to this KB article: How to identify hosts potentially affected by a Windows crash (pdf) or log in to the Support Portal to view it.
Dashboard
Similar to the queries above, a dashboard showing affected channels and CIDs, affected sensors is now available, available in one of the console menus depending on your subscription:
- Next Gen SIEM > Dashboard or;
- Investigations > Dashboard
- Name: hosts_possibly_impacted_by_windows_crashes
Note: Dashboards are not available with the “Live” button
Auto-recovery articles:
Please see this article: Automatic Recovery from Blue Screen on Windows Instances on GCP (pdf) or log in to the Support Portal to view it.
Workaround steps for individual hosts:
- Reboot your host so it can download the channel files you just reverted to. It is highly recommended that you connect your host to a wired network (not WiFi) before rebooting, as your host can get an Internet connection over Ethernet much faster.
- If the host crashes again:
- Boot Windows into Safe Mode or into the Windows Recovery Environment
- Note: Connecting the host to a wired network (not WiFi) and using Safe Mode with Networking can help with repairs.
- Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
- Windows Recovery defaults to X:\windows\system32
- First, navigate to the appropriate partition (default is C:\) and then to the crowdstrike directory.
- C:
- cd windows\system32\drivers\crowdstrike
- Note: In WinRE/WinPE, navigate to the Windows\System32\drivers\CrowdStrike directory on the OS volume.
- Find files matching “C-00000291*.sys” and delete them.
- please do not Delete or change any other files or folders
- Cold boot the host
- Shut down the host.
- Boot the host from a powered off state.
Note: BitLocker encrypted hosts may require a recovery key.
Workaround steps for public cloud or similar environments including virtual:
Option 1:
- Detach the operating system disk volume from the affected virtual server
- As a precaution against unintentional changes, please create a snapshot or backup of your disk volume before proceeding.
- Attach/mount the volume to the new virtual server
- Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
- Find files matching “C-00000291*.sys” and delete them.
- Detach the volume from the new virtual server
- Reattach the pinned volumes to the affected virtual servers
Option 2:
- Roll back to a snapshot before 0409 UTC.
AWS specific documentation:
Azure environment:
Workspace ONE Portal User Access Recovery Key
Enabling this setting allows users to retrieve their BitLocker recovery key from the Workspace ONE portal without contacting the help desk. To turn on recovery keys in the Workspace ONE portal, follow these steps. For more information, see this Omnissa article.