HomeMy WebLinkAbout07.19.2024 Catastrophic Cascading Countywide Computer Systems FailureFrom:Wright, Mike
Subject:Catastrophic Cascading Countywide Computer Systems Failure
Date:Friday, July 19, 2024 3:04:53 AM
Importance:High
Good morning fellow County staff,
I imagine that many of you woke up this morning to the news of catastrophic computer
systems and network failures affecting all sectors and industries worldwide. You don’t have to
look far to find a news story about this massive and impactful event. Butte County was not
spared and I expect many of you will arrive at the office to find your desktops, laptops and
servers sitting on the dreaded “Blue Screen of Death”.
County IS began observing issues a little after 10pm last night and staff from County IS, District
Attorney, Sheriff, Elections Clerk-Recorder and the Department of Behavioral Health all
responded. At the time of our initial response, the cause of the event was unknown, in fact,
we were not even aware of the global impact as we initially responded and took action
according to a ‘worst-case scenario’. Thankfully, after extensive research, we discovered the
root cause and tested and implemented a solution.
I want to stress that any concern that this was any sort of malicious or terror attack is
unfounded. This was in fact a third party software product update malfunction, that caused
a critical conflict with the Windows Desktop and Server Operating systems.
As I type, IS staff from many departments are working diligently to restore systems and
services. Unfortunately, every impacted system requires manual intervention. With as many
as 3,000 systems needing attention, it is going to take a considerable amount of time to fully
recover. We are working through the night to get critical County systems back online as our
priority.
Two other points:
1. IS staff should look for an email shortly from me with instructions.
2. DESS Staff does not use the affected third party software product and is likely not
directly affected. However, DESS staff may experience issues accessing systems and/or
communication with staff from all other departments.
Please be patient as County and Departmental IS staff work to recover from this as quickly as
we possibly can.
Mike Wright, MSIT, CISSP
County Information Security Officer (CISO)
Butte County Department of Information Systems
308 Nelson Avenue, Oroville, CA 95965
Telephone: 530.552.3281
Help Desk: 530.552.3222
Need to schedule a meeting - Bookings