Too late now for canary test updates, says pension fund suing CrowdStrike

CrowdStrike, after suggesting canary testing as a way to ensure it avoids future blunders leading to global computer outages, has been sued in federal court by investors for not using a phased approach in rolling out updates to customers in the first place.

In what will likely be one of many class-action complaints against the embattled IT security firm, a retirement association has accused CrowdStrike, its CEO George Kurtz, and CFO Burt Podbere of defrauding it and fellow shareholders by making false and misleading statements about the biz's Falcon endpoint defense software.

CrowdStrike and its top execs "repeatedly touted the efficacy of the Falcon platform while assuring investors that CrowdStrike's technology was 'validated, tested, and certified,'" the Plymouth County Retirement Association's lawsuit [PDF], filed this week in Texas federal court, reads.

But in reality, the security shop's controls and procedures for updating Falcon weren't up to snuff, the lawsuit argued. And this included not properly testing anti-threat updates before pushing them to all of its tens of millions of customers, all at once.

"This inadequate software testing created a substantial risk that an update to Falcon could cause major outages for a significant number of the company's customers," the Massachusetts-based association alleged. "Such outages could pose, and in fact ultimately created, substantial reputational harm and legal risk to CrowdStrike."

In the antivirus maker's preliminary post-incident review published after it crashed millions of Microsoft Windows boxes around the world with a bad Falcon sensor update, CrowdStrike promised to improve its software testing and deployment by, among other things, implementing a canary deployment strategy, starting with pushing changes to a small subset of users to see how it goes and then gradually deploying to larger portions of customers.

Previously CrowdStrike would automatically distribute files that improved or tweaked the operation of its threat-detection system Falcon to all customer installations at once. In July, one of those files caused CrowdStrike's Windows kernel-level driver to access memory it shouldn't, bringing down the whole operating system and its applications.

What's worse is that CrowdStrike did have some testing procedures in place for updates prior to release, but in this case, the validation system failed to realize the changes were malformed and allowed them to be deployed at scale.

Following that snafu, the software maker vowed to take a more staggered approach, though the pension fund is still unimpressed.

"Since the CrowdStrike outage, publicly revealed evidence indicates that CrowdStrike was taking insufficient precautions regarding such updates," the lawsuit stated.

"For instance, CrowdStrike has promised to take remedial measures to ensure that such a crash does not happen again, including implementing a so-called canary deployment of such updates, meaning a progressive rollout that starts with a subset of users.

"This indicates CrowdStrike was not taking such measures prior to the CrowdStrike outage."

When asked about the lawsuit, a CrowdStrike spokesperson told The Register: "We believe this case lacks merit and we will vigorously defend the company." 

The Falcon update that was heard around the world, and broke IT systems globally, sent CrowdStrike's stock tumbling more than 11 percent, according to the legal complaint, hurting investors including the retirement fund, which is seeking damages.

The association claims it has lost out financially because it was tricked into buying CrowdStrike shares by believing the biz's boasts about itself and its software's reliability. The Falcon-induced outage, caused by a lack of testing, ended up damaging the developer's reputation and stock, and thus the fund's holdings in the firm, it was argued.

A few days after the crash, Congress called on CrowdStrike's Kurtz to testify about the security snafu, and analysts including Guggenheim and BTIG downgraded the biz's rating, both of which allegedly caused CrowdStrike stock to fall even further, dropping more than 13 percent. 

And finally, on Monday the news broke that Delta Air Lines hired famed attorney David Boies to potentially seek as much as $500 million in damages from CrowdStrike and Microsoft after the airline was hit hard by the Falcon-caused outage.

This third nail in the coffin by itself caused $CRwD's stock price to drop almost 10 percent, doing further harm to the association's retirement pot, the class-action suit says.

While its legal battles are unlikely to go away anytime soon, CrowdStrike on Wednesday said it's making progress on getting any straggling Windows devices back online.

"Using a week-over-week comparison, ~99% of Windows sensors are online as of July 29 at 5pm PT, compared to before the content update," the update noted. ®