Don’t Blame Coders for the Iowa Caucuses

The Epic Software Fail was Management‘s Fault
​

Unless you’ve been living under a rock (or perhaps in one of those rural Iowa counties with spotty internet access), you know at the root of the 2020 Iowa Democratic caucus fiasco was the little IowaReporterApp that couldn’t.  Created by Shadow Inc., a company that sounds like the evil nemesis in a James Bond film, this mobile app was built for a very special audience, 1,765 volunteer Iowa Democratic precinct chairs, for a very specialized purpose: reporting and tabulating the results of the Iowa caucuses.  Many of its intended users couldn’t download and sign onto the app, and it didn’t work right for those who could. Another example of sloppy software developers writing buggy code, causing a massive software failure at the worst possible time for the biggest stakes, right?

Not so fast.  It might have been a software failure at the worst possible time for very high stakes (more on that below), but don’t blame the coders for this one.  Blame the managers.

Yes, there was a bug in how the app communicated with the back end system.  It’s something everybody can point to and say, “See?  There was a bug in the code.”  Let’s see a show of hands: any software developers reading this who have created a bug before, please raise your hand.  All of you?  That’s what I thought.

The project plan for launching the IowaReporterApp was fatally flawed.  That a bug made it into the production system is but one of many things that went wrong.  If we look at how a software project like this should have been run and compare it to how it actually was run, we can see several management missteps.  When you’re managing a project to a drop-dead release date like the Iowa caucuses, often the best way to come up with a plan is to start at the end-point and work the timeline back to the beginning.  Let’s do that here, starting with:

7. User Training

A smartphone app handed to 1700+ caucus volunteers, many of them “newer to apps and that kind of stuff”, some still using flip phones (according to the Polk and Buchanan county chairs).  What could possibly go wrong?

Apparently some user training was offered, but that happened before users even received the app.  Many volunteers had never even tried the app before the caucuses.

You can’t blame the users for having issues with an app they’d never been trained on or even seen.  That would be like, uh, blaming software developers for not writing 100% bug-free code.

6. Product Release and User Onboarding

Before training the users you have to get them up and running with the app.  The Shadow project was so late the app was never actually published to Apple’s App Store.  (That requires waiting for Apple to approve the app, which takes 1-2 days — time Shadow apparently didn’t have.)  As a result, users were required to download a developer app called TestFlight which is used to pre-publish apps to beta-testers before general release.

It should have been clear to Shadow a couple of days before the caucuses that they were at risk of not actually being able to release the app to users (and that they weren’t going to be able to get all 1700+ precinct chairs to figure out how to use the developer app). That would have been their last opportunity to say “abort, abort!” and double down on the back-up plan, which was to phone in results.  A couple of days’ notice would have allowed the team to communicate the new plan to volunteers and recruit more operators to staff the phones (avoiding the 90-plus minute hold times for reporting results).

5. Beta Testing and Bug Fixing

Software coding and systems development is complex, with bugs an inevitable byproduct.  Multiple rounds of testing are typically necessary to suss out any significant bugs.  The higher the stakes and the lower the tolerance for bugs, the more rounds are necessary.

Typically the final round(s) of software testing is beta testing, where the product is put in the hands of a relatively small set of engaged (and well-informed) users.  This can be the only way to unearth bugs caused by usage patterns unanticipated by the developers, as well as an important opportunity to gather real user feedback to inform ongoing product development.  (Apple’s previously mentioned TestFlight app is designed to facilitate beta testing feedback.)

Shadow’s CEO said they’d had people beta-testing the app “for weeks”.  The Washington Post, on the other hand, described the app as almost entirely untested.  What’s clear is that Shadow didn’t do beta-testing right.  One or more of the following must be true:

• They gave their beta-testers support and hand-holding they didn’t give the general audience.
• Their beta-testers weren’t representative of their general audience.
• The beta-testers didn’t use the same product as the general audience (were they onboarded through TestFlight?).
• Beta-testing feedback wasn’t collected or given appropriate attention.

Otherwise, the fact that three-quarters of users couldn’t figure out how to log into the app would not have come as a surprise on C-Day.

4. Integration Testing

Typically the final phase before unleashing an emerging software product on beta testers is integration testing.  Unlike earlier phases in which software components may be tested independently, in integration testing the entire system is tested end-to-end.  This is the phase where you should catch bugs in places like “code that transmits results data into the… data warehouse”, which is how Shadow’s CEO described the infamous IowaReporterApp bug.  I don’t know why the bug wasn’t caught during integration testing (or beta testing), but you can’t blame the developers for that.  Maybe management didn’t allow enough time for testing…

3. Initial Development and Testing

I can’t see the app (since it was never publicly released), and nobody has publicly said anything about its development process.  So I can only comment as an outside observer:

• The one-fourth of caucus chairs who were able to download and sign into the app were able to use it as intended.  I have not seen, in broad media coverage of the incident, any complaints about the quality of the app itself.  If the app had usability issues beyond user-onboarding, we’d know about them at this point.  (This is a good thing.)
• Completing development of a typical mobile app in a two-month timeframe would be aggressive even if it were the only phase of the development lifecycle.  Add in all of the other phases in this list, and well, any experienced manager could have predicted the shtuff was bound to hit the fan.

​
2. Design

There have been over 800 cyberattacks of political organizations in the past year alone.  In this climate, security is a fundamental requirement for any online political product, especially one used operationally to tabulate votes, especially votes as high profile as the Iowa presidential caucuses.  Security is not a feature that can be tacked onto a software system; it has to be designed in from the start.

Unfortunately, it is difficult to evaluate the security of the IowaReporterApp system.  The Iowa Democratic party took the approach of keeping security measures themselves a secret.  Prior to the caucuses, Iowa Democratic party chair Troy Price wouldn’t reveal what security measures were being implemented or even who was building the app, saying, "We want to make sure we are not relaying information that could be used against us."  As for any political organizations that might want to use this system in the future, well, that’s their problem.  (He didn’t actually say that last part.)

(Nevada democrats were already planning to use the same app for their February 22 caucuses — not surprisingly, they’ve abandoned those plans.)

Security through Obscurity, as this approach is called, is a discredited practice.  Once the “secret” is out, the “security” is lost.  Truly secure systems, such as the protocol for encrypting and decrypting the content you’re reading right now via HTTPS, are secure despite being open.

1. Signing Up for the Impossible

Working under the brightest of lights, with zero tolerance for slipping the date or allowing any security breach, did the Shadow team ever really have a shot at successfully launching the IowaReporterApp in a little over two months?  Even if they were merely repurposing an existing app (which they weren't), it would have been a tall order to complete multiple rounds of testing, fix bugs, release the app, train users, and get everything locked down in that period of time.  Add in design and development time, and the answer is a resounding “no”.

Signing the contract to build the app — this was the first and biggest mistake made by Shadow management.  This was an impossible ask of the engineering team.  I’ve signed up for the impossible a couple of times in my career and lived to regret it.

Shadow was signed on only about two and a half months before C-Day, judging from public payment records.  Why was the schedule so compressed in the first place?  Apparently Iowa decided to go with an app late in the game. According to the New York Times, “[only] when Iowa Democrats, on the advice of the national party, abandoned plans to have caucus results called in by phone because of security concerns and instead build an app, they chose Shadow from multiple bidders.” (I imagine over the last few days there’s been a lot of kissing of spouses and children by the other bidders.)

It’s been reported that previously Shadow had nearly gone bankrupt after failing to gain traction with an earlier campaign-texting platform.  I can understand why they saw the Iowa caucuses as a chance for redemption, maybe a make-or-break deal for the company.  But when you’re signing up for the impossible, the alternative is always preferable, even if it means running out of money.  At least that gives you a chance at an orderly dissolution that doesn’t result in everybody associated with the company running away like rats from a burning ship.

---

​The impact crater from this fiasco extends far wider than the hapless Shadow employees and their investors.  The chaos of the 2020 Iowa Democratic caucuses muddied the Democratic presidential primary race and made a laughingstock of the Dems (although 49% of Americans might approve).  It also increased distrust in our country’s democratic process — and nobody on either side of the aisle can argue that’s a good thing.

This post is also featured on The Startup.