Unless you’ve been living under a rock (or perhaps in one of those rural Iowa counties with spotty internet access), you know at the root of the 2020 Iowa Democratic caucus fiasco was the little IowaReporterApp that couldn’t.  Created by Shadow Inc., a company that sounds like the evil nemesis in a James Bond film, this mobile app was built for a very special audience, 1,765 volunteer Iowa Democratic precinct chairs, for a very specialized purpose: reporting and tabulating the results of the Iowa caucuses.  Many of its intended users couldn’t download and sign onto the app, and it didn’t work right for those who could. Another example of sloppy software developers writing buggy code, causing a massive software failure at the worst possible time for the biggest stakes, right?

Not so fast.  It might have been a software failure at the worst possible time for very high stakes (more on that below), but don’t blame the coders for this one.  Blame the managers.

Yes, there was a bug in how the app communicated with the back end system.  It’s something everybody can point to and say, “See?  There was a bug in the code.”  Let’s see a show of hands: any software developers reading this who have created a bug before, please raise your hand.  All of you?  That’s what I thought.

The project plan for launching the IowaReporterApp was fatally flawed.  That a bug made it into the production system is but one of many things that went wrong.  If we look at how a software project like this should have been run and compare it to how it actually was run, we can see several management missteps.  When you’re managing a project to a drop-dead release date like the Iowa caucuses, often the best way to come up with a plan is to start at the end-point and work the timeline back to the beginning.  Let’s do that here, starting with:

7. User Training

A smartphone app handed to 1700+ caucus volunteers, many of them “newer to apps and that kind of stuff”, some still using flip phones (according to the Polk and Buchanan county chairs).  What could possibly go wrong?

Apparently some user training was offered, but that happened before users even received the app.  Many volunteers had never even tried the app before the caucuses.

You can’t blame the users for having issues with an app they’d never been trained on or even seen.  That would be like, uh, blaming software developers for not writing 100% bug-free code.

6. Product Release and User Onboarding

Before training the users you have to get them up and running with the app.  The Shadow project was so late the app was never actually published to Apple’s App Store.  (That requires waiting for Apple to approve the app, which takes 1-2 days — time Shadow apparently didn’t have.)  As a result, users were required to download a developer app called TestFlight which is used to pre-publish apps to beta-testers before general release.

It should have been clear to Shadow a couple of days before the caucuses that they were at risk of not actually being able to release the app to users (and that they weren’t going to be able to get all 1700+ precinct chairs to figure out how to use the developer app). That would have been their last opportunity to say “abort, abort!” and double down on the back-up plan, which was to phone in results.  A couple of days’ notice would have allowed the team to communicate the new plan to volunteers and recruit more operators to staff the phones (avoiding the 90-plus minute hold times for reporting results).

5. Beta Testing and Bug Fixing

Software coding and systems development is complex, with bugs an inevitable byproduct.  Multiple rounds of testing are typically necessary to suss out any significant bugs.  The higher the stakes and the lower the tolerance for bugs, the more rounds are necessary.

Typically the final round(s) of software testing is beta testing, where the product is put in the hands of a relatively small set of engaged (and well-informed) users.  This can be the only way to unearth bugs caused by usage patterns unanticipated by the developers, as well as an important opportunity to gather real user feedback to inform ongoing product development.  (Apple’s previously mentioned TestFlight app is designed to facilitate beta testing feedback.)

Shadow’s CEO said they’d had people beta-testing the app “for weeks”.  The Washington Post, on the other hand, described the app as almost entirely untested.  What’s clear is that Shadow didn’t do beta-testing right.  One or more of the following must be true:

• They gave their beta-testers support and hand-holding they didn’t give the general audience.
• Their beta-testers weren’t representative of their general audience.
• The beta-testers didn’t use the same product as the general audience (were they onboarded through TestFlight?).
• Beta-testing feedback wasn’t collected or given appropriate attention.

Otherwise, the fact that three-quarters of users couldn’t figure out how to log into the app would not have come as a surprise on C-Day.

4. Integration Testing

Typically the final phase before unleashing an emerging software product on beta testers is integration testing.  Unlike earlier phases in which software components may be tested independently, in integration testing the entire system is tested end-to-end.  This is the phase where you should catch bugs in places like “code that transmits results data into the… data warehouse”, which is how Shadow’s CEO described the infamous IowaReporterApp bug.  I don’t know why the bug wasn’t caught during integration testing (or beta testing), but you can’t blame the developers for that.  Maybe management didn’t allow enough time for testing…

3. Initial Development and Testing

I can’t see the app (since it was never publicly released), and nobody has publicly said anything about its development process.  So I can only comment as an outside observer:

• The one-fourth of caucus chairs who were able to download and sign into the app were able to use it as intended.  I have not seen, in broad media coverage of the incident, any complaints about the quality of the app itself.  If the app had usability issues beyond user-onboarding, we’d know about them at this point.  (This is a good thing.)
• Completing development of a typical mobile app in a two-month timeframe would be aggressive even if it were the only phase of the development lifecycle.  Add in all of the other phases in this list, and well, any experienced manager could have predicted the shtuff was bound to hit the fan.

​
2. Design

There have been over 800 cyberattacks of political organizations in the past year alone.  In this climate, security is a fundamental requirement for any online political product, especially one used operationally to tabulate votes, especially votes as high profile as the Iowa presidential caucuses.  Security is not a feature that can be tacked onto a software system; it has to be designed in from the start.

Unfortunately, it is difficult to evaluate the security of the IowaReporterApp system.  The Iowa Democratic party took the approach of keeping security measures themselves a secret.  Prior to the caucuses, Iowa Democratic party chair Troy Price wouldn’t reveal what security measures were being implemented or even who was building the app, saying, "We want to make sure we are not relaying information that could be used against us."  As for any political organizations that might want to use this system in the future, well, that’s their problem.  (He didn’t actually say that last part.)

(Nevada democrats were already planning to use the same app for their February 22 caucuses — not surprisingly, they’ve abandoned those plans.)

Security through Obscurity, as this approach is called, is a discredited practice.  Once the “secret” is out, the “security” is lost.  Truly secure systems, such as the protocol for encrypting and decrypting the content you’re reading right now via HTTPS, are secure despite being open.

1. Signing Up for the Impossible

Working under the brightest of lights, with zero tolerance for slipping the date or allowing any security breach, did the Shadow team ever really have a shot at successfully launching the IowaReporterApp in a little over two months?  Even if they were merely repurposing an existing app (which they weren't), it would have been a tall order to complete multiple rounds of testing, fix bugs, release the app, train users, and get everything locked down in that period of time.  Add in design and development time, and the answer is a resounding “no”.

Signing the contract to build the app — this was the first and biggest mistake made by Shadow management.  This was an impossible ask of the engineering team.  I’ve signed up for the impossible a couple of times in my career and lived to regret it.

Shadow was signed on only about two and a half months before C-Day, judging from public payment records.  Why was the schedule so compressed in the first place?  Apparently Iowa decided to go with an app late in the game. According to the New York Times, “[only] when Iowa Democrats, on the advice of the national party, abandoned plans to have caucus results called in by phone because of security concerns and instead build an app, they chose Shadow from multiple bidders.” (I imagine over the last few days there’s been a lot of kissing of spouses and children by the other bidders.)

It’s been reported that previously Shadow had nearly gone bankrupt after failing to gain traction with an earlier campaign-texting platform.  I can understand why they saw the Iowa caucuses as a chance for redemption, maybe a make-or-break deal for the company.  But when you’re signing up for the impossible, the alternative is always preferable, even if it means running out of money.  At least that gives you a chance at an orderly dissolution that doesn’t result in everybody associated with the company running away like rats from a burning ship.

The new year does not look to be a good one for the leading software behemoths, sometimes referred to as FAANG: Facebook, Amazon, Apple, Netflix, and Google.  You probably use services from some if not all of them, but they are increasingly under attack for being too big, too anti-competitive, and knowing too much about us.

I have mixed feelings about proposed solutions such as splitting up some of the FAANGs or regulating them like utilities.  As a technologist, I see them as more than just ruthless corporate entities.  All of the FAANGs have advanced the software industry by taking key, general-purpose software tools and techniques they created in-house and making them freely available.  All of them, that is, but one.

Google and Facebook each released software tools, initially developed internally, that have become two of the hottest front-end frameworks for web software developers.  Google eliminated its famous "Don't be Evil" motto from its employee code of conduct, but not before they released the Angular framework to the world in 2010.  Facebook publicly released the React framework in 2013.  Websites developed using one or both of these technologies (in addition to Google and Facebook) include PayPal, ESPN, Netflix, Twitter, Nike, … it might be easier to list famous brand websites NOT developed using either of these technologies.

Netflix has also made internally developed software freely available to the public, but they’re most well known in software circles for a 125-page HR document defining Netflix’s famously performance-driven corporate culture. Known simply as the Culture Deck, in its various versions it has been read online over 20 million times. Facebook COO Sheryl Sandberg once said, “It may well be the most important document ever to come out of [Silicon] Valley.”

Apple has released software to the public… expressly to enable developers to build applications supporting Apple’s own products.  Still, they make my list of tech largesse with an asterisk, thanks to their User Interface (UI) guidelines.  Apple’s influence on software (as well as hardware) interface design has been profound, dating back to their first publicly published Human Interface Guidelines in the mid-80’s.  Apple converted then-revolutionary UI research into practice, laying out fundamental principles of UI such as consistency, context, and user feedback.  While Apple's guidelines have evolved tremendously over the years, many of their original principles remain tenets of good software design.

That leaves the final FAANG: Amazon.  I’m one of their best customers.  I placed 130 Amazon Prime orders in 2019 alone.  I make extensive use of their “cloud” Amazon Web Services professionally.  But Amazon earns no tech-humanitarian points to balance their arguably ruthless approach to technology and e-commerce.  Like Apple, Amazon contributes to public, “open source” software projects purely to help sell more of their own products and services.  Sorry, Amazon, no good citizen award for you!

​I’ve been managing software development projects since the pre-internet digital dark ages.  Over time I’ve seen software development process change as much as software languages and tools.  (Maybe even more so — the Python language, first released in 1991, is in the middle of a data-science-driven renaissance.)

It’s been fifteen years since I last used a waterfall software development process.  You might think the last place you’d find that kind of “legacy” process would be an early-stage startup, but I still see startups (as well as mature companies) being pitched waterfall contract proposals by outside contractors to develop “Version 1” go-to-market software products.

If you haven’t been doing software development for more than ten years, you may have never done “waterfall” development (or even know what it is).  Regardless of project methodology, most software development follows a sequence of steps that look something like:

1. Define product.
2. Design product.
3. Develop and test product.
4. Deploy and maintain product.

Waterfall methodology refers to planning and scoping all of these steps at the beginning, then executing each step in sequence.  The workflow cascades in one direction, like a waterfall.

Except when external constraints don't allow an agile approach (a government RFP process, say, or a large, hidebound enterprise), I have consistently seen better results through an agile approach: iterative, incremental, customer-conversation driven.  Doubly so for a startup.

Agile looks something like:
​

1. Define product.
2. Get initial customer feedback via mock-ups, prototypes, etc.
3. Incorporate customer feedback into design of product.
4. Develop and test product.
5. Deploy product and measure customer response.
6. Loop back to #3.

A product conceived by an elaborate design process, but not validated by actual customer engagement and feedback, may hit the mark on some product ideas but will inevitably miss the mark on others.  Under a variety of circumstances over thirty-plus years I’ve never seen it play out otherwise.

Several years ago I worked for a well-funded startup alongside very bright product managers, UI designers, and software engineers (several of whom went on to launch their own companies).  Technically we were “agile” because we developed our software in sprints, but our feedback loop was internal.

Rather than validate core assumptions with actual customers, we developed a full-featured “stealth” V1 (Version 1) of our product, a healthcare portal, that launched with much fanfare and a hefty marketing budget.

With an all-star team and resources most startups can only dream of, we had every possible advantage.  But because we did not include customer feedback in our process prior to launch, our product had about the same success as any other V1: some features resonated with users, but too many other features flopped.  I joined this company at founding and saw it rocket to 300+ employees very quickly.  It contracted back down to 20 employees almost as quickly.

So what can you do to promote incremental, customer feedback-driven product definition and development in the construction of your software?

One option is to favor T&M compensation (time and materials, “paid by the hour”) over project-based compensation.  T&M contracts are suitable when scope is uncertain, which it is when taking an “iterate until you validate” approach. It’s worth pointing out that an in-house team, if you have one, is effectively T&M.

Some contractors provide dedicated engineers long-term on a T&M basis, either augmenting your own staff or operating as a self-contained, dedicated team; that’s one model that can jumpstart a new software development effort into an iterative development process.  Make sure your contract allows you to terminate early and quickly if the vendor isn’t performing.
​
If you don’t have in-house resources and you’re uncomfortable with the scope risk of a T&M contract, another option is to minimize the scope of your Version 1 project.  Turn it into a Version .5 project, an incomplete version sufficient to validate your concept, and structure follow-on versions as separate projects.  In addition to lowering your investment in un-validated product development, this approach incentivizes your vendor to perform to win follow-on work.  To avoid misunderstandings, be clear about the incremental, feedback-driven approach you want to take with candidate contractors and make sure they’re amenable.

However you structure your software development project, validate the initial concept with as little engineering investment as possible (consider low-effort approaches such as mockups or prototypes), and continue to vet changes through customer feedback.

Don’t go chasing waterfall, even if you’re outsourcing, and especially if you’re outsourcing V1.  Build customer-driven validation into your design and development process through an incremental approach.  If outsourcing software development, make a phased approach a requirement for your contractor.

​This post was originally published on WeWork Labs Insider and also appears on CTO Craft.  See also Thomas Fuller's take on "extreme" agile development process.