Monday, March 21, 2016

There's No Test Like A Production Test

Dress Rehearsal

Enter stage left. Pacing a slow and deliberate gait to center stage, it's time to deliver your lines. Dress rehearsal is here. After weeks of memorizing lines and working with your follow cast members practicing scene by scene, you're nearly ready for opening night.

As the director walks everyone through each scene of dress rehearsal, small issues come to light that were never noticed before. Whether it's the wrong lighting arrangement, a misplaced prop, or a scene change that doesn't transition smoothly enough, this is the opportunity to make last-minute adjustments to make sure the show is prepared for success.

Opening Night

The cast and crew take their places as the curtain opens. Opening night is packed with eager patrons and critics ready to experience the well-promoted show. The opening act comes to a close to a thunderous applause. Midway through the second act a stage prop malfunctions, causing an awkward pause and a stumbling delivery, breaking the audience's immersion in an otherwise excellent show. The rest of the show goes off without major incident, ending to a satisfying ovation.

Despite all of the hard work and planning by everyone involved, one unexpected or anticipated error can mar an otherwise excellent performance. In a sufficiently complex production, no matter how much you prepare ahead of time, it's impossible to anticipate every possible thing that might go wrong. You must address any issues as they arise and learn to respond better next time

The Software Stage

The analogy is the same in the world of software. It does not matter how many layers of testing suites you wrap around your software. It does not matter that you've tested it in three distinct environments and subjected it to a wide array of contrived and random test data. It does not matter that you've gone over a test deployment twice and three times. You simply cannot know that your software is truly ready for production until it's put to the test with real users and data.

I've been around this game long enough to see well-tested systems run into all kinds of surprising bugs. Many bugs would simply never have been caught, even by adding dozens of programming, testing, and hardware resources prior to production release. These bugs are typically inconsistent, difficult to reproduce, or related to users exercising the software in unexpected ways.

It's a case of diminishing returns -- performing a variety of basic tests will generally pay immediate dividends as obvious, simple bugs are found. Performing additional rounds of testing will often flush out a few more. As more bugs are found, the rare gems, well-hidden beneath the surface, will become harder and more time consuming to find. Often, it is simpler to release the software and fix bugs as they are reported than to waste valuable resources looking for outliers.

Raise the Curtain

As with the theater, it is true in software: the show must go on. Once the production is prepared and the dress rehearsal has worked through the obvious problems, the only thing left to do is to perform for the waiting public and wait for the reviews.

If there's one thing I would suggest you take away from this, it's that there's rarely such a thing as a flawless release. Always expect that your software will fail in a way that you've never seen or anticipated. Therefore, be prepared to respond quickly, lest the critics give you a scathing review and the fickle public refuses to come back for an encore.


Joshua Ganes