Challenge Your Assumptions

Today's brief post was inspired by my recent road trip with my wife from here in Vancouver, BC through the states of Washington and Oregon.

When it comes to road trips, I'm all about efficiency. Pit stops run like clockwork. The guys from NASCAR have nothing on me. That's why I prefer self service with pay-at-the-pump when stopping for gas. My fingers fly with precision as I fetch my card from my wallet, insert it in the machine and punch my way through the menu to start up the pump and select my grade of gasoline to begin fueling.

On my first stop of the trip, everything was running smoothly right up until I was stopped in my tracks by the following prompt:

Enter Zip Code:

 

Cancel         OK

Unfortunately for me, as a Canadian, I don't have a numeric 5-digit zip code. Instead, I have a six character postal code with a mix of numbers and letters. Try though I might, I could not find a way to enter my postal code or skip this prompt.

Edit: I've since been told that you can simply enter the numeric portion of your postal code followed by zeros

After a few moments of poking around at the machine, I eventually resorted to pressing cancel and walked into the station to speak with the cashier. Within a few minutes I was back on track (a lap behind the race leaders).

My experience with the zip code prompt was hardly life altering. Still, it illustrates some of the problems that can come up when your software makes incorrect assumptions (e.g. all gas station customers have a zip code). Challenge your assumptions and do your best to consider all possible users. Question whether all the information you prompt for will be available or match your expected format. If not, your software should gently guide the user on how to proceed.

Cheers,

Joshua Ganes 

P.S. Did you see the seventh inning of today's Blue Jays vs. Rangers game? It was well worth a second look! Go Jays!

Any News? - Working With Third Parties

I Am A Rock

We programmers tend to be an independent bunch. Like a lone wolf, we often prefer our own company to that of others and like to work alone. It's been said that, "no man is an island." I have to agree (unless you happen to be Paul Simon). Even programmers need to interact with their coworkers, managers, other departments, and even third-party contacts to meet the day-to-day demands of their jobs.

While networking and politics may not come naturally to all developers, we're hardly worthy of our titles if we cannot find a way to communicate clearly and accurately. Programmers must express requirements, prerequisites, technical issues, and design ideas in ways that others can understand so that all players can do their jobs and assist us in completing our own work.

Have you ever found yourself in this situation? You're beginning to wrap up a project, but you notice a seemingly simple, yet annoying bug in the library you are working with. Being diligent, you follow up with the library provider to report the issue. You ask if this is a known issue. You ask if you're using their library correctly. You ask if there are any workarounds or alternative approaches they can suggest. You go ahead and make a note to yourself to revisit the problem once the provider has had time to review your questions. Fast forward a few weeks. You've cleaned and polished every other corner of your software, but you really need to address this one remaining issue before you can build a final software release. In the meantime, the only feedback you've received is that they're, "looking into it."

On Dependencies

A wise piece of advice for any project is, "eliminate the dependencies." Just as in the example above, making yourself dependent on someone else's assistance can be a very compromising position. Not being able to deliver a solution because you're waiting for someone else to act can be uncomfortable, awkward, and frustrating. If you can eliminate these dependencies, you may still run into roadblocks of your own, of course, but you will retain the power to work through them and implement a solution dictated by your own schedule and budget.

It's not really fair to say, "eliminate the dependencies", and leave it at that. For many developers, creating a dependency is not really a choice. Maybe your boss, team, or client is demanding a specific technical requirement. Maybe you can't afford any alternatives. Maybe, as is often the case for me, your work involves creating an interface between your own software and a third-party tool. In this case, if you eliminate the third-party tool, you eliminate the need for your own software entirely.

Now, before I go accusing anyone of wrongdoing, let's try the shoe on the other foot. The development team in charge of that third-party library has more than just one bug fix on their plate. They need to balance support and bug fixes against their company's need for revenue-generating new features and routine maintenance. They may have a significant collection of issues that go through a triage system, and your bug report hasn't yet make the cut. I'm sure there's several times when others were waiting on me to complete my fix before they could proceed. Any fingers I point here need to be pointing back at myself also.

Planning For Success

The key to managing the risks of creating project dependencies is to identify those dependencies and understand them. Do your best to perform a reasonable analysis of the risks (if possible). Try to understand the expected flow of communications between both parties. Manage expectations by discussing project timelines and turnaround times for both technical inquiries and potential bug fixes. Understand the implications for your project when you cannot get the other side to take quick action. Are there alternatives? What are they? Will they be just as risky? If the risks are too high, sometimes the only winning move is not to play. Don't allow yourself and your organization to be sucked into wasting resources on an endeavor that will never succeed.

One of the best ways to reduce delays due to third parties is to be prudent and up-front to find and report any issues. Fixing bugs takes time. The sooner a bug is reported, the sooner it will be fixed. Therefore, schedule any development and testing work dependent on third-party support at the earliest opportunity. As soon as an issue is discovered, take time to understand the problem in reasonable detail and communicate it as soon as possible. This will give everyone the most time to respond and allows you to proceed with other features until the bug can be resolved.

Once you have requested support from an outside party, follow up and keep in touch. Remember, the contact at the other end is a person too. Treat them with respect and professionalism. Keep your communications positive and inquisitive. Try not to accuse or pester. There are differing opinions on how frequently you should send messages, so I won't offer specific advice. Instead, I suggest you give your best effort to judge the tone of their replies to see if the other side has become annoyed. Never hesitate to follow up if a written deadline or estimate is missed. Your goal, again, is not to accuse, but to make sure you are fully informed and kept up to date.

If worst comes to worst, don't be afraid to escalate. Give your best effort to work through the default channels provided. If, however, you simply cannot get the answers you need, the time may come where you need to ascend the ladder of power. Do your best to do so as respectfully as possible. Ask your contact if there is an option to expedite or escalate the communication process. I have seen cases where two lower-rung employees went back and forth for weeks. A single brief phone call between two big wigs was all that was required to sort things out. Nobody wants to be disrespected and have someone go over their head. Reserve a power move like this as a last resort, but don't be afraid to pull the trigger if all other options have been exhausted.

The very last resort sometimes required implementing a heroic workaround. For this to work, you need to understand the third-party system well enough to automatically recognize the failure and wield the power to fix it through one or more additional steps. Rather than having the third-party implement a fix, you can choose to live with the existing bug, masking its existing by coding your own workaround fix. On a personal note, while this option is often the best business decision, it often feel like an enabler. We just let the bug continue to fester and bend over backwards to get past it rather than correcting it at its source.

Good Days And Bad Days

In my professional experience, I have worked with a fair number of third parties. Some have been eager and helpful. They are quick to respond and provide clear details needed to solve my problems. Others have been slow to respond and very terse and unclear in their replies. After working through a few issues with a contact, you will quickly develop an opinion of their competence. This can be used in further risk assessment. My advice is to go with your gut. So far, I have never once changed my opinion from that initial impression following subsequent communications.

What's your best or worst story of working with a third party? What went right and what went wrong? What would you do differently? Let me know in the comments.

Cheers,

Joshua Ganes

The Data Structure Barbecue Grill

Charcoal, Fire Starter, Match

Recently, a colleague asked me a question about the implementation of a hash table. I found that I had to stop and think for a moment. My explanation was by no means the smoothest delivery. I tripped over my words a few times and needed to catch myself and backtrack before settling on the correct details. This incident has convinced that it's time for a refresher on one of the fundamental components of computing science: data structures.

Rather than dusting off my old text books and violating a copyright or two, I thought I'd take a new approach inspired by a conversation with yet another colleague of mine. With spring in full bloom here in Vancouver, it's time to pull out a stiff wire brush and begin cleaning the grill. Consider this your invitation to the first ever data structure barbecue.

On A Stick

While my usual preference is for a nice steak, there is one barbecue item that is nearly as much fun as it is delicious: the kebab. The scrumptious kebab is a collection of seasoned meats and vegetables skewered onto a single stick.

How is this tantalizing treat comparable to a data structure? When building a kebab it's impossible to add new pieces to the middle, but poking a new item onto either end is easy. Once cooked, it's far simpler to remove and eat from either end than it is to chew off the center. This matches the properties of a double-ended queue. You can add or remove at either end of the queue, but the items remain in a fixed order and cannot be accessed until found at either end of the queue.

A neat twist on our double-ended kebab is a kebab with a handle on one end. This kebab can only be accessed from a single end; the exact properties of a stack. We push items on one at a time, keeping the initial order until removed. The first item added to the kebab when constructing it is the very last to be removed and eaten. We must wait until all other pieces are gone before finally arrivinge back where we started.

Intestinal Fortitude

Our second course here at the data structure barbecue is a juicy and meaty collection of grilled sausage links. Filled with ground seasoned meats, seasonings, and aromatics, these stuffed links are bursting with flavor. The links of this chain are joined to their nearest neighbors by a thin brand, creating a long collection of heartburn-inducing, artery-clogging satisfaction.

Most of you have probably already figured this one out. This closely resembles the good old-fashioned linked list. Starting at one end, we can journey from link to link all the way to the other end. The order is fixed, but severing just one connection breaks the list into two. Losing a handle on the second half causes the whole chain to be dropped on the floor causing a mess and making it unusable (a memory leak).

Fresh From Chilliwack

I made a special trip down the Fraser Valley for our final course here at the data structure barbecue. Today's finale is grilled corn on the cob. I hope you held onto the toothpick-like skewer from the kebabs, because you're about to sink your teeth into some fresh, sweet corn. This final starchy treat contains a fixed collection of ears of corn. You can start wherever you'd like and easily access any area on demand, choosing to stay in one area or revisit places you've already covered.

The cob of corn closely resembles the classic array. It's very rigid in total size and arrangement of values, but easy to access any part at any time. The constraints of an array are limiting, but enable incredibly quick access to any item within the data structure.

Where's The Hash Table?

Over there next to the table with the macaroni salad, the snacks and all the fixings...

I guess not all things are fit to be grilled. I'll cover hash tables here because they are the essential building block for more advanced data structures such as associative arrays.

A hash table is a clever way to treat non-ordinal values (e.g. strings) as indices. The first thing to do is to define a simple to compute hash function taking the key values as input and computing a numeric hash value. A very terrible hash function could simply return a fixed value (this would result in a data structure equivalent to a linked list in most implementations). An ideal hash function given any random input would output any hash value with an equal chance. Similar input values would result in very different hashes.

Each computed hash value corresponds to a bucket containing all previously stored values. Each bucket may contain many results with the same hash value. Though there are a variety of implementations, from here the collection should be small and much more manageable. Traversing the collection, we look for a matching index and the associated value. In this way, hash tables allow us to quickly store and access a wide variety of data types using limited memory and direct object comparisons.

Come Again Soon

I hope you all enjoyed the first ever data structure barbecue grill. Please feel free to take home seconds and share with your friends. I hope the analogies weren't too labored. Please don't try to push them too far; I'm sure they'll break.

What are your favorite clever data structure analogies? Let me know in the comments.

Cheers,

Joshua Ganes

National Programming League Draft 2015

Did You Say Hockey?

One of the great things about publishing this blog is that it lets me write on topics I'm passionate about. While I blog primarily about software development, today I feel like discussing another passion of mine: hockey -- specifically the National Hockey League and the Edmonton Oilers. For those who say that hockey's not your game, please bear with me as I'm headed somewhere with this.

As a red-blooded Canadian, I've been raised in a hockey culture. Hockey in Canada is unavoidable. Like (American) football in the USA or soccer (football) in Europe, it permeates our culture. Nearly everyone is aware of the stars of the game and the latest exploits of the local team. For the die-hard fans, it goes much deeper and becomes a part of our identity. Unfortunately for me, the Oilers have been taking me through a bit of an identity crisis since I moved to Vancouver in 2007.

Office Pool

My love of hockey has led me to run the office hockey pool for both the regular season and the Stanley Cup playoffs. Our regular season pool is a draft pool where each owner drafts a fixed number of forwards, defencemen, and a goaltender. Each poolie collects points throughout the season based on his team and is allowed a limited number of trades. With more than half of this NHL season complete, it's interesting to see how our draft choices compare to the real numbers.

The first surprise is the busts: those players whose performance failed to live up to expectations. Prior to our pool draft, I distributed a spreadsheet of last season's top scorers. Most of our poolies based their picks (at least in part) on this list. Milan Lucic, and Jordan Eberle, and most notably James Neal were all dropped from our pool within the first few months due to their poor performance. It appears that a player's historical statistics are not always a sure indication of his future results.

Perhaps more surprising is the missed opportunities. Many players were overlooked during our draft and turned into great late-round steals. Several of the top players in the league were passed over and remained completely undrafted. This includes top-rate goaltender Pekka Rinne, and the current league-leading forward Jakub Voracek. The collective knowledge of our pool was not enough to identify these high-scoring gems before draft day.

Perhaps we're just terrible at hockey pools, you say. Maybe so, but before we wallow in self pity at our own ineptitude, let's see how the professionals fared.

The Real Deal

Each season, the NHL hosts its annual entry draft wherein each team attempts to select the league's next superstars. Teams spend massively in time and money to scout young prospects in preparation for the draft. Their one objective is to choose the best player available in each round. Excellent young draft picks develop into the core of an elite NHL squad, and are considered mandatory to build the next championship team. With so much on the line, how good is their track record?

When it comes to drafting forwards, it seems to me that the professionals have done a solid job. Of the top thirty scorers this season, the vast majority were selected in the first round, several were taken number one overall. There are a few notable exceptions such as Henrik Zetterberg and Joe Pavelski, each taken in the seventh round of their own draft year. As for goaltenders, the professional record appears much shakier. Many of the top goaltenders were selected well into the late rounds including two of this season's leaders in the wins column: Pekka Rinne and Jarolav Halak, who were selected in the eighth and ninth rounds respectively.

As for the busts, I feel the best place to look is the number one overall seed. The most coveted position in the draft, the privilege of the first overall pick allows one team first choice of hundreds of young up-and-coming players. While we can hardly expect the scouts to nail it every time, it seems fair to anticipate that anyone taken so early in the draft would surely develop into a NHL staple with consistent performance for years to come.

One of the all-time great busts was the number one selection of Alexandre Daigle by the Ottawa Senators in the 1993 NHL entry draft. After a promising rookie season, he consistently fell below expectations before bouncing from one team to another and leaving the NHL for Europe following the 2006 season.

Then there's my personal favorite draft bust: Patrik Stefan. Stefan was chosen first overall by the Atlanta Thrashers in the 1999 draft. After averaging less than half a point per game over his NHL career, the number one pick left the league after only seven seasons. He is now unfortunately best known for this debacle, missing a wide open empty net and allowing my Edmonton Oilers to tie the game with only seconds to left on the clock.

It appears that the NHL scouts, whose entire job consists of watching, interviewing, and evaluating young players still get it wrong sometimes. I give them all due credit for a tough job where the successes go mostly unnoticed and the failures expose them to public scrutiny, with the critics calling for their heads. Still, these scouts occasionally miss golden opportunities or unwittingly place their faith in the wrong player.

Programmer Draft

Imagine, if you will, that the first ever National Programming League (NPL) draft is a mere three weeks from today. Your company was lucky enough (or perhaps it just performed poorly last season) to be given the privilege of the first overall pick. On draft day you will have your choice of any active programmer in the world to build a your dream team around. How do you choose?

You could start by looking at famous programmers. There's not too many who have wide-reaching fame, but if you dig hard enough, you could probably come up with a list of maybe a few hundred programmers who could be recognized by several people they had never met. Does fame really equate with talent? Not where I come from. Certainly some programmers might be famous because of their talents, others may also be famous despite them.

Another place to look would be job sites and headhunters. Browsing through candidate profiles from a collection of sources, you could try to boil down all the varied types and qualities of information into a short list of candidates. From there you could further research each programmer on the short list and try to come up with your definitive number one pick.

An interesting approach would be to follow chains of references and recommendations. Start with a reasonably large collection of randomly selected programmers and ask each one to recommend the smartest programmers they know. By repeating this process recursively, you should eventually bump into the same highly-praised candidates time and again. Again, from that pool of recommended candidates, you could perform some final detailed analysis and evaluation to settle on one final choice.

With the strategies above, you could probably come away with a solid programmer, but how close could you come to identifying the true "best" programmer on earth? I think you might still miss by thousands of places.

I'll bet that some of my readers have even more clever schemes up their sleeves. Can you think of any better strategies, or are you just hoping that you'll be selected in the early rounds?

Programming Is Different

For hockey and other elite-level sports, all the work is done out in the open and under the public's watchful eye. Every shift, every hit, and every goal is recorded and tallied for later retrieval and analysis. We can watch healthy NHL players play 82 games each season and play back the tape in slow motion from every angle imaginable. Yet, somehow we still find it hard to predict tomorrow's score.

Unlike the world of sports, programming work is largely an independent activity and is typically done behind closed doors. Private companies protect their source code and their employee information. With open source, of course, you can see the fruits of their labor, but it is still very difficult to measure performance in a meaningful and statistically relevant way. There's no great programmer database where you can slice, dice, and graph statistics to your heart's content.

From my own experience, I know that employees who have worked at the same company for years can know little about their colleagues talents. Only by working closely with another developer on a variety of projects, can I even begin to gauge how truly talented (or not) they might be. How much harder would it be to evaluate the performance of someone I've never met?

To The Point

The thing that has me all fired up is that people sharing their "ideal" process for hiring a programmer.

At this point I must mention that my experience in this area is incredibly limited and I'm not really qualified to be giving out advice on hiring. (Now ignore this disclaimer and listen to me!)

I've seen this advice take many forms. Some propose a simple bullet-point test that can be completed in minutes, while others have devised an elaborate series of excruciating tests spanning multiple days and consisting of so much actual work that it might not even be legal anywhere on earth.
(P.S. please don't do this)

What I'm desperately trying to get across is that there is no such thing as the ideal hiring process for programmers.

There are certainly some tips and tricks that you can apply to weed out the riffraff and narrow the field to those with a higher chance of success. You can ask candidates to write simple functions on a whiteboard or using software. You could discuss the advantages of different data structures in a specific scenario. You might give candidates basic design problems and ask them to draw diagrams of their solution using boxes and arrows. You might discuss work history and how they've dealt with people problems. I'm sure if I sat here long enough, I could rattle off dozens more.

All of these hiring tips and tests will only get your so far, eventually you have to settle on a candidate who seems competent and talented given limited information. Even if you are lucky enough to hire an elite performer, it may take years before you can confirm that they are truly a cut above the rest.

Sorry To Disappoint 

Just as it is for the professional scouts, finding and hiring the right people for your technical team is an essential part of your product's success. I understand that this is an important problem, so I'm sorry that all I have to say is, "this problem has no ideal solution". I'll let you in on a secret though: if I ever discover a solution, it will be much too valuable to share -- at least until I can use it to earn some cold, hard cash.

Do you believe in an ideal hiring process? What tricks and tips do you use to identify promising candidates? Let me know in the comments.

Cheers,

Joshua Ganes

Porting To Windows CE - How Hard Could It Be?

What Tiggers Do Best

I've always considered myself someone who is quick on the uptake. If you throw a new challenge at me, I expect that I should quickly engage myself and learn the basics. Soon I will master the material and be unstoppable in future pursuits. There may be setbacks, certainly, but I'll learn from my mistakes and begin to climb the ladder to climb to a peak of any elevation.

A few years ago, I was tasked with just such a challenge. I was asked to take an existing application (one that I was not very familiar with) and port the code to run on Windows CE version 4.2. "How hard could it be?" I remember thinking. Porting code to run on Windows CE, that's what Tiggers do best!

After a bit of  stumbling around, I found Microsoft eMbedded Visual C++ 4.0 available for download. This seemed to be exactly the tool I needed. A bit more searching allowed me to find some service packs and an SDK with support for my target platform. Tools in hand ...or computer, I set off to modify the code.

I began a new Windows CE project and attempted to copy our existing structure using the new tool set. Wielding the power of preprocessor macros, I updated the code. A few functions were not defined. I was forced to use similar functions with the arguments slightly modified. There were a couple of places where I needed to implement my own replacement or grab existing libraries from the web.

All in all, the porting process progressed smoothly. I learned a few lessons and got away from the project reasonably unscathed. My work was installed sometime later and ran without any significant issues for quite some time.

You see, the real thing that Tiggers (i.e. programmers) do best is to learn and adapt to new technology. We are learners, problem solvers, and excellent at pattern recognition, logic, and technological intuition.

The Version Explosion

Some time later, I was tasked with porting the same application to Windows CE 5. With a bit more searching, I was able to find an SDK that allowed me to continue using the old development environment. I now had multiple platform configurations for the same application, but everything was still okay.

It seems that every six months to a year we found the need to support yet another flavor of Windows CE. Beyond Windows CE 5, it seems that every new OS version requires a new development environment. Worse yet, for platform developers, the platform builder requires yet another environment from the application development environment.

The following table was compiled by Werner Willemsens and lists the development tools required for targeting various versions of Windows CE:

Version	Platform (OS) Builder -> NK.BIN	Smart Device application
Windows CE 4.x	Windows CE Platform Builder 4.x	Embedded Visual Studio 4
Windows CE 5.0	Windows CE Platform Builder 5.0	Visual Studio 2005 + SDK
Windows CE 6.0	Plugin for Visual Studio 2005	Visual Studio 2008 + SDK
Windows CE 7.0	Plugin for Visual Studio 2005	Visual Studio 2008 + SDK
Windows CE 8.0	Plugin for Visual Studio 2012	Visual Studio 2012 + SDK

Take careful note that you will require the Microsoft Visual Studio "Professional" edition or higher in order to target Windows CE. As an alternative to Visual Studio 2005 in the table above, I was able to find a Windows CE 5.0 Standard SDK for use with Microsoft eMbedded Visual C++ 4.0 SP4.

In the link above, Werner goes on to explain how to limit the number of necessary development environments in far more detail than I wish to cover here. I would like to say a big thank you to Werner and the valuable information provided.

P.S. Did I mention that the tools above are not all compatible with the same desktop versions of Windows?

On CE's Struggles

Through my own experiences with Windows CE, I've developed my own theory as to why the platform struggled to gain traction. The theory is simple and kind of sad: Microsoft's development tools are so poorly strung together as to feel that they are actively fighting the developer.

I developed for Windows CE because our business needs directly required it. After spending many hours researching and following many paths leading to dead ends, I was able to piece together a working development environment. Even after doing so, maintaining an application for an array of different OS versions becomes unpleasant and expensive as you need to buy multiple costly software licenses when it would seem that the newer tools should be compatible with the older OS versions.

If, knowing nothing, I were starting a new project and given the choice between multiple platforms, I would probably not choose the one that feels like pulling teeth even before I write a simple "Hello World" application. I believe that this frustrating barrier to entry contributes significantly to developers looking at alternative choices.

A Faint Light

This is highly subjective, but I have a feeling that with each new iteration, Microsoft is beginning to clean up its act and making Windows CE development simpler and more streamlined. They may finally be turning the corner and arriving at a new golden age of embedded development.

As with many programming tasks, getting started with Windows CE development was much more difficult than I had originally anticipated. That said, if you tread carefully and find the right resources, you can create beautiful applications on this platform, too. Hopefully posting some of the information I've discovered along the way will help make the journey just a bit simpler for the next guy.

Cheers,

Joshua Ganes

The Cinnamon Twist Alert - Handling Complex Boundary Conditions

The Old Ways Are Not Always Best

A new bug report came in. After reading through the report, the problem was clear. Our system did not completely support single-day batches of over 999 transactions.

The problem boiled down to a single counter used to track transactions throughout the day. The field for the counter has a fixed width of three digits. The sequence counter field begins with 001 and increments by one for each transaction all the way up to 999. This value is used to help identify and correct communication errors. When a terminal attempts to send a request but encounters an error, we resubmit the transaction using the same sequence counter. If the other side receives two similar (duplicate) transaction records with matching sequence counters (more on this later), the earlier of the two submissions is reversed and is not funded.

The problem with the sequence counter becomes obvious when you think about what happens when the terminal needs to go beyond 999 transactions in a day. The sequence counter will spill over the three digits available, reusing a value from earlier in the same batch or the forbidden value 000. Our software was not so silly as to ignore this problem. The solution, as it was implemented, was to increment the batch number by one and reset the sequence counter for the newly-created batch to 001.

Unfortunately for us, one of our clients began frequently exceeding the magic 999 transactions per day and was experiencing problems with this approach. Processing multiple batches on the same day led to reconciliation and accounting issues, while causing delays in the deposits of funds to the client's bank account. Obviously, these were problems we wanted to deal with swiftly, once and for all.

Edit for clarity: Some have asked why I didn't simply increase the width of the sequence counter field to more than three digits. This width was defined in a third party specification and was not under my control. My software had to deal with this somehow.

A New Approach

After consulting the documentation and our technical contacts and running through a couple of false starts, we formulated a new approach to the three-digit sequence counter problem. Instead of creating a new batch to deal with the overflow, we would simply roll the sequence counter from 999 back to 001 and continue processing everything normally.

Our biggest concern with this new approach was related to the special duplicate transaction checking mentioned previously. The duplicate checking logic considers the following criteria when determining whether a subsequent transaction request matches an earlier one:

• Sequence Counter
• Card Account Number
• Total Dollar Amount

If all three of these values match, the earlier of the two transaction requests is silently reversed, leaving the transaction totals out of balance and the client short of money. For some silly reason, people get very upset when their money goes missing unexpectedly.

Mr. Cinnamon Twist

To describe a plausible scenario where I thought this might actually happen, I decided to write a brief

story about a man I dubbed Mr. Cinnamon Twist.

Cinnamon Bun
Cinnamon Twist
...whatever

Mr. Cinnamon Twist is a businessman with a sweet tooth. Knowing he has a long day of meetings ahead of him, he stops in at the busy corner coffee shop looking for a morning treat. He spies a delicious, gooey cinnamon twist (with double frosting). His growling, empty stomach simply cannot resist. Leaving the shop with cinnamon twist in hand, he takes two bites and wraps up the rest as he hurries to catch a train heading downtown. Through the early morning, Mr. Twist savors his treat as he goes about his work. He prepares his materials for the big afternoon presentation for a prospective new client. After a long and stressful day of work, Mr. Cinnamon Twist boards the train heading towards home. Worn out and feeling exhausted, his mind wanders back to his early morning treat. He decides that he will treat himself to another (just this once) before dragging his tired body home,

In this scenario, it's plausible that our sweet-toothed protagonist used the same credit card to pay the same amount for both a very low and very high sequence number. If all the stars aligned and these transactions happened to reuse the exact same sequence counter, this would mean that Mr. Cinnamon Twist magically received his first treat of the day without being charged for it. Great news for Mr. Twist, not so good for the coffee shop who would be out the cost of a scrumptious cinnamon-flavored treat.

Back Of The Envelope

The first problem with the above scenario is simply noticing it. Detecting this type of situation on the fly is hard enough. With the requirement to be fast, high volume, and redundant between multiple data centers, this becomes complicated very quickly. The second problem is how to correct the situation once an error has been identified. I could think of a few tricks that I might consider, but I saw no obvious trivial approach for this problem.

While trying to avoid tackling this complex condition, I paused to look at some data. How likely is the above scenario? I looked at some rough numbers to try to get an idea. I looked at the number of clients exceeding the magic 999 transaction limit. I looked at the number of transactions using the same card at the same merchant on the same day. Using the classic back-of-the-envelope approach, I calculated that we would likely only see this situation a handful of times in a year.

It seemed to me that we were looking at a lot of complicated and error-prone work to save the cost of a tray of delicious cinnamon treats each year.

The Compromise

As it turns out, there is a manual process available to correct these types of transactions. By picking up the phone and talking to a real live human being, we are able to manually single out a transaction request and force it through.

Knowing that this manual correction process was available and fearing the work required to fully automate every possibility, I proposed a compromise. We would create a scheduled script to run daily and search the database for requests matching the duplicate transaction scenario above. If any duplicates were found, we would fire off an email alert message (subject line: "Cinnamon Twist") identifying the key transaction details and describing the process for manual corrections. The worst case, I thought, was that the alert would fire too often and I would have to implement the complex solution later anyway. The best case, on the other hand, was that the alert would basically never fire, saving a great deal of time and effort.

Sounding The Alarm

The first week after installing my script, there were still no email alerts. I was beginning to feel optimistic that we may never see the alert fire in practice. They say that trouble shows up when you least expect it. The day after sharing my optimism with my coworkers, we received our first Cinnamon Twist alert.

No problem, I thought. We followed the manual procedure only to discover that both transaction requests were good and no corrective action was required. This contradicted the documentation and our general understanding of how the system should work, but who am I to look a gift horse in the mouth?

Another week or two went by before the next alert fired. It seems that my back-of-the envelope calculations were a bit off. We were receiving more alerts than I had expected. This alert, too, turned out to be a false positive when we followed up manually.

We asked our technical contacts for clarification. After our messages got passed around a few times, our contacts eventually got back to us saying that this behavior was by design. It seems that we had worried ourselves over a problem that didn't actually exist.

We disabled the Cinnamon Twist Alert script a short time later. My "lazy" approach had saved me from implementing a lot of complicated logic for no reason.

An Ounce Of Cure

What is my point? What can we learn from these events? Perhaps it's time to spin the wheel of morality to tell us the lesson we should learn.

Maybe I was lucky. My calculations turned out to be somewhat (but not excessively) optimistic. There was a risk that we would need to handle these manual corrections frequently, leaving me scrambling to implement a complex change to relieve pressure from the rest of my team as quickly as possible. My approach was a calculated gamble, but it paid dividends even larger than I had anticipated.

To me, this is a turnabout on the old adage saying, "an ounce of prevention is worth a pound of cure." In this case, an ounce of cure (the alert and manual correction) was quicker, safer, and simpler to implement than a pound of prevention (a fully automated solution). In rare cases, the easiest way to deal with complex boundary conditions is not to. Instead, find a way to look for the errors and tidy up after they happen. Don't forget to calculate the risk and the cost, but you may just discover that you were about to make much ado about nothing.

Cheers,

Joshua Ganes

Institutional Knowledge Is The Default

This article is a follow up to my previous post on the topic of institutional knowledge.

Do As I Say, Not As I Do

Please don't interpret my recent post as a claim of personal innocence when it comes to accumulating institutional knowledge. I have completed many projects in my time that are completely devoid of, or seriously lacking in adequate documentation.

I realized long ago that the only way to avoid becoming emotionally paralyzed by constant feelings of inadequacy is to acknowledge my own shortcomings and work hard to improve myself day by day. By staying disciplined and focusing on continuous improvement, my recent projects have been more thoroughly documented than those from only a few years ago.

The Pit Of Despair

Eric Lippert writes about the pit of despair as a place where the traps are easy to fall into and difficult to climb out of. Unfortunately, institutional knowledge fits this description to a tee.

We constantly pick up valuable little nuggets of information as we go about our duties. Sometimes these are technical details about the systems we're working with. Other times, it may simply be the knowledge of who is already an expert in a given area.Tapping into the institutional knowledge of others can be more valuable than struggling to discover everything for yourself.

There is nothing wrong with this knowledge in and of itself. This knowledge can be used to unlock further discoveries and make key decisions that allows us to avoid disasters and achieve success. The problem is that the knowledge is trapped inside a lone individual's head. Without further action, we end up continuously accumulating more and more institutional knowledge. Institutional knowledge is the default and we must act deliberately if we intend to avoid it.

Why We Despair

Knowledge is tremendously valuable. As G.I. Joe has taught us, "knowing is half the battle." This is why distributing institutional knowledge is so important to any group of people working towards a common goal. When knowledge is trapped within a single mind, its potential is limited to that one individual. Time is wasted, uninformed decisions are made, and existing work is duplicated unnecessarily. From a business perspective, institutional knowledge is clearly bad for the bottom line.

I am about to draw a moral line in the sand. Neglecting to share institutional knowledge is regrettable, but intentionally hoarding knowledge to the detriment of the team in order to further one's own selfish ends is reprehensible. This is comparable to the salesman who viciously defends his "territory" from his coworkers to protect his own commissions. Not only does it reduce the collective effectiveness of the team, but it fosters and air of hostility and inhibits sharing important details needed to succeed.

Scaling The Walls

How then, do we climb out of the pit of despair and tiptoe around the pitfalls waiting to drag us back down? I'm no expert on this topic, but I'll share some of the things I do in my attempt to scale the walls and share my knowledge with my coworkers.

One of the best tools available at my workplace for sharing knowledge is our internal company wiki. Any pages I create on the wiki are immediately available to be searched, read, and modified by our entire company. These days, whenever I start a new project I will immediately create a new wiki page describing the basic purpose of the project and how it will work. As I continue to develop the project, I frequently edit the page with the most up-to-date understanding of the available details. As for my writing, I try to follow many of Joel Spolsky's excellent tips for writing functional specifications.

Another great way to ensure you're not accumulating institutional knowledge is to pay attention to the questions people ask. Sometimes people ask lazy questions. When they ask about something you've already covered, simply point them to the relevant documentation. If, on the other hand, they've done their homework and still require missing details or clarification, consider this a flaw in your documentation. Recognize the flaw, modify the documentation, and think about how to improve for the next time around.

On the same token, any time you find yourself asking for assistance, it's a likely sign that someone else has a collection of hidden institutional knowledge. Ask them if there's documentation, and suggest (or insist) that they write some. If nothing else, write down whatever lessons you've learned from your interactions.

Do You Validate?

Words of caution: just because you wrote some documentation, it doesn't mean that it's adequate.

When it comes to documentation, if I can't find it, it doesn't exist. You may have written a 500-page treatise covering every last detail of uses and maintenance of your paper clip system including a full bibliography, glossary, and footnotes on every page. It pays me exactly zero benefit if I can't find the document after giving an honest effort to search for it in all the expected places.

Just because instructions are clear to you, that doesn't necessarily mean that they will be clear to everyone. Each person is familiar with his own style. Things that appear straightforward to you may be ambiguous or unclear to others. Instructions that seem obvious to an experienced user may involve hidden steps unknown to a novice.

A great way to check for these flaws is to ask someone to validate your documentation for you. I find this particularly effective in the case of a documented procedure. In the spirit of hallway usability testing, ask a coworker to start from scratch and try to achieve your documented goal. Watch from a distance and note every time that they get stuck or confused. Later, add additional notes for clarification. Once another person can follow your documentation with minimal fuss, then you can be confident that someone else can perform the task when you're gone.

Still No Expert

As noted previously, I am not to be considered an expert in these matters. Listed above are some tips that I've found useful in sharing my institutional knowledge with my coworkers. What are are the best tips and tricks you have for avoiding the pit of despair and sharing your own institutional knowledge? Tell me in the comments.

Cheers,

Joshua Ganes