tag:blogger.com,1999:blog-66311271024891303222024-03-05T20:13:09.866-08:00Amish ProgrammerMusings and gripes about software, design, and the world in general.Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.comBlogger29125tag:blogger.com,1999:blog-6631127102489130322.post-18030091208651715672019-06-11T06:54:00.001-07:002019-06-11T06:54:51.686-07:00As Soon As Possible and Shifts in Priority<h3>
Managing Priorities</h3>
<br />
I have been lucky so far in my career to rarely encounter unreasonable deadlines. Every so often something will force me to weather the storm and bear the pressure as the clock winds down. Some deadlines are soft and self-imposed (<i>e.g.</i> hoping to deliver on a new feature by next month). Others are firmer and come from promises made to clients, or fixed dates mandated by third parties.<br />
<br />
If you've been in the same position for a few years, it's likely that you balance your time between developing new software and features and supporting existing products with troubleshooting, bug fixes, and other software maintenance. This, of course, means making choices about which tasks to take on first. The act of prioritizing between live issues and valuable new features is by no means trivial. Simply spending the time to analyze the potential impact, costs, and benefits of each task can take as much time as a simple bug fix. Even after events have played out, it can be difficult to evaluate whether you made the correct choice.<br />
<br />
As a team leader, I am also tasked with building and maintaining an effective team. Part of that responsibility is prioritizing and assigning work for the members of my team. This includes communicating project status within my own team, and answering to those in other departments, who also have their own requests and expectations.<br />
<h3>
<br />As Soon As Possible</h3>
<br />
Every so often, someone will ask for something "as soon as possible" (ASAP!). This is a slightly less demanding way of saying that they want it <i><b>right now</b></i>! This can be understandable if it happens rarely. Every once and a while deadlines sneak up or catch a person off guard -- the unexpected may happen and needs to be dealt with quickly. The problem comes when ASAP is used too often or by too many parties all at the same time. As the villain, Syndrome, from <i>The Incredibles</i> says, "when everyone's super, no one will be."<br />
<br />
A big problem with ASAP is that there is an implied caveat that comes with its use. ASAP generally means as soon as possible without causing unreasonable harm. The problem with this caveat is that everyone has their own definition of what unreasonable harm might look like.<br />
<br />
This kind of request is practically guaranteed to cause friction between the two parties, with great potential for tension and conflict. The person requesting help may see their own problem as far more important than a mild interruption to the other party, while the person whose time and attention is being requested is likely to see the request as a significant imposition on their already busy schedule.<br />
<br />
This implied caveat is also dangerous due to its implications towards prioritizing two simultaneous ASAP requests. Not causing unreasonable harm also implies not interfering with something that is of greater importance or higher priority. Unfortunately, when I see two ASAP requests without much context, they can both appear to have equal status. This conflict cannot be resolved without gathering more information beyond, "because I said so." The burden of sorting out priorities often falls to the person who's attention is already stretched thin.<br />
<br />
Interrupting a focused worker with an unexpected urgent task can have terrible results. This breaks their flow, the ramifications of which are not to be underestimated. It also prevents them from whatever they were working on. Presumably, a task at the top of their to do list because it is a high priority. In addition to interfering with the task in progress, unexpected urgency and stress can impact the quality of work on the new task as well. This often leads to rushed work, cut corners, and overlooked errors that might otherwise be avoided.<br />
<h3>
<br />Setting Deadlines</h3>
<br />
One method to avoid the perils of ASAP requests is to set clear deadlines. This allows the requested party extra flexibility to fit the task into their schedule where it makes sense and provide feedback or ask for assistance if they feel the deadline will be missed.<br />
<br />
Deadlines seem a reasonable suggestion when there is a clear motivation for a specific delivery time and enough lead time to allow for flexibility. They are not always reasonable when the required delivery time is unclear or is obviously needed sooner than can allow for the flexibility of lead time. It's not reasonable to set a deadline for the ambulance to arrive when someone has been severely injured in a car accident. This is the truest definition of ASAP.<br />
<br />
Effectively setting a deadline is a difficult problem. It is much simpler to set a deadline when a project dependency has a clearly defined deadline already. For example, task A must be completed before task B and task B is due three weeks from today. If task B is estimated at one week, then task A must be completed no later than two weeks from today. However, if task B is simply an interesting new feature, then there is no compelling reason to set a hard deadline for task A. Yes, we want this, but we don't necessarily need it "right now". Setting a deadline In this example, and for many others, becomes difficult.<br />
<br />
When missing a deadline does not have immediate negative effects, prefer a soft deadline or target date. Distinguishing between hard and soft deadlines helps to express a rough prioritization of tasks, allowing for flexibility to accommodate unexpected changes and requirements. When it becomes apparent that a soft deadline might be missed, there is an opportunity for evaluation. Perhaps the original project estimates were too optimistic, or a different priority interfered with the schedule. This gives all interested parties an opportunity to observe the situation without creating undue pressure and interfering with focused work<br />
<br />
<h3>
Team Building</h3>
<br />
Teams are emotional beasts. We sometimes like to think of our coworkers like cogs in a machine, with each piece working together to produce the functions of the whole. While this might be a nice analogy, the truth is usually far messier. Each team member is a human with emotions and needs. Some days team members show up in a grumpy mood or become distracted by the many details of their complex lives. Just keeping team members satisfied, focused, and productive can be a full time job and requires not just technical expertise, but excellent social skills and awareness. Like it or not, the morale of the team is strongly connected to the quality of work they will produce.<br />
<br />
A leader's actions can make or break effective teams. Choosing to interrupt a project already in progress signals that a deeply engrossed team member's work is not really that important. Choosing to do so frequently or reversing decisions every other day can kill team morale and compromise the team's perception of their leader. Think hard before reassigning an urgent task to someone who is already engaged and focused. There are certainly circumstances that call for an urgent shift in priorities, but this should be exceedingly rare.<br />
<br />
The words we choose are often just as important. There are usually many ways to phrase something. Your choice of words, tone, and even medium (<i>e.g.</i> email, phone, in person) can send important signals. Choosing appropriate words and methods of communication can build rapport and improve morale. Handing down ever-changing edicts from above can erode even the best of teams.<br />
<br />
My recommendation is to avoid ASAP like the plague. In the rare case where priorities must shift urgently, take the time to communicate with your team Choose appropriate words to frame the situation and help your team understand the rationale for the change. Discuss this in person, if possible. If you are not able to address questions and provide reasonable feedback, the minds of your team may begin to imagine things are more dire than they truly are. Use tact and draw on all of your social skills to reassure the team that this is a rare emergency and the time has come to pull together, not fall apart.<br />
<br />
<br />
<br />
Do you have any stories of an ASAP gone wrong? Can you think of a time where you successfully navigated an unexpected shift in priorities? What happened? Tell me your story in the comments.<br />
<br />
Cheers,<br />
<br />
Joshua Ganes<br />
<br />
<br />Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-40004956688270911162018-12-05T20:57:00.000-08:002018-12-05T20:59:06.316-08:00The Tyranny of the Lowest Common DenominatorOne of my great frustrations in developing software that needs to interact with multiple versions, flavors, and configurations of external components and hardware is finding the simplest way to work with all components, each with varied capabilities.<br />
<br />
A simple and familiar real world example is credit card receipts and printers that support different character width receipts. Receipts include basic transaction details such as the time, amount, card information, processing details, and merchant identification. Many of these fields are mandated by the payment processor. Additional optional details may also be included, such as the employee or order details. This information must fit within different receipt widths such as standards of 24, 32, or 40 characters.<br />
<br />
In an ideal world with infinite time and resources, one would design software with flexible layouts and configurations to support a wide variety of printers with varying character widths, and even more advanced features such as colored, bold, or different sized text. Unfortunately, we're not living in an ideal world. In the real world, we sometimes need to make tradeoffs to save time and money, and make compromises to make things work adequately in multiple environments.<br />
<br />
Making short term tradeoffs and concessions often leads to a poorer final product due to the tyrrany of the lowest common denominator. In the case of credit card receipts, we can support multiple receipt width printers if we focus first on the narrowest 24 character width receipts and simply reuse this format for the 32 and 40 character printers by adding a little extra padded space. By restricting ourselves, we are left with a cramped space and receipts that look less than visually appealing. We save effort in terms of supporting all environments at the cost of the final receipt layout, hopefully finding time to one day return and improve things.<br />
<br />
Beyond this simple example, there are many cases where it may be simpler to throw a feaure away that cannot be supported everywhere. We may choose to reduce a feature's complexity and power in order to support multiple environments. Business is business, and these choices are not necessarily wrong, but it can be difficult to extract myself from the emotional impact of purposely selecting an inferiror design to accommodate shortcomings of a system outside of my control. The tyranny part comes in when the most inferior system dictates the experience for the superior systems, holding them to provide less than their full potential.<br />
<br />
To overcome this tyranny, sometimes we need to reject the inferior system that is holdng back the rest. When this is not possible, sometimes we can find a way to downgrade a superior solution and wedge it into the lesser environment. Given enough time we can define a system of support for different feature sets, and simply disable advanced features for systems that cannot handle them, but continue to support those features for better environments. And sometimes, when all the stars align, we find the time and resources to do it the ideal way simply because it's great.<br />
<br />
Cheers,<br />
<br />
Joshua Ganes<br />
<br />Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-83331882852873984352016-03-21T22:04:00.002-07:002016-03-22T06:49:53.631-07:00There's No Test Like A Production Test<h3>
Dress Rehearsal</h3>
<br />
Enter stage left. Pacing a slow and deliberate gait to center stage, it's time to deliver your lines. Dress rehearsal is here. After weeks of memorizing lines and working with your follow cast members practicing scene by scene, you're nearly ready for opening night.<br />
<br />
As the director walks everyone through each scene of dress rehearsal, small issues come to light that were never noticed before. Whether it's the wrong lighting arrangement, a misplaced prop, or a scene change that doesn't transition smoothly enough, this is the opportunity to make last-minute adjustments to make sure the show is prepared for success.<br />
<h3>
<br />Opening Night</h3>
<br />
The cast and crew take their places as the curtain opens. Opening night is packed with eager patrons and critics ready to experience the well-promoted show. The opening act comes to a close to a thunderous applause. Midway through the second act a stage prop malfunctions, causing an awkward pause and a stumbling delivery, breaking the audience's immersion in an otherwise excellent show. The rest of the show goes off without major incident, ending to a satisfying ovation.<br />
<br />
Despite all of the hard work and planning by everyone involved, one unexpected or anticipated error can mar an otherwise excellent performance. In a sufficiently complex production, no matter how much you prepare ahead of time, it's impossible to anticipate every possible thing that might go wrong. You must address any issues as they arise and learn to respond better next time<br />
<br />
<h3>
The Software Stage</h3>
<br />
The analogy is the same in the world of software. It does not matter how many layers of testing suites you wrap around your software. It does not matter that you've tested it in three distinct environments and subjected it to a wide array of contrived and random test data. It does not matter that you've gone over a test deployment twice and three times. You simply cannot know that your software is truly ready for production until it's put to the test with real users and data.<br />
<br />
I've been around this game long enough to see well-tested systems run into all kinds of surprising bugs. Many bugs would simply never have been caught, even by adding dozens of programming, testing, and hardware resources prior to production release. These bugs are typically inconsistent, difficult to reproduce, or related to users exercising the software in unexpected ways.<br />
<br />
It's a case of diminishing returns -- performing a variety of basic tests will generally pay immediate dividends as obvious, simple bugs are found. Performing additional rounds of testing will often flush out a few more. As more bugs are found, the rare gems, well-hidden beneath the surface, will become harder and more time consuming to find. Often, it is simpler to release the software and fix bugs as they are reported than to waste valuable resources looking for outliers.<br />
<h3>
<br />Raise the Curtain</h3>
<br />
As with the theater, it is true in software: the show must go on. Once the production is prepared and the dress rehearsal has worked through the obvious problems, the only thing left to do is to perform for the waiting public and wait for the reviews.<br />
<br />
If there's one thing I would suggest you take away from this, it's that there's rarely such a thing as a flawless release. Always expect that your software will fail in a way that you've never seen or anticipated. Therefore, be prepared to respond quickly, lest the critics give you a scathing review and the fickle public refuses to come back for an encore.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-43941806683950116842015-12-01T21:38:00.003-08:002015-12-01T21:41:25.786-08:00Security Month - Episode 3: Social Engineering<h3>
End Of The Month</h3>
<br />
This past month has sure flown past for me. I intended to create one more security month entry, but life got in the way and it will have to wait for another day. Today's post is based on a presentation I gave at work a few years ago on a topic that is probably the most important of all security: the human component. It doesn't matter how good your locks and alarm systems are if you neglect to turn on the alert and lock the front door as you leave your house. Similarly, it doesn't matter if your system is designed properly if your users do not know how to protect the security of the system and their own credentials.<br />
<br />
<h3>
The Conundrum</h3>
<br />
When designing a secure system, you must walk a fine line between burdening your users with a barrage of security measures and keeping attackers at bay. Legitimate users should be able to access the system quickly, easily, and with minimal interruptions while at the same time keeping it "impossible|" to access the software without permission.<br />
<br />
Typical applications rely on single-factor authentication (username / password). When authentication relies only on a secret key (the password), when a password becomes compromised, the system becomes compromised. The more users a system has, the more opportunities exist that at least one user will become compromised.<br />
<br />
<h3>
Social Engineering</h3>
<br />
It's often simpler to manipulate people than machines. While machines may be rigid in terms of interactions and enforcement of rules, humans tend to be more flexible -- especially when subjected to emotional influences. Social engineering is the process of "hacking" people to gain access to secure systems. This may involve tricking or convincing them to give up sensitive information such as passwords, or manipulating users into performing actions on the attacker's behalf.<br />
<br />
<div style="text-align: center;">
Mandatory xkcd Reference</div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAVOaBlCyxMEqxwjGZ91ap3LyhwqCdV9pFtszdyveIXbzVQZCOd15PH_rJDXtVkW7uMxSfNGKBvuqmU5RjDWdmDeK-N-NYT-1B-Zc2RjEZ-oyGwNGKgClkMtRm1BwA2O8V8hr4PoObftQ/s1600/xkcd+crypto.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="195" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAVOaBlCyxMEqxwjGZ91ap3LyhwqCdV9pFtszdyveIXbzVQZCOd15PH_rJDXtVkW7uMxSfNGKBvuqmU5RjDWdmDeK-N-NYT-1B-Zc2RjEZ-oyGwNGKgClkMtRm1BwA2O8V8hr4PoObftQ/s320/xkcd+crypto.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">https://xkcd.com/538/</td></tr>
</tbody></table>
<br />
<br />
<h3>
Pretexting</h3>
<br />
In order to manipulate users into making rash and emotional decisions, attackers may pose as authority figures and create plausible urgent scenarios where sensitive information needs to be divulged. This may include authentication credentials, or other details to be used in subsequent attacks. It's important to inform users to never give out their credentials to an unknown party regardless of urgency.<br />
<br />
<h3>
Phishing</h3>
<br />
In a phishing attack, a hacker will send a link to an illegitimate web site impersonating a legitimate page. A classic example is an email claiming to be from PayPal, and telling you that your account may have been compromised. It asks you to visit a link and enter your username and password. Ironically for those who are duped, the very attempt to save their credentials is what ultimately divulged them.<br />
<br />
It is very simple to copy a legitimate site using copy/paste tools and modify it for devious purposes. Many people are not savvy enough to know to look for a legitimate secure (https) address. Teaching users not to "click the dancing bunnies" and how to verify a legitimate web page is a battle we are constantly waging.<br />
<br />
<br />
<h3>
Baiting</h3>
<br />
A rarer and more targeted attack vector is baiting. In this attack, a hacker will leave a physical device such as a USB thumb drive somewhere to be found. They may leave it in a company parking lot with an enticing label in the hopes that curiosity gets the better of their victim. Upon inserting the device into a secure system, the payload is delivered to either steal information or run a program or script to do something on the attacker's behalf. Users must learn never to connect hardware of suspicious origin to a secure system.<br />
<br />
<br />
<h3>
Quid Pro Quo</h3>
<br />
<br />
A hacker may attempt to butter up users by giving them something, either physical or performing a service, in return for sensitive information. While this may come in the form of a bribe, it may be as simple as establishing a friendly relationship over several days and requesting a simple favor.<br />
<br />
Perhaps the most devious attempt I've heard described is to call random employees at a large company posing as "technical support". Many non-technical people will accept support on their latest computer problem and can be tricked into entering commands on the attacker's behalf.<br />
<br />
<h3>
All Good Things...</h3>
<div>
<br /></div>
<div>
If there's one lesson to be learned from this post, it's to always be on your guard. "Trust no one" is bad advice to live by, but essential advice when it comes to sensitive credentials. Computers do not have human judgement and will gladly accept validated credentials from the shadiest of characters. As is demonstrated so well by social engineering, humans and machines do not behave the same way and your interactions with them ought not follow the same rules.</div>
<div>
<br /></div>
<div>
That brings us to the end of Amish Programmer security month. I hope you learned something new, or at least refreshed yourself.</div>
<div>
<br /></div>
<div>
Here's wishing you a merry Christmas and a happy New Year. I hope you return next year for some non-security topics here at Amish Programmer.</div>
<div>
<br /></div>
<div>
Cheers,</div>
<div>
<br /></div>
<div>
Joshua Ganes</div>
Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com1tag:blogger.com,1999:blog-6631127102489130322.post-11546562275009875292015-11-02T22:05:00.002-08:002015-11-03T06:38:53.705-08:00Security Month - Episode 2: Storing Passwords<h3>
Trust No One</h3>
<br />
Two weeks after being promoted from Junior to Intermediate Programmer at Paranoid Software Ltd. you've been tasked with implementing a secure sign-in system for the company's new up-and-coming website. Due to "government conspiracies" and myriad other reasons, your tinfoil-hat-wearing boss has mandated that you use no third-party libraries for passwords and authentication. Only his most trusted and loyal employees are qualified to create something so important and sensitive. He'll be reviewing your work at a later time in his secure bunker...<br />
<br />
Not being allowed to research industry best practices on this subject via the corporate internet connection (again, "conspiracies"), you sit down to draw up designs for a new database table as follows:<br />
<br />
<table border="1">
<tbody>
<tr><th>UserName</th><th>Password</th></tr>
<tr><td>WayneGretzky</td><td>Gr8One99</td></tr>
<tr><td>SidneyCrosby</td><td>SidTheKid87</td></tr>
</tbody></table>
<br />
Each user will sign using a unique user name and corresponding password. When the user submits the sign in form, the submitted credentials will be compared against the database values found in the table and access will be granted or denied accordingly. This design allows for passwords to be verified while remaining flexible enough to allow for additional new columns for features such as temporary passwords, expiration, and more.<br />
<br />
<h3>
Plain Text Woes</h3>
<br />
You set to work building a new sign in page based on this design. Everything is coming along fine until forgetful Tom drops by your desk after your coffee break. "Looking for something, Tom?", you ask casually. "Maybe...", says Tom, "... I can't remember what I'm looking for, but that's not the only reason I'm here..." Tom scratches his head for a moment before his face changes in a moment of sudden recollection. He begins to explain how he can only remember one password for everything, and he's afraid of what certain other employees might do if they copied his password out of the new database table. They could potentially use it to access his account on another system.<br />
<br />
Tom's a nice enough guy with some serious clout and connections around the office. You don't want to upset him, so you revise your design as follows:<br />
<br />
<table border="1"><tbody>
<tr><th>UserName</th><th>EncryptedPassword</th></tr>
<tr><td>WayneGretzky</td><td>Te8Bar99</td></tr>
<tr><td>SidneyCrosby</td><td>FvqGurXvq87</td></tr>
</tbody></table>
<br />
In this revised design, the passwords are stored encrypted using the "ultra secure" <a href="https://en.wikipedia.org/wiki/ROT13">ROT-13</a> algorithm. When a user's sign in credentials are submitted, the database value is decrypted and compared against the submitted password to authenticate the user and grant or deny access to the system.<br />
<br />
<h3>
Clever Estelle</h3>
<br />
As your project nears completion it needs to pass a review from Estelle, the company's oldest programmer. Estelle got her start in the industry working with VAX machines in the '70s. Nobody knows the details of her top-secret projects, but her office is strewn with archaic servers and mainframes cobbled together with assorted wires and duct tape. She left a barely-legible handwritten note on your desk asking you to stop by for a review of your project.<br />
<br />
A light blue haze of smoke oozes out from under the door as you approach and knock briskly. You hear a loud cough before Estelle's unusually deep and raspy voice beckons you to enter. As you enter, you notice the yellow-brown stained walls of her private office. Distracted, you trip over an old server, yanking loose a bundle of wires. "Don't worry", Estelle reassures you, "that machine deserves it!"<br />
<br />
Estelle likes your design, but due to her advanced technical savvy, she's figured out how to decrypt the passwords stored in your database and suggests that you use one-way hashes instead. She hands you a ream of fan-fold paper retrieved from an old dot-matrix printer. "Here's some materials I've prepared for you," see grunts as she urges you towards the exit.<br />
<br />
After reading her notes about a wide variety of one-way hash algorithms, you once again draft up a design for your database table:<br />
<br />
<table border="1"><tbody>
<tr><th>UserName</th><th>HashedPassword</th></tr>
<tr><td>WayneGretzky</td><td><span style="font-family: "courier new" , "courier" , monospace;">$2a$12$/Ox24ETwuRKwqCASxD40x.O</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">7Y1HOuaLzVp7nwf6FwUQjnRGnJok0m</span></td></tr>
<tr><td>SidneyCrosby</td><td><span style="font-family: "courier new" , "courier" , monospace;">$2a$12$o7chUYiQh0htw463/m5df.o</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">lmVQzkIjINY9CyyQmtC7xHV53OxtEC</span></td></tr>
</tbody></table>
<br />
This time there is a subtle change to the password verification logic. Instead of decrypting the stored password, the submitted password is passed through a one-way hash (in this case bcrypt with 12 iterations) and the hashed values are compared to see if they match.<br />
<br />
A good hash algorithm is easy to calculate, but nearly impossible to reverse. They also distribute hashes effectively to avoid collisions (two similar strings hashing to the same value), which makes them statistically secure for use in verifying that you have a matching password.<br />
<br />
<h3>
Tom's Back!</h3>
<br />
Having passed Estelle's second review, your project is mere days away from being installed. You're enjoying a well-earned break in the lunch room when Tom pokes his head in looking confused. "I'm looking for someone," says Tom, "but I can't quite remember who... Oh yeah, it's you!" You notice that Tom is holding the same dot-matrix print-out about hashes in his hand. He explains that he's found a way to break your password system. He's created a script to generate every string up to an arbitrary length, calculate its corresponding bcrypt hash, and store those in a new indexed database. With this upfront work complete, he can use the newly indexed table to perform a reverse password look-up from the hash of any password short enough to be covered by the script. He calls this generated table a "rainbow table". This could potentially be used to reveal thousands of user's passwords.<br />
<br />
After racking your brain late into the evening, you modify your design one last time:<br />
<br />
<table border="1"><tbody>
<tr><th>UserName</th><th>RandomSalt</th><th>HashedSaltedPassword</th></tr>
<tr><td>WayneGretzky</td><td><span style="font-family: "courier new" , "courier" , monospace;">PQw#{hC*bDvyHnhVeOGaVOnG</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">dcsiAYETeNBbannZbdtu?kLN</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">dDXjbfDe?slovDjJfw$m2uGf</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ZRneg-7bDJeeVwSwDGEsonI4</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Cini1p*zuJ&HwJRgC 3grzzU</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">c!EKxrSt</span></td><td><span style="font-family: "courier new" , "courier" , monospace;">$2a$12$tasiR59CA7uefis0.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">4r4CuTjo2gD7MkP3Mzzn77yr</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">/QWuRbV8kyf.</span></td></tr>
<tr><td>SidneyCrosby</td><td><span style="font-family: "courier new" , "courier" , monospace;">;ZjmYbJeNMstnqu\q|gbLpfs</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">zb@6PVyOoglbBuLUb-Hulgau</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">nwCR.qvCUikOEkEQroqi8lRY</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">J4gVA{VwlWwh0nyQnZG5JYCm</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">k.cStNNyy|hzgiIqPjSdTlkh</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">zxaZIi_C</span></td><td><span style="font-family: "courier new" , "courier" , monospace;">$2a$12$o3aiDUMmSDqX9/fpA</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">SJ/VOB483GsYvtfESeZbrON.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Gz080gVtlBqW</span></td></tr>
</tbody></table>
<br />
In this final incarnation, each password gets assigned its own unique random "salt" value. This value is first concatenated with the password. This "salted" password value is then run through the hashing algorithm to populate the database. When the a user submits a password at sign in, this process is repeated with the submitted password. It is first concatenated ("salted") with the stored random salt, then hashed. The final result is compared the hash stored in the database to grant or deny access.<br />
<br />
Your tinfoil-hat-wearing boss emerges from his bunker to shake your hand for a job well done and asks you if you wouldn't mind helping him recover his password...<br />
<h3>
<br />Transition</h3>
<br />
The best way to secure your users' passwords is to protect them in such a way that even a programmer with full knowledge and access to the system cannot get at them. A corollary of this is that if you ever submit a lost password request to a website that returns your password in plain text, you know they're doing something wrong.<br />
<br />
Several years ago I came across a collection of sites with multiple sign in systems that were at various stages of progression from the story above. Some were storing passwords in plain text, some were storing them using different forms of simple encryption, and others were hashing the passwords (using varying strengths of hashing algorithms) without salting them first. It was my job to bring these all up to snuff without breaking live systems with active users.<br />
<br />
The plain-text passwords were the simplest. I already had the passwords -- I salted and hashed them and moved along.<br />
<br />
The encrypted passwords were almost as simple. I decrypted the passwords to their plain text form, then treated them as in the plain-text case above.<br />
<br />
The hashed passwords were the most challenging. I created an extra database column to list the hash algorithm and imported the hash values into the database with no salt value. When one of these users signed in and was authenticated, I could then use their submitted password value to populate a strong salted hash value on their very first visit under my new system.<br />
<br />
There were also some accounts that were not so active. We conveniently had a 90-day password expiration policy. After 90 days, I ran a script to reset all of the remaining passwords to new temporary values and purge our system of all insecurely stored user credentials. Any users who had not signed in within 90 days would have to use our password recovery system to regain access to their accounts.<br />
<br />
<h3>
Take It Away</h3>
<br />
If there's one thing you should take away from this episode it's that you should do your best to store passwords such that they cannot even be recovered by someone with inside knowledge.<br />
<br />
If you know of any code that stores passwords without securely salting and hashing them, please do your best to correct the situation. If that code belongs to you, please go fix it. If not, please write a friendly but firm reminder to get those passwords under control and securely stored.<br />
<br />
Cheers,<br />
<br />
Joshua Ganes<br />
<br />
P.S. In case you missed the joke, ROT-13 just rotates the letters through the alphabet by 13 characters. Please don't use this for any form of security.Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com1tag:blogger.com,1999:blog-6631127102489130322.post-50526776751926822812015-10-27T20:35:00.000-07:002015-10-29T21:24:32.416-07:00Security Month - Episode 1: Injection<h3>
Starting With A Bang</h3>
<br />
Injection attacks, in all their various flavors, are now the number one most exploited security flaw in software. Their impact can range from something as innocuous as displaying extra text on a web page to the full compromise of your host machine.<br />
<br />
<h3>
Setting A Bad Example</h3>
<br />
Do you see anything wrong with the following snippet of PHP code?<br />
(Note: this is a simplified example for demonstration purposes only)<br />
<br />
<code>
// Initializes necessary variables, functions, etc.<br />
require_once( "important.libs.php" );</code><br />
<code><br /></code>
<code>$accessLevel = GetAccessLevel();<br />
<br />
<snip><br />
<br />
// Defaults to empty string if not present<br />
$name = Posted( "name" );<br />
<br />
// Find contacts with matching first name in database<br />
$sql = "SELECT * FROM Contacts WHERE ".</code><br />
<code> "FirstName = '$name' AND ".</code><br />
<code> "AccessLevel <= $accessLevel";<br />
$result = $db->query( $sql );<br />// We should really do some error checking here<br /><br />
// Print details for each match found<br />
while( $row = $result->fetch_assoc() ) {<br />
<br />
$firstName = $row[ "FirstName" ];<br />
$lastName = $row[ "LastName" ];<br />
$phone = $row[ "Phone" ];<br /><br />
print "Name: $firstName $lastName, Phone: $phone<br />";<br />
<br />
}<br />
<br />
<snip> </code><br />
<code></code><br />
As you might have guessed from the title of this article, this code is vulnerable to injection. In fact, it is subject to two common forms of injection: SQL injection, and cross-site scripting (XSS).<br />
<br />
<h3>
What Is Injection?</h3>
<br />
Injection happens every time we receive input from an untrusted source and submit it to an interpreter without first sanitizing the input. In the case of SQL injection, that interpreter is the database interpreting the text of our SQL query. In the case of cross-site scripting, the interpreter is the browser attempting to display our HTML page. Whatever the scenario, allowing potentially dangerous user input to be interpreted as a trusted command or request can have disastrous .consequences.<br />
<br />
<h3>
What Could Go Wrong?</h3>
<br />
In the scenario above, imagine that a hacker wants to get a list of all contacts in the database. They are currently restricted only to those contacts whose AccessLevel value is lower than a configured value. How could they use SQL Injection to gain access to everything?<br />
<br />
What if, in the form that requests this page, they entered "<i><b>' OR 1 --</b></i>" as the name to search? What would our query look like now? (Assuming AccessLevel = 2)<br />
<br />
<code><b>SELECT * FROM Contacts WHERE FirstName = '' OR 1 </b><i><span style="color: #93c47d;">--' AND AccessLevel <= 2</span></i></code>
<br />
<code><br /></code>
With a new terminating quote and comment, suddenly our query is changed to be much more permissive. The attacker has tricked our script into revealing more information than intended. What if instead of a SELECT, we were working with a DELETE query? The entire table full of valuable data might be lost.<br />
<br />
The second vulnerability is to a cross-site scripting attack. What if, instead of entering their name, an attacker entered some raw HTML such as "<b><i><script>alert('XSS!')</script></i></b>"? Suddenly, they've inserted their own elements and even javascript code into your web page. If they can get this to happen on a page displayed to another user, the attacker can hijack your page to show you them anything they want and it will look like it's coming directly from your page.<br />
<br />
<h3>
It Could Be Worse</h3>
<br />
Imagine a different scenario where input is passed to an exec() call and run on the system. Now instead of limiting damage to the database system, a clever attacker can run commands directly on the server. Depending on the privilege of the process and security of the underlying system, the attacker could potentailly gain root access to the machine. They can use this to access sensitive data, change important settings, or even run any program of their choosing. The entire system can be compromised and placed at the mercy of a malicious hacker.<br />
<br />
<h3>
Don't Forget Secondary Sources</h3>
<br />
Here's one that can really trip you up. Imagine an application that performs a task based on values stored in the database. It's easy to forget that despite the fact that the database belongs to you, the source of your database values may not always be clean. "We've traced the call, and it's coming from inside the house." Always assume any source of data is dangerous unless you can verify its source and security.<br />
<br />
<h3>
How Can We Fix It?</h3>
<div>
<br /></div>
The good news is that there is a simple way to protect yourself from injection attacks: sanitize user input using readily-available libraries.<br />
<br />
If your language of choice is PHP, there's plenty of built in functions to safely encode all varieties of user input:<br />
<ul>
<li><a href="http://php.net/manual/en/mysqli.real-escape-string.php">mysqli::real_escape_string()</a> - Because you "really" want to escape your SQL inputs.</li>
<li><a href="http://php.net/manual/en/mysqli.prepare.php">mysqli::prepare()</a> / <a href="http://php.net/manual/en/mysqli-stmt.bind-param.php">mysqli_stmt::bind_param()</a> - Use this to insert input paramaters into the middle of your SQL query similar to the classic C <b>printf()</b> function.</li>
<li><a href="http://php.net/manual/en/function.htmlentities.php">htmlentities()</a> - Used to escape user input to include in your HTML page or web app.</li>
<li><a href="http://php.net/manual/en/function.escapeshellarg.php">escapeshellarg()</a> - Always be careful when interacting directly with the shell from any application. Don't pass a shell argument without it.</li>
</ul>
<div>
Please remember that this is not a comprehensive list. There are so many different languages and environments out there. Whenever you're wielding user input, be careful to consider whether your code may be subject to injection attacks and research appropriate measures to counter such attacks.</div>
<div>
<br /></div>
<h3>
Where Do We Go From Here?</h3>
<div>
<br /></div>
<div>
If you feel this post is yesterday's news, I hope you take it as a friendly reminder and refresher on the topic. If any of this is new to you, I suggest you take the next opportunity to study up on the subject. Read some other blog posts. Follow some examples online. Revisit a piece of recent code and try to break it (in a test environment please!). Once you've found some vulnerable code, learn the proper way to secure it and and run your tests again to prove that it's been fixed.</div>
<div>
<br /></div>
<div>
If there's one thing to take away from this it's this: always treat user input as suspect. Never let raw user input pass from your application to another system without sanitizing it first.</div>
<div>
<br /></div>
<div>
Now go and write some secure code.</div>
<div>
<br /></div>
<div>
Cheers,</div>
<div>
<br /></div>
<div>
Joshua Ganes</div>
Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-77118004867286626022015-10-22T21:10:00.001-07:002019-06-12T09:43:46.399-07:00Announcing Amish Programmer Security Month<h3>
From Humble Beginnings</h3>
<br />
I started programming from a young age. I distinctly recall being in grade 5 and being curious about how some of the games in our school's computer lab worked. Having absolutely zero clue, I checked out a book from the school library on programming for the <a href="http://en.wikipedia.org/wiki/Apple_IIe">Apple IIe</a>. My first programming experience was typing out Apple BASIC programs straight from the book and watching them fail as my two-fingered hunt-and-peck typing resulted in multiple typos.<br />
<br />
My exposure to computers was reasonably limited until my sister and I pooled our money to buy a 100MHz 486 (with TURBO on for maximum fastness). I'm pretty sure I got the better end of that deal as I began spending a lot of time both playing video games and experimenting with <a href="https://en.wikipedia.org/wiki/QBasic">QBASIC</a> and its elaborate but confusing help files.<br />
<br />
In high school, I took a few courses in Turbo Pascal and C++ programming. This mostly involved running through a bunch of easy exercises in each chapter and then spending the rest of our class time playing in the lab or working on our frighteningly terrible video game group project. That Pascal text book may be the only high school textbook that I've ever read from cover to cover.<br />
<br />
After putting four years of university and another year's paid internship under my belt, I was sure that I knew almost everything I would ever need to know as a professional software developer. I had no idea just how much more I still had to learn...<br />
<br />
<h3>
What Do You Mean This Is Insecure?</h3>
<br />
With so many courses behind me, I was sure that I had covered most of the important topics. If security was so important and difficult, surely someone would have covered it in one of my course by now. I was surprised when one of my colleagues pointed out security flaws in my code. I did not even recognize it as a flaw when it was pointed out to me.<br />
<br />
Over time, with the help of more experienced mentors, bloggers, and some job-sponsored training, I began to learn a lot more about secure coding techniques. For many of my experienced readers, these have likely become second nature. Still, everyone can stand a refresher on the basics now and again. For those who, like I once was, may still be in the dark about secure coding, you have a great deal to learn.<br />
<br />
<h3>
Amish Programmer Security Month</h3>
<br />
With that in mind, I declare the next month to be security month here at Amish Programmer. I will strive to get back on board the posting train and publish a brief post covering such security topics as injection, buffer overflows, and more.<br />
<br />
These topics have all been covered in depth by others. I will try to present a fresh take on these topics using code snippets and straightforward examples. You can tune in in the coming weeks for a short discussion of several fundamental topics in secure coding. I hope to see you back here soon.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-87563583608868509212015-10-14T21:22:00.000-07:002015-10-22T21:20:47.111-07:00Challenge Your AssumptionsToday's brief post was inspired by my recent road trip with my wife from here in Vancouver, BC through the states of Washington and Oregon.<br />
<div>
<br /></div>
<div>
When it comes to road trips, I'm all about efficiency. Pit stops run like clockwork. The guys from NASCAR have nothing on me. That's why I prefer self service with pay-at-the-pump when stopping for gas. My fingers fly with precision as I fetch my card from my wallet, insert it in the machine and punch my way through the menu to start up the pump and select my grade of gasoline to begin fueling.</div>
<div>
<br /></div>
<div>
On my first stop of the trip, everything was running smoothly right up until I was stopped in my tracks by the following prompt:</div>
<div>
<br />
<b>Enter Zip Code:</b><br />
<br />
<span style="border-style: solid; border-width: 1; margin: 1em; padding-left: 2.8em; padding-right: 2.8em;"> </span>
<br />
<br />
<b>Cancel OK</b><br />
<br />
Unfortunately for me, as a Canadian, I don't have a numeric 5-digit zip code. Instead, I have a six character postal code with a mix of numbers and letters. Try though I might, I could not find a way to enter my postal code or skip this prompt.<br />
<br />
<i><span style="font-size: x-small;">Edit: I've since been told that you can simply enter the numeric portion of your postal code followed by zeros</span></i></div>
<div>
<br /></div>
<div>
After a few moments of poking around at the machine, I eventually resorted to pressing cancel and walked into the station to speak with the cashier. Within a few minutes I was back on track (a lap behind the race leaders).</div>
<div>
<br /></div>
<div>
My experience with the zip code prompt was hardly life altering. Still, it illustrates some of the problems that can come up when your software makes incorrect assumptions (e.g. all gas station customers have a zip code). Challenge your assumptions and do your best to consider all possible users. Question whether all the information you prompt for will be available or match your expected format. If not, your software should gently guide the user on how to proceed.</div>
<div>
<br /></div>
<div>
Cheers,</div>
<div>
<br /></div>
<div>
Joshua Ganes </div>
<div>
<br /></div>
<div>
P.S. Did you see the seventh inning of today's Blue Jays vs. Rangers game? It was well worth a second look! Go Jays!</div>
Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-26858770089860982082015-05-27T20:41:00.002-07:002015-05-29T08:10:53.916-07:00Any News? - Working With Third Parties<h3>
I Am A Rock</h3>
<br />
We programmers tend to be an independent bunch. Like a lone wolf, we often prefer our own company to that of others and like to work alone. It's been said that, "no man is an island." I have to agree (<a href="https://www.youtube.com/watch?v=My9I8q-iJCI">unless you happen to be Paul Simon</a>). Even programmers need to interact with their coworkers, managers, other departments, and even third-party contacts to meet the day-to-day demands of their jobs.<br />
<br />
While networking and politics may not come naturally to all developers, we're hardly worthy of our titles if we cannot find a way to communicate clearly and accurately. Programmers must express requirements, prerequisites, technical issues, and design ideas in ways that others can understand so that all players can do their jobs and assist us in completing our own work.<br />
<br />
Have you ever found yourself in this situation? You're beginning to wrap up a project, but you notice a seemingly simple, yet annoying bug in the library you are working with. Being diligent, you follow up with the library provider to report the issue. You ask if this is a known issue. You ask if you're using their library correctly. You ask if there are any workarounds or alternative approaches they can suggest. You go ahead and make a note to yourself to revisit the problem once the provider has had time to review your questions. Fast forward a few weeks. You've cleaned and polished every other corner of your software, but you really need to address this one remaining issue before you can build a final software release. In the meantime, the only feedback you've received is that they're, "looking into it."<br />
<br />
<h3>
On Dependencies</h3>
<br />
A wise piece of advice for any project is, "eliminate the dependencies." Just as in the example above, making yourself dependent on someone else's assistance can be a very compromising position. Not being able to deliver a solution because you're waiting for someone else to act can be uncomfortable, awkward, and frustrating. If you can eliminate these dependencies, you may still run into roadblocks of your own, of course, but you will retain the power to work through them and implement a solution dictated by your own schedule and budget.<br />
<br />
It's not really fair to say, "eliminate the dependencies", and leave it at that. For many developers, creating a dependency is not really a choice. Maybe your boss, team, or client is demanding a specific technical requirement. Maybe you can't afford any alternatives. Maybe, as is often the case for me, your work involves creating an interface between your own software and a third-party tool. In this case, if you eliminate the third-party tool, you eliminate the need for your own software entirely.<br />
<br />
Now, before I go accusing anyone of wrongdoing, let's try the shoe on the other foot. The development team in charge of that third-party library has more than just one bug fix on their plate. They need to balance support and bug fixes against their company's need for revenue-generating new features and routine maintenance. They may have a significant collection of issues that go through a triage system, and your bug report hasn't yet make the cut. I'm sure there's several times when others were waiting on me to complete my fix before they could proceed. Any fingers I point here need to be pointing back at myself also.<br />
<br />
<h3>
Planning For Success</h3>
<br />
The key to managing the risks of creating project dependencies is to identify those dependencies and understand them. Do your best to perform a reasonable analysis of the risks (if possible). Try to understand the expected flow of communications between both parties. Manage expectations by discussing project timelines and turnaround times for both technical inquiries and potential bug fixes. Understand the implications for your project when you cannot get the other side to take quick action. Are there alternatives? What are they? Will they be just as risky? If the risks are too high, sometimes <a href="http://en.wikipedia.org/wiki/WarGames">the only winning move is not to play</a>. Don't allow yourself and your organization to be sucked into wasting resources on an endeavor that will never succeed.<br />
<br />
One of the best ways to reduce delays due to third parties is to be prudent and up-front to find and report any issues. Fixing bugs takes time. The sooner a bug is reported, the sooner it will be fixed. Therefore, schedule any development and testing work dependent on third-party support at the earliest opportunity. As soon as an issue is discovered, take time to understand the problem in reasonable detail and communicate it as soon as possible. This will give everyone the most time to respond and allows you to proceed with other features until the bug can be resolved.<br />
<br />
Once you have requested support from an outside party, follow up and keep in touch. Remember, the contact at the other end is a person too. Treat them with respect and professionalism. Keep your communications positive and inquisitive. Try not to accuse or pester. There are differing opinions on how frequently you should send messages, so I won't offer specific advice. Instead, I suggest you give your best effort to judge the tone of their replies to see if the other side has become annoyed. Never hesitate to follow up if a written deadline or estimate is missed. Your goal, again, is not to accuse, but to make sure you are fully informed and kept up to date.<br />
<br />
If worst comes to worst, don't be afraid to escalate. Give your best effort to work through the default channels provided. If, however, you simply cannot get the answers you need, the time may come where you need to ascend the ladder of power. Do your best to do so as respectfully as possible. Ask your contact if there is an option to expedite or escalate the communication process. I have seen cases where two lower-rung employees went back and forth for weeks. A single brief phone call between two big wigs was all that was required to sort things out. Nobody wants to be disrespected and have someone go over their head. Reserve a power move like this as a last resort, but don't be afraid to pull the trigger if all other options have been exhausted.<br />
<br />
The very last resort sometimes required implementing a heroic workaround. For this to work, you need to understand the third-party system well enough to automatically recognize the failure and wield the power to fix it through one or more additional steps. Rather than having the third-party implement a fix, you can choose to live with the existing bug, masking its existing by coding your own workaround fix. On a personal note, while this option is often the best business decision, it often feel like an enabler. We just let the bug continue to fester and bend over backwards to get past it rather than correcting it at its source.<br />
<br />
<h3>
Good Days And Bad Days</h3>
<br />
In my professional experience, I have worked with a fair number of third parties. Some have been eager and helpful. They are quick to respond and provide clear details needed to solve my problems. Others have been slow to respond and very terse and unclear in their replies. After working through a few issues with a contact, you will quickly develop an opinion of their competence. This can be used in further risk assessment. My advice is to go with your gut. So far, I have never once changed my opinion from that initial impression following subsequent communications.<br />
<br />
What's your best or worst story of working with a third party? What went right and what went wrong? What would you do differently? Let me know in the comments.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-53159852041182967712015-03-30T22:03:00.001-07:002015-03-30T22:06:18.939-07:00The Data Structure Barbecue Grill<h3>
Charcoal, Fire Starter, Match</h3>
<br />
Recently, a colleague asked me a question about the implementation of a hash table. I found that I had to stop and think for a moment. My explanation was by no means the smoothest delivery. I tripped over my words a few times and needed to catch myself and backtrack before settling on the correct details. This incident has convinced that it's time for a refresher on one of the fundamental components of computing science: data structures.<br />
<br />
Rather than dusting off my old text books and violating a copyright or two, I thought I'd take a new approach inspired by a conversation with yet another colleague of mine. With spring in full bloom here in Vancouver, it's time to pull out a stiff wire brush and begin cleaning the grill. Consider this your invitation to the first ever data structure barbecue.<br />
<br />
<h3>
On A Stick</h3>
<div>
<br /></div>
<div>
<a href="http://www.pd4pic.com/images/kebab-grill-skewers-shish-kebab-skewers-beef.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://www.pd4pic.com/images/kebab-grill-skewers-shish-kebab-skewers-beef.jpg" height="213" width="320" /></a>While my usual preference is for a nice steak, there is one barbecue item that is nearly as much fun as it is delicious: the kebab. The scrumptious kebab is a collection of seasoned meats and vegetables skewered onto a single stick.</div>
<div>
<br /></div>
<div>
How is this tantalizing treat comparable to a data structure? When building a kebab it's impossible to add new pieces to the middle, but poking a new item onto either end is easy. Once cooked, it's far simpler to remove and eat from either end than it is to chew off the center. This matches the properties of a double-ended queue. You can add or remove at either end of the queue, but the items remain in a fixed order and cannot be accessed until found at either end of the queue.</div>
<div>
<br /></div>
<div>
A neat twist on our double-ended kebab is a kebab with a handle on one end. This kebab can only be accessed from a single end; the exact properties of a stack. We push items on one at a time, keeping the initial order until removed. The first item added to the kebab when constructing it is the very last to be removed and eaten. We must wait until all other pieces are gone before finally arrivinge back where we started.</div>
<div>
<br /></div>
<h3>
Intestinal Fortitude</h3>
<div>
<br /></div>
<div>
Our second course here at the data structure barbecue is a juicy and meaty collection of grilled sausage links. Filled with ground seasoned meats, seasonings, and aromatics, these stuffed links are bursting with flavor. The links of this chain are joined to their nearest neighbors by a thin brand, creating a long collection of heartburn-inducing, artery-clogging satisfaction.</div>
<div>
<br /></div>
<div>
Most of you have probably already figured this one out. This closely resembles the good old-fashioned linked list. Starting at one end, we can journey from link to link all the way to the other end. The order is fixed, but severing just one connection breaks the list into two. Losing a handle on the second half causes the whole chain to be dropped on the floor causing a mess and making it unusable (a memory leak).</div>
<div>
<br /></div>
<h3>
Fresh From Chilliwack</h3>
<div>
<br /></div>
<div>
<a href="http://upload.wikimedia.org/wikipedia/commons/7/79/VegCorn.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://upload.wikimedia.org/wikipedia/commons/7/79/VegCorn.jpg" height="210" width="320" /></a>I made a special trip down the Fraser Valley for our final course here at the data structure barbecue. Today's finale is grilled corn on the cob. I hope you held onto the toothpick-like skewer from the kebabs, because you're about to sink your teeth into some fresh, sweet corn. This final starchy treat contains a fixed collection of ears of corn. You can start wherever you'd like and easily access any area on demand, choosing to stay in one area or revisit places you've already covered.</div>
<div>
<br /></div>
<div>
The cob of corn closely resembles the classic array. It's very rigid in total size and arrangement of values, but easy to access any part at any time. The constraints of an array are limiting, but enable incredibly quick access to any item within the data structure.</div>
<div>
<br /></div>
<h3>
Where's The Hash Table?</h3>
<div>
<br /></div>
<div>
Over there next to the table with the macaroni salad, the snacks and all the fixings...</div>
<div>
<br /></div>
<div>
I guess not all things are fit to be grilled. I'll cover hash tables here because they are the essential building block for more advanced data structures such as associative arrays.</div>
<div>
<br /></div>
<div>
A hash table is a clever way to treat non-ordinal values (<i>e.g.</i> strings) as indices. The first thing to do is to define a simple to compute hash function taking the key values as input and computing a numeric hash value. A very terrible hash function could simply return a fixed value (this would result in a data structure equivalent to a linked list in most implementations). An ideal hash function given any random input would output any hash value with an equal chance. Similar input values would result in very different hashes.</div>
<div>
<br /></div>
<div>
Each computed hash value corresponds to a bucket containing all previously stored values. Each bucket may contain many results with the same hash value. Though there are a variety of implementations, from here the collection should be small and much more manageable. Traversing the collection, we look for a matching index and the associated value. In this way, hash tables allow us to quickly store and access a wide variety of data types using limited memory and direct object comparisons.</div>
<div>
<br /></div>
<h3>
Come Again Soon</h3>
<div>
<br /></div>
<div>
I hope you all enjoyed the first ever data structure barbecue grill. Please feel free to take home seconds and share with your friends. I hope the analogies weren't too labored. Please don't try to push them too far; I'm sure they'll break.</div>
<div>
<br /></div>
<div>
What are your favorite clever data structure analogies? Let me know in the comments.</div>
<div>
<br /></div>
<div>
Cheers,</div>
<div>
<br /></div>
<div>
Joshua Ganes</div>
Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com1tag:blogger.com,1999:blog-6631127102489130322.post-40513054744859142392015-01-22T06:48:00.001-08:002015-01-22T06:48:53.041-08:00National Programming League Draft 2015<h3>
Did You Say Hockey?</h3>
<div>
<br /></div>
One of the great things about publishing this blog is that it lets me write on topics I'm passionate about. While I blog primarily about software development, today I feel like discussing another passion of mine: hockey -- specifically the <a href="http://www.nhl.com/">National Hockey League</a> and the <a href="http://oilers.nhl.com/">Edmonton Oilers</a>. For those who say that hockey's not your game, please bear with me as I'm headed somewhere with this.<br />
<br />
As a red-blooded Canadian, I've been raised in a hockey culture. Hockey in Canada is unavoidable. Like (American) football in the USA or soccer (football) in Europe, it permeates our culture. Nearly everyone is aware of the stars of the game and the latest exploits of the local team. For the die-hard fans, it goes much deeper and becomes a part of our identity. Unfortunately for me, the Oilers have been taking me through a bit of an identity crisis since I moved to Vancouver in 2007.<br />
<br />
<h3>
Office Pool</h3>
<div>
<br /></div>
My love of hockey has led me to run the office hockey pool for both the regular season and the <a href="http://en.wikipedia.org/wiki/Stanley_Cup">Stanley Cup</a> playoffs. Our regular season pool is a draft pool where each owner drafts a fixed number of forwards, defencemen, and a goaltender. Each poolie collects points throughout the season based on his team and is allowed a limited number of trades. With more than half of this NHL season complete, it's interesting to see how our draft choices compare to the real numbers.<br />
<br />
The first surprise is the busts: those players whose performance failed to live up to expectations. Prior to our pool draft, I distributed a spreadsheet of last season's top scorers. Most of our poolies based their picks (at least in part) on this list. <a href="http://www.nhl.com/ice/player.htm?id=8473473&docid=TeamPlayerBio:291">Milan Lucic</a>, and <a href="http://www.nhl.com/ice/player.htm?id=8474586&docid=TeamPlayerBio:36358">Jordan Eberle</a>, and most notably <a href="http://www.nhl.com/ice/player.htm?id=8471707&docid=TeamPlayerBio:47027">James Neal</a> were all dropped from our pool within the first few months due to their poor performance. It appears that a player's historical statistics are not always a sure indication of his future results.<br />
<br />
Perhaps more surprising is the missed opportunities. Many players were overlooked during our draft and turned into great late-round steals. Several of the top players in the league were passed over and remained completely undrafted. This includes top-rate goaltender <a href="http://www.nhl.com/ice/player.htm?id=8471469">Pekka Rinne</a>, and the current league-leading forward <a href="http://www.nhl.com/ice/player.htm?id=8474161">Jakub Voracek</a>. The collective knowledge of our pool was not enough to identify these high-scoring gems before draft day.<br />
<br />
Perhaps we're just terrible at hockey pools, you say. Maybe so, but before we wallow in self pity at our own ineptitude, let's see how the professionals fared.<br />
<div>
<br /></div>
<h3>
The Real Deal</h3>
<div>
<br /></div>
Each season, the NHL hosts its annual entry draft wherein each team attempts to select the league's next superstars. Teams spend massively in time and money to scout young prospects in preparation for the draft. Their one objective is to choose the best player available in each round. Excellent young draft picks develop into the core of an elite NHL squad, and are considered mandatory to build the next championship team. With so much on the line, how good is their track record?<br />
<br />
When it comes to drafting forwards, it seems to me that the professionals have done a solid job. Of the <a href="http://www.nhl.com/ice/playerstats.htm?fetchKey=20152ALLSASAll&sort=points&viewName=summary">top thirty scorers</a> this season, the vast majority were selected in the first round, several were taken number one overall. There are a few notable exceptions such as <a href="http://www.nhl.com/ice/player.htm?id=8468083">Henrik Zetterberg</a> and <a href="http://www.nhl.com/ice/player.htm?id=8470794">Joe Pavelski</a>, each taken in the seventh round of their own draft year. As for goaltenders, the professional record appears much shakier. Many of the top goaltenders were selected well into the late rounds including two of this season's leaders in the wins column: <a href="http://www.nhl.com/ice/player.htm?id=8471469">Pekka Rinne</a> and <a href="http://www.nhl.com/ice/player.htm?id=8470860">Jarolav Halak</a>, who were selected in the eighth and ninth rounds respectively.<br />
<br />
As for the busts, I feel the best place to look is the number one overall seed. The most coveted position in the draft, the privilege of the first overall pick allows one team first choice of hundreds of young up-and-coming players. While we can hardly expect the scouts to nail it every time, it seems fair to anticipate that anyone taken so early in the draft would surely develop into a NHL staple with consistent performance for years to come.<br />
<br />
One of the all-time great busts was the number one selection of <a href="http://alexandre%20daigle/">Alexandre Daigle</a> by the Ottawa Senators in the 1993 NHL entry draft. After a promising rookie season, he consistently fell below expectations before bouncing from one team to another and leaving the NHL for Europe following the 2006 season.<br />
<br />
Then there's my personal favorite draft bust: <a href="http://www.nhl.com/ice/player.htm?id=8467055">Patrik Stefan</a>. Stefan was chosen first overall by the Atlanta Thrashers in the 1999 draft. After averaging less than half a point per game over his NHL career, the number one pick left the league after only seven seasons. He is now unfortunately best known for <a href="https://www.youtube.com/watch?v=RCWpE_cqnuU">this debacle</a>, missing a wide open empty net and allowing my Edmonton Oilers to tie the game with only seconds to left on the clock.<br />
<br />
It appears that the NHL scouts, whose entire job consists of watching, interviewing, and evaluating young players still get it wrong sometimes. I give them all due credit for a tough job where the successes go mostly unnoticed and the failures expose them to public scrutiny, with the critics calling for their heads. Still, these scouts occasionally miss golden opportunities or unwittingly place their faith in the wrong player.<br />
<br />
<h3>
Programmer Draft</h3>
<br />
Imagine, if you will, that the first ever National Programming League (NPL) draft is a mere three weeks from today. Your company was lucky enough (or perhaps it just performed poorly last season) to be given the privilege of the first overall pick. On draft day you will have your choice of any active programmer in the world to build a your dream team around. How do you choose?<br />
<br />
You could start by looking at famous programmers. There's not too many who have wide-reaching fame, but if you dig hard enough, you could probably come up with a list of maybe a few hundred programmers who could be recognized by several people they had never met. Does fame really equate with talent? Not where I come from. Certainly some programmers might be famous because of their talents, others may also be famous despite them.<br />
<br />
Another place to look would be job sites and headhunters. Browsing through candidate profiles from a collection of sources, you could try to boil down all the varied types and qualities of information into a short list of candidates. From there you could further research each programmer on the short list and try to come up with your definitive number one pick.<br />
<br />
An interesting approach would be to follow chains of references and recommendations. Start with a reasonably large collection of randomly selected programmers and ask each one to recommend the smartest programmers they know. By repeating this process recursively, you should eventually bump into the same highly-praised candidates time and again. Again, from that pool of recommended candidates, you could perform some final detailed analysis and evaluation to settle on one final choice.<br />
<br />
With the strategies above, you could probably come away with a solid programmer, but how close could you come to identifying the true "best" programmer on earth? I think you might still miss by thousands of places.<br />
<br />
I'll bet that some of my readers have even more clever schemes up their sleeves. Can you think of any better strategies, or are you just hoping that you'll be selected in the early rounds?<br />
<br />
<h3>
Programming Is Different</h3>
<br />
For hockey and other elite-level sports, all the work is done out in the open and under the public's watchful eye. Every shift, every hit, and every goal is recorded and tallied for later retrieval and analysis. We can watch healthy NHL players play 82 games each season and play back the tape in slow motion from every angle imaginable. Yet, somehow we still find it hard to predict tomorrow's score.<br />
<br />
Unlike the world of sports, programming work is largely an independent activity and is typically done behind closed doors. Private companies protect their source code and their employee information. With open source, of course, you can see the fruits of their labor, but it is still very difficult to measure performance in a meaningful and statistically relevant way. There's no great programmer database where you can slice, dice, and graph statistics to your heart's content.<br />
<br />
From my own experience, I know that employees who have worked at the same company for years can know little about their colleagues talents. Only by working closely with another developer on a variety of projects, can I even begin to gauge how truly talented (or not) they might be. How much harder would it be to evaluate the performance of someone I've never met?<br />
<br />
<h3>
To The Point</h3>
<br />
The thing that has me all fired up is that people sharing their "ideal" process for hiring a programmer.<br />
<br />
At this point I must mention that my experience in this area is incredibly limited and I'm not really qualified to be giving out advice on hiring. (Now ignore this disclaimer and listen to me!)<br />
<br />
I've seen this advice take many forms. Some propose a simple bullet-point test that can be completed in minutes, while others have devised an elaborate series of excruciating tests spanning multiple days and consisting of so much actual work that it might not even be legal anywhere on earth.<br />
(P.S. please don't do this)<br />
<br />
What I'm desperately trying to get across is that there is no such thing as the ideal hiring process for programmers.<br />
<br />
There are certainly some tips and tricks that you can apply to weed out the riffraff and narrow the field to those with a higher chance of success. You can ask candidates to write simple functions on a whiteboard or using software. You could discuss the advantages of different data structures in a specific scenario. You might give candidates basic design problems and ask them to draw diagrams of their solution using boxes and arrows. You might discuss work history and how they've dealt with people problems. I'm sure if I sat here long enough, I could rattle off dozens more.<br />
<br />
All of these hiring tips and tests will only get your so far, eventually you have to settle on a candidate who seems competent and talented given limited information. Even if you are lucky enough to hire an elite performer, it may take years before you can confirm that they are truly a cut above the rest.<br />
<br />
<h3>
Sorry To Disappoint </h3>
<br />
Just as it is for the professional scouts, finding and hiring the right people for your technical team is an essential part of your product's success. I understand that this is an important problem, so I'm sorry that all I have to say is, "this problem has no ideal solution". I'll let you in on a secret though: if I ever discover a solution, it will be much too valuable to share -- at least until I can use it to earn some cold, hard cash.<br />
<br />
Do you believe in an ideal hiring process? What tricks and tips do you use to identify promising candidates? Let me know in the comments.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-65895117566941693602014-12-15T22:06:00.003-08:002015-01-22T17:31:20.772-08:00Porting To Windows CE - How Hard Could It Be?<h3>
What Tiggers Do Best</h3>
<br />
I've always considered myself someone who is quick on the uptake. If you throw a new challenge at me, I expect that I should quickly engage myself and learn the basics. Soon I will master the material and be unstoppable in future pursuits. There may be setbacks, certainly, but I'll learn from my mistakes and begin to climb the ladder to climb to a peak of any elevation.<br />
<br />
A few years ago, I was tasked with just such a challenge. I was asked to take an existing application (one that I was not very familiar with) and port the code to run on Windows CE version 4.2. "How hard could it be?" I remember thinking. Porting code to run on Windows CE, that's what Tiggers do best!<br />
<br />
After a bit of stumbling around, I found <a href="http://www.microsoft.com/en-ca/download/details.aspx?id=16683">Microsoft eMbedded Visual C++ 4.0</a> available for download. This seemed to be exactly the tool I needed. A bit more searching allowed me to find some service packs and an SDK with support for my target platform. Tools in hand ...or computer, I set off to modify the code.<br />
<br />
I began a new Windows CE project and attempted to copy our existing structure using the new tool set. Wielding the power of preprocessor macros, I updated the code. A few functions were not defined. I was forced to use similar functions with the arguments slightly modified. There were a couple of places where I needed to implement my own replacement or grab existing libraries from the web.<br />
<br />
All in all, the porting process progressed smoothly. I learned a few lessons and got away from the project reasonably unscathed. My work was installed sometime later and ran without any significant issues for quite some time.<br />
<br />
You see, the real thing that Tiggers (<i>i.e.</i> programmers) do best is to learn and adapt to new technology. We are learners, problem solvers, and excellent at pattern recognition, logic, and technological intuition.<br />
<br />
<h3>
The Version Explosion</h3>
<br />
Some time later, I was tasked with porting the same application to Windows CE 5. With a bit more searching, I was able to find an SDK that allowed me to continue using the old development environment. I now had multiple platform configurations for the same application, but everything was still okay.<br />
<br />
It seems that every six months to a year we found the need to support yet another flavor of Windows CE. Beyond Windows CE 5, it seems that every new OS version requires a new development environment. Worse yet, for platform developers, the platform builder requires yet another environment from the application development environment.<br />
<br />
The following table was compiled by <a href="http://geekswithblogs.net/WernerWillemsens/archive/2013/09/13/building-windows-ce-6-or-7-smart-device-application-with.aspx">Werner Willemsens</a> and lists the development tools required for targeting various versions of Windows CE:<br />
<br />
<table border="1" style="border-collapse: collapse; width: 100%;">
<tbody>
<tr><th>Version</th><th>Platform (OS) Builder -> NK.BIN</th><th>Smart Device application</th></tr>
<tr><td>Windows CE 4.x</td><td>Windows CE Platform Builder 4.x</td><td>Embedded Visual Studio 4</td></tr>
<tr><td>Windows CE 5.0</td><td>Windows CE Platform Builder 5.0</td><td>Visual Studio 2005 + SDK</td></tr>
<tr><td>Windows CE 6.0</td><td>Plugin for Visual Studio 2005</td><td>Visual Studio 2008 + SDK</td></tr>
<tr><td>Windows CE 7.0</td><td>Plugin for Visual Studio 2005</td><td>Visual Studio 2008 + SDK</td></tr>
<tr><td>Windows CE 8.0</td><td>Plugin for Visual Studio 2012</td><td>Visual Studio 2012 + SDK</td></tr>
</tbody>
</table>
<br />
Take careful note that you will require the Microsoft Visual Studio "Professional" edition or higher in order to target Windows CE. As an alternative to Visual Studio 2005 in the table above, I was able to find a <a href="http://www.microsoft.com/en-ca/download/details.aspx?id=17310">Windows CE 5.0 Standard SDK</a> for use with Microsoft eMbedded Visual C++ 4.0 SP4.<br />
<br />
In the link above, Werner goes on to explain how to limit the number of necessary development environments in far more detail than I wish to cover here. I would like to say a big thank you to Werner and the valuable information provided.<br />
<br />
P.S. Did I mention that the tools above are not all compatible with the same desktop versions of Windows?<br />
<br />
<h3>
On CE's Struggles</h3>
<br />
Through my own experiences with Windows CE, I've developed my own theory as to why the platform struggled to gain traction. The theory is simple and kind of sad: Microsoft's development tools are so poorly strung together as to feel that they are actively fighting the developer.<br />
<br />
I developed for Windows CE because our business needs directly required it. After spending many hours researching and following many paths leading to dead ends, I was able to piece together a working development environment. Even after doing so, maintaining an application for an array of different OS versions becomes unpleasant and expensive as you need to buy multiple costly software licenses when it would seem that the newer tools should be compatible with the older OS versions.<br />
<br />
If, knowing nothing, I were starting a new project and given the choice between multiple platforms, I would probably not choose the one that feels like pulling teeth even before I write a simple "Hello World" application. I believe that this frustrating barrier to entry contributes significantly to developers looking at alternative choices.<br />
<br />
<h3>
A Faint Light</h3>
<br />
This is highly subjective, but I have a feeling that with each new iteration, Microsoft is beginning to clean up its act and making Windows CE development simpler and more streamlined. They may finally be turning the corner and arriving at a new golden age of embedded development.<br />
<br />
As with many programming tasks, getting started with Windows CE development was much more difficult than I had originally anticipated. That said, if you tread carefully and find the right resources, you can create beautiful applications on this platform, too. Hopefully posting some of the information I've discovered along the way will help make the journey just a bit simpler for the next guy.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-73044315000836710562014-09-25T20:41:00.000-07:002014-09-26T16:21:56.775-07:00The Cinnamon Twist Alert - Handling Complex Boundary Conditions<h3>
The Old Ways Are Not Always Best</h3>
<div>
<br /></div>
<div>
A new bug report came in. After reading through the report, the problem was clear. Our system did not completely support single-day batches of over 999 transactions.</div>
<br />
The problem boiled down to a single counter used to track transactions throughout the day. The field for the counter has a fixed width of three digits. The sequence counter field begins with 001 and increments by one for each transaction all the way up to 999. This value is used to help identify and correct communication errors. When a terminal attempts to send a request but encounters an error, we resubmit the transaction using the same sequence counter. If the other side receives two similar (duplicate) transaction records with matching sequence counters (more on this later), the earlier of the two submissions is reversed and is not funded.<br />
<br />
The problem with the sequence counter becomes obvious when you think about what happens when the terminal needs to go beyond 999 transactions in a day. The sequence counter will spill over the three digits available, reusing a value from earlier in the same batch or the forbidden value 000. Our software was not so silly as to ignore this problem. The solution, as it was implemented, was to increment the batch number by one and reset the sequence counter for the newly-created batch to 001.<br />
<br />
Unfortunately for us, one of our clients began frequently exceeding the magic 999 transactions per day and was experiencing problems with this approach. Processing multiple batches on the same day led to reconciliation and accounting issues, while causing delays in the deposits of funds to the client's bank account. Obviously, these were problems we wanted to deal with swiftly, once and for all.<br />
<br />
<span style="font-size: x-small;">Edit for clarity: Some have asked why I didn't simply increase the width of the sequence counter field to more than three digits. This width was defined in a third party specification and was not under my control. My software had to deal with this somehow.</span><br />
<br />
<h3>
A New Approach</h3>
<div>
<br /></div>
After consulting the documentation and our technical contacts and running through a couple of false starts, we formulated a new approach to the three-digit sequence counter problem. Instead of creating a new batch to deal with the overflow, we would simply roll the sequence counter from 999 back to 001 and continue processing everything normally.<br />
<br />
Our biggest concern with this new approach was related to the special duplicate transaction checking mentioned previously. The duplicate checking logic considers the following criteria when determining whether a subsequent transaction request matches an earlier one:<br />
<br />
<ul>
<li>Sequence Counter</li>
<li>Card Account Number</li>
<li>Total Dollar Amount</li>
</ul>
<br />
If all three of these values match, the earlier of the two transaction requests is silently reversed, leaving the transaction totals out of balance and the client short of money. For some silly reason, people get very upset when their money goes missing unexpectedly.<br />
<br />
<h3>
Mr. Cinnamon Twist</h3>
<br />
To describe a plausible scenario where I thought this might actually happen, I decided to write a brief<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
story about a man I dubbed Mr. Cinnamon Twist.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRqbT2F0PAqAQ2Eo9s54a1NxaIsmJnGo5Ng6Qm2l_Wbonvdp3_OV9D6BwHqLqSueAenjFQwxvp1P_OJl6C3U5gURoBzOkG8TWDY4iXtTVEWCHIOTx9dOidESOTNO2ZyMKe-W2MP5NPbZw/s1600/cinnamon.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRqbT2F0PAqAQ2Eo9s54a1NxaIsmJnGo5Ng6Qm2l_Wbonvdp3_OV9D6BwHqLqSueAenjFQwxvp1P_OJl6C3U5gURoBzOkG8TWDY4iXtTVEWCHIOTx9dOidESOTNO2ZyMKe-W2MP5NPbZw/s1600/cinnamon.png" height="200" width="198" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Cinnamon Bun<br />
Cinnamon Twist<br />
...whatever</td></tr>
</tbody></table>
Mr. Cinnamon Twist is a businessman with a sweet tooth. Knowing he has a long day of meetings ahead of him, he stops in at the busy corner coffee shop looking for a morning treat. He spies a delicious, gooey cinnamon twist (with double frosting). His growling, empty stomach simply cannot resist. Leaving the shop with cinnamon twist in hand, he takes two bites and wraps up the rest as he hurries to catch a train heading downtown. Through the early morning, Mr. Twist savors his treat as he goes about his work. He prepares his materials for the big afternoon presentation for a prospective new client. After a long and stressful day of work, Mr. Cinnamon Twist boards the train heading towards home. Worn out and feeling exhausted, his mind wanders back to his early morning treat. He decides that he will treat himself to another (just this once) before dragging his tired body home,<br />
<br />
In this scenario, it's plausible that our sweet-toothed protagonist used the same credit card to pay the same amount for both a very low and very high sequence number. If all the stars aligned and these transactions happened to reuse the exact same sequence counter, this would mean that Mr. Cinnamon Twist magically received his first treat of the day without being charged for it. Great news for Mr. Twist, not so good for the coffee shop who would be out the cost of a scrumptious cinnamon-flavored treat.<br />
<br />
<h3>
Back Of The Envelope</h3>
<br />
The first problem with the above scenario is simply noticing it. Detecting this type of situation on the fly is hard enough. With the requirement to be fast, high volume, and redundant between multiple data centers, this becomes complicated very quickly. The second problem is how to correct the situation once an error has been identified. I could think of a few tricks that I might consider, but I saw no obvious trivial approach for this problem.<br />
<br />
While trying to avoid tackling this complex condition, I paused to look at some data. How likely is the above scenario? I looked at some rough numbers to try to get an idea. I looked at the number of clients exceeding the magic 999 transaction limit. I looked at the number of transactions using the same card at the same merchant on the same day. Using the classic <a href="http://en.wikipedia.org/wiki/Back-of-the-envelope_calculation">back-of-the-envelope</a> approach, I calculated that we would likely only see this situation a handful of times in a year.<br />
<br />
It seemed to me that we were looking at a lot of complicated and error-prone work to save the cost of a tray of delicious cinnamon treats each year.<br />
<br />
<h3>
The Compromise</h3>
<br />
As it turns out, there is a manual process available to correct these types of transactions. By picking up the phone and talking to a real live human being, we are able to manually single out a transaction request and force it through.<br />
<br />
Knowing that this manual correction process was available and fearing the work required to fully automate every possibility, I proposed a compromise. We would create a scheduled script to run daily and search the database for requests matching the duplicate transaction scenario above. If any duplicates were found, we would fire off an email alert message (subject line: "Cinnamon Twist") identifying the key transaction details and describing the process for manual corrections. The worst case, I thought, was that the alert would fire too often and I would have to implement the complex solution later anyway. The best case, on the other hand, was that the alert would basically never fire, saving a great deal of time and effort.<br />
<br />
<h3>
Sounding The Alarm</h3>
<br />
The first week after installing my script, there were still no email alerts. I was beginning to feel optimistic that we may never see the alert fire in practice. They say that trouble shows up when you least expect it. The day after sharing my optimism with my coworkers, we received our first Cinnamon Twist alert.<br />
<br />
No problem, I thought. We followed the manual procedure only to discover that both transaction requests were good and no corrective action was required. This contradicted the documentation and our general understanding of how the system should work, but who am I to look a gift horse in the mouth?<br />
<br />
Another week or two went by before the next alert fired. It seems that my back-of-the envelope calculations were a bit off. We were receiving more alerts than I had expected. This alert, too, turned out to be a false positive when we followed up manually.<br />
<br />
We asked our technical contacts for clarification. After our messages got passed around a few times, our contacts eventually got back to us saying that this behavior was by design. It seems that we had worried ourselves over a problem that didn't actually exist.<br />
<br />
We disabled the Cinnamon Twist Alert script a short time later. My "lazy" approach had saved me from implementing a lot of complicated logic for no reason.<br />
<br />
<h3>
An Ounce Of Cure</h3>
<br />
What is my point? What can we learn from these events? Perhaps it's time to spin the wheel of morality to tell us the lesson we should learn.<br />
<br />
Maybe I was lucky. My calculations turned out to be somewhat (but not excessively) optimistic. There was a risk that we would need to handle these manual corrections frequently, leaving me scrambling to implement a complex change to relieve pressure from the rest of my team as quickly as possible. My approach was a calculated gamble, but it paid dividends even larger than I had anticipated.<br />
<br />
To me, this is a turnabout on the old adage saying, "an ounce of prevention is worth a pound of cure." In this case, an ounce of cure (the alert and manual correction) was quicker, safer, and simpler to implement than a pound of prevention (a fully automated solution). In rare cases, the easiest way to deal with complex boundary conditions is not to. Instead, find a way to look for the errors and tidy up after they happen. Don't forget to calculate the risk and the cost, but you may just discover that you were about to make much ado about nothing.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-80086194902139806842014-09-17T21:52:00.000-07:002014-09-17T21:56:42.267-07:00Institutional Knowledge Is The DefaultThis article is a follow up to my<a href="http://amish-programmer.blogspot.ca/2014/09/what-people-say-when-youre-gone.html"> previous post</a> on the topic of institutional knowledge.<br />
<br />
<h3>
Do As I Say, Not As I Do</h3>
<br />
Please don't interpret my recent post as a claim of personal innocence when it comes to accumulating institutional knowledge. I have completed many projects in my time that are completely devoid of, or seriously lacking in adequate documentation.<br />
<br />
I realized long ago that the only way to avoid becoming emotionally paralyzed by constant feelings of inadequacy is to acknowledge my own shortcomings and work hard to improve myself day by day. By staying disciplined and focusing on continuous improvement, my recent projects have been more thoroughly documented than those from only a few years ago.<br />
<br />
<h3>
The Pit Of Despair</h3>
<br />
<a href="http://ericlippert.com/about-eric-lippert/">Eric Lippert</a> writes about the <a href="http://blogs.msdn.com/b/ericlippert/archive/2007/08/14/c-and-the-pit-of-despair.aspx">pit of despair</a> as a place where the traps are easy to fall into and difficult to climb out of. Unfortunately, institutional knowledge fits this description to a tee.<br />
<br />
We constantly pick up valuable little nuggets of information as we go about our duties. Sometimes these are technical details about the systems we're working with. Other times, it may simply be the knowledge of who is already an expert in a given area.Tapping into the institutional knowledge of others can be more valuable than struggling to discover everything for yourself.<br />
<br />
There is nothing wrong with this knowledge in and of itself. This knowledge can be used to unlock further discoveries and make key decisions that allows us to avoid disasters and achieve success. The problem is that the knowledge is trapped inside a lone individual's head. Without further action, we end up continuously accumulating more and more institutional knowledge. Institutional knowledge is the default and we must act deliberately if we intend to avoid it.<br />
<br />
<h3>
Why We Despair</h3>
<br />
Knowledge is tremendously valuable. As G.I. Joe has taught us, "<a href="https://www.youtube.com/watch?v=pele5vptVgc">knowing is half the battle.</a>" This is why distributing institutional knowledge is so important to any group of people working towards a common goal. When knowledge is trapped within a single mind, its potential is limited to that one individual. Time is wasted, uninformed decisions are made, and existing work is duplicated unnecessarily. From a business perspective, institutional knowledge is clearly bad for the bottom line.<br />
<br />
I am about to draw a moral line in the sand. Neglecting to share institutional knowledge is regrettable, but intentionally hoarding knowledge to the detriment of the team in order to further one's own selfish ends is reprehensible. This is comparable to the salesman who viciously defends his "territory" from his coworkers to protect his own commissions. Not only does it reduce the collective effectiveness of the team, but it fosters and air of hostility and inhibits sharing important details needed to succeed.<br />
<br />
<h3>
Scaling The Walls</h3>
<br />
How then, do we climb out of the pit of despair and tiptoe around the pitfalls waiting to drag us back down? I'm no expert on this topic, but I'll share some of the things I do in my attempt to scale the walls and share my knowledge with my coworkers.<br />
<br />
One of the best tools available at my workplace for sharing knowledge is our internal company <a href="https://www.mediawiki.org/wiki/MediaWiki">wiki</a>. Any pages I create on the wiki are immediately available to be searched, read, and modified by our entire company. These days, whenever I start a new project I will immediately create a new wiki page describing the basic purpose of the project and how it will work. As I continue to develop the project, I frequently edit the page with the most up-to-date understanding of the available details. As for my writing, I try to follow many of <a href="http://www.joelonsoftware.com/AboutMe.html">Joel Spolsky</a>'s excellent <a href="http://www.joelonsoftware.com/articles/fog0000000033.html">tips for writing functional specifications</a>.<br />
<br />
Another great way to ensure you're not accumulating institutional knowledge is to pay attention to the questions people ask. Sometimes people ask lazy questions. When they ask about something you've already covered, simply point them to the relevant documentation. If, on the other hand, they've done their homework and still require missing details or clarification, consider this a flaw in your documentation. Recognize the flaw, modify the documentation, and think about how to improve for the next time around.<br />
<br />
On the same token, any time you find yourself asking for assistance, it's a likely sign that someone else has a collection of hidden institutional knowledge. Ask them if there's documentation, and suggest (or insist) that they write some. If nothing else, write down whatever lessons you've learned from your interactions.<br />
<br />
<h3>
Do You Validate?</h3>
<br />
Words of caution: just because you wrote some documentation, it doesn't mean that it's adequate.<br />
<br />
When it comes to documentation, if I can't find it, it doesn't exist. You may have written a 500-page treatise covering every last detail of uses and maintenance of your paper clip system including a full bibliography, glossary, and footnotes on every page. It pays me exactly zero benefit if I can't find the document after giving an honest effort to search for it in all the expected places.<br />
<br />
Just because instructions are clear to you, that doesn't necessarily mean that they will be clear to everyone. Each person is familiar with his own style. Things that appear straightforward to you may be ambiguous or unclear to others. Instructions that seem obvious to an experienced user may involve hidden steps unknown to a novice.<br />
<br />
A great way to check for these flaws is to ask someone to validate your documentation for you. I find this particularly effective in the case of a documented procedure. In the spirit of <a href="http://www.joelonsoftware.com/news/fog0000000118.html">hallway usability testing</a>, ask a coworker to start from scratch and try to achieve your documented goal. Watch from a distance and note every time that they get stuck or confused. Later, add additional notes for clarification. Once another person can follow your documentation with minimal fuss, then you can be confident that someone else can perform the task when you're gone.<br />
<br />
<h3>
Still No Expert</h3>
<br />
As noted previously, I am not to be considered an expert in these matters. Listed above are some tips that I've found useful in sharing my institutional knowledge with my coworkers. What are are the best tips and tricks you have for avoiding the pit of despair and sharing your own institutional knowledge? Tell me in the comments.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-40909507041763886522014-09-14T09:29:00.004-07:002014-09-17T21:52:39.729-07:00What People Say When You're Gone<h3>
Parental Leave</h3>
<div>
<br /></div>
My wife and I are pleased to announce the birth of our second daughter, Isla. She was born at the very end of May and has been providing us with baby snuggles and depriving us of sleep ever since.<br />
<br />
I was fortunate enough to be in a position to take a decent length of paternity leave to help my wife with our two young girls and to enjoy some time together as a family. We made good use of our time by showing off Isla to our friends and family scattered across western Canada.<br />
<br />
If you are in a position where you can manage and afford to take parental leave, I would urge you to take hold of the opportunity. Getting away from my work routine for a while was a great way to recharge and reflect on my current situation and goals. The precious first months and years of a child's life pass just as fast as the cliches say. Slowing down to experience and savor this special time with my children while I'm able is a privilege that I wouldn't want to pass up.<br />
<br />
<h3>
Returning To Work</h3>
<div>
<br /></div>
When I came back to work, my coworkers greeted me with in a variety of ways. There were those who (hopefully) jokingly told me, "I though you were fired." There were a lot of pleasant and generic, "welcome back", or "how's the family?" responses. There were also a few who genuinely expressed that they missed me and how glad they were that I was back.<br />
<br />
Of course, on an emotional level, we all want to be missed. It's a wonderful feeling to know that you're missed and appreciated while you're gone. From a personal standpoint, being missed is great. That got me to thinking about whether I want to be missed on a professional level as well.<br />
<br />
<h3>
Professionally Speaking</h3>
<div>
<br /></div>
Imagine if not one colleague missed you during an extended absence. That would mean they don't need or desire your assistance to do their work, or worse, that you may actually stand in their way. Imagine if your boss didn't miss you either. It would mean that your job is irrelevant or that you are so unproductive that your absence is barely noticed. Either way, it sounds like your job security is in perilous danger. Obviously, you want to be missed at least a bit.<br />
<br />
To be successful professionally, you need to become indispensable to your team. I believe that there are two varieties of indispensability -- one good and one bad. Let me illustrate using a couple of examples and see if you agree.<br />
<br />
Mr. Smith is indispensable to his team. When the OIU system goes down (as it often does), he is the one who knows just how to diagnose the problem and get things back up and running. Last month when he was on vacation, it took his coworker three days to fix the problem. Mr. Smith can usually sort out issues in a matter of hours. The operations team loves Mr. Smith, because he's always so quick to dive in and troubleshoot their problems as soon as they call.<br />
<br />
Mr. Brown is indispensable to his team. He is always ready to lend his expertise to help a colleague solve a technical issue or discuss a design question. His software is always high quality, well documented, and easy to maintain. The junior developers love him because of his valuable mentoring. They prefer to maintain and enhance Mr. Brown's projects because the code is clear, well designed, and easy to work with.<br />
<br />
You can probably see where I'm going with this. Mr. Smith and Mr. Brown are both considered indispensable for wildly different reasons. Mr. Smith uses something called "institutional knowledge". Over the years, he has become an expert in the internal systems of his company (institution). This knowledge, while valuable, can sometimes even be hoarded. With all of this valuable information held only in his own head, Mr. Smith essentially holds the information for ransom. He ensures his own job security while maintaining a charade of expertise and talent.<br />
<br />
Mr. Brown, on the other hand, tries to offload institutional knowledge. Instead of hoarding it, he documents the details someplace where anyone can easily find it. Sure, it may take an intern new to the project some time to get up to speed, but that's only natural. Instead of banging his head against the wall or <a href="http://www.joelonsoftware.com/articles/fog0000000022.html">interrupting Mr. Brown with unending questions</a>, our intern can simply read and reference the documentation as he stumbles his way through the project. This leaves Mr. Brown free to concentrate on his own work, while empowering others to do great things.<br />
<br />
<h3>
What I Hope They're Saying</h3>
<div>
<br /></div>
I hope I was missed during my absence. I also hope that people weren't asking when I'll be back because I'm the only one who knows about a specific system. Instead, I hope that they were simply discussing their projects, confident in their understanding based on my documentation. I hope that my boss was missing me because I'm the best man for the task, not because everyone else is struggling to keep my work on track.<br />
<br />
What do you want people to say when you're gone? Tell me in the comments.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com1tag:blogger.com,1999:blog-6631127102489130322.post-57578065974037741822014-05-08T16:40:00.000-07:002014-05-08T21:44:29.737-07:00Desperate Measures - A Time To Hack<h3>
A Case Study</h3>
<br />
Sometimes the features that seem simplest on the surface are the hardest to implement. This statement rang true for me in the past few weeks. In today's post I will walk you through the stages of my latest feature request and the challenges I encountered. This example is far from my typical experience, but is an extreme example that illustrates the kind of problem solving and outside-the-box thinking required of professional software developers.<br />
<br />
For a bit of background, my work involves payment card processing in various forms and across a wide range of systems and third-party applications. One of my most successful projects to date has been a full payment solution integrated with a third-party point of sale system. This new feature was an enhancement to that system and required customization of receipt printing as requested by a client.<br />
<br />
In a typical setup, the system will print three receipts for each credit card payment: a merchant copy, a customer copy, and an itemized receipt listing all of the items and totals ordered by the customer. Our client must be environmentally friendly (or, perhaps, simply cost-conscious) because they asked to eliminate as much receipt waste as possible. When configured for this new feature, the system would print nothing by default. Instead, it would cache the receipt details for later printing if requested. "Hey, could I get a copy of my receipt?"<br />
<br />
At first glance, this feature sounded pretty simple. Creating a cache to store data for later retrieval is the bread and butter of a developer's work with many tried and true solutions. I would have to research the point of sale system scripting language and communications protocols to install a new button that would fetch the receipt details from our cache and print the receipt on demand. This, I figured, would be the challenging part of the project. While challenging in its own right, this paled in comparison to one challenge that I had overlooked.<br />
<br />
The overlooked piece was the itemized receipt. These details are normally printed automatically by the third-party system. "No matter", I thought, "there is an API call to access these details". I had used it in a couple of different places previously. It wasn't until I went to implement the API call that I discovered the problem. The API did not work when a payment was in progress or after the check had been closed on the system. This meant that by the time my software got involved with the payment, it was already too late to retrieve the itemized receipt details.<br />
<br />
Desperate times call for desperate measures. We contacted the vendor to discuss possible workarounds. After some back and forth discussion, we learned that the system has a print-to-file receipt option. I figured this would be a good option. My software could monitor the receipt file, waiting for new data to be written. I could then parse and cache the latest receipt details for retrieval. I spent some time restructuring the code to monitor a file for updates and scrape out the itemized receipt details.<br />
<br />
After struggling to learn <a href="http://blogs.msdn.com/b/oldnewthing/archive/2014/05/07/10523172.aspx">how to monitor a file for changes</a>, all went well with my initial test runs. The receipt text was written to the file. My software read the new data and cached it. Hitting the receipt button triggered the system to print out all of the required receipt details. I figured it was just a matter of tidying the code and running it through a battery of tests. I sat back and admired my clever solution to the problem, proud of implementing a reasonably complex solution to my seemingly simple problem. It wasn't until we brought a second machine into the mix that signs of a flaw in my plan appeared.<br />
<br />
The point of sale system runs with a central server and multiple client workstations. I had performed the majority of my development work on the server for simplicity. On the server, everything was working seamlessly. On the workstations, we saw random delays from the time of processing to the time the file was actually modified. Sometimes it would take a few seconds. On other tests, the delay climbed nearly to the 1-minute mark. Waiting a whole minute for a receipt simply wouldn't fly with a busy merchant.<br />
<br />
Further discussion with our vendor contact led us to conclude that this was simply the nature of the system. The file-based printing was designed as a record-keeping system, not a live processing environment. It was time to go back to the drawing board. Something about the best laid plans...<br />
<br />
While discussing the problem with a colleague, I joked about it saying, "maybe we should just write a printer driver." My colleague took my joke a bit more seriously than I had expected and started playing around with the printing configurations on our test lab. He found a printer option identified as TCP/IP printer and pointed it at our software's listen port. We ran a test transaction and saw that some unknown request data had been logged by our software.<br />
<br />
Somewhat bewildered and whimsically bemused at what I was about to try, I visited everyone's favorite search engine and started looking around for TCP/IP printer message format specifications. With a bit of trial and error and some luck, I stumbled upon a specification that looked similar to the messages that my application was receiving. "This might just work...", I thought.<br />
<br />
According to the specification, the first message I saw was a printer status message. I updated my software to respond with a "ready" response. When I received a second request following my response, I knew we were onto something. I worked through the commands one by one until I saw the first line of the receipt header in our logs - plain as day.<br />
<br />
There's something about this particular hack that is both wonderful and horrifying at the same time. I get a thrill out of the fact that we are simulating a printer simply to retrieve the information we need. At the same time, I boggle at the fact that something this simple shouldn't have to be so hard. There is also, unfortunately, a nagging risk that my software will be incompatible with some requests sent to it in the field. We will do our best to reduce this risk, but doing something that's not officially sanctioned and ordained by a vendor is generally a bad idea. Whatever way you slice it, this solution works!<br />
<br />
Perhaps my favorite part of this solution is how I imagine the message exchange to determine the printer status:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">THIRD PARTY: Do you have paper?</span><br />
<span style="font-family: Courier New, Courier, monospace;">MY SOFTWARE: I have a roll long enough to span the country.</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">THIRD PARTY: Is the paper door closed?</span><br />
<span style="font-family: Courier New, Courier, monospace;">MY SOFTWARE: It's glued shut.</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">THIRD PARTY: Do you have ink?</span><br />
<span style="font-family: Courier New, Courier, monospace;">MY SOFTWARE: A well of ink so deep you could swim in it.</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">THIRD PARTY: Is your cutter enabled?</span><br />
<span style="font-family: Courier New, Courier, monospace;">MY SOFTWARE: The blade is sharp enough to cut diamonds.</span><br />
<br />
<h3>
Hacks</h3>
<br />
Why do I have two simultaneous and opposing reactions to this hack? Why would some developers turn their nose up at my solution, while others would describe it by saying, "that's brilliant"?<br />
<br />
The software industry has <a href="http://paulgraham.com/gba.html">a long tradition of describing both clever and ugly solutions as "hacks"</a>. This is, I believe, because most programmers are very calculating people. We tend to have strong feelings about the "correct" solution to a problem. When something deviates drastically from that ideal, we describe it as a hack. At the same time, when there is no correct solution to a problem, the curious among us try to come up with a "clever" solution to work around the barriers set in front of us. We will often describe this as a hack as it deviates from the theoretical ideal solution in the same way.<br />
<br />
Whether or not you like what I've done in the case study above, it's hard to argue with results. If an ugly hack is what stands between myself and my goals, I'll choose the hack every time (barring the ethically questionable). When there is no correct way to do things and I can find a workaround, you can bet that I will take the workaround. On the other hand, if there is a correct way to do things that cannot be accomplished under some immediate time pressure, you can be sure that I will want to come back and set things right when I get the chance. Remember, the customer is always right. The client doesn't care about your academic ideals -- they want results.<br />
<br />
Quality developers usually need to think inside the box. The best developers will look at standard approaches to their problems. They will use proven technologies and techniques to get things done in a way that is robust and maintainable for the future. The best developers, however, have the ability to think outside the box when necessary. The resourcefulness and ability to bend a software system to your will is what separates the great from the merely good.<br />
<br />
What is craziest hack you have seen or implemented? Tell me your story in the comments.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com6tag:blogger.com,1999:blog-6631127102489130322.post-14326195172422358252014-03-12T21:08:00.001-07:002014-05-14T23:13:34.817-07:00My Love / Hate Relationship With Stack Overflow<h3>
From Day One</h3>
<div>
<br /></div>
I was an avid reader of <a href="http://blog.codinghorror.com/about-me/">Jeff Attwood's</a> <a href="http://blog.codinghorror.com/">Coding Horror</a> blog from the time I left university and ventured into the world of professional software development in 2006. I had just recently discovered the ever insightful <a href="http://www.joelonsoftware.com/">Joel On Software</a> by <a href="http://www.joelonsoftware.com/AboutMe.html">Joel Spolsky</a> when the two bloggers <a href="http://blog.codinghorror.com/help-name-our-website/">announced a joint venture</a> to create a new resource for programmers to collaborate using a question-and-answer format. Providing a desperately needed service backed by the talent, reputation, and influence of two widely known developers, I had great confidence that this venture would be a success. I am proud to say that while I had nothing to do with the actual coding or design of Stack Overflow, at very least I had my say on the name. I participated in <a href="http://www.addpoll.com/results?14407">this poll</a> and counted my vote among the 1721 supporters of the eventual winner: <a href="http://stackoverflow.com/">stackoverflow.com</a>.<br />
<br />
<h3>
Falling In Love</h3>
<div>
<br /></div>
Perhaps my favorite thing about Stack Overflow (and the family of Stack Exchange sites) is when I try to ask a question that has already been answered by the community. I can quickly find similar questions to my own with all of the answers ranked based on how helpful the community found them. I can't tell you how many times I've paid a visit to the Stack, only to find that my question was already answered clearly enough to proceed immediately with my work.<br />
<br />
In the rare case where I have not found the information I was looking for, I have been surprised by just how eager the community was to leap to my rescue. Helpful answers began to flow within minutes of posing my question. Intelligent developers from around the world competed with each other to offer helpful answers to my question in return for nothing more than my gratitude and some (nearly) meaningless reputation points.<br />
<br />
<h3>
The Hatred Begins</h3>
<div>
<br /></div>
There was once a time where open-ended and opinionated questions would be asked and discussed ad nauseam. These questions would divide the community into cliques, each supporting their own point of view and uniting in defiance of all who would disagree. I, personally, found many of these discussions fascinating. It was exciting to see the varying perspectives and how viciously they were defended by their champions.<br />
<br />
While some (like myself) loved these types of discussion and debate questions, others wanted these questions to fade away. The most adamant ones wanted them dragged out into the street and shot. They didn't like having questions with multiple conflicting answers or no real answer at all. Eventually, word came down from on high that every question needs to have a real answer.<br />
<br />
This requirement did not destroy the world (or the site), but it began to slowly erode part of the community and its nature. Developers, as one might expect, tend to be nerdy. Many of us are pedantic (guilty as charged) and love to enforce stringent rules to the letter of the law. Those who are most eager to do so often find their way to elevated positions of authority also known as moderators. I am under the impression that some of today's Stack Overflow moderators are on a quest to smite questions (be they useful or no) before they can be answered. They close questions as off topic or not constructive or a wide variety of other resolutions if they even hint at straying from the true and acceptable path.<br />
<br />
The biggest offenders I've seen are of the form: "what is the best tool for X?" These are usually perfectly sensible questions. I need to accomplish X. I've probably done some searching and found a couple of tools that claim to help with X, but I have no experience with any of them. I attempt to consult the community for its collective experience with these or other tools in the quest for X. I find that these questions often have a couple of brief answers and have been ruthlessly closed as not constructive. The very nature of this type of question makes it subjective, but there is still a lot of valuable information to be shared here. Instead, the community is stifled by an overzealous moderator too eager to close the question to see the tremendous value that its discussion would bring to the site.<br />
<br />
The other casualty of this is those active members of the community who lost interest following the change of course. Instead of participating in an exciting discussion with intellectual equals, they were challenged to answer technical questions with correct answers. I guarantee you that some of these members reduced their level of activity on the site as a result.<br />
<br />
<h3>
Sites vs. Tags</h3>
<div>
<br /></div>
My next beef is with the <a href="http://stackexchange.com/">Stack Exchange</a> family of sites. As more and more groups begin to create Q&A communities on Stack Exchange, it becomes harder and harder to find the right site to pose my questions or look for answers. Depending on the nature of my search, there may be a fair number of relevant sites to look at.<br />
<br />
Imagine that you run into a library issue while programming a GUI application primarily targeting Ubuntu Linux. Where should you go first? <a href="http://askubuntu.com/">Ask Ubuntu</a>? <a href="http://unix.stackexchange.com/">Unix & Linux</a>? <a href="http://programmers.stackexchange.com/">Programmers?</a> <a href="http://stackoverflow.com/">Stack Overflow</a>? <a href="http://ux.stackexchange.com/">User Experience</a>? You might think that some of my suggestions are more probable than others. My question is: why should you have to think about it?<br />
<br />
Rather than running many similar but separate sites and migrating questions between them, why not put it all in one place and tag each question as belonging to multiple relevant categories? Stack Exchange can still maintain communities centered around each category. The moderators can still callously and judiciously kick inadequate questions out of their domain. Why should I have to choose a site? Just let me ask my question and allow the community as a whole help direct it to the place where it belongs.<br />
<br />
<h3>
Crawling Back For More</h3>
<br />
Even with the problems I've discussed above, Stack Overflow is still a fantastic resource. When I have a programming question, I consistently find the best answers there -- even if I make my way there by searching Google. The fact of the matter is that with a large, intelligent, and active community, Stack Overflow is the best resource I know for all flavors of programming minutiae. I simply could not perform my job as efficiently without it. I guess I have to accept its flaws and admit that I still love it after all.<br />
<br />
What are the best and worst features of Stack Overflow for you? Tell me your thoughts in the comments.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-68088195629443833982013-12-10T20:41:00.002-08:002013-12-10T20:44:08.361-08:00Pulling The Plug - Software End-of-LifeRecently, the company I work for underwent a serious hardware overhaul -- upgrading a large swath of 32-bit servers to 64-bit processors and updating to the latest operating system and application versions. All in all, this upgrade went very smoothly. We planned out this transition and tested our software in our development environment before deploying to production. We ironed out a few issues that cropped up within the first few days. The problems ranged between minor incompatibilities and simple human error. Unfortunately, hiding in the shadows was a long-forgotten application that was still in use by a handful of clients.<br />
<br />
This piece of software is the kind of application legends are made of. Written many years ago, it has been performing well enough to be forgotten about and ignored. The original developer and documentation are long since gone, and it required a great deal of asking around just to find out who used this software and why. Unfortunately, the software, which had been running smoothly for years, did not transition well onto the new servers.<br />
<br />
One of our developers (not me) was tasked with getting this service up and running on the new architecture. After spending a few hours digging through the old source code he was able to get it compiling and passing a few basic test cases. We figured that we could move ahead with a production deployment and shift our focus to more pressing matters. We figured wrong.<br />
<br />
The next day, the support calls came in once more. Our software worked well enough for a few hours, but something had gone wrong overnight. The application was fully locked up and unresponsive to any requests. A quick restart did the trick, but this was only a temporary band-aid fix and was not acceptable as a long-term solution. The same developer began trying to reproduce the behavior and fix the problem for good. This became the routine over the next few days.<br />
<br />
Now, the part I haven't yet told you is that this system has long since been replaced with a newer, shinier, and more fully-featured solution. This new solution is still active with many customers and regular maintenance and enhancements. Only a handful of customers remained on the old system. The reason for staying was simple: don't fix what ain't broke. This system continued to serve their needs for all these years with no updates or known issues.<br />
<br />
The way we saw things, we were left with three choices:<br />
<ul>
<li>Install the old server hardware back into our production systems to support these customers</li>
<li>Assign developers to debug the old code and get it working on the new hardware</li>
<li>Offer these customers the choice between upgrading to the new system or lose this service (with a reasonable amount of time to decide and transition their systems)</li>
</ul>
The idea of reverting back to the old hardware was dismissed pretty quickly. The reasons we decided to upgrade our hardware were related to security, consistency, and future maintenance. We did not want to have to support 31 flavors of hardware in our production environment.<br />
<br />
The remaining choice seemed to come down to simple math. Weigh the revenues against the costs and you'll find the correct decision. Is the math really so simple? How do you come up with a formula for this. I don't know the answer myself, but I'll take a shot at offering a few observations.<br />
<br />
Let's start with some obvious factors. Software developers don't come cheap, and having developers spend a few days fighting with a bug can quickly run into the hundreds and even thousands of dollars. Estimate the required effort in time and multiply that by the developer's pay rate and come up with the cost of fixing this bug. Losing a paying customer is contrary to the goals of running a successful business. Continuing to support this solution means that the company can continue to feed off of these revenue streams. Keep in mind that these customers have been using these services with minimal maintenance for many years. Multiply the regular fees by the expected service time to calculate revenue. As with all things, estimates may not accurately reflect reality.<br />
<br />
Simple revenues and resource costs don't really cover some of the more complicated factors. If cancelling the service means losing a customer, it may also mean losing future streams of revenue from that customer. When the customer needs to expand their services in the future, who will they turn to? Will they look to maintain a relationship with a company that they already know and trust? If so, terminating that relationship today may mean waving goodbye to new opportunities in the future. What about word of mouth? Damaging your company's reputation may have wide ranging effects in one of the most powerful and underestimated forms of marketing. What about future bug fixes and maintenance? Who is to say that the next round of updates won't expose more bugs in what becomes a money pit of maintenance work for only a handful of customers? What about the confusion caused by the need to support two similar but disparate systems?<br />
<br />
To be honest, I am no expert on this matter. The choice was not mine to make and we decided to pull the plug on this legacy application. From a development perspective, I was pleased with this choice. Out with the old and in with the new, I say! I'd much rather have everyone up to date and on the same platform instead of trying to support a creaky old piece of legacy software. From the perspective of a businessman, I'm still not sure.<br />
<br />
Have you or your company pulled the plug on legacy software? What were the circumstances? What triggered your decision? What factors did you consider before making the decision? Let me know in the comments.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-5347524510851987332013-08-08T22:28:00.004-07:002013-08-08T22:28:39.936-07:00Backwards Compatibility III - Planning For The Future<h3>
Dreaming Big</h3>
<div>
<br /></div>
<div>
I like to daydream about my software projects from time to time. In my dreams, my software would become a worldwide sensation. It would be run on millions of computers operated by millions of users using various platforms and each having unique needs. My software would interact with thousands of third-party applications. Of course, my software would be flawless. Nonetheless, I would be forced to pander to the inept masses whose software is not worthy to interface with my magnificent technical achievements.</div>
<div>
<br /></div>
<h3>
<b>Down to Earth</b></h3>
<div>
<br /></div>
<div>
In reality, most software that is written today will not see such widespread adoption. There are many software developers earning a reasonable living writing in-house software applications for use by a handful of employees in their own office. Others may have a wider impact, but will not go beyond a few thousand users in their niche market. While backwards compatibility is important for all developers to understand, it is rarely the highest priority for these types of software projects.</div>
<div>
<br /></div>
<div>
One example of software targeting the kind of compatibility required on a global scale is web servers. Web servers must be able to interact with an incredibly wide variety of devices and applications. They should also run on as many varied systems as possible to avoid limiting themselves to a single market. A web server needs to provide your grandmother's garage-sale Pentium II running Internet Explorer on Windows 95 with pictures of her family. It needs to provide content to yuppies browsing social sites on their iPhones during Saturday brunch. It needs to allow 14-year old Linux fanboys to post to their favorite hacker forums.</div>
<div>
<br /></div>
<h3>
Identify Yourself</h3>
<div>
<br /></div>
<div>
Perhaps the most straightforward way to plan for future compatibility is to add an identification protocol to each interaction. This lets applications on side of the interface know who they're dealing with and to compensate for known issues in their counterpart.</div>
<div>
<br /></div>
<div>
This is really not so dissimilar from interactions with people. You know not to mention your recent SUV purchase to your tree-hugging hippie friend named Rainbow. You probably avoid talking about your contempt for Justin Bieber to your teen-aged niece. In the same way, you can build up a list of known compatibility problems over time and compensate for them by adjusting your own software to avoid the problem or fix it in another way.</div>
<div>
<br /></div>
<div>
Identifying a unique name for each piece of software is not as simple as it may first seem. There are several key pieces of information required to uniquely identify each application: name, version, platform, and configuration.<br />
<br />
The name distinguishes similar products with obviously different implementations and opportunity for discrepancies in behavior. For example, consider web browsers provided by different companies such as IE, Firefox, Chrome, etc.<br />
<br />
The version is generally a sequential numbering system that can help distinguish older (buggy and limited) versions from newer (corrected and fully-featured) versions.<br />
<br />
The platform describes the operating system and architecture on which a cross-platform product is running. This distinguishes the same product built for 32-bit Windows XP, 64-bit Windows 7, or 64-bit Ubuntu Linux.<br />
<br />
Finally, the configuration may contain additional information about modifications, plug-ins, and settings that enable special features or trigger certain buggy behavior. This information can be very wide ranging, complex, and difficult to fully capture.<br />
<br />
Once each application has verified the identity of its neighbor, it is able to accommodate known compatibility issues and limitations by changing its own behavior. Unfortunately, not all players are always working towards a common goal. These version identification techniques can be <a href="http://en.wikipedia.org/wiki/User_agent#User_agent_spoofing">turned on their heads</a> and used to limit or disable certain applications.<br />
<br />
<h3>
Expect New Features</h3>
<br />
With most software, there is a great incentive to add new bells and whistles and little incentive to trim the fat. New features are used to attract new customers and provide new capabilities. Old features may be missed if removed, and are generally simpler to retain than to completely remove.<br />
<br />
A widely-used standard designed with this in mind is XML. This, of course, stands for Xtreme Markup Language (or eXtensible Markup Language if you're a real stickler).<br />
<br />
XML is designed as a tree of nested tags with attributes and text. Well-designed XML libraries provide the ability to parse XML text into a data structure and to query for desired components within the tree. The beauty of this design is that adding new tags should not interfere with old logic. Older applications are completely free to ignore tags they do not understand and simply proceed as if nothing is different.<br />
<br />
By using XML or similar extensible formats for data interchange, applications will be better prepared to accommodate any of the inevitable new features the future has in store.<br />
<br />
<h3>
Be Prepared</h3>
<br />
As the Boy Scout motto says, "be prepared." If you are designing a software product with the dreams of being compatible with a diverse range of other products and amenable to new features, you must plan for the future. A little planning up front will save you a great deal of grief later. Expect that incompatibilities will crop up and need to be addressed. Expect that new features beyond the original design will be required. Use an interface that is designed to help deal with these problems even before they arise.<br />
<br />
Keep in mind that designing for the future can be challenging and time consuming. If you are writing an internal application with no plans for data exchange and targeting a single platform, you're probably best to stick with the "You Aren't Gonna Need It" (<a href="http://en.wikipedia.org/wiki/You_aren't_gonna_need_it">YAGNI</a>) approach. Make sure you understand which type of application you are creating before you dive in too quickly and shoot yourself in the foot.<br />
<br />
Cheers,<br />
<br />
Joshua Ganes</div>
Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-48495787370064244382013-07-21T17:17:00.003-07:002014-09-18T16:26:00.075-07:00Bowling Is Not A Sport<h3>
Rules To Live By</h3>
<div>
<br /></div>
<div>
Back in 2003-2004 I completed an internship with IHS Energy in Calgary, Alberta. Not to spark too much controversy, but as an Edmontonian at heart, I cannot advise anyone else to do the same. The company had a lot going for it, many of the people were great, but I couldn't get over the fact that I felt like I was a double agent living within the walls of the enemy base. For those who have not yet sworn allegiance to the Edmonton side of <a href="http://battle%20of%20alberta/">the battle of Alberta</a>, it was a great experience.</div>
<div>
<br /></div>
<div>
While I was there, I was privileged enough to work with a bright and charismatic coworker named Demetrio (I hope I've spelled that correctly). Originally from the Bronx in New York, he was one of those people that everyone seems to like instantly. I admire people with that gift and wonder where that special quality comes from. I came to see him as somewhat of a mentor over my time in Calgary.</div>
<div>
<br /></div>
<div>
Among work-related talks and intense games of foosball, Demetrio would share nuggets of wisdom and humor. On one such occasion, he shared that he and a few of his college friends each came up with a list of three rules to live by. I can no longer remember his first two rules, but the third was so profound that it has stuck with me to this day: "bowling is not a sport".</div>
<div>
<br /></div>
<div>
Obviously, this rule was meant in jest. Yet, somehow, I couldn't get the idea out of my head. Meditating on this thought led me to ask the question, "How do I define a sport?" This question is tougher than it seems. Pause here for a moment and think about your answer it before proceeding.<br />
<br />
(I'll wait here ...patiently)</div>
<div>
Stay tuned and I will share my thoughts on this important topic.</div>
<div>
<br /></div>
<h3>
A Spectrum Of Games</h3>
<div>
<br /></div>
<div>
There are some games that the vast majority of people would define as a sport. Whether or not you enjoy it, very few would claim that football (European or American, take your pick) is not a sport. Balanced competitions between opposing teams with a great deal of strength, speed, agility, and strategy are almost universally accepted as sports.
</div>
<div>
At the other end of the spectrum are games that have no physical component to them, but may contain some amount of strategy. Most board games and card games fall within this category. Indeed, some of these games require great time and commitment to master. Chess is the definitive example of a complex challenge for great minds. Some people include these games in their definition of sport, but most will protest their lack of a physical component.
</div>
<div>
In the middle lay games and competitions whose validity as a sport are hotly contested. Is, as Demetrio would deny, bowling a sport? What about equestrian events? The jockey clearly has skill and athleticism, but the heavy lifting is done by the horse. What about car racing? Again, drivers have great skill and a long race can be a great test of endurance and concentration. What about individual contests like golf, where the player faces the course, not any opponent. What about judged competitions such as figure skating, dance, or gymnastics?
<br />
<br /></div>
<h3>
Formal Definitions</h3>
<br />
<a href="http://dictionary.reference.com/browse/sport">The definition of the word 'sport'</a> as found on <a href="http://dictionary.reference.com/">dictionary.reference.com</a> contains a list of activities that I would not personally include using my definition of sport (more on that later). It even includes Demetrio's maligned bowling in its list of sports.<br />
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
The <a href="http://www.olympic.org/ioc">International Olympic Committee</a> recognizes sports as activities that are based in physical athleticism or physical dexterity. This guiding rule seems to be widely accepted by many.
<br />
<br />
<h3>
My Definition</h3>
<span style="font-family: inherit;"><br /></span>
To find my own definition of sport, I examined games and competitions that I thought were not deserving of the title. My first target was games without a physical component. If the game can be adequately simulated by a computer, that is a bad sign. If an able-bodied man in his twenties has no advantage over a seventy-year-old retiree with arthritis, I cannot consider the game to be a sport. This rules out most board games and card games. Sorry poker, I love you, but the sports networks are just plain wrong.<br />
<br />
I generally like the definition from the International Olympic Committee, but I feel it is still too broad. Prepare yourself for my first controversial criterion. True sports must include an element of strategy. Part of the joy of being a sports pundit is speculating on how I would do things differently if I were calling the shots. Nobody ever told a short-distance sprinter to hold back and lull his opponents into a false sense of security. No, the only strategy is to run as hard and fast as possible.<br />
<br />
Another requirement is that the structure of the game must seek to put opponents on an even footing. By this, I mean that a game is designed to be symmetrical with both competitors trying to achieve equivalent goals. While games that break this structure are rare, this requirement becomes necessary for some of the more esoteric competitions out there. This eliminates most events from <a href="http://en.wikipedia.org/wiki/American_Gladiators">American Gladiators</a>.<br />
<br />
My final rule, which may also be controversial, requires a form of direct opposition. While one of the competitor tries to accomplish a physical task, the opponent must have a way to try to stop or impede his progress. A great deal of the glory of sport derives from a struggle of man <i>vs.</i> man. This rule puts the final nail in the coffin of bowling's claims to sporthood in my mind.<br />
<br />
<h3>
Applying The Tests</h3>
<br />
My criteria for a sport are quite clear and easy to apply. Even so, there are some events which sit on the border line. The first one that comes to my mind is curling. Let's take a closer look:<br />
<br />
1. Curling requires weight control, precise aim, and physical sweeping. <b>CHECK</b><br />
2. Curling has many strategic components from guards, to freezes, to raises. <b>CHECK</b><br />
3. Each team has the same number of players and stones and uses the same playing surface and target. <b>CHECK</b><br />
4. The only "direct" opposition consists of sweeping stones out the back half of the house. <i>check?</i><br />
<i><br /></i>
Is curling a sport? I'm still not sure.<br />
<br />
<h3>
Please Don't Hurt Me</h3>
<br />
Before I find myself impaled by javelins, being thrown across the room by weightlifters, or having my face used as a surface for ice dancers, I would like to express my respect for athletes of all forms. You train long and hard to achieve excellence and distinguish yourself in your field. You perform fantastic feats of of strength, skill, and elegance that are exciting and inspirational. Please take my writing with a grain of salt.<br />
<br />
My definition of sport comes from the concepts that comes to my mind when I hear the word. I picture hockey, baseball, football, soccer, tennis, and wrestling. These other competitions are wonderful too, but they just seem different to me. For those who see things differently, more power to you.<br />
<br />
<h3>
What Do You Think?</h3>
<br />
I asked you to come up with your own definition of a sport first. Now that you've heard my theories, how well does it hold up? What do you disagree with? Is there something I've missed?<br />
<br />
Demetrio, if you're reading this, I hope you've kept the faith. Thank you for all the good advice.<br />
<br />
Cheers,<br />
<br />
Joshua Ganes</div>
Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com4tag:blogger.com,1999:blog-6631127102489130322.post-12642059989580245572013-07-08T21:52:00.000-07:002013-07-10T22:04:54.631-07:00Stop Whining And Get Over Yourself<h3>
A Disturbing Theme</h3>
<div>
<br /></div>
There has been a recurring theme in programming blogs over the past decade or so that continues to get my ire up each time I encounter it. Some entries tiptoe around this theme without ever coming out and state it. Others state it so boldly that I am left with no choice but to roll my eyes or to fume at them and leave a nasty YouTube-style comment. Many authors with incredibly wide readership and influence have repeated this theme using different words.<br />
<br />
Can you guess what the theme is?<br />
<br />
<div>
Maybe a few quotes will help:</div>
<br />
<ul>
<li>
<a href="http://steve-yegge.blogspot.ca/2007/01/pinocchio-problem.html">"I think all software is crap"</a> - <a href="http://www.blogger.com/profile/14812997485690838920">Steve Yegge</a>
</li>
<li>
<a href="http://www.codinghorror.com/blog/2004/10/we-make-shitty-software-with-bugs.html">"all my software is shitty"</a> - <a href="http://www.codinghorror.com/blog/2004/02/about-me.html">Jeff Atwood</a>
</li>
<li>
<a href="http://davmac.wordpress.com/about/">"Software is Crap"</a> - davmac
</li>
<li>
<a href="http://www.hanselman.com/blog/EverythingsBrokenAndNobodysUpset.aspx">"it's all a mess. Everyone sucks, equally and completely."</a> - <a href="http://www.hanselman.com/blog/AboutMe.aspx">Scott Hanselman</a>
</li>
</ul>
<br />
<div>
Shall I go on?
</div>
<br />
It seems that everyone with an opinion and access to the internet is writing a post bemoaning the sad state of affairs in the world of software. They complain that every single system is utterly flawed at the deepest level and is at risk of crashing immediately if we dare even look at it the wrong way.<br />
<br />
<h3>
A Dose Of Reality</h3>
<br />
Now, don't get me wrong. There is software out there that is crap. I've even written some of it myself. I feel, however, that saying that <em>all</em> software is crap, or even that <em>most</em> software is crap is far from the truth.<br />
<br />
In the process of creating this post I have already interacted (both directly and indirectly) with many different software systems. I used my operating system, its user interface, and various hardware drivers to launch and control a web browser. I used my browser to search for and visit many different websites. Each of these sites is running at least a web server, and most have additional scripting and database systems that work with it. All of that data passes through multiple systems in transit over the internet. Guess what? I have not encountered a single noticeable error up to this poin...NO CARRIER<br />
<br />
...<br />
<br />
To my readers old enough to get the above reference, I apologize for subjecting you to such a terrible joke. To those younger readers who are scratching their puzzled heads, I apologize for subjecting you to such a terrible joke that you did not even understand.<br />
<br />
The fact of the matter is that most of us use a wide variety of software each and every day. Occasionally we encounter a problem that interrupts us or prevents us from accomplishing the task at hand. It may be the blue screen of death disrupting our career-defining presentation to a high-profile client, or our browser freezing halfway through a video of adorable cats diving head first into slightly-too-small boxes. The rest of the time, our software is quietly plodding along in the background and doing its job so well that we barely recognize it's doing anything at all.<br />
<br />
It's like that old caretaker at your company that most people were aware of, but nobody could quite tell you what he did. He worked after hours when everyone else had gone home for the night. You might have run into him that time you returned to collect something you had left in your office. Maybe he vacuumed the hallway outside your door that time you stayed late to finish your essential project before the deadline. When he retired, everyone suddenly noticed how so many subtle niceties around the office had changed . When a system is working well, we may not even realize how much we take it for granted until it is gone.<br />
<br />
Most software that gets noticed is like the annoying guy at the office who all the managers praise, but nobody wants to work with. He feels the need to add his input on everything and boasts about his "expertise". Trying to integrate his work with the rest of the team sets everyone back as they struggle to correct all of the problems he's caused. He manages to assign the blame to the soft-spoken fellow who sits by himself in the lunch room. When he manages to take credit for the eventual success of the project and parlay it into a promotion, everyone else on the team breaths a grudging sigh of relief that at least they don't need to deal with that guy anymore. This is the bad kind of getting noticed.<br />
<br />
The fact of the matter is that for every annoying software system that stands out in the wrong way, there are dozens of others that you barely even notice. You don't notice them because they perform their designated task in silence while you're busy watching groan-inducing videos of skateboarders landing crotch first on the railing instead of stylishly grinding down it.<br />
<br />
<h3>
Professional Pride</h3>
<br />
I don't know about everyone else, but when I step back from a challenging project that required lots of hard work, time, and energy, I take a certain amount of pride in it. I enjoy seeing efficiency improve as people use my system to accomplish tasks that used to take an order of magnitude longer and require tedious, manual effort. I enjoy creating new software realities out of mere ideas and possibilities.<br />
<br />
In any system large enough to be proud of, there will inevitably be some problems. When the bug reports roll in, I don't just throw in the towel and bemoan the fact that I've created yet another crappy system. No, I roll up my sleeves and get to work. I fire up my IDE and get started debugging my code and developing a fix. Depending on the operating environment and the nature of the bug, I can have a fix ready and deployed within hours - delivering a working solution to an eager customer who is (hopefully) pleased with such prompt resolution for their issue.<br />
<br />
When I watch users struggle to control my software, I don't just sit back and whine about how I've created yet another clunky user interface. I go back to the drawing board and come up with new ideas on how to make the user interface easier and more intuitive. I perform usability tests with ordinary folks who are trying my software for the first time. I iterate over new designs until I find a solution that is user friendly and simple to understand.<br />
<br />
<h3>
What They Really Mean</h3>
<br />
Perhaps I'm taking these people too literally. If their point is to acknowledge that most software has bugs and it's nearly impossible to create "perfect" software, then they've got it right. Developing software is challenging. Just when you think you've achieved a new level of competence, a new bug rears its ugly head. The bug mocks you as you sweat and furrow your brow, struggling to figure out what has gone wrong. When you finally figure it out, panting and exasperated, you smack your forehead for overlooking such an obvious flaw.<br />
<br />
The good news is that with each boneheaded bug you create and fix, there's actually a chance you might be able to avoid making the same mistake again in the future. If you keep your head down and work hard for many years, you may just learn to create software that isn't a flaming ball of garbage. That said, even the best of us have days when our brains are just not on their game and we write a fancy version of <code>while( 1 ) fork();</code> Don't lose faith. Keep going and continue striving to become a better developer. In time, you will gain a new level of competence and, at very least, you'll know enough not to let your guard down the next time you're feeling like a project was too easy.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com3tag:blogger.com,1999:blog-6631127102489130322.post-59645783554754013282013-06-07T21:16:00.003-07:002013-06-28T16:46:07.629-07:00Is It Easier To Write Code Than To Read It?<h3>
The Premise</h3>
<div>
<br /></div>
<div>
<a href="http://programmers.stackexchange.com/questions/200812/is-writing-software-easier-than-reading-and-understanding-it-from-scratch">A recent Programmers Stack Exchange post</a> reminded me of a seemingly widespread belief among programmers - writing code is easier than reading code. In his post, <a href="http://www.joelonsoftware.com/articles/fog0000000069.html">"Things You Should Never Do, Part I"</a>, about not writing software from scratch, Joel Spolsky writes, "It’s harder to read code than to write it". Most other posts I have found on this topic seem to reference Joel's article. They nod along in agreement as if to say that, once revealed, this premise is gospel truth and cannot be denied.<br />
<br />
<h3>
Horror Stories</h3>
<br />
Any programmer who's been around the block a few times can remember some godawful snippet of source code that warped their minds and nearly broke their will. Foreign (AKA not mine) source code may be difficult to understand for a variety of reasons. It may be oddly formatted, use poorly named variables and functions, have confusing organization, or seriously lack comments. Other than code full of "clever" tricks (why multiply by 8 when you can just use the bit shift operator?), I believe the most difficult code to read is simply at a level of complexity beyond the expectations of the reader.<br />
<br />
Since everyone seems to recall a story about an undecipherable code base from their past, it becomes easy to latch onto Joel's premise. We know that if we aren't careful about our code choices, it can grow into an unholy mess full of badly-named functions full of side effects that run over multiple screens. It's easy to get code into this state and it's difficult to deal with maintenance once the damage has been inflicted. In this sense, Joel's premise is correct.<br />
<br />
<h3>
Defining Difficulty</h3>
<div>
<br /></div>
If we ignore "bad" code and focus on "good" code, I believe that the picture changes. Writing good code is hard. Choosing meaningful and descriptive names for variables and functions is difficult. Structuring and documenting code so it's easy to understand and modify is a constant struggle. Self-documenting code contains all kinds of clues about the purpose of each component and demonstrates how each piece interacts with the other.<br />
<br />
I think that part of the problem comes with the vague way we define difficulty. What makes one task hard and one task easy? There are several ways to measure difficulty. Some are more meaningful than others.<br />
<br />
I think some of us choose to define difficulty based on our interest in performing a task. Clearly, watching TV is easier than washing the dishes. This measurement breaks down when a task is both challenging and interesting. I'd much rather climb a rugged mountain trail than take out the garbage, but scaling the mountain provides a much bigger challenge than dragging the trash to the corner. Programmers love writing code because of the intellectual challenge and the joy of creating something from nothing. Understanding the work of another coder does not generally hold the same appeal.<br />
<br />
When it comes to business, a much better way to measure the difficulty of a task is in the cost in resources. In this case, we're talking about the time and effort of a programmer. It may not be fantastically interesting to decipher someone else's code, but given a well-written code base, understanding comes with perseverance. Rewriting the same piece of code (larger than a few hundred lines) to the same level of quality will require an order of magnitude in extra time.<br />
<br />
<h3>
Discipline</h3>
<br />
Here's the bad news: every field has unpleasant tasks. As a programmer, I prefer writing new code over maintaining old code. When someone proposes an unpleasant task, people tend to come up with all kinds of excuses to avoid it. This is as true for programmers as for anyone else. The excuses pile up until we talk our way out of doing the unpleasant task.<br />
<br />
Unfortunately, as professional software developers, we're not being paid for pleasant. We're being paid for results. Great developers recognize this and exercise discipline. They buckle down and read the code carefully. They are able to understand it much more quickly than it could be rewritten. Once they understand the code, they make their changes and move on.<br />
<br />
The good news is that if you follow this approach, you'll get things done quickly and be able to move on to new projects. Reading good code instead of rewriting it will result in a higher level of productivity. Your boss may just take notice and reward you for your efforts.<br />
<br />
Cheers,<br />
<br />
Joshua Ganes</div>
Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-14723323294301223592013-04-10T21:24:00.002-07:002013-04-10T21:24:10.057-07:00Standard vs. Real-Time Unix Signals<h3>
Software Architecture</h3>
<br />
Imagine the following software architecture: a single parent process manages multiple child processes. The parent and child processes send Unix signals to manage and trigger activities between the two processes.<br />
<br />
You've tested this basic functionality under reasonable load and all looks well. After initial deployment, everything is looking great. It's time to turn up the volume and bring more activity and clients on board.<br />
<br />
After some time, a new bug report comes in saying that some requests are being lost in transit. Your logs show activity hitting the child process. You see the child signaling the parent to perform its next action. You see the parent handling a variety of signals from other processes, but the one from the relevant child process is lost to the mysterious void of /<i>dev/null.</i><br />
<br />
<h3>
The Reliability of Standard Signals</h3>
<br />
It turns out that there are two classes of Unix signals: standard and real-time. Standard signals run from 1-32 and real-time signals typically run from 33-64 (Be sure to use SIGRTMIN and SIGRTMAX to avoid compatibility issues).<br />
<br />
It also turns out that standard signals, while guaranteed to be delivered to a process by someone, are not guaranteed to be delivered by each process that sends them. This means that if multiple child processes send the same standard signal in a short period of time, the parent process may only receive one signal.<br />
<br />
This is where the magic of real-time signals comes in. These signals can be queued and will be delivered for each process individually. Use <a href="http://man7.org/linux/man-pages/man3/sigqueue.3.html">sigqueue()</a> to add signals to the queue of waiting signals. Be sure to check the return value for errors to prevent creating new bugs.<br />
<br />
Cheers,<br />
<br />
Joshua Ganes<br />
<br />Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0tag:blogger.com,1999:blog-6631127102489130322.post-53867711081996349642013-03-15T21:46:00.004-07:002013-07-09T22:14:38.547-07:00When Advertising Crosses The Line<h3>
Exploration</h3>
<br />
After publishing my first few posts on this blog I became curious about all of the available options to control it. I ran through the list of tabs reading the options and thinking about which ones I might adjust or tweak. I got to the "Earnings" tab and suddenly perked up a bit. I know that successful bloggers with thousands of regular readers can earn a tidy profit by allowing advertising space on their blogs. Now, my readership is nowhere near those numbers. My earnings, if any, wouldn't add up to more than a few dollars a month. Still, in my ever-optimistic mind, I smiled at the thought of my blog achieving critical mass and earning me a steady stream of cold, hard cash.<br />
<br />
<h3>
Human Behavior</h3>
<br />
What is the number one extension or add-on for web browsers? Well, that partially depends on who you ask. Most top-10 lists will be sure to feature <a href="http://adblockplus.org/">AdBlock</a> somewhere, with many placing it in the topmost spot.<br />
<br />
Do you fondly remember the early days of <a href="http://www.youtube.com/">YouTube</a>? Early on, you could view a video without the need to hover your mouse for five seconds before rapidly dismissing an advertisement. Admittedly, not all videos have this problem. <a href="http://google.com/">Google</a> (<a href="http://googlepress.blogspot.ca/2006/10/google-to-acquire-youtube-for-165_09.html">who owns YouTube</a>) allows flexibility for channel owners to decide how invasive the ads should be.<br />
<br />
Have you noticed that since <a href="http://en.wikipedia.org/wiki/Personal_video_recorder">PVR</a>s have become commonplace, watching live TV seems, somehow, more painful than it used to be? Those commercial breaks we used to tolerate are now prolonged interruptions leading to impatience and frustration (unless you needed the bathroom break). Many people have become self-professed experts at fast-forwarding over the commercials and hitting play just in time for the show to return.<br />
<br />
It seems, based on our behavior, that people don't like advertising much. We block it, we skip it, and we change our habits just to avoid it. Some of us have even begun changing our brains and started developing <a href="http://en.wikipedia.org/wiki/Banner_blindness">banner blindness</a>. All of these facts start to make a person wonder how anyone can advocate advertising at all. Surely, those people must be trying to justify their greed for those shiny advertising dollars.<br />
<br />
(...<a href="http://www.youtube.com/watch?v=0A5t5_O8hdA">and don't call me Shirley</a>)<br />
<br />
<h3>
The Other Side</h3>
<br />
<em>Quick</em>: What are the ingredients in a <a href="http://www.mcdonalds.ca/ca/en.html">McDonald's</a> <a href="http://en.wikipedia.org/wiki/Big_Mac">Big Mac</a>? <strong>Go!</strong><br />
<br />
(highlight to view the answer):<br />
<div style="background-color: #249fa3; color: #249fa3;">
Two all-beef patties, special sauce, lettuce, cheese, pickles, onions on a sesame-seed bun.
</div>
<br />
Did you get it?<br />
<br />
How about this one? Complete the following jingles:<br />
<br />
Oh, I wish I were an <span style="background-color: #249fa3; color: #249fa3;">Oscar Mayer wiener.</span><br />
Plop, plop. Fizz, fizz. Oh, what a <span style="background-color: #249fa3; color: #249fa3;">relief it is.</span><br />
<br />
Each one of these marketing phrases was broadcast several years before I was born. How could I possibly be so familiar with them? It seems that some advertisements have a certain <i>je ne sais quoi</i> that allows them to endear themselves to us and become part of popular culture. They have become burned into the cultural psyche.<br />
<br />
One of my personal favorites was <a href="http://www.molsoncanadian.ca/">Molson Canadian</a>'s "<a href="http://en.wikipedia.org/wiki/I_Am_Canadian">I Am Canadian</a>" advertising campaign. This series of ads inspired a wave of national pride and fraternity for Canadians (this was before the <a href="http://www.cbc.ca/news/business/story/2004/07/22/molsonmerger_040722.html">Canadian Molson and US Coors companies merged</a>). These ads connected with me personally and helped to fill me with pride in my country.<br />
<br />
When I was a kid, say seven or eight, I used to like some of the commercials more than the television shows. I would play in our living room while my parents watched TV and I would perk up only to watch my favorite commercials. I loved to sing along with the catchy jingles. I loved the brightly-colored and animated commercials most of all.<br />
<br />
I don't think you would find many people complaining that a restaurant puts a simple sign outside of its doors inviting customers inside. I certainly can't complain when they slap enticing pictures of their tasty dishes in the window. Some people love to window shop. They love just looking at the stylish displays in the store windows which beckon consumers to come inside and check out their other products and deals. Most people won't balk when an acquaintance offers them a business card if it suits the topic at hand. This sort of available enticement tends to be overlooked as a form of advertising.<br />
<br />
It seems to me that it's not as simple as judging advertising as good or bad; right or wrong. Clearly, some ads bother people more than others. Some may even inspire them, or at least amuse and entertain them. Others manage to get their point across without ruffling too many feathers. How, then, can we know which are good and which are not? When does advertising cross the line between acceptable and questionable?<br />
<br />
<h3>
Crossing The Line</h3>
<br />
I think we can all agree that we don't want to be treated like a sucker. Nobody likes to play the fool. Advertising should not bend the truth, and most certainly should not lie to you outright. This sentiment is even <a href="http://www.ipvancouverblog.com/2010/05/canadiancompetitionlaw-misleadingadvertisingupdate/">built into our legal system</a>. Perhaps most abhorrent of all is the ads that are misleading in their purpose. They entice you with an offer that sounds too good to be true. Those foolish enough to fall for the bait will be taken for a ride and dumped unceremoniously out on the curb, their wallets lighter for the experience. Don't forget web page ads that disguise themselves as content.<br />
<br />
Have you ever met a salesman who won't take no for an answer? Did you enjoy the experience? When someone puts information out in plain sight for me to see, I can choose to engage with him or choose to ignore him (recommended). If that person then begins to hound me as if I just wasn't paying attention, it can really get my hackles up. Guess what, I saw you just fine the first time and chose to walk past you. How likely do you think you are to get anything good out of me now that I have been interrupted?<br />
<br />
In much the same way, aggressive in-your-face advertising tends to receive very negative reactions. People begin to curse not just the products being sold, but the people who made them and their ads. Most people are content to ignore modest web advertising and continue browsing the site. When the advertising becomes more intrusive, the reaction becomes stronger. Who hasn't thrown at least a minor fit trying to find the browser tab with the obnoxious smiley face ad complete with sound effects?<br />
<br />
My final guideline to avoid crossing the line is directed at service and content providers: don't double-dip. No, I'm not talking about the time <a href="http://en.wikipedia.org/wiki/George_Costanza">George Costanza</a> <a href="http://www.youtube.com/watch?v=YWuSi00CcNk">double-dipped his tortilla chip</a>. I am talking about when providers try to accept money from both advertisers and consumers. This can be seen in various forms: a movie theater showing ads before the movie that patrons paid $12 a piece to see; magazines or web sites charging a large monthly subscription fee, but still cramming tons of ads between the content; DVD or Blu-ray discs that play <a href="http://en.wikipedia.org/wiki/User_operation_prohibition">unskippable ads</a> when they start up. Why should consumers pay good money to watch your advertising? Who are you serving? Make up your minds already.<br />
<br />
<h3>
Getting It Right</h3>
<br />
It seems that the path to advertising with a clear conscience is fraught with pitfalls. With so many ways it can be done wrong, we will need some guidelines for getting it right. Where can we start?<br />
<br />
If you're going to interrupt me, make it worth my time. As I mentioned earlier, some forms of advertising are more invasive than others. Some forms are a pure and unadulterated interruption. If you're going to interrupt me, please do whatever possible to appease me. Do something amusing, entertaining, inspiring, or compelling. Present your message if you must, but do not linger longer than the task requires. Your interruption will leave an emotional impact regardless - make sure it's positive and not one of frustration or anger.<br />
<br />
Be forthcoming. Most advertising is trying to drum up business so that the advertiser can sell more widgets, or get more people to use their service. The bottom line is money. Most people understand that people need to make a living. They won't hold it against you if you make it clear that you're asking them to participate in a business transaction. It's not that complicated, really. I will give you X if you give me money. Don't try to hide the fact that you are asking to be paid.<br />
<br />
My final guideline is to be accessible. We tend to accept advertisements that are intrusive but explain how the advertiser can meet our needs right now. While car ads at the movie theater may alienate moviegoers, ads suggesting that we <a href="http://www.youtube.com/watch?v=vfDXlgmKFyU">all go to the lobby to get ourselves a treat</a> are more acceptable. It's better to choose to advertise to consumers who are interested in your product or service now, than to those who have no need of it until the unforeseeable future.<br />
<br />
I wish that I could say more on how to get it right. Frankly, it's not too difficult to find ads that avoid pissing people off. Inspiring them, on the other hand, is an incredible challenge. When everything comes together just right, <a href="http://www.youtube.com/watch?v=8UZV7PDt8Lw">a brief segment of airtime can transcend advertising and become its own cultural icon</a>.<br />
<br />
<h3>
What Do You Think?</h3>
<br />
As I explained at the beginning of this post, monetizing my blog, given my current readership, would barely be worth my time. I'm pleased to find that the ads available to me from Google AdSense tend to be of the mild variety. They exist on the periphery of the blog and don't tend to interrupt the reader. They are even intended to be geared to the interests of the user and the contents of this blog.<br />
<br />
Imagine with me, if you will, that you have a successful blog that could potentially rake in an additional $1000 every month. Would you enable ads? Would it be selling out? What would you be concerned about?<br />
<br />
What are some things that advertisers do that cross the line for you? Let me know what you think in the comments.<br />
<br />
Cheers,<br />
<br />
Joshua GanesJoshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com1tag:blogger.com,1999:blog-6631127102489130322.post-20570498887593803282013-03-12T16:00:00.003-07:002013-08-08T22:30:16.878-07:00Backwards Compatibility II - Fixing Bugs<h3>
A Bug Report</h3>
<br />
Picture the following scenario:<br />
<br />
You are in charge of maintenance for a system that has been deployed in production at many different locations for over a year. A new bug report comes in saying that one location is having difficulties with a specific request. This particular site has been running well for several months up to this point, but one new request has started causing issues. After a little investigation, you realize that the problem is due to a longstanding bug in your software that no one has encountered before. Your server application is sending an ampersand ('&') character that is getting misinterpreted by the client application. The client is simply incapable of handling this message and crashes every time.<br />
<br />
What do you do?<br />
<br />
Well, you can start by fixing the client application so that this is not an issue going forward for any new installations and for anybody who is capable of updating their application software. Sometimes, this is not good enough.<br />
<br />
(By the way, why does software always go months without issue before receiving three bug reports in the same week?)<br />
<br />
In certain circumstances the client application is outside of your control and may be impossible to update at all the sites for various reasons: the corporate policymakers have frozen the solution and will not sign off on any updates; the software is installed at a remote location and nobody has the necessary permissions or skills to perform an update; <a href="http://reviewtimes.shinydevelopment.com/">some hardware governing body requires a drawn-out review process for you to release your client application</a>; or many other absurd reasons.<br />
<br />
Now what do you do?<br />
<br />
<h3>
Fixing Your Own Bugs</h3>
<br />
The key to designing for backwards compatibility is to embrace the fact that you cannot change the past. Since you cannot change the software that already exists in the field, you need to update the software that you do control to compensate.<br />
<br />
Before releasing your updated client application, add a new version indicator to the message format (if it doesn't exist already). Modify the server application to distinguish between the old client application (version 1) and the new client application (version 2). When the server is ready to send a version 1 response, it will replace any ampersands with a suitably-safe substitute. When it encounters a version 2 client, no substitution is required.<br />
<br />
The following example C++ code snippet demonstrates this logic:<br />
<br />
<code>
// Generate a new response object for the given request<br />
Response* ServerApp::handleRequest( const Request& request ) {<br />
<br />
Response* response = new Response();<br />
<br />
...<br />
<br />
string content = this->getResponseContent( request );<br />
<br />
// Remove ampersands for clients below version 2</code><br />
<code> // See bug 19243<br />
if( request.getVersion() < 2 ) {<br />
<br />
content = this->replaceAmpersands( content );<br />
<br />
}<br />
<br />
response->setContent( content );</code><br />
<code><br /></code>
<code> ...<br />
<br />
return response;<br />
<br />
}<br />
</code>
<br />
<br />
This update allows the existing client applications to continue operating in a limited way (sans-ampersand) while allowing new clients to use the full features of the application.<br />
<br />
<h3>
Fixing Other People's Bugs</h3>
<br />
Not to spark a religious operating system debate, but I feel that I have to mention the fantastic efforts of Microsoft to maintain backwards compatibility when developing Windows 95. As described by <a href="http://blogs.msdn.com/b/oldnewthing/">Microsoft employee and blogger Raymond Chen</a>, the Windows 95 development team <a href="http://blogs.msdn.com/b/oldnewthing/archive/2005/08/24/455557.aspx">went to great lengths to make sure old software still ran seamlessly on their new operating system</a>. They knew that most businesses and even individual users have one or more deal-breaker application that <strong>must</strong> work when they agree to upgrade their software. If those essential applications failed, Windows 95 would not fly.<br />
<br />
The engineers at Microsoft came up with a clever way to build backwards compatibility directly into Windows. They created <a href="http://technet.microsoft.com/en-ca/library/dd837644(v=ws.10).aspx">Application Compatibility Shims</a> - named, as I understand, after door shims used in construction. Just like a door shim allows for adjustments between the framing wall and the door frame, these software shims allow for adjustments between Windows and incompatible software applications. Microsoft wrote many custom shims to compensate for subtle and not-so-subtle bugs found when they tested old software running on their new operating system.<br />
<br />
When Windows 95 finally launched, there were surprisingly few programs with serious compatibility issues. When issues were discovered, there was rarely a need to dig into the scary underbelly of the operating system. Instead, Microsoft could just configure an existing shim or whip up a new one to fix the bug. This design was so successful that very few people ever gave a second thought to upgrading Windows. This gave Microsoft market dominance of both the corporate and personal desktop - a position that other platforms are still struggling to pry away.<br />
<br />
<h3>
Something to Think About</h3>
<br />
As I explained earlier, the key to backwards compatibility is embracing the fact that you cannot always change what already exists. By cleverly designing new software to compensate for old mistakes, we can mitigate their issues and sometimes even pretend that the old mistakes never existed. All of this comes at a cost, but that's a topic that I will have to address in a future post.<br />
<br />
Cheers,<br />
<br />
Joshua Ganes<br />
<br />
<br />
<span style="font-size: xx-small;">This post was the second part of a series on the topic of backwards compatibility. </span><span style="font-size: xx-small;">My next article on this topic: </span><a href="http://amish-programmer.blogspot.ca/2013/08/backwards-compatibility-iii-planning.html" style="font-size: x-small;">Backwards Compatibility III - Planning For The Future</a><span style="font-size: xx-small;">.</span>Joshua Ganeshttp://www.blogger.com/profile/02344437586406159667noreply@blogger.com0