Don’t let Somebody Else’s Technical Debt take you Under

There’s a lot written about technical debt: what technical debt is and the different kinds of technical debt, how to avoid taking on debt unnecessarily when designing and coding and changing code, how much technical debt is costing your organization, and why and how and how much and when to pay these debts off.

But all of this ignores massive amounts of technical debt and risk in your systems that have nothing to do with your developers taking short cuts in design or sloppy coding practices or not writing unit tests. What it doesn’t make clear is that there may be much more technical debt in all the code that you didn’t write, than in the code that you did.

In the same way that financial debt problems in Spain, Greece and Portugal are dragging down the global economy, debt problems in other people’s software can drag you down into the pit even if you've been doing a responsible job of managing debt in your own code.

Problems in software from Oracle or IBM or Microsoft, or all that nice free Open Source software that your developers keep downloading from the Internet, become your problems. If you are using other people’s software (and we’re all doing this), you have to pay for those other people’s mistakes and oversights, the decisions that they made to put time-to-delivery ahead of stability or security. You are at the mercy of their development and testing and security programs – or lack thereof. And you’re at the mercy of the third party and Open Source code that they used, and the people who wrote that software too.

Their bugs become your bugs. Their security holes become your security holes. Their bad decisions become your bad decisions.

The amount of code – and the amount of debt – involved can be huge, much bigger than the code that you’ve written yourself. According to a study by Sonatype, 80% of a typical java app is assembled from open source components and frameworks, and a big system can contain as many as 30 or more different libraries or other components.

But your exposure to third party debt is bigger than frameworks and libraries and controls. There is also all of the other software that your system depends on: the operating system, virtual machines and the rest of the run-time stack, your containers and caching technology and data stores, and all of the tools that you need to deploy and run the system.

In 2010, Gartner guesstimated that the total amount of global “IT Debt” – the cost of bringing all of this software up to date and fully supported in every organization – was around $500 billion, and that this could reach $1 trillion by 2015.

We keep taking more of this debt on all of the time. Aspect Security studied more than 113 million Open Source Software downloads, and found that more than 1/3 of open source libraries downloaded had known vulnerabilities.

Our exposure to problems in other people’s software is a serious enough – and common enough – problem that OWASP is now explicitly calling out organizations that use out of date software components as one of the Top 10 software security risks:

Virtually every application has these issues because most development teams don’t focus on ensuring their components/libraries are up to date. In many cases, the developers don’t even know all the components they are using, never mind their versions. Component dependencies make things even worse…

How much of other people’s debt are you carrying?

This kind of technical debt is much harder problem to understand and manage than debt in your own code. Because it is code that you don’t understand, code that you can’t look at to understand how bad it is, or code that you could look at but don’t. It’s often code that you don’t have control over, especially common platform technology shared across multiple systems, technology that everybody uses but nobody is responsible for. And it can be code that you don’t even know that you have – code that has been downloaded by somebody and has become part of your system without anyone else knowing about it.

To get some idea how big this debt problem could be, you need to audit your code for third party and Open Source packages and dependencies. Big companies can (and probably need to) use something like Black Duck or Palamida or OpenLogic to scan and build an inventory of Open Source and other libraries and components, to check on licensing issues and to keep up to date on reported vulnerabilities in this software.

Smaller companies can do this by hand, or try WhiteSource, a SaaS scanning platform that is free for startups and small companies, and that offers a free source code scanner for license discovery called JNinka; or check out the OWASP Dependency Track Project (an Open Source project to track third party software that is still in an early, pre-release state).

Keeping Up to Keep Debt Down

Once you understand how of other people’s software you have to worry about, it’s your responsibility to assess the risks and problems you have inherited, and to keep up with vulnerabilities and bug reports and with vendor patches and upgrades. This is harder than it sounds.

As OWASP explains:

In theory, it ought to be easy to figure out if you are currently using any vulnerable components or libraries. Unfortunately, vulnerability reports do not always specify exactly which versions of a component are vulnerable in a standard, searchable way. Further, not all libraries use an understandable version numbering system. Worst of all, not all vulnerabilities are reported to a central clearinghouse that is easy to search, although sites like CVE and NVD are becoming easier to search.

Determining if you are vulnerable requires searching these databases, as well as keeping abreast of project mailing lists and announcements for anything that might be a vulnerability. If one of your components does have a vulnerability, you should carefully evaluate whether you are actually vulnerable by checking to see if your code uses the part of the component with the vulnerability and whether the flaw could result in an impact you care about.

There are also some general-purpose services that can help. Consolidated security vulnerability feeds like SANS @Risk or the SecurityFocus Vulnerabilities feed provide frequent notification of security vulnerabilities in common software packages. And Industry-specific information sharing and analysis centers like FS-ISAC and IT-ISAC provide members with notifications of problems with common third party and open source software.

Patching and Patching and Upgrading

Knowing how big the problem is, is one thing. Fixing it is another.

Patching to the latest dot release is a hassle that we all try to put up with. It’s better to be safer than sorry with patching outside-facing technology even if the patches don’t always apply to how you use the software: Apache, Tomcat, web services libraries, client components. Back-end code can be – and usually has to be – managed more conservatively. If you really really have to install every Linux or Oracle RDBMS patch as soon as it comes out, there’s probably something wrong with your architecture.

Upgrades are a much bigger problem. There’s usually little upside, but a lot of costs and risks involved with upgrading to the latest major OS or DBMS release or VM or other critical software. This is the kind of work that you take on because you’re held hostage by your suppliers – because the only way that you are going to keep getting support is by moving to their latest and greatest release.

While you may gain some advantages in scalability or manageability or a feature or two, upgrade projects are often more about managing the potential downsides: functional regressions, compatibility problems, changes to operations procedures and dependencies on other systems. Most software gets bigger, not better – more features, even if they are features that you don’t need or don’t want, still means more work to install, configure and test when upgrading. You need to work through everything that has changed and re-do all of your testing and tuning, re-finding and checking old workarounds and finding new workarounds for new problems or improvements that you don’t want, updating documentation and scripts and procedures. And the testing that you have to do is the worst kind of testing: manual system testing and operational testing and stress testing, expensive while giving you low confidence that you will find all of the problems. Everybody needs to be dragged into this: development, QA, Ops, and other teams who share dependencies.

So after all of this work and cost you still need to be prepared to deal with problems in production. And you haven’t accomplished anything of clear business value. You’ve taken on short-term operational costs and risks in order to minimize other operational costs and risks in the future. It’s a low return on investment at best.

Open Source – Be a Lazy Customer, Join the Community, or Just Fork It

Almost everybody is using Open Source Software in development – it’s too compelling not to.

But most organizations do a bad job of managing this software:

While nearly 80 percent of companies rely on open-source components for their development efforts, more than three-quarters lack any meaningful controls over the usage of such libraries and frameworks, according to the annual Open Source Software Development Survey conducted by Sonatype, a manager of a large repository of open-source components.

Another vendor, WhiteSource Software, has found that 85% of all software projects contain out-of-date open source components.

It’s your responsibility to understand what risks you are exposed to and to minimize them when choosing and using Open Source Software. This means doing some detective work to ensure that a project is active and to see if it stays alive, periodically monitor check-ins and forum activity, watch out for the authors fading away or changing direction, for people forking off down separate paths.

Black Duck’s Ohloh database is one place to start: a public directory that has information on thousands of Open Source projects including the size of the code base, the size of the community, the latest commit, and licensing information. Or there’s Freecode (formerly Freshmeat) for information on Linux, Unix and cross-platform software and mobile applications, although these directories are not always comprehensive or up-to-date.

Another useful source of information is the database that Coverity maintains on bugs found by scanning Open Source projects for security and quality problems.

For every piece of Open Source Software that you decide to use

1. you can be a user, take advantage of the free software and hope that the community will keep the project alive for you.
2. you can take an active role, contribute fixes and improvements back to the community and see which changes are accepted. If they don’t accept your changes, then you will have to worry about reconciling them with any future updates that the community makes.
3. if the software is important enough to you (and if the license permits it), you can choose to fork it and support it yourself. This is not a trivial commitment to make, even if you seem to have the necessary technical skills in-house. A lot of Open Source software is highly specialized and requires highly-specialized skills and knowledge – which is why it’s worth using in the first place. Even smart experienced developers will take time to figure it out well enough to extend it and fix it, and now you have to take this responsibility and these costs on over the long term.

You can bury your head in the sand or…

In the short-term, ignoring the hassles and costs and putting them off into the future sounds good. No payments until 2015. Of course, when the interest payments kick-in, you’ll wish you were somewhere – or somebody – else.

But consciously taking on debt and putting it off into the future can be a useful strategy – as long as you understand the trade-offs. You may be doing the best thing for your business today by consciously taking on debt and sticking with what you know and what works today, keeping everything stable and planning to upgrade later. As long as everyone understands that the longer you put upgrading off, the more that costs and risks could add up.

But this is a path that can lead you into a legacy trap if you put this work off for too long, leaving you with a system that nobody can understand and nobody can support properly.

The Tip of the Iceberg

Like the iceberg that sunk the Titanic, a lot of your technical risk may be hidden or ignored until it is too late. You need to understand how big the risks are and take responsible steps to manage them.

Large companies that have the resources and leverage to influence their vendors can be more proactive. For example, Veracode offers a third party scanning service for enterprises that want to do scanning of third party binary code for vulnerabilities and bugs.

The rest of us can at least check out the software that we are using and the people who wrote it, keep up with vulnerabilities and bugs, and recognize the risks of using somebody else’s software. Because we can’t pretend that the risks and costs aren’t out there somewhere, or that they aren't serious enough to take us down.