I’m writing this post as a rant. I am tired of hearing people who have never worked in Federal IT try and come forward with ideas about what was wrong about the way Healthcare.gov was developed. I have one statement for all of you who think you could have done better.
You would have ALL failed miserably.
Federal IT is broken. Hell, all of Federal contracting is broken from what I’ve seen, but I want to focus on the IT side for now.
Before I get started, a quick reminder of my background. My first Federal project was back in the late 90s as the tech lead for the Secretary of the Air Force’s correspondence tracking system. Over the years, I have worked on a multitude of projects and managed many more while I was the Director of Technology Solutions for Washington Consulting . I’ve responded to many proposals and run Federal IT projects through the wide variety of hurdles that they face.
I can tell you right now, I am impressed that Healthcare.gov even boots up.
Success is Rare
I can quickly list for you all the Federal IT projects priced at over $5 million that were delivered on-time.
Zero.
Did you know that from 2003 to 2012, only 6.4% of projects with labor costs over $10 million in Federal IT were successful? Over 40% were never even completed they were such abject failures.
Projects of smaller scale than Healthcare.gov fail all the time. CIO.com has a nice little article about the futility that is Federal IT. They sum it up nicely by saying that Healthcare.gov Didn’t have a Chance in Hell.
Let’s face it. Healthcare.gov is a very public, politically-charged, system that is large and complex by nature. When you put out an inflexible go-live date that is in the middle of a political battle to delay or cancel the program completely, failure was certain.
What Went Wrong?
Let’s forget the obvious facts surrounding a system that is interfacing with over 36 states, a multitude of insurance carriers, and several federal agencies that think they have the best IT shop in the land. I want to talk about the over-arching process.
In Federal IT, typically nobody with both knowledge and authority owns all the components of a system. Even on smaller efforts, one contractor owns the data center, another runs the database infrastructure, one is developing the actual system, and a fourth contractor is in charge of making sure all the rules are followed.
I hate that fourth contractor. From what I can tell, they believe they won’t get paid if they don’t find problems in the documentation, the project plan, or your accent. I’ve seen projects held up because the columns in a table weren’t perfectly aligned. What should have been, Looks good, just fix the alignment before the next review, becomes, Go fix that table and we’ll reevaluate the entire package again.
In retrospect, hate is too mild. I despise them. I despise them like I despise the guy in the movie theater that has to call his friend to tell him that yes, he saw that hidden logo in the film this time and he now agrees that it does foreshadow the death of the main character at the end of the film.
The Federal IT process is not just a waterfall development cycle. That is an understatement. It is the Victoria Falls of software development processes. There are multiple reviews at every stage of the development lifecycle. Each and every interface needs it’s own collection of documentation. If you are talking to another Government agency, there are separate security documents that explain how the communication channels will be authenticated and protected. That documentation has to be completed before any holes in a firewall are even considered for opening.
Remember the part about 36 states? Multiply that out.
Let’s go back to the Data Center team for a minute. Do you need 300 Terabytes of storage for Content? No problem. We’ll go buy this huge NAS array. Do you need better performance for your database system? No worries. The vendor assures us that this storage system can handle the performance needs of any database system. Maybe you need to tune your database queries.
Keep in mind, this is for a small system and isn’t even counting the subcontractors involved. After all, there are small business quotas that have to be met in every procurement and if your team can’t check every single box in the RFP then you will be disqualified.
I’ve seen requirements for 5 years of experience with software that was only 2 years old.
It all works out though because the best qualified team always gets awarded the contract. Who wants to use those other guys when they promised to do it for 30% less. That would be everyone because if their proposal checks all the boxes, the procurement officer won’t get fired for selecting them. After all, who can lodge a legitimate protest to an award that will save 30% over the competition even if it promises failure.
How does that winning firm plan to deliver on the project while saving the government money? They just hire younger staff who have never done anything more challenging than fixing bugs for a 10 year old system in another agency and mix them in with aging IT architects who are just so happy to have a job that they will gleefully overdesign the system just to prove that they know how to use this new technology called a whiteboard.
If there was competent leadership on the Federal side there might be some hope. As I alluded to in the beginning of this rant, there isn’t. The best and the brightest usually get scooped up by a Federal contractor. They gladly make the change when they realize that they can do similar work for a lot more money. There is an IT labor shortage after all.
This is capped off when that talent gets hired away and then put into the same role at the same agency. The difference is that now they are a contractor. They are paid more money for the same job, and have less authority because they aren’t a Federal employee anymore.
In this environment, find a project manager that can effectively control the purported 55 contractors that worked on this effort. Keep in mind, contractors only listen to the people that sign the checks for their contract.
Like I said. I’m impressed Healthcare.gov even turns on.
Healthcare.gov is likely salvageable. Federal IT isn’t. Federal IT needs a do-over and the freedom from politics to let that happen.
Word of Pie 
That RFP process is broken here in Washington DC. The Nonprofit Times had a good discussion about that in this recent report: http://www.thenonprofittimes.com/wp-content/uploads/2013/10/10-15-13_ExecutiveSession.pdf (PDF)
LikeLike
Great rant. Thank you.
LikeLike
Great essay – thank you, thank you, thank you for putting this out there. I work in IT myself, and I would like to disabuse your readers of the notion that private sector IT functions any differently. I work for a large company, and like any large organization, public or private, we experience all of the issues that you describe above. At its best, distributed ownership and responsibility is complex and inefficient and balloons costs and timelines, but in most cases these conditions also add silos and turf wars to the challenges.
LikeLike
I’ve worked on both Federal and large commercial projects including large Insurance, Banking, and Pharmaceutical firms. It is a different scale of crazy in Federal IT. I’m not saying that there aren’t a lot of very dysfunctional private sector IT shops out there. I’ve seen a few so I know they exist. However, all the ones I’ve seen would favorably compare to every Federal IT shop that I’ve seen.
I think the major reason why is that an innovative leader can make a big difference in the private sector. In Federal IT, they don’t just have rules and guidelines, they have laws that can’t be readily circumvented. Also, when was the last time the Private Sector had to report their budget to Congress?
Difference scale of crazy.
LikeLike
When you put it that way, it makes sense that the crazy is on a different scale! I’ve never worked on a government contract, but I have friends who have, so I have a little window into that world. My only point is that many of the issues you enumerate above aren’t unique to government – private industry also faces them. On a different scale, as you say, but the private sector is not immune.
LikeLike
California’s system is excellent. Worked from the start. Why don’t the Feds just use CA’s system?
LikeLike
Simpmly put, the Federal site has to integrate with 36 States, who likely don’t want their data in California, and more insurance carriers, not to mention directly to Federal agencies. From a basic level, it would be seem good idea but it isn’t quite a 1:1 mapping.
LikeLike
The CA system works by zipcode. What I meant was for the Feds to get a copy of the CA web software and then just attach their databases to it. I’m talking about the Fed’s broken web interface. I’m pretty sure that’s a 1:1 map of user requirements. Swapping databases shouldn’t be all that hard. There’s a saying in IT that the Feds have never understood: “A complex system designed from scratch for implementation in total will not work and cannot be fixed. You have to start with a simple system that works, and then build on that.” Or, in commercial software product development, we call it “iterative prototyping”. The California system was also up and running a month before opening day. That’s the king of stuff that accounts for the multiple, massive systems that gov IT folks continually have to throw away.
LikeLike
Still have to communicate with the states. California has the data that it needs. That system isn’t designed to communicate with other systems. It is likely going to be faster to fix the current issues than to modify the California website.
Keep in mind, that is assuming you could get through all the procurement and security audits that would be required.
LikeLike
I agree that the system (both federal IT and federal contracting) is broken and in need of overhaul. However, there are arguably certain advantages to the current system, in its broken state, that any fix would need to preserve in some way: (1) that no one company holds all the power and control over government IT, (2) that “good ol’ boys networks” do not dominate vendor selection, (3) that the cost to taxpayers is lower (theoretically, at least) due to competitive bidding and outsourcing, and (4) the use of contractors prevents the government from being overly large (and thus costly for taxpayers). So, given that you have experience in the complexity of federal IT and insight into government boundaries, security issues and regulations, how would you fix it? Who should be tapped to fix it? I am very interested in how we lower failure rates and give taxpayers more value.
LikeLike
Two comments. 1) Contractors cost money and while they do not officially add to the Government’s size, they do in reality. Of course, Contractors are more readily trimmed than Federal employees. 2) Most of what you are referring to are procurement rules, which while problematic, isn’t the biggest problem with Federal IT. It is part of a broader discussion.
The primary way that procurement may have contributed to the current problem, aside from the large number of contractors involved, is the time it took to execute the procurement. 1 month of saved time is 1 month for testing and development.
LikeLike
Having worked in IT for over 30 years, as a contractor, a retail product developer, and an employee of a major computer mainframe manufacturer, this is what I have seen.
First, I do think politics of one sort or another are involved. Why? Because I sold multi-million dollar systems, and I did most of it through connections, lots of wining and dining etc. I also had my *incredible* expertise! as well, of course. But staff managers are pretty vulnerable to cronyism.
Second, the in-house staff and the vendors have incredibly incestuous relationships. It was a major problem for me as an account manager. If the in-house programmers screwed up, the vendor covered for them, to strengthen the relationship. When the vendor screwed up, that was not disclosed by in-house staff. So by the time top management found out about the real status of things, it was already so badly messed up and behind schedule that it was a disaster.
Third, the whole bidding and pricing process is a black hole. No $400 million contract should EVER be awarded to anyone. It’s too big for accountability. Projects need to proceed in phases, with each phase clearly defined, and phases should remain open for reassignment or shifting to other contractors as performance may dictate. No guarantees beyond any phase.
There’s more, but here is my solution. There needs to be much more time and effort spent on project management, and planning in general, all the way through. It should begin well before the RFP even goes out. The RFP needs to map out a series of benchmarks that are subsequently managed by an independent project auditor, along with the contracted or in-house Project Manager. Benchmarks must be met. And if one group is behind, resources can be shifted appropriately.
The additional pre-planning, project phase definition, and firm benchmarks, also cut the cost of the project. The bidding process is much less of a black hole. Both the vendor and the buyer are much better able to scope the requirements. Too often, when people are in a hurry, the thing they cut is the pre-planning. That, and they want results that simply can’t be achieved by the date requested.
So. Pre-planning, even before bidding, is key. An independent auditor needs to be assigned from the very beginning. No vendor should be awarded the whole project without explicit performance requirements at least every 90 days at which they nay lose some or all of the contract based on performance.
Ta Da!
P.S. Our government doesn’t audit anything. They do not audit any of their budgets. No department, including the military, is required to perform an annual audit by an accredited auditing firm. No one really keeps track of income, expenses, or assets in a credible way. If we simply began implementing the auditing process that every American public company does every year, we would save TRILLIONS of dollars annually.
LikeLike
Couldn’t agree more. With 50+ contractors, 36 states and a slew of federal agencies involved it’s a wonder this thing works at all. The number of potential communication and interface breakdowns is staggering with all those variables. Too many people think an agile approach would have fixed all these problems or that open sourcing the initiative would have saved the day. The problem here is much more complex than that. I think Shammy’s comments offer some sound guidance. Smaller chunks with real audit and control would incentivize the contractors to close scopes and delivery quality.
One more thought on some of the other commentary I’ve read blaming the “waterfall”. Writing down requirements first is not a bad thing if they define the “what” and not the “how”. The primary output objective of this project has not changed since ACA was passed in 2009.
LikeLike
Heard something astonishing on ABC News last night. The contracts awarded for the Affordable Care Act website totaled at least $387 million. In comparison, Apple developed the iPhone for $150 million.
LikeLike