HealthCare.gov: Boat of Gloat v.5b

The failure of HealthCare.gov is epic and groWING. We are 25 days past the live date and the number of enrollments are still at a trickle. The administration is trying to stifle all reports of actual enrollments because they are in a fog and because it is embarrassing.

The House of Representatives held a hearing on what went wrong this week and there were several vendors present, all pointing fingers at one another and at the government, who was apparently in charge of integration testing, or end-to-end testing. We've also learned that there are major flaws in every major part of the system regardless of the claims of the vendors. Worse than the technical problems, the management failures at all levels of this project are classic and numerous. A nice case study for academics to pour over for a generation.

There is so much goodness to discuss (by goodness, I mean badness), it can make one dizzy. Today, I want to focus on the data corruption issues with the electronic enrollments being sent to insurers, for those lucky few that make it all the way. To understand what is happening, you need to start in 1996 with the enactment of HIPPA. HIPPA was supposed to simplify and speed up health care transactions, and improve patient privacy. A set of data standards were published for all health care organizations to use for standard transactions like: enrollment, disenrollment, claims, payments, benefit inquiries, etc. Each transaction was assigned a number. An electronic enrollment is an 834 transaction. A claim is an 837 transaction. An eligibility inquiry is 270 and the response is a 271 transaction.

The data specification for each transaction type, when printed, fills a large binder. I implemented and maintained an EDI claims system for dental billing from 2003-2004 so I know many of these formats intimately. In general, each transaction consists of a set of variable length records. There is usually a header record to start, followed by a variable number of information record types or segments, followed by a trailer record. The headers and trailers are easy. The segments are variable length, with a variable number of elements, separated by asterisks (*). Segments are separated by tildes (~). Beyond deciphering the correct format, each insurer may or may not have been in 100% compliance with the standard. Some required extra segments or special values in certain elements or they would reject the transaction. It was not enough to be in 100% compliance with the standard, you also had to handle the quirks of each insurer's IT and transmission requirements. All transactions were supposed to be encrypted (we used PGP) and files were exchanged via FTP or dial up. I worked with about 25 different insurance companies and it was a nightmare. HealthCare.gov has to deal with hundreds or maybe a thousand on the back end. Major nightmare.

Here is what a partial HIPPA transaction looks like:

ISA*00* *00* *ZZ*CALIFORNIA-DHCS*ZZ*777888999 *111025*1127*^*00501*000000001*0*T*:~GS*BE*CALIFORNIA-DHCS*777888999*20111025*112755*1*X*005010X220A1~ST*834*0001*005010X220A1~BGN*00*DHCS834-DA-20110712-777888999-001*20111025*11275500****2~QTY*TO*6~N1*P5*California Department of Health Care Services *FI*68-0317191~N1*IN*Medical Plan 123*FI*777888999~INS*Y*18*001*AI*A*C**AC~REF*0F*12345678C~REF*1L*111111111~REF*23*0;20100204;~REF*3H*11101111111373;;~REF*6O*Q;;A;Y;A;11;78~REF*F6*010203040A~REF*DX*B5555;E777;20110323~NM1*IL*1*MCGEORGE*GEORGE*G~PER*IP**TE*7025551212~N3*777 GEORGE AVENUE APT 222~N4*LOS ANGELES CA*CA*900201111**CY*19~DMG*D8*19570709*M**:RET:2106-3~NM1*31*1~N3*PO BOX 12345~N4*LOS ANGELES CA*CA*900481234~HD*021**HLT*123;01~DTP*348*D8*20110901~DTP*349*D8*20110930~REF*17*D;49101;;;~REF*9V*3;2;2~REF*CE*10;001;80;301;;;;~REF*RB*10~REF*ZX*19~HD*021**HLT*123;01~DTP*348*D8*20110801~DTP*349*D8*20110831~REF*9V*3;2;2~REF*CE*10;301;80;301;;;;~REF*RB*10~REF*ZX*19~HD*001**HLT*123;P4~DTP*348*D8*20110701~DTP*349*D8*20110630~REF*9V*3;1;2~REF*CE*14;001;80;891;;;;~REF*ZX*19~INS*Y*18*001*AI*A*E**AC~REF*0F*23456789E~REF*1L*222222222~REF*17*201205;~REF*23*1;20061214;~REF*3H*193NXXX7772101;SMITH AMY;038~REF*6O*A;;A;Y;;11;~NM1*IL*1*PLANK*FRAN*M~PER*IP**TE*2135551212~N3*THIS IS LINE ONE OF THE ADDRESS*123 FRANCIS LANE~N4*LOS ANGELES CA*CA*900391111**CY*19~DMG*D8*19841125*F**:RET:2106-3~HD*021**HLT*123;01~DTP*348*D8*20110901~DTP*349*D8*20110930~REF*17*N;49201;;;~REF*CE*3N;401;;;;;;~REF*RB*3N~REF*ZX*19~HD*021**HLT*123;01~DTP*348*D8*20110801~DTP*349*D8*20110831~REF*CE*3N;401;;;;;;~REF*RB*3N~REF*ZX*19~HD*021**HLT*123;01~DTP*348*D8*20110701~DTP*349*D8*20110731~REF*CE*3N;401;;;;;;~REF*RB*3N~REF*ZX*19~HD*021**HLT*123;01~DTP*348*D8*20110601~DTP*349*D8*20110630~REF*CE*3N;401;;;;;;~REF*RB*3N~REF*ZX*19~HD*021**HLT*123;01~DTP*348*D8*20110501~DTP*349*D8*20110531~REF*CE*3N;401;;;;;;~REF*RB*3N~REF*ZX*19~INS*Y*18*001*AI*A*E**AC~REF*0F*34567890D~REF*1L*333333333~REF*17*201203;~REF*23*8;20050531;~REF*3H*197JQQQQ555103;WHITE JESSICA;051~REF*6O*A;;W;Y;;56;~NM1*IL*1*PUNTER*GUNTHER~PER*IP**TE*9165551212~N3*69001 BRIAN WAY~N4*SACRAMENTO CA*CA*958141234**CY*19~DMG*D8*19900817*M**:RET:2054-

When you read about insurers getting bad data for enrollments, they are getting 834 HIPPA EDI transactions that don't work in their systems. In some cases it is bad data, meaning the format might be OK, but the data is incorrect. In some cases, the format might be right, and the data might be right, but the insurer is not 100% compliant so it fails in their system. In some cases, the format might be wrong, but the data is right. In other cases, the format might be wrong and the data may be wrong. Multiplied by as many insurers as receive transactions from HeathCare.gov. And this is only one small part of the processing on the back end.

Kathleen, this boat's for you!

HealthCare.gov: Boat of Gloat v.5a

I found this article, Bad Government Software, at baselinescenario.com from James Kwak:

Ezra Klein, one of the biggest supporters of Obamacare the statute, has already called the launch of Obamacare a disaster, and it looks like things are now getting worse: as people are actually able to buy insurance, the data being passed to health insurers are riddled with errors (something Klein anticipated), in effect requiring applications to be verified over the phone.

Oh, my. I hope the insurance comapanies staffed up to verify millions of applications by phone.

I wanted to find out who built this particular example, and I found Farhad Manjoo’s WSJ column, which fingered CGI, a big old IT consulting firm (meaning that they do big, custom, software development projects, mainly for big companies).

Why is so much software so bad? There are lots of reasons. Writing good software is hard to begin with.* Big, custom projects are unique by definition, so they are sold as promises, not as finished products. Every vendor promises the same thing, so the one who promises to do it at the lowest cost often wins; when the project turns out late, bad, and over budget, too many executives have too much invested in its success to admit defeat. Consulting firms, which bill by the hour, make money by staffing projects with lots of people at relatively low cost, which is absolutely the wrong way to develop software; the productivity differentials in software are so vast that you can often get ten times as much output (of quality software) for less than twice the price, while a bad developer will do more harm than good to a project.

All completely accurate, echoing many of my observations and repeated real life experience. What happens to the most awful big systems? The best outcome is it gets shut down and abandoned with lessons learned and a few careers ended. That is possible with in-house systems because the rest of the world doesn't have to know. With a system that has been unleashed on the unsuspecting public, it can't just be abandoned. It has to be fixed or replaced while running, neither of which is ideal or easy, Both are expensive. CGI, and each vendor involved will probably rake in mad stacks trying to fix the bugs, depending on how their contracts werw structured.

As others have noted, the failure of healthcare.gov is not unique in the annals of government technology projects. But it is surprising that the Obama administration—which has tried to build a reputation for competence—did so spectacularly badly on its flagship project. Most likely, there were just not enough people in the chain of command who had enough understanding of technology to realize that things were going horribly wrong, which is a pretty clear management failure on the part of the administration.

Bazinga!

To make matters so much worse, the President still doesn't understand the scope of the software disaster. He thinks the vendors hired by HHS to create this frankenstein can slip in a couple of patches, like an Apple IOS update, and everything will start working. That analogy shines a spotlight on his lack of understanding of software development. A better analogy would be Duke Nukem Forever.

Extraordinary Mismeasures

source: http://www.treasurydirect.gov/NP/debt/current

Since May, the official debt of the US government was frozen. It didn't change a penny despite billions in new bills, notes, and bonds issued. Then, from October 17 to October 18, it jumped $328 billion overnight. Instead of adding to the official debt, the Treasury used what it calls "extraordinary measures" to report money owed to other programs, like federal and military pensions, to fill in the hole not reported as debt. As shown during the shutdown and debt ceiling scare, the government cannot operate any longer without massive additional borrowing. It is accepted that large deficits (despite the shrinking YoY deficit) are essentially a permanent feature of the US government, that US debt can always be rolled over easily (unless Congress prevents it), and that all this debt issuance is good for the world.

If that was true, how did the world survive before 1975 when US deficits became permanent (except for one bubble stock capital gains year at the end of the 1990s)?

HealthCare.gov: Boat of Gloat v.5

Ten days after the open enrollment started on the ACA (Obamacare) HealthCare.gov web site, Politico describes it as a "train wreck" (full disclosure: the piece was written by a National Review editor). News organizations of every political slant can plainly see the disaster it has been and continues to be. From the Politico link:

The rollout of Obamacare has been so disastrous that even “Daily Show” host Jon Stewart was plainly mystified and unconvinced when Sebelius came on his show the other day to offer soothing explanations and reassurances. Stewart gently expressed his frustration that there is “a level of incompetence that is larger than what it should be,”

and...

Nancy Pelosi infamously said that we had to pass the law to find out what’s in it. But the then-House speaker erroneously assumed, evidently, that people would be able to get onto the government-run exchanges created by the law. So far the law’s implementation has been as ugly as its passage.

As a professional software developer (among other things) with 28 years of experience in both private and public sectors, I happen to have expertise in this field. In fact, four years ago, I wrote an open enrollment system for a public agency with complex flexible benefits covering health insurance, dental insurance, vision insurance, health and dependent savings accounts, deferred compensation, long term disability insurance, and life insurance. The system is still running today, and while the scales are not comparable, my system has enrolled more people this year that the states of Maryland (326), Iowa (5), and Hawaii (0) combined. I understand system development, analysis, design, testing, tuning, documentation, and deployment from conception to production. Creating good software is very hard. If you haven't done it for a living, it is easy to underestimate and hard to appreciate.

I've had the unfortunate experience to be pulled into a large failing government project (Blue Cross and Blue Shield of Florida, Medicare Part B) many years ago. I witnessed amazing incompetence, especially at the management level, that torpedoed the 90 hour/week efforts the programmers were suffering. In that project, out of 9 different virtual systems, no one could identify which was production. There were 6 test systems, all out of sync with each other, and 3 remaining systems, any of which "might" have been the production system. Programmers would apply their updates to the 3 possible production systems, frequently overwriting updates from other programming teams. The ticketing system worked intermittently and databases went down without warning. It was chaos when I started the engagement, and chaos when I left. The firm I worked for at the time collected huge consulting fees for the effort, but very little got permanently fixed over that 6 month period.

This kind of scenario might be playing out behind the scenes at Healthcare.gov. There are probably numerous contractors that built various pieces of the system (I know Experian is one of the culprits) at a cost, so far, of over $600 million. But contractors sometimes have little control or incentive to get the get the whole system working. They are focused on their part and collecting time and materials on a can't lose contract. What I'm suggesting is that it could easily be 6-12 months before the system is stable and performing as expected. Bringing in outside experts at this stage won't help and might even be counterproductive. Blame is being passed around and the work environment is probably toxic.

Obama might have saved himself a huge embarrassment if he had allowed the individual mandate to be delayed a year as the Republicans first wanted. Instead, he forced America to use the half baked, wheezing, hot mess that is the current system. For that, the President and Department of Health and Human Services have earned the Boat of Gloat.

Other opinions:

Obamacare Disaster Predictions Coming True - Even Howard Dean Agrees

Why Obama’s tech-savvy team couldn’t make Obamacare glitch-free

Obamacare Is a Disaster and Democrats Own It

Another Democrat Calls ObamaCare Implementation a "Real Disaster"

Obamacare ‘disaster,’ or routine maintenance?

Will glitches derail Obamacare?