Friday, December 6, 2013

Outsourcing Federal IT = Silver Bullet: Don’t Let The Truth Interfere With A Good Story


 
Sparked by the Healthcare.gov rollout “disaster”, pundits and politicians have pointed to “outsourcing” to the private sector as a solution to developing and operating Federal IT systems.  To paraphrase Ronald Reagan: There they go again! 

 I worked on a study for an agency looking into outsourcing software development and operations.  The CIO wanted examples of where outsourcing was being done and whether it was achieving the organizations technical and cost goals.  We enlisted the help of Professor Mary Lacity:  http://www.umsl.edu/~lacitym/vita.htm of the University of Missouri-St. Louis who is widely known for her teaching, data collection and 18 books and countless journal articles on outsourcing.  She has been collecting global data on outsourcing for many years, and has tracked deals through their life cycles.

 At the time of our study the idea of outsourcing had taken hold in the IT industry.  Australia’s Inland Revenue Service (their IRS), Dow Chemical, Kodak, and other large organizations were attempting to focus their own staff on direct mission functions and jettison the IT work to (presumably) more productive and cost effective contractors. 

 While there is a vast body of work which I do not want to misrepresent, it’s fair to say that contractor vs inhouse skills isn’t as much of an issue as is the ability of the organization to craft and execute contracts that meet their REAL and evolving needs at predictable, perhaps lower costs.  It turns out that many organization’s employees actually do things as a part of what they perceive as their JOB that the official organization charts and job descriptions do not document.  So, it is problematic for an organization to specify “exactly” the duties, functions, procedures, and tasks to be performed under contract, at a price, with performance criteria.  It is also difficult to specify future needs and to specify, measure and enforce cost and productivity characteristics.

 Unless perfectly specified, the work performed by a contracted source will either: a) not meet the organization’s ACTUAL requirements, or b) will (and should) cost more than originally priced.  On the other hand, to achieve an “exact” specification and contracting documents acceptable to both parties take a very, very long calendar time (multiple years) and thousands of hours for both parties in lawyers, accountants, business and IT managers to arrive at an acceptable set of contractually binding documents.

 The real world experience has yielded very mixed results.  Lacity’s books and papers chronicle these results and are mandatory reading for those contemplating outsourcing.  The myth that outsourcing is a silver bullet is partially due to the fact that most who have attempted it, and failed, do not discuss or publicize those failures, leaving the positive impressions that were left from the big media deal announcements (even Time Magazine covers).  Lacity has published data and analyses showing the true story.

 In the Federal IT environment, requirements are continually evolving (including new legislation); contracting process is proscribed; deals must be done in public; and there are more antagonists than protagonists. Outsourcing federal IT functions is probably a fool’s errand if the goal is to save dollars and achieve higher overall productivity. 


Follow this blog for updates and new items.

Monday, December 2, 2013

Federal I.T. Lesson Learned and the Hero of Katrina: National Finance Center’s Gil Hawk

It’s been 8 years since Katrina, but a good story about the Federal Government working well can’t be repeated too often and serves as a lesson to CMS and others operating Federal systems.  The agency that worked extremely well during and in the immediate aftermath of Katrina was the U.S. Coast Guard.  The USGG didn’t need anyone is Washington D.C. to direct them.  Their rescue mission is well defined and executed 24/7/356 without hesitation or confusion.  

But there is another Federal organization that actually worked extremely well before, during and after Katrina. The National Finance Center (NFC) is a part of the Department of Agriculture and processes the personnel and payroll for many other agencies and organizations.  NFC’s massive data center is located in New Orleans on a spit of land between Lake Pontchartrain and Lake Borgne. Before Katrina it had over 1,100 employees, processed payroll/personnel for 600,000 federal employees as well as processing health benefits for ~2M participants.  Other organizations at the data center had about 350 employees and made over 2.5M payments back then and ran several financial and administrative systems.

Sounds like a disaster in the making huh?  Somehow Gil Hawk and his teams managed to keep ALL MAJOR FEDERAL FUNCTIONS running in spite of the affects of the storm on the NFC employees lives, families and homes. How’d they do that?  Through contingency planning, practice drills of those plans, continuity of operations (COOP) preparations, establishing alternate operational sites, off-site data storage and backup infrastructure, and a host of other well-known and recognized techniques for ensuring continuity of services under the most adverse conditions.

If the Healthcare.gov team at CMS hasn’t reached out to Gil Hawk yet, they should.  After the current “glitches” of the Exchange portal are fixed, and the back-end systems for subsidy calculation and payment to insurance companies are rolled out. They need to quickly focus on Preparedness Planning.   The Healthcare.gov data centers aren’t surrounded by lakes…. but they’re probably on the same electrical grid, nor’easter area, communications cable trunk, ‘’snowmageddon’ zone, or eastern-earthquake area. Since planning doesn't seem to be CMS's strong suit, it needs to ensure that they redouble efforts to prepare for these or other conceivable natural or man-caused major disruptions. They couldn't go wrong by seeking advice from Mr. Hawk.

Give kudos to Gil Hawk and the dedicated employees at NFC.

The media ‘talking heads’ who suggest “outsourcing” to contractors as a solution to developing and operating federal systems are wrong. 


Follow this blog and read about the Myth of Successful Outsourcing in the next post.

Friday, November 29, 2013

Healthcare.gov: There’s NO app for that.

 In the beginning of an automated system project, business owners envision all of the automated features that they’ve seen before, and even some that they’ve just imagined. Some, if not all of the White House pols and speech writers “let” the president compare the Exchange to buying things from Amazon and the price comparison experience to Travelocity.  Was this an intentional over-simplification for sound-bites and “talking points”, or was it a gross underestimation of the complexity of the undertaking?

After pouring over left and right media accounts, internal emails, memos, a red team briefing and several progress reports, I’ve concluded that the administration at the high levels were close to clueless as to the technical complexity and the time required to develop as system with complex and difficult technical requirements.  Among these 'difficult technical requirements' are:
  •     Identity Proofing: associating an internet-based transaction with a real person and validating that association with little-to-no doubt.  This is required for both privacy and subsidy/tax purposes.  There’s no app for that.
  •   Agency Interfaces: to validate identity and to determine eligibility and amount of tax subsidy the Exchange has to compare information submitted to it by the prospective customer to The Social Security Administration, Internal Revenue Service, The Department of Veterans Affairs, The Department of Homeland Security and others as a part of determining eligibility (citizenship, eligibility for other insurance programs. Annual income, and other eligibility criteria. This means that the Exchange needed to interface with existing, OLDER agency systems.  Or it meant that these agencies needed to build a new system to provide such information to the Exchange’s data format and content specifications. There’s no app for that.
  •        State and Insurance Company Interfaces: All 36 states, and some 170 insurance companies also need to interface with the Exchange.  The states that built their own web-marketplaces need to use the part of the Exchange that performs the Identity Proofing and Agency Interface capabilities.  The insurance companies need to upload information about their offerings and pricing of some 4,500 plans, There’s no app for that. 
  •   Volume:  The idea that tens of millions of customers might show up at one time even scares Amazon on “cyber monday”…. and they’ve been in the business since they went online in 1995.  There’s no capacity or precedent and definitely no app for that or even for calculating it.


So it appears that the political folks and the policy folks that were ‘in charge’ of this complex undertaking did not obtain or trust technical and management staff, expertise or advice from proven sources.  Instead it seems that they relied on gross technical oversimplification, hubris, mismanagement or misinformation.  At this point it almost doesn’t matter which, and it was probably a mix of all of those.  On Dec 1, we’ll see if their attempts to recover will work.  Unfortunately there is no RECOVER app either.

Wednesday, November 20, 2013

Is healthcare.gov suffering from new, modern I.T. problems or is it the same-old, same-old?

 After reading the first three posts, someone asked me (thanks dear) why the themes listed focus on negative aspects, rather than on positive ideas about how things should be done.  For me, the answer is that I have neither interest in regurgitating what is already documented in texts, nor interest in creating a new one.  There are libraries of excellent texts; many professional disciplines and certifications; an ample number of proven life-cycle methodologies; industry-wide standards and practices; and educated, seasoned professionals ‘out there’. Perhaps, there are too many sources and methods from which to choose.

 What sparked me to write about systems projects gone awry was watching the healthcare.gov project crash. As more information about its problems became public, I expected, more accurately, HOPED to see some new category of problem, some unforeseeable root cause, some hidden threat that emerged late in the project.  I reasoned confidently: that the system at the center of the 100-years-in-the-making legacy achievement for the Administration would get the ‘best and brightest’, and would get high level detailed, perhaps even brutal scrutiny. This was not to be.  Instead of repeating the successes of the high-flying new generation of IT gurus made legendary by their historically brilliant application of IT during the political campaigns, America got the same-old, same-old IT project failure scenario.

Why?  How?  Who?

 In this blog space I want to explore both the nature and behavior of organizations and the people inside of them when building automated systems.  Why do (we) IT and management professionals fall into the same 40 year-old traps?  How and why does management miss or ignore warning signs?  Who makes up the typical team – and does that contribute to this apparent blindness?

 Since 1963 I’ve worked in what I’ll call Information Technology. Back then it was called Automated Data Processing (ADP) and was punch card based (a term so old that spell check doesn’t want me to use it). The processing and logic was split across different machines (one for each function) each having a unique hand-wired control panel for each step. All of the technology has changed drastically. What it took eight of us to do on a dozen machines back then, can be done by one person on an iPad today   What hasn’t changed enough is the ability of organizations and the people in them to consistently apply proven methods in building systems.


 This blog is dedicated to exposing these issues more fully in the hopes that someone will learn something that will help avoid the mistakes that continue to plague IT projects in particular.

Tuesday, November 19, 2013

Healthcare.gov and Irrational Optimism

On Nov 19, 2013, 50 days after launch of Healthcare.gov, a ‘red team’ report by McKinsey was made public.  The (undated) red team report, was apparently briefed to White House, HHS, and CMS officials in April.  The briefing addressed several areas of the project, but one set of observations in the 14 page briefing that struck my interest is the part that compared an ‘ideal situation’ for projects of this type to the ‘current situation’. The current situation was described as having:
  • ·      Evolving requirements
  • ·      Multiple definitions of success
  • ·      Significant dependency on external parties/contractors
  • ·      Parallel “stacking” of all phases
  • ·      Insufficient time and scope of end-to-end testing, and
  • ·      Launch at full volume 

In the systems engineering world, these are among the deadly sins of projects:
  • ·      No technical target          (specification ambiguity)
  • ·      No project goals              (scope creep and confusion)
  • ·      No single authority          (managerial chaos)
  • ·      No proven methodology (technical chaos)
  • ·      No time                           (mandated 'big bang' date)
  • ·      No live testing                 (everything must work for everybody on day 1)

So, what happened in the five months between the red team’s diagnosis and launch?  One politically oriented notion is that those closest to the President understood the risks and decided to ignore them or not raise attention to them. A technically oriented notion is that they didn’t understand these serious risks, or anyone’s ability to actually mitigate them.  In either case, 'everyone' involved seems to have suffered critical, if not fatal, cases of Irrational Optimism: a belief based on something other than rational thinking.  This kind of magical thinking includes hoping that the diagnosis was overly critical, hoping that the teams would "fix it", and hoping that history would not repeat itself. 

What else could make the apparent optimism even more irrational?: The need to service, organize, interface and/or manage:
  • ·      17,000,000 users
  • ·      55 contractors
  • ·      9 government agencies
  • ·      36 states depending on the federal system
  • ·      14 states building their own exchange
  • ·      170 insurance companies
  • ·      4,500 insurance plans


On December 1, 2013 we will discover whether this system development project team can redeem themselves of the MAJOR DEADLY SINS committed.  Can they ultimately avoid being added to the list of FEDERAL FAILED SYSTEMS PROJECTS? 

We’ll be watching, analyzing and reporting.