Federal IT Dashboard: Visibility Into Complex Data or Eye Candy?

The "tree map" below shows the relative size (dollars) of Information Technology (IT) investments in the various Federal agencies. The Defense Department is the largest chunk and the tiny bit in the bottom right corner is the Smithsonian.  This is one of the many displays available to the public on the Federal IT Dashboard.

The idea behind the IT Dashboard is great: Require all major IT investment programs to post and update information about their state and status on a publicly available website.  The information would be available to be scrutinized by interested press, public, auditors, contractors, and those INSIDE the federal government. 

 So, how’s the Federal IT Dashboard working out? 

Two Points.

POINT ONE: the website is a well designed, organized, professional, interactive, and had appealing…. I’d even say ‘gorgeous’ ways of presenting extremely complex, voluminous data.  There are pull down lists for filtering data and for selecting the method of display (tables, graphs, even animated timelines), and there are great visualizations that you can design for yourself that shows the trends in spending, progress and other factors over the last 10+ years.

To understand how (our) ~$76,000,000,000 is being spent on almost 7,400 IT projects, you can and should visit and explore for yourself: https://itdashboard.gov.  You will find information on individual programs and projects as well.  I find that part the most interesting, especially for projects that I was involved in, or read about in the press.  That’s where “POINT TWO” comes in.

POINT TWO: Bureaucracies and the people in them tend NOT to deliver bad news up the chain.  Whether it’s wishful thinking that things will improve, or thinking that nobody is watching, or simply fear of negative consequences is immaterial.  The effect is that rather than having actual insight into a program, those who depend on such data actually have flawed insight.  Well, maybe they have great insight into programs that have no troubles, and little insight into those that NEED intervention from higher up in the chain.  

The data that feeds the IT dashboard is supposed to be reviewed and ‘blessed’ by Chief Information Officers in the reporting agencies, but there also is perhaps a tendency for the CIOs to avoid ‘help’ from up the chain.  If the IT Dashboard had the Healthcare.gov project identified as behind schedule, who cared? If it didn’t identify issues…. What happened?

One way to avoid getting ‘help’ is to modify a program’s basic elements of reporting (cost, schedule, functionality).  All or most reporting systems properly allow changes to these ‘baseline’ factors.  The issue is who keeps up with the rolling and cumulative changes over time?  Too often, a change is the baseline basically becomes a new start for reporting purposes. OR… is so confusing in the end, that it becomes almost impossible to figure out what capability the project will deliver….  at what cost….at what date.

In the project shown below, the baseline has been changed 5 times since 2009 (each Δ is a baseline change).  I found this one by searching for a project that was “ALL GREEN”.  But when I looked into the details and the "Evaluation Explanation", I found that they acknowledge having program management problems and their cost profile is RED (on the green, yellow, red scale). In this case I also notice that an adjustment of the baseline was followed by a DECREASE in score rather than what I would expect to be an INCREASE in score (a baseline lets a project reset it's goals, cost estimates, and schedule).

In another graphic, same program, it appears to me that the part of the project that is active is actually RED, but the future parts of the project that are not yet active are GREEN.  I guess the FIVE GREEN cost/schedule scores overwhelm the ONE RED cost score? Even if the RED one is the same cost as the GREEN ones combined?  Jumpin' Jimminy.

From personal experience I can tell you that there are many dedicated people feeding the many data sources that drive the IT Dashboard.  But as that data gets interpreted or adjusted or scored/colorized as it works it's way up the chain leaves lots of room for creeping, serial … . ummm…errrr…ummm.... overly optimistic reinterpretation of the data by the management chain during the data aggregation phase of reporting.

Fortunately there is a lot of semi-raw data on the website for one to explore.  Unfortunately the eye candy charts and graphs don't always reflect the data and may distract and mislead observers who do not dig deeper into the details. This effort is 5 years old, and I guess is still a crawling or walking toddler.  I hope it matures and begins running soon.

Fixed Price Contracting for a Total System: An Attempt – Part 2 of 2 How The Plan Worked Out.

PART 2

The software development and systems integrator community took interest in the agency’s procurement.  They attended the vendor day briefings, asked questions, reviewed documents in the reading room, requested copies and otherwise showed interest in bidding.  One of the questions most often asked had to do with the engineering change proposal process.  The government made it clear that they did not anticipate using a change process due to the limitation of funds on the program. And made it clear that they understood that it was indeed tradition to refine specifications during development, but would not add funds to the contract or approve changes during the development process. The government made it clear that any changes required would be made AFTER delivery of what was specified, rather than DURING the development of the deliverable.  The government acknowledged that the conventional wisdom and experience was that catching and fixing logic and other errors earlier were cheaper than fixing them AFTER completion.

The bidders bid, the proposal evaluation panel selected a winner and made the announcement.  There were no protests.  After award ceremonies and a little partying on both sides, the government and the contractor held a series of meetings at the highest levels, to discuss the projects goals, philosophy and…. No changes!  No technical spec changes, no documentation requirements changes, no changes to the outlines of the documents, no changes to anything.

The contractor’s team hunkered down in office space in Greenbelt Maryland and got to work.  The government got office space nearby to cut the time required for meetings, status updates and questions or issues that the contractor may have.  The government was under orders to NEVER authorize or even suggest that they WOULD authorize any changes to anything.

After a month or so, the contractor’s project manager began submitting requests for changes.  First it was seemingly small things like changes to the outline of several documents.  Request denied? Then they found several logic errors or inconsistencies in flows between screens and processes. They argued that the cost of fixing the problem now would be orders of magnitude cheaper now, rather than after all was coded.  The government said: ‘implement it with that error, and we’ll deal with it AFTER you delivery everything…with errors.  NO CHANGES! 

The honeymoon was officially over.

The PM made many other attempts to get the government to authorize changes.  The government wouldn’t.  The government had another round of high-level meetings with executives, lawyers, contracting officers from both sides.  The PM was replaced, with apologies from the contractor’s executives.

The contractor acquired and installed hardware and database management software and other tools into the development facility.  That kept the appearance of progress and optimism high.

At some point the contractor started missing key deliverable dates, and the quality of deliverables slipped.  There were more high-level meetings, more apologies, and another project manager.

It became clearer that whatever was happening inside the contractor’s firm was working against the government’s interests, and against the ability and will of the contractor to meet its contractual obligations. Clearly their capture strategy was an underbid and ‘getting well’ on requirement modifications. Memory tells me that they went through three or four project managers before all was said and done.

I don’t remember exactly what event triggered the government’s decision to terminate the contract, but the only remaining decision was to terminate for default or to terminate for the convenience of the government.  The government’s project office kept great records that could have supported a strong legal default process (where the government could recover monies paid). But the decision from the government’s legal shop was that they didn’t have the horsepower to engage in a legal battle with the contractor. So the legal office pushed the program to terminate the contract ‘for convenience’ (where the government would neither recover monies paid nor seek money damages). 

The deal that was eventually negotiated is that the government took ownership of the hardware and some of the commercial software products, and the contractor walked away with whatever revenue they had been paid in the form of progress/deliverables payments.

The hardware was given to another program in the agency and the program began a new attempt at a much more modest effort to automate using internal IT staff.  While the government learned much about the inner workings of its program through the years of analysis and design that it could leverage in it’s ongoing modernization effort.  This effort was a huge failure.

Years later, one of our team met one of the contractor’s deputy project managers.  He told the story of how the legal department had advised the cognizant VP that the contract terms and conditions were too tight and risky to bid on. And advised a no-bid.  The VP decided to ignore that advice.

It is hard to imagine a more diligent government effort at attempting a fixed price IT contract, and a contractors equally diligent effort at attempting to weaken the terms and conditions of the contract to which they were legally bound.  But it is standard fare for contractors to exploit or create weaknesses in contracts.  In this case they didn’t succeed.  Nor did the government get a system. 

For most people, I guess this ONE program, ONE contractor, ONE set of circumstances isn’t proof of anything about the viability of fixed price contracting for IT solutions. But I think that it was a perfect experiment that showed that the contracting community has problems of its own.  These problems and practices do not get enough exposure and discussion.  If they did, the dialog about ‘fixing’ federal IT contracting would be more analytical about the IT contractor side of the federal-private sector equation.  While governments' failures receive much public exposure, the failures of some of the best in the business receive very little exposure (or quickly fade).  

More on comparing government IT failures to private IT failures in a future post.

EPILOG:  The contractor was Martin Marietta Data Systems (MMDS), which no longer exists as far as I can tell. At the time they had a good reputation (so it wasn’t just some little inexperienced company, it was part of a very large corporation). Looking at public records now, it seems that it’s parent company was involved in defending against a hostile takeover during the time of this effort. 

It is not clear to me how independent the MMDS business unit was from the other MM entities, or how, or if they were effected by the takeover activities. But even when the government chooses a reputable firm, nothing guarantees against negative effects of internal corporate issues, incentives, and bad decisions.

Please post comments and ideas on this topic.

Fixed Price Contracting for a Total System: An Attempt – Part I of 2 - Executing The Government’s Plan.

In the aftermath of the 'Obamacare' rollout problems, there has been renewed talk of using fixed price contracting as a solution to some of the Federal IT community's contracting woes. " Why can't the government just specify what they need, then let the contractor just build it without interference"?  "The government just needs to get out of the way of the experts".  Here's what happened when we tried it.

This is a true story.  And I was a manager and technical contributor deeply involved in it from beginning to end.

A Federal agency wanted to evolve their claims processing from a totally paper and mostly manual process one to an automated standardized one. While there was some automated record keeping such as case tracking, the problems with the existing capabilities and recordkeeping were many.  There were district offices that processed claims differently, some produced with much better speed and quality than others.  At the high level, neither headquarters nor anyone else could ascertain the status of a claim without finding the physical inbox and file folder of whichever claims examiner in whichever district office was responsible for the case. And there was much unjustifiable and legally unsustainable inconsistency in claims decision making.  We were hired to analyze the problems and design and specify an automated system in collaboration with the agency.  The specification would be put out for competitive bid.

Over several years, before everything was moving at ‘internet speed’, we visited and assessed key district offices, documented and applied the best practices and methods, noted the best practices that we found and thoroughly analyzed the policies and procedures manuals, as well as the legislation, regulations and other internal policies, manuals and memoranda.  I don't remember the numbers from back then, but now the program has a nearly $3B budget, and processes millions of claims each year. It was never a trivial program.

While one of our two teams developed extremely detailed technical specifications, the other team worked with the government on creating the other documents that specified the deliverables that would be required from the winning contractor.  In addition to the request for proposal documents, we developed the specifications for testing, training, user manuals, maintenance manuals, code documentation and other technical documents.  We specified literally every document that the government wanted and required.

The governing philosophy was that we would specify ‘everything’ and the contractor would ‘only’ need to provide the hardware and write the program code following the specifications. And complete all of the required documentation following the government’s specifications. Behind this philosophy was the program’s mandate that this effort needed to be a fixed price effort. This would help protect the program’s funding for the system if the funds were obligated through a fixed price contract rather than through a cost-reimbursable type contract.  The contracting office (and others including me) were very resistant due to the lack of a history of successful fixed price software development projects.

As a condition of approving the overall fixed price approach, the contracting officer mandated that we develop ‘liquidated damages’ (LDs) essentially for every deliverable to protect the government against non-delivery,  and to send the message to the contractor community that fixed price indeed meant FIXED PRICE.  It was an attempt to dissuade bidders from the usual behavior of underbidding then using a change process to earn back revenue.  This ‘get well’ strategy was (and is) typical in government contracting.

The resulting printed notebooks of technical specifications were in boxes about four feet high and were written in a pseudo code called Program Design Language (PDL).

Program Design Language (or PDL, for short) is a method for designing and documenting methods and procedures in software. It is related to pseudocode, but unlike pseudocode, it is written in plain language without any terms that could suggest the use of any programming language or library. ¹

The other specifications for documents were in a few more boxes, and were closely in alignment with the standards being used by the Department of Defense at that time.  We considered this a ‘best practice’ of its time.

The Request for Proposal document(s) were written and reviewed over many months until all issues were resolved between the program managers, contracts shop, legal folks, and executive leadership.  A part of the contract document that received much ‘design’ and review were the Liquidated Damages section of the RFP.  We created a matrix where the rows contained EVERY deliverable and the columns contained potential compliance faults (e.g. unacceptable quality; non delivery; late delivery; and other fault scenarios).  For each cell we developed legal language and penalty formulas tied to the value of the deliverable (each of which was to be priced by the bidders) and a formula for the damage that the government would sustain if the deliverable was not received in accordance with the government’s specification and the winning bidder’s proposed schedule.

The RFP was announced and published.  All documents made available in a reading room. Copies provided upon request.  Briefing sessions were conducted with all interested parties.  All vendor community questions were answered in writing.  Adequate time was allotted for vendor responses.  Requests for an extension to the proposal due date were granted.

We tried to limit the government’s risks of cost and ‘scope creep’ while virtually eliminating the contractor’s unknowns.

¹ http://en.wikipedia.org/wiki/Program_Design_Language

Next: Part 2 of 2 - How The Plan Worked Out