Putting Testing and Evaluation Back into R&D

23 April 2009 (Last Updated April 23rd, 2009 18:30)

Funding research into the right technologies suitable for today's warfare is no easy task. Richard B Gasparre reports on how the Pentagon needs to pay attention to its processes.

Putting Testing and Evaluation Back into R&D

In announcing his defence budget recommendations for the 2010 financial year, US Secretary of Defense Robert Gates stated that the Pentagon 'must reform how and what we buy', and called for a 'fundamental overhaul of our approach to procurement, acquisition and contracting'.

The Department of Defense (DoD) has never been a paragon of development efficiency but the procurement problem has become intolerable during this decade, as the Government Accountability Office (GAO) demonstrated in a report released last year. It showed in FY2000 acquisition programme schedules were slipping by an average of 16 months – by 2008, the average delay had increased to 21 months.

In addition, in FY2000, the aggregate acquisition costs of big-ticket programmes were running 6% over their initial estimate. Seven years later, cost growth had mushroomed to 26%. For the report, GAO measured 72 weapons programmes against its efficiency standards yet couldn't find even one programme that qualified as a managerial success.

Track records this abysmal almost always have several causes, and tracking their roots is not a trivial task in the vast jungle of the US military-industrial complex. Still, every journey must begin somewhere, and to the extent that weapons development in the US pushes the scientific and technological envelope, a natural starting point is all the way upstream – with the R&D process.

A rose by any other name (in the Pentagon) might not smell as sweet

"In FY2000, acquisition programme schedules were slipping by an average of 16 months."

DoD breaks out its budget into seven categories, the first three of which fall in the 'science and technology' (S&T) sub-account. Basic research (called 6.1) and applied research (6.2) fund 'pure knowledge' programmes that potentially apply to many military (and civilian) uses. Together, 6.1 and 6.2 represent the 'science' half of S&T, although they are collectively called 'total research'. Based on its name, advanced technology development (6.3) would seem to belong in the development bucket, but in DoD's scheme, even applied technology buildouts are research until they are paired with a specific weapons system.

Leaving aside management support (6.6), which is reserved for funding of the R&D infrastructure itself, the remaining categories cover the development of specific weapons systems. Through FY2005, these categories had the following definitions, according to a RAND Corporation monograph:

  • Demonstration and validation (6.4) included 'all efforts necessary to evaluate integrated technologies in as realistic an operating environment as possible to assess [their] performance or cost reduction potential'.
  • Engineering and manufacturing development (6.5) applied to 'projects…for service use [that had] not received approval for full rate production'.
  • Operational systems development (6.7) covered projects for acquisitions 'still in engineering and manufacturing development, but which have received Defense Acquisition Board (DAB) or other approval for production, or for which production funds have been [budgeted]'.

In other words, the distinction between 6.5 and 6.7 was more bureaucratic than substantive. In contrast, 6.3's emphasis on 'field experiments and tests' gave it much in common with the 6.4 demo phase, which specifically emphasised realistic testing of integrated technologies with an eye on cost as much as performance – before the technologies moved to 6.5 and 6.7 to become weapons projects, with their own constituencies and momentum.

In 2006, however, DoD altered the middle of its R&D process: 6.4 became advanced component development and prototyping, and 6.5 morphed into system development and demonstration. Of course, bureaucracies often put old wine in new bottles, but given then-Secretary Rumsfeld's emphasis on 'transforming' the way the Pentagon did business, as well as the actual evolution of programmes that were initiated or restarted this decade, such as the Arapaho helicopter, the new phase titles seemed to reflect actual procedural changes.

"The new process seemed to reflect the Pentagon's longstanding spending proclivities."

At face value, therefore, the new R&D categories implied three hazardous changes. Firstly, the new paradigm was 'build the parts' (6.4), then 'fit them into the whole' (6.5). At the cutting edge of advanced innovation, however, the parts-to-whole relationship more closely resembles a loop rather than a straight line.

Component / system iteration becomes especially necessary when the designers are pushing the technological envelope on each and every component, leaving no slack for them to fit together.

In pursuing this holy grail on every project it could, Pentagon and industry managers forgot the shipper's version of Murphy's Law: a truck rated at xft³ of cargo space never holds xft² of cargo in practice.

Equally important, demonstration slipped toward the back of the process. When demonstration occurs concurrently with (or more likely after) system integration, developers end up spending more, rather than less, on designs that subsequently must be modified at greater time and cost when, inevitably, something doesn't work.

Lastly (and most disturbingly) validation simply disappeared. Whereas demonstration shows how a technology works, validation determines if the technology works as intended. In other words, if it does what it is supposed to do under reasonable conditions and in the context of the programme's goals.

That something works in principle doesn't mean it will work in practice. But the new process assumed that developers could make any desired technology fit into any platform and implied that defence technologists had become 'too cool for school'.

DoD's field of dreams

These changes may have been more semantic than real, but the new process seemed to reflect the Pentagon's longstanding spending proclivities, and some statistics from the American Association for the Advancement of Science (AAAS) reflect this.

"R&D spending by service need not be symmetrical to threat."

In FY2004 and FY2005 – the last two years in which 6.4 was demonstration and validation – the Pentagon actually underspent their D&V budget by an average of $520m a year.

In FY2006 and FY2007, the first two years of advanced component development, actual 6.4 spending exceeded the Pentagon's budget in this area.

More generally, as annual AAAS reports suggest, the Pentagon routinely tries to shift spending downstream. 'In what has now become an annual ritual', according to the 2009 edition, 'the Pentagon proposes sharp cuts' in the S&T budget and 'Congress adds [back] billions of dollars in the appropriations process'.

Although gamesmanship and politics on both sides doubtless explain some of the ritual, the Pentagon is as reluctant as any other institution to give up funding, simply on bureaucratic principle. In particular, this thriftiness is absent in the development phases, where the Pentagon routinely asks for more, leaving Congress to trim requests.

Other aspects of R&D spending imply a preset agenda. For example, from FY2002 through FY2009, army R&D spending grew by 71% (8% compound annual growth). Navy spending also grew by 74% but US Air Force spending grew by 86% cumulatively – at a time when the US was fighting hot wars against adversaries in Iraq and Afghanistan who essentially had no navy and no air force.

Apparently, R&D spending followed preconceived notions of what the services and contractors wanted to build, rather than what the field operators needed. To some extent this inertia reflected the technocentric, airpower-heavy 'shock-and-awe' paradigm that Rumsfeld's regime promoted. The Pentagon, therefore, might have needed to undergo the post-invasion Iraq grind in order to see the light. By this reasoning, however, DoD still seeks enlightenment: from 2007 to 2009, Army R&D
spending increased by 6.3% but the air force spending increased by 10.1%.

"Asymmetric warfare is often argued to be more efficient."

Granted, there is a philosophical argument that researchers should not modify R&D portfolios to reflect every change in the current environment. After all, the limits of science at any particular time bear no necessary relationship to current threats and the mere desire for something doesn't make it feasible.

The Pentagon also had more on its plate than Iraq and Afghanistan – if one assumed the worst about Iranian, North Korean, and Chinese ballistic missile progress, then the Missile Defense Agency arguably deserved more than a 23% increase in R&D funding over the past eight years. Finally, R&D spending by service need not be symmetrical to threat if, as strategists like to argue these days, asymmetric warfare is more efficient.

In any event, one thing is clear: sometime this decade, DoD forgot that its umbrella term for 6.1-6.7 spending is total RDT&E: research development, testing, and evaluation. As Murphy might have put it, the more your process assumes things will go right, the more likely they are to go badly wrong.