We seek to understand how the use of Research, Development, and Engineering
(R,D&E) metrics can lead to more effective management of R,D&E. This paper
combines qualitative and quantitative research to understand and improve t
he use of R,D&E metrics. Our research begins with interviews of 43 represen
tative Chief Technical Officers, Chief Executive Offices, and researchers a
t 10 research-intensive international organizations. These interviews, and
an extensive review of the literature, provide qualitative insights. Formal
mathematical models attempt to explore these qualitative insights based on
more general principles.
Our research suggests that metrics-based evaluation and management vary acc
ording to the characteristics of the R,D&E activity. For applied projects,
we find that project selection can be based on market-outcome metrics when
firms use central subsidies to account for short-termism, risk aversion, an
d scope. With an efficient form of subsidies known as "tin-cupping," the bu
siness units have the incentives to choose the projects that are in the fir
m's best long-term interests. For core-technological development, longer ti
me delays and more risky programs imply that popular R,D&E effectiveness me
trics lead researchers to select programs that are not in the firm's long-t
erm interest. Our analyses suggest that firms moderate such market-outcome
metrics by placing a larger weight on metrics that attempt to measure resea
rch effort more directly. These metrics include standard measures such as p
ublications, citations, patents, citations to patents, and peer review. For
basic research, the issues shift to getting the right people and encouragi
ng a breadth of ideas. Unfortunately, metrics that identify the "best peopl
e" based on research success lead directly to "not-invented-here'' behavior
s. Such behaviors result in research empires that are larger than necessary
, but lead to fewer ideas. We suggest that firms use "research tourism" met
rics, which encourage researchers to take advantage of research spillovers
from universities, other industries, and, even, competitors.