The Paradyn performance-measurement tool uses several novel technologi
es to scale to long-running programs and large systems and automate mu
ch of the search for performance bottlenecks. The tool is based on a d
ynamic notion of performance instrumentation and measurement. Applicat
ion programs are placed into execution, and performance instrumentatio
n is inserted into the running programs and modified during execution.
The performance consultant module automatically directs instrumentati
on placement and associates bottlenecks with specific causes and speci
fic program parts. Paradyn controls its instrumentation overhead by mo
nitoring the cost of its data collection and limiting its instrumentat
ion to a user-controllable threshold. Paradyn's instrumentation can be
configured to accept new operating systems, hardware, and application
-specific performance data. Paradyn also provides an open interface fo
r performance visualization and a simple programming library to interf
ace visualizations, Paradyn can gather and present performance data in
terms of high-level parallel languages (such as data-parallel Fortran
) and can measure programs on massively parallel computers, workstatio
n clusters, and heterogeneous combinations of these systems. The artic
le presents two examples that illustrate Paradyn's use in finding perf
ormance problems in real applications.