TIMING TRIALS, OR THE TRIALS OF TIMING - EXPERIMENTS WITH SCRIPTING AND USER-INTERFACE LANGUAGES

This paper describes some basic experiments to see how fast various po pular scripting and user-interface languages run on a spectrum of repr esentative tasks. We found enormous variation in performance, dependin g on many factors, some uncontrollable and even unknowable. There seem s to be little hope of predicting performance in other than a most gen eral way; if there is a single clear conclusion, it is that no benchma rk result should ever be taken at face value. A few general principles hold: Compiled code usually runs faster than interpreted code: the mo re a program has been 'compiled' before it is executed, the faster it will run. Memory-related issues and the effects of memory hierarchies are pervasive: how memory is managed, from hardware caches to garbage collection, can change runtimes dramatically. Yet users have no direct control over most aspects of memory management. The timing services p rovided by programs and operating systems are woefully inadequate. It is difficult to measure runtimes reliably and repeatably even for smal l, purely computational kernels, and it becomes significantly harder w hen a program does much I/O or graphics. Although each language shines in some situations, there are visible and sometimes surprising defici encies even in what should be mainstream applications. We encountered more than a few bugs, size limitations, maladroit features, and total mysteries. (C) 1998 John Wiley & Sons, Ltd.