PAGING TRADEOFFS IN DISTRIBUTED-SHARED-MEMORY MULTIPROCESSORS

Massively parallel processors have begun using commodity operating sys tems that support demand-paged virtual memory. To evaluate the utility of virtual memory, we measured the behavior of seven shared-memory pa rallel application programs on a simulated distributed-shared-memory m achine. Our results (1) confirm the importance of gang CPU scheduling, (2) show that a page-faulting processor should spin rather than invok e a parallel context switch, (3) show that our parallel programs frequ ently touch most of their data, and (4) indicate that memory, not just CPUs, must be ''gang scheduled.'' Overall, our experiments demonstrat e that demand paging has limited value on current parallel machines be cause of the applications' synchronization and memory reference patter ns and the machines' high page-fault and parallel context-switch overh eads.