SOFTWARE OVERHEAD IN MESSAGING LAYERS - WHERE DOES THE TIME GO

Despite improvements in network interfaces and software messaging laye rs, software communication overhead still dominates the hardware routi ng cost in most systems. In this study, we identify the sources of thi s overhead by analyzing software costs of typical communication protoc ols built atop the active messages layer on the CM-5. We show that up to 50-70% of the software messaging costs are a direct consequence of the gap between specific network features such as arbitrary delivery o rder, finite buffering, and limited fault-handling, and the user commu nication requirements of in-order delivery, end-to-end flow control, a nd reliable transmission. However, virtually all of these costs can be eliminated if routing networks provide higher-level services such as in-order delivery, end-to-end flow control, and packet-level fault-tol erance. We conclude that significant cost reductions require changing the constraints on messaging layers: we propose designing networks and network interfaces which simplify or replace software for implementin g user communication requirements.