Evaluating computerized health information systems: hardware, software andhuman ware: experiences from the Northern Province, South Africa

Despite enormous investment world-wide in computerized health information s ystems their overall benefits and costs have rarely been fully assessed. A major new initiative in South Africa provides the opportunity to evaluate t he introduction of information technology from a global perspective and ass ess its impact on public health. The Northern Province is implementing a comprehensive integrated hospital i nformation system (HIS) in all of its 42 hospitals. These include two menta l hearth institutions, eight regional hospitals (two acting as a tertiary c omplex with teaching responsibilities) and 32 district hospitals. The overa ll goal of the HIS is to improve the efficiency and effectiveness of health land welfare) services through the creation and use of information, for cl inical, administrative and monitoring purposes. This multi-site implementat ion is being undertaken as a single project at a cost of R130 million (whic h represents 2.5 per cent of the health and welfare budget on an annual bas is). The implementation process commenced on 1 September 1998 with the intr oduction of the system into Mankweng Hospital as the pilot site and is to b e completed in the year 2001. An evaluation programme has been designed to maximize the likelihood of suc cess of the implementation phase (formative evaluation) as well as providin g an overall assessment of its benefits and costs (summative evaluation). T he evaluation was designed as a form of health technology assessment; the s ystem will have to prove its worth tin terms of cost-effectiveness) relativ e to other interventions. This is more extensive than the traditional form of technical assessment of hardware and software functionality, and moves i nto assessing the day-to-day utility of the system, the clinical and manage rial environment in which it is situated (humanware), and ultimately its ef fects on the quality of patient care and public health. In keeping with new South African legislation the evaluation process sought to involve as many stakeholders as possible at the same time as creating a methodologically r igorous study that lived within realistic resource limits. The design chosen for the summative assessment was a randomized controlled trial (RCT) in which 24 district hospitals will receive the HIS either earl y or late. This is the first attempt to carry out an RCT evaluation of a mu lti-site implementation of an HIS in the world. Within this design the eval uation will utilize a range of qualitative and quantitative techniques over varying time scales, each addressing specific ai ms of the evaluation prog ramme. In addition, it will attempt to provide an overview of the general i mpact on people and organizations of introducing high-technology solutions into a relatively unprepared environment. The study should help to stimulat e an evaluation culture in the health and welfare services in the Northern Province as well as building the capacity to undertake such evaluations in the future.