Management by consent in human-machine systems: When and why it breaks down

This study examined the effects of conflict type, time pressure, and displa y design on operators' ability to make informed decisions about proposed ma chine goals and actions in a management-by-consent context. A group of 30 B 757 pilots were asked to fly eight descent scenarios while responding to a series of air traffic control clearances. Each scenario presented pilots wi th a different conflict that arose from either incompatible goals contained in the clearance or inappropriate implementation of the clearance by autom ated flight deck systems. Pilots were often unable to detect these conflict s, especially under time pressure, and thus failed to disallow or intervene with proposed machine actions. Detection performance was particularly poor for conflicts related to clearance implementation. These conflicts were mo st likely to be missed when automated systems did more than the pilot expec ted of them. Performance and verbal protocol data indicate that the observe d difficulties can be explained by a combination of poor system feedback an d pilots' difficulties with generating expectations of future system behavi or. Our results are discussed in terms of their implications for the choice and implementation of automation management strategies in general and, mor e specifically, with respect to risks involved in envisioned forms of digit al air-ground communication in the future aviation system. Actual or potent ial applications of this research include the design of future data link sy stems and procedures, as well as the design of future automated systems in any domain that rely on operator consent as a mechanism for human-machine c oordination.