Observation and judgment in psychology: Assessing agreement among markingsof behavioral events

In a sequence from a videotaped dialogue between two persons (e.g., psychot herapist and patient), observers are given the task of searching for, marki ng, and commenting on conspicuous events. The observers receive no informat ion on where to look for conspicuous events and no information on what even ts are worth marking. Moreover, nothing is known about the observers' marki ng preferences. Marking is therefore very spontaneous and subjective. This marking freedom leads to the problem that a superimposition of all observer s' markings yields complicated configurations of mutually overlapping inter vals. To address this difficulty, a new definition of the concept of markin g agreement is proposed that is based upon relational considerations. Accor ding to this definition, a set of agreeing marking subsets of various sizes can be found in marking configurations. A significance concept is proposed to allow comparison of different configurations, and standardization on th e basis of random configurations is performed using Monte Carlo simulations .