Latent Class Models for the Analysis of Rater or Test Agreement


Latent class models for agreement data have become increasingly popular. The premise of this approach is that cases in a population belong to two or more latent classes--"latent class" simply means that the class membership of a given case is not directly observed. A latent class may be thought of as a case subtype or "genotype."

A case's probability of being assigned a given rating level is assumed to depend on the case's latent class. For example, a medical population may consist of two latent classes: disease-negative (normal) cases and disease-positive cases. The probability that case is diagnosed "positive" by a rater or procedure depends on which latent class the case belongs to.

Analysis aims to estimate (1) the proportion of cases in each latent class, and (2) for each latent class, the probability that a member of that latent class will elicit each rating level from a given rater. When the latent class model fits, these parameters can be used to assess rater agreement, provide information about rating accuracy, or or be put to other practical uses.

Advantages of the latent class approach are as follows:

  1. The model is relatively simple, reasonably plausible, and statistically testable.

  2. It allows one to analyze agreement among more than two raters or procedures simultaneously.

  3. It may permit inferences about the accuracy of ratings or diagnoses (e.g., their Sensitivity and Specificity) when a "gold standard" is unavailable (Walter & Irwig, 1988; Uebersax, 1988; Hui & Zhou, 1998).

  4. Latent class models can be estimated with easily available software (Clogg, 1977; van de Pol, Langeheine & de Jong, 1989; Vermunt, 1998; Vermunt & Magdison, 2000).

  5. One can use results to construct Bayesian decision rules for the optimal diagnosis or classification of cases using ratings by two or more raters or procedures (Uebersax & Grove, 1990).

Limitations of the approach are as follows:

  1. With dichotomous data, the method generally requires that there be at least three raters or procedures.

  2. The usual latent class model assumes that, within each latent class, the rating level assigned by one rater is statistically independent of the rating levels assigned by other raters. This assumption--"conditional independence" or "local independence" (independence of ratings, conditional on latent class)--can be unrealistic. For example, in medical applications, some disease-positive cases may have a stronger or more salient disease level; such cases are more likely to elicit a positive diagnosis by all raters, violating the conditional independence assumption. Simple extensions of the basic latent class model, however, relax this limiting assumption (see A Practical Guide to Local Dependence in Latent Class Models.

    Other methods for avoiding this limitation are discussed by Qu, Tan and Kutner (1996) and Uebersax and Grove (1993). These fall generally into the category of random effects latent class models for the analysis of rater or test agreement. This area has seen increased interest in recent years. For examples of recent articles, see the section of the Bibliography below.

  3. Special provisions are required for ordered-category ratings (Uebersax, 1993; Uebersax & Grove, 1993; Vermunt, 1998; Uebersax, 2000).

For a great deal more information on Latent Class Modeling, including FAQs, Software information, Links, and detailed References, visit the Latent Structure Analysis pages.

(Top of page)
(Back to Agreement Statistics)

Bibliography: Latent Class Models for Agreement

(Top of page)
(Back to Agreement Statistics)

Where to Start

    Clogg CC. Latent class models. In: Arminger G, Clogg CC, Sobel ME (Eds.), Handbook of statistical modeling for social and behavioral sciences (Ch. 6; pp. 311-359). New York: Plenum, 1995.

    Uebersax JS, Grove WM. Latent class analysis of diagnostic agreement. Statistics in Medicine, 1990, 9, 559-572.

(Top of Bibliography)

Latent Class Analysis in General

    Clogg CC. Latent class models. In: Arminger G, Clogg CC, Sobel ME (Eds.), Handbook of statistical modeling for social and behavioral sciences (Ch. 6; pp. 311-359). New York: Plenum, 1995.

    Goodman LA. Exploratory latent structure analysis using both identifiable and unidentifiable models. Biometrika, 1974, 61, 215-231.

    Heinen T. Latent class and discrete latent trait models: Similarities and differences. Thousand Oaks, California: Sage, 1996.

    Lazarsfeld PF, Henry NW. Latent structure analysis. Boston: Houghton Mifflin, 1968.

    McCutcheon AC. Latent class analysis. Beverly Hills: Sage Publications, 1987.

(Top of Bibliography)

Latent Class Models for Agreement Data

(Top of Bibliography)

Software for Latent Class Analysis

    Clogg CC. Unrestricted and restricted maximum likelihood latent structure analysis: a manual for users. Working paper 1977-09, Pennsylvania State University, Population Issues Research Center.

    van de Pol F, Langeheine R, de Jong W. PANMARK user manual. Netherlands Central Bureau of Statistics, Voorburg, The Netherlands, 1989.

    Vermunt J. LEM users manual. Department of Methodology, Tilburg University, Tilburg, The Netherlands, 1998.

    Vermunt JK, Magidson J. Latent GOLD User's Guide. Belmont, Mass.: Statistical Innovations, Inc., 2000.

(Top of Bibliography)

(Top of Page)
Go to Agreement Statistics
Go to Latent Structure Analysis

This page maintained by

John Uebersax PhD

rev: 27 Sep 2000
rev: 14 Oct 2009 (updated bibliography)

Copyright © 2000-2009 John Uebersax. All rights reserved.