gordon jury learning chi22
The paper introduces jury learning, a supervised machine learning framework that explicitly models each annotator in a dataset and lets practitioners compose a "jury"—a weighted set of annotators defined by demographic or other characteristics—to determine a classifier’s prediction. The authors present a deep learning architecture that jointly trains a content encoder (BERTweet) with a Deep & Cross Network (DCN) to predict individual annotator labels on unseen inputs, enabling sampling of jurors and aggregation via a median‑of‑means estimator. An interactive interface lets users explore jury compositions, view dissent visualizations, and generate counterfactual juries. Technical evaluation on a large toxicity‑detection dataset shows the model predicts individual annotator labels more accurately (MAE 0.61 vs 0.90 baseline) and improves jury‑level predictions (MAE 0.27 vs 0.41). A user study with content moderators demonstrates that participants construct more diverse juries (2.9× non‑White, 31.5× non‑binary representation) and that these juries flip 14% of classification outcomes, especially on highly divisive comments. The paper discusses design goals, related work, extensions (conditional and counterfactual juries), ethical considerations (procedural justice, privacy, potential misuse), limitations, and future directions.
Topics
Jury Learning Metaphor: From Majority Vote to Explicit Juror Selection
Reframes supervised learning as a jury process where practitioners explicitly choose which annotators—or demographic groups—determine model predictions.