2024.naacl-long.439
The paper investigates how large language models (LLMs) can be used to automatically generate continuous emotion‑intensity labels, a task traditionally requiring costly human annotation. The authors compare four annotation strategies—single‑instance rating scales (RS), rating‑scale tuples (RS‑T), paired comparisons (PC), and best‑worst scaling (BWS)—by prompting GPT‑3.5‑turbo (and, for comparison, GPT‑3 and Llama‑2) on the Affect‑in‑Tweets (AIT) dataset. They describe a full workflow (tuple creation, prompting, score calculation) and train a RoBERTa‑base regressor on the automatically generated labels. Direct evaluation (Pearson correlation with human scores) and indirect downstream evaluation (regressor performance on a held‑out test set) show that BWS consistently yields the most reliable automated annotations, approaching human‑trained performance, while PC is inefficient and RS/RS‑T perform worse. Scaling the number of BWS tuples (3N, 6N, 12N) modestly improves both direct and downstream scores. Model comparison reveals GPT‑3.5 outperforms GPT‑3 and Llama‑2 by large margins. Error analysis highlights systematic differences in how GPT treats explicit versus implicit emotion mentions. A secondary validation on the SemEval‑2007 Affective Text dataset shows that BWS excels when the original data were created with BWS, whereas rating‑scale‑tuple prompts can match or exceed BWS on rating‑scale‑native data. The authors conclude that BWS is the preferred automated annotation method for novel emotion‑intensity corpora, discuss limitations (task generality, reliance on closed‑source LLMs, token limits), and outline future work on other regression tasks and open‑source models.
Topics
Motivation: Reducing the Bottleneck of Manual Emotion‑Intensity Annotation
Explains why labeling continuous emotion intensity is costly, how weak supervision and LLMs have mitigated categorical annotation bottlenecks, and why continuous regression remains under‑explored.
Comparative Annotation Paradigms: Rating Scales, Paired Comparisons, and Best–Worst Scaling
Defines the three major human‑annotation strategies for continuous values, their cognitive load, and prior findings on reliability (e.g., BWS > rating scales).
Research Questions Guiding the Study
Two explicit RQs: (1) Does BWS outperform RS and PC when using LLMs? (2) Can models trained on LLM‑generated labels match those trained on human labels?
Methodology Overview: Tuple Creation, Prompt Engineering, and Score Computation
Presents the end‑to‑end pipeline (Figure 1) – generating 4‑tuples, designing role‑based prompts for each annotation type, extracting raw LLM outputs, and normalizing scores to [0,1].
Prompt Designs for RS, RS‑T, PC, and BWS
Shows the side‑by‑side prompt templates (Table 1), including role description, task description, scale specification, formatting rules, and example outputs.
Dataset and Evaluation Protocols
Details the Affect‑in‑Tweets (AIT) corpus statistics, split‑half reliability background, and the two evaluation modes: direct correlation with human scores and indirect downstream regression performance (Pearson).
Experimental Results: Direct Comparison of Annotation Methods
Reports Pearson correlations (Table 3) showing BWS achieving the highest alignment with human labels across all emotions, with RS‑T improving over single‑instance RS, and PC lagging due to cost.
Experimental Results: Downstream Regression Performance
Presents RoBERTa‑based regressor outcomes (Table 4), where models trained on BWS annotations approach but do not surpass those trained on human annotations; RS and PC lag further behind.