2024-embedding-democratic-values-cscw
The paper introduces a framework called societal objective functions that translates rigorously vetted social‑science constructs into algorithmic objectives for social‑media feed ranking. Focusing on anti‑democratic attitudes—operationalized as an eight‑variable construct (partisan animosity, support for undemocratic practices, support for partisan violence, support for undemocratic candidates, opposition to bipartisanship, social distrust, social distance, and biased evaluation of politicized facts)—the authors develop a detailed manual coding scheme and annotate 405 real Facebook posts, achieving high inter‑coder reliability (Krippendorff’s α = .895 overall). They then convert the codebook into zero‑shot prompts for GPT‑4, creating a democratic attitude model that reproduces human scores with Spearman ρ ≈ .75 and variable‑level Krippendorff’s α ranging from .506 to .806. Using these scores, three feed‑ranking interventions are built: (1) downranking anti‑democratic content, (2) adding content‑warning overlays, and (3) removing anti‑democratic posts and replacing them with pro‑democratic content. Three preregistered online experiments evaluate the interventions. Study 1 (N = 1,380) with manual labels shows downranking and remove‑and‑replace reduce partisan animosity (effect sizes d ≈ .20–.25) without harming engagement or satisfaction; content warnings increase perceived threats to free speech. Study 2 validates the GPT‑4 model, confirming strong alignment with human annotations. Study 3 (N = 558) replicates Study 1 using the automated model, again finding reduced partisan animosity (d ≈ .25) with no engagement loss. The appendix provides extensive inter‑rater reliability tables for each variable, detailed prompt templates used for GPT‑4 (and GPT‑3.5) rating, and performance metrics (including classification accuracy, F1 scores, and mean absolute error) for both models. The discussion situates the contribution as a top‑down, value‑centered intervention that can be extended to other societal values (e.g., well‑being, equity, sustainability), while acknowledging limitations such as U.S. partisan sample bias, short‑term measurement, potential LLM bias, and freedom‑of‑speech concerns. The extensive bibliography situates the work within literature on algorithmic moderation, political polarization, LLM evaluation, and value‑aligned AI.
Topics
Societal Objective Functions & Top‑Down Value‑Centered Intervention
A methodological framework that translates rigorously vetted social‑science constructs (e.g., anti‑democratic attitudes) into algorithmic objective functions for social‑media feed ranking, positioning value‑aligned AI as a top‑down alternative to engagement‑only metrics.
Democratic Attitude Model: Manual Coding and LLM Scaling
Development of an eight‑variable codebook for anti‑democratic attitudes, manual annotation of 405 Facebook posts (overall Krippendorff’s α = .895), and zero‑shot prompting of GPT‑4 that reproduces human scores (Spearman ρ ≈ .75; variable‑level α ≈ .506–.806). Includes model performance metrics such as accuracy and F1.
Feed‑Ranking Interventions
Three algorithmic interventions built on democratic attitude scores: (1) downranking anti‑democratic content, (2) overlaying content‑warning labels, and (3) removing anti‑democratic posts and replacing them with pro‑democratic material. Compared against engagement‑based, ideologically balanced, chronological, and null feeds.
Empirical Evaluation of Interventions (Studies 1 & 3)
Preregistered online experiments testing the interventions. Study 1 (N = 1,380) with manual labels showed downranking and remove‑and‑replace reduced partisan animosity (d ≈ .20–.25) without harming engagement or satisfaction; content warnings raised perceived free‑speech threats. Study 3 (N = 558) replicated these findings using the GPT‑4‑based model, confirming similar effect sizes and no engagement loss.
Model Validation with GPT‑4 (Study 2)
Evaluation of the zero‑shot GPT‑4 prompts on a held‑out set (n = 200). Reported overall correlation ρ ≈ .75, Krippendorff’s α ≈ .78, per‑variable agreement .506–.806, and classification accuracy/F1 scores, demonstrating strong alignment with human annotations.
Supplementary Materials (Appendices)
Includes detailed inter‑rater reliability tables for each anti‑democratic variable, exact zero‑shot prompt templates for GPT‑4 and GPT‑3.5, and comprehensive performance‑metric tables (accuracy, F1, MAE, Spearman) for both models.
Limitations, Ethical Risks, and Societal Impact
Acknowledges constraints such as U.S. partisan sample bias, short‑term measurement horizon, potential LLM bias, freedom‑of‑speech concerns, and the need for stakeholder governance to mitigate ethical risks.