2021.bionlp-1.15
The paper addresses the lack of resources for detecting biomedical claims on social media by introducing a manually annotated Twitter corpus of 1,200 English tweets covering COVID‑19, measles, cystic fibrosis, and depression. Claims are defined as the central, potentially controversial conclusion of an argument and are categorized as explicit (with token‑span annotation) or implicit. The authors detail a four‑step data collection process, a multi‑iteration guideline development, and report inter‑annotator agreement (final κ: claim/non‑claim .56, explicit/implicit/non‑claim .48, span .38). Corpus analysis shows that 45 % of tweets contain a claim, with 68 % of those being explicit, and reveals patterns such as longer tweets being more likely to contain claims. Baseline classification experiments employ Naïve Bayes, logistic regression, BiLSTM, and BERT (fine‑tuned) using FastText embeddings or pretrained language models. Binary claim detection reaches an F1 of .70 (logistic regression), while multiclass detection (explicit vs. implicit) is harder, with the best explicit F1 of .59 in a pipeline setting and implicit F1 around .36. Error analysis highlights difficulties with argumentative intention and implicitness. Cross‑domain experiments using persuasive‑essay data show high in‑domain performance but a drop when models are transferred, indicating limited generalizability. The discussion emphasizes the challenging nature of biomedical Twitter claims, the moderate annotation agreement, and proposes future work on domain‑robust models, expert annotators, and deeper topic analysis.
Topics
Motivation: Biomedical Misinformation on Twitter and the Need for Claim Detection
Explains why biomedical claims on social media are valuable yet risky, and positions claim detection as a prerequisite for fact‑checking and argument mining.
Corpus Construction: Targeted Retrieval and Sampling of Biomedical Tweets
Details the Twitter API collection, query term categories (disease names, hashtags, combinations, drug names), filtering steps, and the final 1,200‑tweet sample across four health topics.
Annotation Scheme: Defining and Labeling Explicit vs. Implicit Claims with Span Annotations
Presents the claim definition, distinction between explicit and implicit claims, token‑span marking for explicit claims, and the iterative guideline refinement process.
Inter‑Annotator Agreement and Annotation Quality Metrics
Reports Cohen's κ scores across four development iterations and the final corpus, highlighting challenges especially for span annotation.
Corpus Statistics and Qualitative Insights into Claim Characteristics
Provides distribution of claim vs. non‑claim tweets, explicit/implicit ratios, length correlations, and manual analysis of argument completeness, blending, anecdotal evidence, and implicitness.
Baseline Classification Models and Experimental Design
Describes the binary and multiclass tasks, model architectures (NB, logistic regression, BiLSTM, BERT), embedding choices, training/validation splits, and oversampling of implicit claims.
Results: Performance of Claim Detection Across Tasks
Presents precision, recall, and F1 scores for each model in binary and multiclass settings, noting the best binary F1 (.70) and the difficulty of implicit claim detection (F1 ≈ .36).
Error Analysis and Cross‑Domain Transfer Experiments
Analyzes typical false positives/negatives, the role of argumentative intention, and reports cross‑domain experiments with persuasive essays, showing performance drops when transferring.