When the Data Doesn’t Tell the Story You Expected (1) (1)
The article, authored by Erica Pauly and Steven Morgeson, reports on an experiment involving roughly 30 financial advisors who were asked to estimate key performance indicators (KPIs) for their own businesses without consulting any reports. The advisors’ estimates were compared to actual performance data. The researchers expected that certain advisor characteristics—such as practice size, production level, or experience—would predict estimation accuracy. Statistical analysis, however, revealed no significant relationships between any of the tested variables and accuracy. The authors argue that a null result is itself a valuable story, challenging the common myth that only statistically significant findings are worthwhile. They discuss how confirmation bias can lead firms to assume that high‑producing or experienced advisors inherently know their numbers, and they emphasize that data should be used to test assumptions, not merely validate them. The article explores possible reasons for the lack of significant findings, including small sample size, unmeasured factors (e.g., reporting frequency, office culture, personality), and the notion that metric awareness is an individual habit rather than a demographic trait. Practical recommendations follow: advisors should regularly review their metrics, reporting should be simple and accessible, and estimation exercises can serve as coaching conversation starters. The authors also suggest future research avenues, such as examining whether frequent dashboard review improves estimation accuracy. Throughout, the piece frames the absence of clear patterns as evidence that data‑driven cultures must embrace all outcomes—positive, negative, or nuanced—to reduce uncertainty and make better decisions.
Topics
Experiment on Advisors' Metric Awareness
A study where ~30 financial advisors estimated their own business KPIs without consulting reports, and their estimates were compared to actual performance data.
Null Findings as a Storytelling Asset
The argument that a lack of statistically significant relationships is itself an informative narrative, countering the myth that only positive findings matter.
Debunking Myths About Negative Results
Discussion of common misconceptions that data analyses are only valuable when they produce significant results, and clarification of statistical testing purpose.
Intuition vs. Objective Measurement
Exploration of how advisors' gut instincts can misrepresent actual performance, highlighting the limits of intuition without data.
Confirmation Bias and Data as a Decision Tool
Explanation of how organizations often seek data that confirms pre‑existing beliefs, and the recommendation to use data to test, not just validate, assumptions.
Potential Explanations for the Null Result
Consideration of factors such as small sample size, unmeasured variables (e.g., reporting frequency, culture, personality), and the hypothesis that metric awareness is an individual habit.
Practical Recommendations for Advisors
Actionable advice: regular metric review, simplify reporting, use estimation exercises as coaching prompts, and foster consistent data engagement across all advisors.
Implications for Universal Metric Education
Argument that education around business metrics should be offered to all advisors rather than targeting specific segments, given the lack of measurable differences.