Surprisingly popular voting

Recovering the ground truth from crowds of experts and non-experts. NeurIPS 2024, WWW 2025 and KDD 2026.

FAIR Lab, Penn State, Jan 2023 to Dec 2024

Advised by Dr. Hadi Hosseini; co-advised by Dr. Debmalya Mandal.

When most of a crowd is misinformed, aggregating their votes amplifies the error. The surprisingly popular (SP) algorithm asks each person for their own answer and for a prediction of how others will answer, then picks the answer that is more popular than predicted. That lets it recover the truth even when the experts are a minority. My MS work extended SP from single questions to rankings over many alternatives.

  • Scaling SP to partial preferences (NeurIPS 2024) (Hosseini et al., 2024). Designed Partial-SP and Aggregated-SP, which ask each voter for a vote and a prediction over only a small subset of the alternatives, as a top choice, an approval set or a partial ranking. In a crowdsourcing study with 432 participants on Amazon Mechanical Turk, across geography, movies and paintings, both outperformed classical rules such as Borda, Copeland, Maximin and Schulze at recovering the true ranking. Voter behavior was modeled with a concentric mixture of Mallows models, fit by Bayesian inference (No-U-Turn sampling), and the sample complexity was shown to grow by a factor of k! rather than m! for subsets of size k out of m alternatives.
  • When SP voting works (WWW 2025) (Hosseini et al., 2025). Proposed concentric mixtures of Mallows and Plackett-Luce models with any number of voter groups, identified expert, intermediate and non-expert groups in real data, and derived the conditions and number of samples under which SP voting recovers the true ranking with high probability.
  • SP-Rank dataset (KDD 2026) (Hosseini et al., 2026). Released the first large public dataset that pairs ranked votes with predictions of others’ votes: 12,384 datapoints from 1,152 participants in nine elicitation formats, with benchmarks for full and subset rank recovery and for modeling voter behavior. Dataset and code on GitHub.

This work formed my MS thesis (Puhan, 2024), which won the Best MS Thesis Award in Artificial Intelligence at the Penn State AI Awards. Implementations of the classical voting rules are on GitHub.

References

2026

  1. KDD 2026
    SP-Rank: A Dataset for Ranked Preferences with Secondary Information
    Hadi Hosseini, Debmalya Mandal, and Amrit Puhan*
    In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), Jeju Island, Republic of Korea, Datasets and Benchmarks Track, Poster. Dataset: SP-Rank on GitHub , Aug 2026

2025

  1. WWW 2025
    Surprisingly Popular Voting with Concentric Rank-Order Models
    Hadi Hosseini, Debmalya Mandal, and Amrit Puhan*
    In Proceedings of the ACM Web Conference 2025 (WWW ’25), Sydney, Australia, Main Track, Poster, Apr 2025

2024

  1. NeurIPS 2024
    The Surprising Effectiveness of SP Voting with Partial Preferences
    Hadi Hosseini, Debmalya Mandal, and Amrit Puhan*
    In Advances in Neural Information Processing Systems (NeurIPS), Main Track, Poster, Dec 2024
  2. MS Thesis 2024
    Recovering Ground Truth Rankings When the Majority Is Misinformed
    Amrit Puhan
    The Pennsylvania State University. Best MS Thesis Award in Artificial Intelligence, Penn State AI Awards , Aug 2024