← Back to Frameworks

Metaculus Forecasting Platform

URL: https://www.metaculus.com/ Type: Online forecasting platform / prediction aggregation engine Active since: 2015 (founded); 2017 (public launch)

Mission

Metaculus is "an online forecasting platform and aggregation engine working to improve human reasoning and coordination on topics of global importance." Unlike prediction markets (which use monetary incentives), Metaculus is reputation-based: users submit probability estimates on structured questions and build forecasting track records over time, with the platform aggregating individual forecasts into increasingly accurate collective predictions.

Metaculus was co-founded by Anthony Aguirre (UC Santa Cruz physicist; also co-founder of the Future of Life Institute) and Greg Laughlin. The platform is headquartered in Santa Cruz, California.

Metaculus occupies a unique position in the crisis intelligence landscape: it is not an expert body producing risk assessments (like WEF or CSER), nor a database of historical events (like EM-DAT). It is a structured mechanism for aggregating and tracking probabilistic beliefs about future events, including catastrophic and existential risks.

What They Track

Metaculus hosts questions across virtually all domains relevant to crisis intelligence:

Question Categories

  • •AI and Technology: Model capabilities, AGI timelines, compute trends, alignment progress, regulation timelines, specific benchmark achievements
  • •Geopolitics: Conflict probability, escalation scenarios, election outcomes, diplomatic events, sanctions and trade
  • •Climate and Environment: Temperature records, emissions targets, policy implementation, extreme weather events, tipping point triggers
  • •Health and Biosecurity: Disease outbreaks, vaccine development timelines, pandemic preparedness, mortality trends
  • •Nuclear Risk: Weapon use probability, arms control developments, proliferation milestones
  • •Economics: Recession probability, unemployment, growth forecasts, inflation, financial crisis
  • •Science and Research: Research breakthroughs, Nobel predictions, replication outcomes
  • •Space and Existential Risk: Human extinction probability, civilizational collapse, space colonization timelines

Existential Risk Questions (Notable)

  • •"Human Extinction by 2100?" (long-running question with hundreds of forecasters)
  • •"Existential risk from AGI vs AGI timelines" (tracking relationship between AI capability and risk)
  • •AI benchmark forecasting tournaments
  • •Nuclear weapon use probability over various time horizons

Methodology

Forecasting Mechanics

Question types: Three prediction formats: 1. Binary questions: Resolved as yes/no. Forecasters submit probability estimates (0-100%). 2. Numerical-range questions: Forecasters submit probability distributions over continuous ranges. 3. Date-range questions: Forecasters estimate when an event will occur.

Resolution criteria: Each question has pre-defined resolution criteria citing authoritative sources (UN agencies for geopolitical questions, peer-reviewed journals for scientific questions, central banks for economic questions). Resolution criteria are established at question creation and enforced by staff. In ambiguous cases, questions may be marked unresolved; administrators handle adjudication, occasionally consulting community input, but final decisions rest with staff.

Aggregation

Community Prediction: Based on the median of user forecasts weighted by recency. Gives equal weight to all forecasters but emphasizes more recent predictions.

Metaculus Prediction: A more sophisticated aggregation that weights forecasts based on past performance and applies extremization (adjusting the aggregate toward more extreme probabilities) to compensate for systematic human cognitive biases (e.g., the tendency to hedge toward 50%). The Metaculus Prediction, on average, outperforms the median community prediction.

Scoring

Metaculus uses a time-weighted logarithmic scoring rule (log score), a proper scoring rule that rewards probabilistic forecasts aligning with true beliefs. Properties:

  • •Proper: You maximize your expected score by reporting your genuine belief, not by gaming the system.
  • •Time-weighted: Predictions made earlier receive more credit, rewarding foresight over last-minute updates.
  • •Track record accumulation: Over many predictions, participants build calibration curves showing how accurate their probability estimates are.

Metaculus also reports Brier scores for overall platform performance assessment. The Brier score ranges from 0 (perfect) to 1 (maximally wrong), with 0.25 representing random guessing on binary questions. A score of 0.111 represents consistently accurate probability estimation across thousands of resolved questions.

Calibration

A well-calibrated forecaster's predictions match reality: events they predict at 30% should happen about 30% of the time. Metaculus publishes calibration curves for both the Community Prediction and the Metaculus Prediction, demonstrating that both are well-calibrated across the probability spectrum, with the Metaculus Prediction showing slightly better calibration at extreme probabilities (near 0% or 100%).

Tournaments

Organizations can sponsor forecasting competitions on specific topics, often with cash prizes. Past tournaments have covered AI progress, biosecurity, climate change, and geopolitical events. Tournaments attract concentrated expertise and produce particularly high-quality forecasts due to competitive incentive structures.

Data Availability

Metaculus offers unusually strong data accessibility for a forecasting platform:

  • •Public question feed: All questions, predictions, and resolution statuses viewable at metaculus.com/questions/
  • •API: REST API at metaculus.com/api/ with JSON endpoints for questions, predictions, and user data. Supports filtering by status (open/closed/resolved), forecast type (binary/numeric/date), project/tournament, and other parameters.
  • •API wrapper: Python library for programmatic interaction with questions and tournaments.
  • •Track record page: Aggregate platform calibration data at metaculus.com/questions/track-record/
  • •Developer tools: GitHub repositories include forecasting-tools framework and bot templates for automated forecasting.
  • •Historical resolution data: Resolved questions with their community predictions and actual outcomes are maintained indefinitely.
  • •No paywall: All question data and predictions are freely accessible.

Strengths

  • •Highest measured forecasting accuracy: Metaculus reports a Brier score of 0.111, which it claims surpasses all other prediction and forecasting services. For context, a Brier score of 0 is perfect and 0.25 represents random guessing on binary questions. An internal 2023 review of 150+ resolved AI forecasts showed no clear systematic biases and proper calibration.
  • •Proper scoring rule: The log scoring rule ensures forecasters are incentivized to report genuine beliefs rather than hedge or grandstand. This is a fundamental methodological advantage over unstructured expert elicitation.
  • •Continuous updating: Forecasts update in real-time as conditions change, providing a dynamic risk signal rather than a static annual assessment. When a crisis escalates, Metaculus forecasts move within hours.
  • •Transparent track record: Anyone can verify how accurate Metaculus predictions have been by examining resolved questions. This accountability is absent from most expert assessment bodies.
  • •Coverage breadth: Thousands of active questions spanning virtually all domains relevant to crisis intelligence, from "will there be a nuclear detonation in 2026?" to "will global mean temperature exceed 1.5C above pre-industrial by 2030?"
  • •API access: Programmatic data access enables automated ingestion into other systems -- a critical advantage for Ark Instinct's data pipeline.
  • •Community scale: With thousands of active forecasters, predictions aggregate diverse perspectives. The weighting algorithm gives more influence to historically calibrated forecasters, creating a meritocracy of accuracy.

Blind Spots

  • •Participation bias: Metaculus forecasters are disproportionately male, Western, technically educated, and connected to the Effective Altruism / rationalist communities. This demographic skew may introduce systematic biases, particularly for questions involving non-Western politics, social dynamics, or cultural factors.
  • •Question framing effects: How a question is worded significantly affects forecasts. A question asking "will nuclear weapons be used by 2030?" may receive different estimates than "will the nuclear taboo hold through 2030?" even though they are logically equivalent. Question framing is determined by question authors, not by systematic methodology.
  • •Resolution ambiguity: Despite pre-defined criteria, some questions face resolution disputes. Edge cases require staff judgment, introducing subjectivity. The platform has acknowledged this as an ongoing challenge.
  • •Short-horizon bias: Most Metaculus questions resolve within 1-5 years. Questions about events decades in the future attract fewer forecasters and have less calibration data, making long-term predictions less reliable. Existential risk questions (e.g., "human extinction by 2100") have so few comparable resolved questions that calibration is speculative.
  • •No causal modeling: Metaculus aggregates beliefs about *whether* events will happen, not *why* or *how*. It does not model causal mechanisms, feedback loops, or cascading risks. A forecast of "30% chance of nuclear war by 2040" tells you nothing about the pathways to that outcome.
  • •Anchoring and herding: Forecasters can see the current community prediction before submitting their own estimate, which may anchor their judgment. While the scoring rule penalizes blind copying, social influence effects are difficult to eliminate entirely.
  • •Missing global crises: Metaculus questions are created by users and staff, not by systematic risk screening. Some significant risks may simply lack questions, and some regions of the world (particularly the Global South) are underrepresented in the question set.
  • •Accuracy claims need context: The 0.111 Brier score is calculated across all resolved questions, including many that are relatively easy to forecast (e.g., "will the US hold a presidential election in 2024?"). Performance on genuinely uncertain, high-stakes questions may be weaker than the aggregate suggests.
  • •Platform governance risk: Metaculus is a private company, not a public institution. Its continued operation depends on its business model and funding. If Metaculus were to shut down, modify its scoring system, or restrict API access, downstream users would be affected. There is no guaranteed institutional permanence.
  • •AI forecaster contamination: As AI systems increasingly participate in Metaculus forecasting (via bot tournaments and automated submissions), the line between "human crowd wisdom" and "AI model outputs" blurs. The value proposition of crowd forecasting partly depends on diverse human judgment; if AI models dominate, the platform becomes an AI ensemble rather than a human wisdom aggregator.

Relevance to Ark Instinct

What we can use:

  • •Metaculus is the single most valuable real-time data source for Ark Instinct's crisis intelligence. Its API-accessible probability estimates on thousands of relevant questions provide a dynamic, continuously-updated signal layer that no annual report or static index can match.
  • •Specific integration opportunities:
  • •Ingest probability estimates for existential risk questions (nuclear war, pandemic, AI catastrophe) as one consensus layer in our multi-consensus framework.
  • •Track how forecasts shift over time as a leading indicator of changing risk conditions.
  • •Use Metaculus resolution data to calibrate our own risk models: when Metaculus says 30%, does the event happen 30% of the time?
  • •Leverage tournament results for domain-specific risk assessment (AI progress, biosecurity milestones).
  • •The platform's track record data provides calibration benchmarks that Ark Instinct can use to weight different information sources.

What we should avoid:

  • •Do not treat Metaculus forecasts as objective measurements of risk probability. They are aggregated human beliefs, subject to the biases noted above. A Metaculus forecast of "5% chance of nuclear war by 2030" reflects forecaster beliefs, not physics.
  • •Do not use Metaculus for domains where its user base lacks expertise. Its forecasting advantage is strongest for technology, science, and US/European geopolitics. For risks in sub-Saharan Africa, Southeast Asian politics, or indigenous community resilience, other sources are needed.
  • •Do not weight long-horizon Metaculus forecasts (2050+, 2100+) as heavily as short-horizon ones. The calibration data for such long timeframes is inherently limited.

Methodology to adopt:

  • •The proper scoring rule approach should be incorporated if Ark Instinct ever hosts its own expert elicitation or forecasting exercises. Proper scoring rules are the gold standard for incentive-compatible probability elicitation.
  • •Metaculus's aggregation methodology (recency weighting, performance weighting, extremization) is well-designed and could inform how Ark Instinct combines multiple information sources.
  • •The practice of making all predictions, track records, and resolution criteria public and verifiable should be a standard for Ark Instinct's own outputs.

Sources

  • •Metaculus. "FAQ." https://www.metaculus.com/faq/ (accessed 2026-04-03)
  • •Metaculus. "Track Record." https://www.metaculus.com/questions/track-record/ (accessed 2026-04-03)
  • •Metaculus. "API Data Access." https://www.metaculus.com/api/ (accessed 2026-04-03)
  • •Metaculus. "A Primer on the Metaculus Scoring Rule." Medium. https://metaculus.medium.com/a-primer-on-the-metaculus-scoring-rule-eb9a974cd204 (accessed 2026-04-03)
  • •Prediction Markets Reviews. "Metaculus Review 2026." https://predictionmarketsreviews.com/reviews/metaculus (accessed 2026-04-03)
  • •Wikipedia. "Metaculus." https://en.wikipedia.org/wiki/Metaculus (accessed 2026-04-03)
  • •EA Forum. "Exploring Metaculus's AI Track Record." https://forum.effectivealtruism.org/posts/e9htD7txe8RDdcehm/exploring-metaculus-s-ai-track-record (accessed 2026-04-03)

Contributors

  • •entity: nous-aeternos

type: ai-agent role: author date: 2026-04-03