Publications
You can also find my articles on my Google Scholar profile.
2026
- FutureSim: Replaying World Events to Evaluate Adaptive Agents.
Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, Jonas Geiping.
AI Forecasting Workshop, ICML, 2026.
Best Paper Award (Top 1 / 187) Abstract | Arxiv | Website | Bibtex
We introduce FutureSim, a benchmark that replays real-world events to evaluate how forecasting agents adapt their beliefs as new information arrives over real-world time. Agents make predictions about world events beyond their knowledge cutoff while interacting with a chronological replay of the world through real news articles arriving and questions resolving over the simulated period. Evaluating frontier agents on their ability to predict world events over a three-month period (January–March 2026), we find a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse Brier skill score than making no prediction at all. FutureSim provides a testbed for emerging techniques such as test-time adaptation, search, memory, and uncertainty reasoning in realistic long-horizon scenarios.
@article{goel2026futuresim,
title={FutureSim: Replaying World Events to Evaluate Adaptive Agents},
author={Goel, Shashwat and Chandak, Nikhil and Arun, Arvindh and Prabhu, Ameya and Staab, Steffen and Hardt, Moritz and Andriushchenko, Maksym and Geiping, Jonas},
journal={arXiv preprint arXiv:2605.15188},
year={2026}
}
2025
- Scaling Open-Ended Reasoning to Predict the Future.
Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, Jonas Geiping.
International Conference on Machine Learning (ICML), 2026.
(accepted as Curating the Future: A Scalable Recipe for Training Open-Ended Forecasters) Abstract | Arxiv | Website | Code | Bibtex
We train language models to make predictions on open-ended forecasting questions, scaling up training data by synthesizing novel forecasting questions from global events reported in daily news using a fully automated curation recipe. We release OpenForesight, our training data with 52k forecasting questions, and train OpenForecaster 8B, a specialized model that matches much larger proprietary models. Training improves the accuracy, calibration, and consistency of predictions, with calibration gains transferring to other benchmarks. All models, code, and data are open-sourced to make research on language model forecasting broadly accessible.
@article{chandak2025scaling,
title={Scaling Open-Ended Reasoning to Predict the Future},
author={Chandak, Nikhil and Goel, Shashwat and Prabhu, Ameya and Hardt, Moritz and Geiping, Jonas},
journal={arXiv preprint arXiv:2512.25070},
year={2025}
}
- Answer Matching Outperforms Multiple Choice for Language Model Evaluation.
Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, Jonas Geiping.
Preprint, 2025. Abstract | Arxiv | Code | Bibtex
Multiple choice benchmarks have long been the workhorse of language model evaluation because grading multiple choice is objective and easy to automate. However, we show that multiple choice questions from popular benchmarks can often be answered without even seeing the question. This shortcut arises from a fundamental limitation of discriminative evaluation not shared by evaluations of the model's free-form, generative answers. We propose answer matching: give the candidate model the question without the options, have it generate a free-form response, then use a modern language model with the reference answer to determine if the response matches the reference. Answer matching achieves near-perfect agreement with human grading, outperforming multiple choice and LLM-as-a-judge evaluations.
@article{chandak2025answer,
title={Answer Matching Outperforms Multiple Choice for Language Model Evaluation},
author={Chandak, Nikhil and Goel, Shashwat and Prabhu, Ameya and Hardt, Moritz and Geiping, Jonas},
journal={arXiv preprint arXiv:2507.02856},
year={2025}
}
2024
- Proportional Aggregation of Preferences for Sequential Decision Making.
Nikhil Chandak, Shashwat Goel, Dominik Peters.
AAAI Conference on Artificial Intelligence (AAAI), (in print), 2024.
Outstanding Paper Award (Top 3 out of 12000+) Abstract | Arxiv | Video | Bibtex
We study the problem of fair sequential decision making given voter preferences. In each round, a decision rule must choose a decision from a set of alternatives where each voter reports which of these alternatives they approve. Instead of going with the most popular choice in each round, we aim for proportional representation. We formalize this aim using axioms based on Proportional Justified Representation (PJR), which were proposed in the literature on multi-winner voting and were recently adapted to multi-issue decision making. The axioms require that every group of α% of the voters, if it agrees in every round (i.e., approves a common alternative), then those voters must approve at least α% of the decisions. A stronger version of the axioms requires that every group of α% of the voters that agrees in a β fraction of rounds must approve β⋅α% of the decisions. We show that three attractive voting rules satisfy axioms of this style. One of them (Sequential Phragmén) makes its decisions online, and the other two satisfy strengthened versions of the axioms but make decisions semi-online (Method of Equal Shares) or fully offline (Proportional Approval Voting). The first two are polynomial-time computable, and the latter is based on an NP-hard optimization, but it admits a polynomial-time local search algorithm that satisfies the same axiomatic properties. We present empirical results for these rules based on synthetic data and U.S. political elections. We also run experiments using the moral machine dataset about ethical dilemmas. We train preference models on user responses from different countries and let the models cast votes. We find that aggregating these votes using our rules leads to a more equal utility distribution across demographics than making decisions using a single global preference model.
@article{chandak2023proportional,
title={Proportional Aggregation of Preferences for Sequential Decision Making},
author={Chandak, Nikhil and Goel, Shashwat and Peters, Dominik},
journal={arXiv preprint arXiv:2306.14858},
year={2023}
}
2023
- Informed Steiner Trees: Sampling and Pruning for Multi-Goal Path Finding in High Dimensions.
Nikhil Chandak, Kenny Chour, Sivakumar Rathinam, R. Ravi.
IEEE Transactions on Automation Science and Engineering (T-ASE), 2023.
ICAPS PlanRobo Workshop 2023. Abstract | Arxiv | Code | Video | Bibtex
We interleave sampling based motion planning methods with pruning ideas from minimum spanning tree algorithms to develop a new approach for solving a Multi-Goal Path Finding (MGPF) problem in high dimensional spaces. The approach alternates between sampling points from selected regions in the search space and de-emphasizing regions that may not lead to good solutions for MGPF. Our approach provides an asymptotic, 2-approximation guarantee for MGPF. We also present extensive numerical results to illustrate the advantages of our proposed approach over uniform sampling in terms of the quality of the solutions found and computation speed.
