INTERACTCOMP — Interaction Benchmark

Motivation & Setup

InteractComp targets real-world ambiguity. Agents can search, interact (yes/no), and answer. We compare Answer-only, Answer+Search, and Answer+Search+Interact, and include a forced-interaction variant to reveal latent capabilities.

Metrics: Round, IR, Accuracy, Calibration Error, Cost.
Grader: GPT-4o (binary correctness); responder simulation for interact.
Finding: With full context, accuracy rises to ~68–71%, showing interaction—not reasoning—is the primary bottleneck.

Key Findings

Higher interaction rates often improve calibration (lower CE) even without consistent accuracy gains.
Open-weight vs proprietary models differ in interaction willingness; excessive caution or ineffective questioning hurts performance.
Allowing more rounds does not automatically increase interaction: models exhibit premature convergence.

Paper Links

arXiv PDF Hugging Face Dataset BibTeX

Chart

Table

Accuracy by Model

Main Results