Summary of the OS-Marathon Benchmark Research
This research introduces OS-Marathon, a new benchmark designed to evaluate the performance of Conversational Understanding Agents (CUAs) on long-horizon, repetitive tasks.Existing benchmarks primarily focus on short-term tasks, leaving a gap in assessing agents’ ability to handle complex, multi-step workflows common in real-world professional settings.
Key Findings & Features:
* Failure Modes: Leading CUAs struggle with three primary issues:
* Logical Incoherence: Incorrect ordering of tasks.
* Hallucination: Generating actions not grounded in reality.
* Inconsistency: Failing to maintain context and consistency across repetitive sub-workflows.
* Domains: The benchmark utilizes two realistic domains:
* Expense Report System: Automating expense report processing.
* GPA Calculator: Automating GPA calculation from transcripts.
* Difficulty Levels: Tasks are designed with varying horizon lengths and document complexity (Levels 1-4) to provide fine-grained evaluation.
* Execution Environments: Agents operate within both web-based systems and local spreadsheet applications, creating a realistic testing ground.
* Novel Metric: Sub-Workflow Accuracy (SWA): A new metric (n/N) to measure performance on extended action sequences, going beyond simple success/failure rates.
* Exhibition Method: A technique to create condensed demonstrations from few examples, enabling agents to generalize to larger, unseen datasets.
* Improved Performance: Combining the demonstration method with AgentS2.5 and GPT-5 significantly improved agent performance.
* Data Generation: Synthetic receipts were generated using LLMs,and the Transcript domain includes both real and synthetic data.
OS-Marathon provides a standardized and challenging benchmark for evaluating and improving CUAs’ ability to handle complex, long-horizon tasks, highlighting the need for advancements in long-horizon reasoning and consistency.
Resources: The project website is available at github. io/.
Keep reading