arXiv cs.AIThursday · May 28, 2026FREE

DynaSchedBench: Calibrated Dynamic Scheduling Benchmarks and Observability Paradox in LLM-based Scheduling Agents

Published on arXiv on May 28, 2026, DynaSchedBench is a new diagnostic framework designed to address methodological challenges in neural combinatorial optimization for the Dynamic Flexible Job Shop Scheduling Problem (DFJSP). The framework aims to overcome issues like benchmark overfitting from static benchmarks and obscured algorithmic capabilities due to stochastic noise from uncalibrated generators. Its core innovation is the Sequential Event-Space Calibrator (SESC), which computes a novel Schedule Stress Index (SSI) to stratify scheduling instances by difficulty, moving beyond simple parameter sampling. The authors demonstrate that SESC is significantly more computationally efficient than evolutionary baselines while reliably converging to target metrics. DynaSchedBench features modular components for instance generation, snapshot-based simulation, agent integration, evaluation, and visualization, enabling rigorous testing of both reactive and lookahead-based scheduling policies. Leveraging this calibrated environment, the research identified specific limitations of LLM-based scheduling agents, particularly concerning their performance in step-wise online decision-making for dynamic scheduling scenarios. This suggests that while LLMs show promise, their application in complex, real-time combinatorial optimization problems requires more robust evaluation and development.