Accepted poster · VISA 2026 at IEEE/ACM MICRO 2026
What Makes a Virtual ISA Agent-Programmable? A Controlled Study of Agent-Facing Documentation for StableHLO
01
Abstract
AI agents increasingly write code at the virtual instruction-set architecture (ISA) layer, while benchmarks score the agent, not the interface the ISA exposes. Isolating that interface is hard because documentation variants must carry the same facts, examples and length. We fix one intermediate representation (IR), StableHLO, with a pinned verifier and executor, the tasks and a harness-guided generate-and-repair loop, and vary only how documentation reaches the agent: none, syntax only, prose, a length-matched JSON contract with the same facts and examples, or retrieval through a tool (free, forced, or with cross-cutting rules upfront). With a protocol frozen beforehand, we evaluated 267 held-out tasks (114 synthetic, 153 from KernelBench) with independent oracles and three GPT-4.1 snapshots (5,005 pre-registered trials, the largest model on 91 tasks; 625 post-hoc ablation trials). Even syntax-only documentation added 16 to 33 percentage points (pp) of first-attempt correctness over none. With matched content, the tested JSON contract scored below prose for every model (-6.0, -7.9, -12.1 pp; significant for the two larger), reversing our earlier, unmatched comparison; an exploratory JSON-with-prose ablation also scored below prose, suggesting that the wrapper, grouping or added length may contribute. On-demand access trailed upfront delivery by 4-32 pp: the strongest model never called the tool unless forced, and partial upfront documentation suppressed lookup. Over 99% of observed failures were rejected statically, even on real-world tasks, and a stronger oracle re-grade (265 tasks) changed 1 of 1,069 accepted first attempts: well-formedness was the binding bottleneck.