https://agentdiagonstics.com/ Newly added papers, benchmarks, and tools on AI agent diagnostics and optimization, curated by CausaMind. en Tue, 20 Oct 2026 12:00:00 GMT https://arxiv.org/abs/2603.03116 https://agentdiagonstics.com/resources/beyond-task-completion Tue, 20 Oct 2026 12:00:00 GMT Evaluation An audit of tau-bench with procedure-aware evaluation: re-scoring reported successes against the procedures agents actually followed finds 27–78% of them corrupt — success without the required steps. Under gated Pass^4, headline scores collapse (GPT-5 0.58 → 0.24; Mistral 0.46 → 0.03). Because the audit cannot recall every corrupt run, these rates are lower bounds. https://arxiv.org/abs/2610.23599 https://agentdiagonstics.com/resources/trajectories-to-differentiated-skills Mon, 19 Oct 2026 12:00:00 GMT Agent harness Treats each skill as four coordinated parts (Meta, Experience, Tool, Script) updated separately from success and failure evidence. Finds most real skill packages blur these roles; the split method topped all 36 benchmark, model and harness settings tested. https://arxiv.org/abs/2610.21412 https://agentdiagonstics.com/resources/jevagentharness Mon, 19 Oct 2026 12:00:00 GMT Agent harness Frequent small decisions inside a harness, like which tool next, can go to a cheap decision model instead of an LLM call. A 4B model trained on post-mistake states made agents much cheaper and faster with little success loss. https://arxiv.org/abs/2610.21458 https://agentdiagonstics.com/resources/provenance-marks-copied-harnesses Mon, 19 Oct 2026 12:00:00 GMT Agent harness Harnesses leak, and a plagiarist can launder one through a self-evolver. Measured marks mostly fail to survive; survival depends on the rewriting model. The authors' alternative keys a mark to the ordering of near-tied entries in a task-critical lookup table. https://arxiv.org/abs/2610.22869 https://agentdiagonstics.com/resources/ms-eval Mon, 19 Oct 2026 12:00:00 GMT Agent harness An evaluation kit for main agents that delegate to specialised sub-agents over MCP: benchmark, execution harness, delegation metrics and a composite index. Specialisation helped in practice; delegation raised cost, helping most open models but sometimes hurting closed ones. https://arxiv.org/abs/2610.21170 https://agentdiagonstics.com/resources/memscope Mon, 19 Oct 2026 12:00:00 GMT Agent design A patch agent reads execution traces, runtime memory state and the memory architecture's own code to locate failures and apply small bounded repairs instead of rewrites: gains on GAIA, xBench and unseen ALFWorld with a median code change of 4.4%. https://arxiv.org/abs/2610.22469 https://agentdiagonstics.com/resources/testing-planning-claims Mon, 19 Oct 2026 12:00:00 GMT Agent design Agent planning claims checked against hidden PDDL ground truth. The best setup solves 78% of solvable tasks offline but 40% online and recognises unsolvable tasks under 10% of the time; differences trace to the model more than to prompting or architecture. https://arxiv.org/abs/2610.23255 https://agentdiagonstics.com/resources/sample-mix-or-verify Mon, 19 Oct 2026 12:00:00 GMT Optimization How to split a fixed test-time budget between more samples, model mixing and verification when the verifier itself errs. SPLIT uses a small labelled pilot and an exact success-probability formula that accounts for correlated verifier mistakes. https://arxiv.org/abs/2610.21791 https://agentdiagonstics.com/resources/rsiflow Mon, 19 Oct 2026 12:00:00 GMT Optimization A meta-agent reads agent trajectories and feedback to diagnose bottlenecks, picks a component-specific improvement workflow, and distils results into skills and principles that evolve its own harness. Best average across six benchmarks and two backbones in the paper's tests. https://arxiv.org/abs/2610.23574 https://agentdiagonstics.com/resources/darwin-claw Mon, 19 Oct 2026 12:00:00 GMT Optimization Instead of replaying raw trajectories, this framework consolidates experience hierarchically: fine-grained error attributions distilled into updatable prototype vectors that steer skill and memory refinement. Reports gains on WildClawBench. https://arxiv.org/abs/2610.21893 https://agentdiagonstics.com/resources/gacc-bench Mon, 19 Oct 2026 12:00:00 GMT Evaluation 500 goals and 2,571 questions test whether a compressor keeps what matters for a stated goal under a length budget. Even the best of ten compressors trails full context; the right goal helps, the wrong goal actively hurts. https://arxiv.org/abs/2610.23254 https://agentdiagonstics.com/resources/ignored-not-refused Mon, 19 Oct 2026 12:00:00 GMT Evaluation Across 33,000 trajectories from eleven small agents, most failed injections were silently ignored rather than refused, and system-prompt changes alone swung attack success widely. Argues injection evals should report how attacks fail, not just whether they succeed. https://arxiv.org/abs/2610.22511 https://agentdiagonstics.com/resources/auditkit Mon, 19 Oct 2026 12:00:00 GMT Evaluation One result representation for benchmark accuracy, judge scores, runtime behaviour, tool-use traces and retrieval quality, keeping verified outcomes separate from trajectory diagnostics and marking unobservable evidence as unscored rather than failed. https://arxiv.org/abs/2610.23612 https://agentdiagonstics.com/resources/sit-eval Mon, 19 Oct 2026 12:00:00 GMT Evaluation Behaviour specified as situational if-then rules; at each turn an interlocutor enacts a sampled situation and a judge scores the reply against every applicable rule. Exposes per-rule failure modes that aggregate benchmark scores hide. https://arxiv.org/abs/2610.23233 https://agentdiagonstics.com/resources/agentloupe Mon, 19 Oct 2026 12:00:00 GMT Observability An eBPF-based observability layer that tags each LLM interaction with a causal label at kernel level and follows it across processes, threads, files and sockets, so delayed or cross-process side effects can still be attributed to the interaction that caused them, at under 8% average overhead. https://arxiv.org/abs/2610.23284 https://agentdiagonstics.com/resources/annotrace Mon, 19 Oct 2026 12:00:00 GMT Observability Failure studies start with hand-labelled traces, and every team rebuilds its own labelling tool. Annotrace imports six trace formats, splits runs into nested steps, and lets humans and models label failure points against a shared codebook with measured agreement. https://arxiv.org/abs/2610.22779 https://agentdiagonstics.com/resources/trust-agents-own-verifier Mon, 19 Oct 2026 12:00:00 GMT Failure analysis A stress-test method for agents that verify their own rationales: negate each claimed benefit and re-check, scored by a pressure ratio and falsifiable probes. Applied to the authors' own verifier, early oracle access quietly degraded held-out faithfulness. https://arxiv.org/abs/2610.22984 https://agentdiagonstics.com/resources/self-evolving-agents-repair-failures Mon, 19 Oct 2026 12:00:00 GMT Failure analysis When agents learn only from failed interactions, the experience fixes the tasks that produced it but rarely transfers to new ones - a repair-generalization gap that success trajectories do not show. A caution for any self-improvement loop trained on failure logs. https://arxiv.org/abs/2610.23342 https://agentdiagonstics.com/resources/robust-function-calling-interface-drift Mon, 19 Oct 2026 12:00:00 GMT Failure analysis Tools change after deployment: descriptions, schemas, candidate lists. This paper names that interface drift, measures which changes hurt most (distractor tools in the candidate set), and adds a training-free validation-and-repair layer that recovers much of the lost accuracy. https://github.com/ruvnet/ruflo https://agentdiagonstics.com/resources/review-rest-340 Fri, 11 Sep 2026 12:00:00 GMT Agent harness Adds coordination, persistent memory, tools, and workflow controls around coding agents; project performance claims need workload-specific validation.