AI Coding Agents: Better at Finding Files, Still Struggling with Fixes

AI coding assistants are getting better at finding their way around complex codebases, but they're still fumbling when it comes to actually fixing bugs. That's the key takeaway from new research on 'probe-and-refine tuning,' a technique that significantly improves an AI's ability to locate the right files for a task, without making its code changes any more accurate.

Here's the problem: AI coding agents need more than just raw code to work effectively. They require high-level knowledge about a repository's structure, testing procedures, and past pitfalls. Traditionally, this comes from README files, but their usefulness has been debated. This new research shows that how we create and refine this guidance is crucial.

The probe-and-refine process works like this:

Using synthetic bug-fix 'probes,' the system iteratively improves a repository's guidance file through single-shot large language model (LLM) calls. Crucially, this happens without complex agent loops or tool use during tuning.

The results are clear:

• Probe-and-refine guidance: 33.0% mean resolve rate
• Static knowledge base: 28.3%
• Unguided baseline: 25.5%

These differences are statistically significant (p < 0.001). But here's the kicker: the improvement comes entirely from better coverage, not precision. With refined guidance, the AI produced evaluable patches for 14.5 percentage points more instances. However, the per-patch precision stayed flat at about 59% (p = 0.119).

In plain English: The AI gets better at finding the right files to change, but not at making the right changes once it's there.

This exposes a fundamental limitation of current AI coding assistants. They can be guided to the correct part of a codebase more reliably, but their ability to write correct fixes hasn't improved. It's like giving a novice programmer a better map of the codebase, they'll find the right file more often, but they're still just as likely to introduce new bugs when they try to fix things.

Two other findings are worth noting:

1. Better guidance lets AI agents use larger 'step budgets' (number of allowed attempts) more effectively.
2. The tuning process degrades with models that can't generate sufficiently diagnostic output, though per-patch precision remains constant even then.

For software teams, the implications are clear: Invest in creating and refining high-quality repository guidance. It will significantly boost an AI's ability to focus on the right files. However, human oversight remains crucial, as the AI's actual code changes aren't becoming more accurate.

This 'probe-and-refine' approach could become a standard part of repository maintenance, alongside documentation and test updates. But the real challenge lies ahead: extending this success from file selection to actual code quality improvement. Until then, the task of writing correct, nuanced code fixes remains firmly in human hands.