AI Assistant Overhauls sqlite-utils 4.0, Catching Critical Bugs

Simon Willison's experiment with AI-assisted software development has yielded surprising results, showcasing both the potential and the pitfalls of using large language models in the development process. Willison employed Claude Fable, an AI assistant, to review the release candidate of sqlite-utils 4.0, a popular Python library for working with SQLite databases. The outcome? A thorough overhaul that caught critical bugs and improved the library's design, but at a cost that raises questions about the economics of AI-assisted coding.

The most striking finding was a severe bug in the delete_where() method that could have led to data loss. This function failed to commit its changes and left the database connection in an inconsistent state, potentially losing subsequent operations. Willison notes, "That's a really bad bug! Very glad I didn't ship that." This catch alone justifies the review process, as shipping such a bug could have eroded user trust and forced an emergency patch.

Beyond this showstopper, Fable identified several other issues and suggested improvements. The review process resulted in 34 commits, touching 30 files with over 1,500 lines changed. This extensive rework included a comprehensive overhaul of the library's transaction handling, a core feature of the 4.0 release.

The AI assistant's work went beyond mere bug-fixing. It produced detailed documentation on the new transaction model, clarifying the library's behavior for users. This documentation revealed an important compatibility issue with Python 3.12's new autocommit options for SQLite connections, prompting further adjustments to ensure the library works correctly across Python versions.

Interestingly, Willison didn't stop with one AI review. He had GPT-5.5 review Claude Fable's work, a process he admits seemed "somewhat absurd" at first but has proven valuable. This meta-review uncovered two more significant issues related to transaction handling in the query() method, demonstrating that even AI-assisted work benefits from multiple passes and different perspectives.

The experiment's results are impressive, but they come with a sobering price tag. Willison estimates the cost of this AI-assisted review at $149.25, based on standard API pricing. While this cost was covered by his subscription plan, it raises questions about the economics of AI-assisted development. For smaller projects or independent developers, such costs could be prohibitive.

This case study in AI-assisted development reveals several key takeaways:

1. AI can catch critical bugs that humans miss, potentially preventing major issues in production software.
2. The process is not foolproof, multiple reviews, even by different AI models, can uncover additional issues.
3. AI assistants can contribute significantly to documentation and design improvements, not just bug fixes.
4. The cost of AI-assisted development can be substantial, potentially limiting its accessibility.

Willison's experiment with sqlite-utils 4.0 demonstrates that AI can be a powerful tool in the software development process, capable of improving code quality and catching critical issues. However, it also highlights the need for human oversight and the potential economic barriers to widespread adoption of AI-assisted coding practices. As these tools evolve, finding the right balance between AI assistance and human expertise, and managing the associated costs, will be crucial for developers and organizations looking to leverage AI in their development workflows.