Leanstral 1.5: Mistral's Open-Source Lean 4 Model Pushes Automated Theorem Proving Forward

Mistral AI has released Leanstral 1.5, a powerful new code agent model for Lean 4 that significantly advances automated theorem proving and proof engineering. While the release touts several impressive benchmarks, the model's real impact lies in its potential to transform how mathematicians and software engineers approach complex proofs and code verification.

Leanstral 1.5 is built on Mistral's mixture-of-experts (MoE) architecture, boasting 119 billion total parameters with 6.5 billion activated per token. This design allows for a massive model capacity while keeping computational requirements manageable. The model accepts both text and image inputs, with a generous 256,000 token context length that enables it to tackle lengthy, complex proofs.

The headline numbers are certainly eye-catching. Leanstral 1.5 reportedly saturates the miniF2F benchmark, achieving 100% on both validation and test sets. It solves 587 out of 672 problems in the challenging PutnamBench, surpassing the previous leader, Seed-Prover 1.5, by 7 problems. Perhaps more importantly, it does so at a fraction of the cost, about $4 per problem compared to Seed-Prover's estimated$300 or more.

But raw problem-solving ability isn't the whole story. Leanstral 1.5's training process, which included reinforcement learning in simulated environments, has produced a model with genuine agentic behavior. In one environment, it iteratively refines proofs based on compiler feedback. In another, it operates within a file system, using bash commands and interacting with the Lean language server to complete partial proofs and build auxiliary lemmas.

This capability translates into real-world applications that could significantly impact both mathematical research and software development. For instance, Mistral reports that Leanstral 1.5 proved the O(log n) time complexity of an AVL tree implementation, a non-trivial task that required structural induction and monadic time tracking. The proof process spanned 2.7 million tokens and 22 context compactions, demonstrating the model's ability to persist through long, complex reasoning chains.

Even more intriguing is Leanstral 1.5's potential for code verification. In a case study, the model was used to analyze 57 open-source repositories, automatically generating and testing correctness properties. This process uncovered 11 genuine bugs, including 5 previously unreported issues. One such bug, found in a Rust library's zigzag decoding function, could cause silent corruption in release builds, exactly the type of subtle error that traditional testing might miss.

The model's availability is also noteworthy. Released under the Apache 2.0 license, Leanstral 1.5 is accessible through multiple channels: Mistral's free API, downloadable weights for local deployment, or through Mistral Vibe, the company's agent CLI. This openness could accelerate adoption and further research in automated theorem proving and formal verification.

However, it's important to approach these results with some skepticism. While the benchmarks are impressive, they don't tell the whole story of real-world performance. The model's effectiveness on tasks outside its training domain remains to be seen. Additionally, the computational resources required to fully leverage Leanstral 1.5's capabilities may be beyond the reach of many individual researchers or smaller organizations.

Despite these caveats, Leanstral 1.5 represents a significant step forward in the field of automated reasoning. Its combination of raw problem-solving power, agentic behavior, and practical applicability to both mathematical proofs and code verification makes it a tool worth watching. As researchers and developers begin to explore its capabilities, we may see new approaches to long-standing problems in mathematics and software engineering emerge.

The release of Leanstral 1.5 isn't just about solving more math problems faster, it's about potentially changing how we approach formal verification and proof engineering across multiple domains. Whether it lives up to this potential remains to be seen, but it's certainly a development that merits close attention from the mathematical and computer science communities.