This is a follow-up post to my experiments with local models for code optimization. In the previous post, I give more background on how and why the experiment was set up. I mention some limitations of the approach. And some things which, in retrospect, I would do differently. Nevertheless, only two things changed in this experiment: the models, and where inference ran. I picked three frontier models, and three large open-weight models. The inference ran in the cloud, via OpenRouter.
First impressions
The first thing that is immediately clear from these results, is how much more reliable the bigger models are. This picture is much greener than it was for local models:
While the smaller models failed to produce useful results for LLVM IR, the larger models do much better. For the top_words_from_file benchmark, the fastest result was an LLVM IR result:
The choice of models still matters, but the differences between models are less significant than they were with the smaller models:
Guided C++ still gives the best results:
But the multiple iterations make it the most expensive of the strategies in this experiment:
LLVM IR becomes a viable approach
Before I ran this experiment, I uploaded an LLVM IR example into the web interface of some commercial AI agents. These agents run on the same models I used here, but they also use tools which this experiment did not. One of these agents chose to use a decompiler to convert the LLVM IR to C, optimize the C, and convert it back to LLVM IR. This reinforced my suspicion that optimizing LLVM IR directly just wasn’t going to work. But this experiment proved me wrong.
Reliability remains an issue with LLVM IR, but with the extracted IR approach we get almost 2/3 valid candidates. That is better than what Guided C++ achieved in the experiment with the smaller models.
Some of these IR optimizations are quite advanced. For count_matches, Claude implemented its own hash table in LLVM IR.
Another interesting example is this format_list optimization. In a first pass, GPT computes the exact output length. It then reserves enough space in the std::string and writes the result directly into its buffer. To do this efficiently, it even implements its own integer-to-decimal conversion loop.
C++ Examples
The best performing format_list optimization comes from Gemini. It uses resize_and_overwrite and a custom integer formatter with a lookup table for the strings “00” through “99”.
Just like in Claude’s IR example we saw earlier, GLM writes a custom hash table for count_matches. But it also dynamically chooses between three algorithms, depending on the size and the range of the input.
Conclusion
While the local models optimized code at a higher abstraction level, these larger models are much more willing to lower those abstractions and generate specialized low-level implementations. format_list is particularly interesting because it has no obvious algorithmic shortcut. The local models failed to make significant improvements to it. Several cloud models, however, left -O3 comfortably behind. And they did so in very interesting ways.
So will the compiler of the future integrate an LLM? It very well might. And with today’s models, a compiler with LLM optmizations is more likely to be a cloud service than a binary you’ll run on your own hardware.