Can a local LLM beat -O3?

During his keynote at ACCU on Sea in June, Andrei Alexandrescu made some predictions. One of those was the integration of LLMs and compilers. Compilers can generate optimization remarks for LLMs to process. And LLMs can propose optimizations that are difficult for a conventional compiler to discover. If you haven’t yet, I very much recommend you to watch Andrei’s talk. It’s a great talk, and it led me down a rabbit hole of experimenting: what could an LLM do today? Integrated into the compiler pipeline, locally on my hardware?

These experiments resulted in a poster, which I will present today at the registration reception of CppCon. The aim of the poster is to answer these questions:

• How much performance improvements can a small, local LLM achieve as part of the compilation process?
• How likely is the LLM to produce code that is functionally correct?
• Is the IR optimization stage the right place to introduce LLM-based optimizations, or is it better to optimize the C++ code before it goes to the compiler?

This blog post allows me to dive a bit deeper into my project, with links to the code.

ℹ️ This post was written before I tested with larger cloud models. They perform drastically better and make LLVM IR optimization a viable approach. Please also check my follow-up post with those results.

The benchmark programs

The testing library consists of five exported functions, each implemented in a separate C++ file. They each test for a different optimization situation:

• fibonacci: A basic recursive implementation, with an obvious algorithmic optimization opportunity.
• format_list: Array serialization in a for loop, with opportunities for optimizing the string concatenation.
• repeated_sort: Gives the model an opportunity to notice unnecessary repeated work.
• count_matches: Finding elements of one array in another array, with a possible optimization in picking another data-structure.
• top_words_from_file: The most complex of the five: file parsing and word-frequency counting, with a deliberately recursive implementation.

The test cases contain deliberately inefficient implementation choices. The aim was to give both the compiler and the LLM opportunities for optimization. In retrospect, this might have been the choice that had the biggest effect on the outcome of the experiment. If I would repeat the experiment now, I would probably include test code with less obvious algorithmic opportunities, where the optimizations actually have to come from the more low-level tweaks a complier excels at.

This experiment is based on Clang/LLVM because LLVM exposes the intermediate representation used by its optimization passes in a documented textual format: LLVM IR. This conveniently allows the LLM to work directly on a textual representation of the same IR that LLVM itself optimizes.

How much does the compiler matter?

The poster title mentions -O3. But I also bench marked other optimization flags. This shows -O3 does not always give you the most efficient binary. It is however a reasonable starting point. In my results, the performance of -O3 tends to be close to the optimal performance. But in the end, you’ll only know what the optimal flags are for your application, if you measure. Although this is also an area where LLMs might help. Meta’s LLM Compiler work, for example, includes models fine-tuned to predict compiler optimization sequences for code-size optimization.

This is how different compiler flags performed on my test code:

You might notice there is no result for -O0 for top_words_from_file. The recursive implementation hits a stack overflow with the test data used during the benchmark. At -O1 and above, LLVM optimizes enough of the recursion away for the test to complete.

How I measured performance

The final results are only as good as the method used to measure them. I took great care to make the measurements reliable. If you find weaknesses in my approach, I’d be very interested in your feedback. (You’ll find my email address in the heading of my homepage)

Each of the five test functions is optimized separately, either as LLVM IR or as C++ code, as described below. The optimized code is then compiled into a shared library (this library contains all five benchmark functions: one of them optimized, the others with the original code). The librunner application loads this library, first runs a test suite to verify correctness, and only then benchmarks it. The runner itself is compiled once and the same binary is used for all benchmark runs.

The benchmarking code first calibrates each benchmark by estimating how many iterations should take approximately 30 seconds. It then executes that number of iterations, cycling through a deterministic set of input values, and measures the resulting throughput in calls per second. This is the performance value collected by the Python scripts.

For the final benchmark matrix, each benchmark is measured five times. Execution order is randomized in balanced blocks to reduce systematic effects from one implementation consistently running before another. The median calls-per-second value from the five measurements is used as the final result. Because small timing differences can easily be measurement noise, I use a 2% threshold when classifying a candidate as faster.

When inspecting the result, I noticed 3 candidates containing undefined behavior, but still passed my correctness tests. I then added ASan/UBSan validation, and marked these 3 candidates as invalid in the final results on the poster.

The optimization strategies

I set out on this experiment, thinking I would just run 2 LLM optimization passes for each testing function on each model:

• Naive C++: Feed the full C++ test file (each file contains one of the five test functions) to the model, together with a prompt telling it to optimize the code without changing the external API.
• Full LLVM IR: Similar to Naive C++, but sending one full LLVM IR file, created from the C++ code by compiling it with -O1.

The results of this were quite disappointing. Especially for LLVM IR, where only five of the forty tasks produced a valid candidate. Context length was a major failure mode, especially for full LLVM IR. This led me to try out other strategies:

• Extracted LLVM IR: The test function is extracted from the full LLVM IR module, using llvm-extract. The extracted IR is sent to the model, which is instructed to return only a replacement for the target function, without repeating the surrounding module.
• Guided C++: This strategy tries to guide the model by providing it with information from the compiler. The LLM is given some of the compiler optimization hints, together with the code. As long as the compiler generates new hints, the LLM gets up to three iterations, each time with new compiler hints. If compilation fails, the LLM gets up to two chances to fix the code, based on the compiler error.

Guided C++ adds compiler remarks, multiple optimization iterations, performance-based selection, and repair attempts. To investigate the contribution of repairs, I added another strategy:

• Naive C++ + repair: When compilation fails, the model gets a chance to fix its code, based on the compiler error.

Results differed a lot by model. But when combined, this was the overall picture for each of the optimization strategies:

The models

The choice of models matters a lot to the outcome of this experiment. I set myself the restriction that models had to be able to run locally, on my hardware. This hardware has plenty of RAM (128 GB), but no GPU suitable for LLM inference. The LLM had to run only on the CPU (AMD Ryzen 9 5950X).

We are used to compilers being tools that we can download, install, and run locally, often free of charge and open source. LLM-assisted compilation does not necessarily have to follow that model: a future compiler could rely on large models running in the cloud, much as many coding agents already do today.

For this experiment, however, I deliberately limited myself to relatively small local models. I wanted to explore how useful an LLM could be as part of a compiler that still behaves like the compilers we use today: software that you can download and run on your own machine.

These are the eight models I picked:

• Four general/coding models of similar size (12–14B): Qwen3 14B, Qwen2.5-Coder 14B, Gemma 4 12B, Ministral 3 14B.
• Two compiler-specialized models: LLM Compiler 7B and 13B.
• Two larger coding/reasoning models: gpt-oss-20b and Devstral Small 2 24B.

I specifically included the LLM Compiler models from the Meta paper I mentioned earlier. These models are based on Code Llama and were further pretrained on LLVM IR and x86-64, ARM and CUDA assembly. Meta also released FTD variants, further fine-tuned to predict optimization sequences to reduce code size. I tested the foundation variants, hoping their specialized training would make them better at processing LLVM IR than the more general models. However, in my experiment, neither produced a valid optimization candidate. If you managed to get better results out of these models, I’d be interested to hear how.

These are the results by model:

Results

We’ve already seen the models don’t always produce correct results. But when the generated code is valid, it certainly can outperform the code optimized by the compiler:

Also when I benchmark the fastest LLVM optimization against the fastest LLM candidate, the LLM wins:

What did the LLM actually do?

All candidates that produced a meaningful (>2%) speedup came from the C++ strategies. These mostly target the deliberate inefficiencies present in the benchmark programs. The most interesting of them, is probably format_list, where the LLM attempts optimizations that are closer to the kinds of things a compiler might attempt. The LLMs try things like reserving the output string, estimating the required size or manually converting integers. But most of these candidates fail to outperform -O3. And format_list is also the benchmark where the best LLM result was closest to conventional compiler performance. Against the fastest LLVM configuration, the difference was around 2%.

Failure modes

Failures can be roughly grouped into five categories:

Response truncation: With 71 cases, this is the larges group. The LLM burns through the token budget, without producing a useful result. For instance this Gemma 4 run. The model keeps reasoning. 4096 Tokens long (the reasoning_content field in the JSON). And it never generates the actual response.

Context too large: There are 33 cases where the model context was simply insufficient. This was mainly an LLVM IR problem, where sometimes even the prompt alone did not fit in the context. This top_words_from_file run is an extreme example. The context was around 32k tokens, but the prompt alone was 107k tokens long.

Compile/assemble failures: In 20 cases, the generated code did not compile. These are sometimes relatively small mistakes. In this format_list example, Qwen3 used memcpy, but forgot to include <cstring>.

Correctness failure: In fourteen cases, the compiled candidate failed correctness validation or benchmark execution. An example is this top_words_from_file optimization. This code compiles, but contains bugs.

Undefined behaviour / memory-safety errors: the three most subtle cases, compiled fine and passed the validation tests. They benchmarked faster than -O3, but they were still wrong. This example contains UB, including a signed overflow on INT_MIN, and a new[]/free() allocator mismatch.

Limitations

The most obvious limitation is of course the one I set myself for this experiment: using relatively small, local models. Larger context windows and more generous generation budgets might improve completion rates, particularly for IR. Whether larger models improve optimization quality remains untested.

In retrospect, the five benchmark programs I picked, limit the relevance of the experiment. I wrote code that offered different, fairly obvious optimization opportunities, with large potential gains. The test case that is missing, is one with efficient code, without the obvious wins. Would the LLM have picked strategies there that are closer to the kinds of optimizations a compiler would perform? Would the LLM still have outperformed -O3? These remain open questions.

The Naive C++ + repair approach offers the LLM a chance to fix the code after a compilation failure. It does not however offer the LLM multiple attempts at optimizing. Once we have a valid candidate, we move on to the next experiment. In contrast, Guided C++ performs a maximum of 3 iterations in which it feeds the LLM new compiler hints. It also benchmarks each candidate, and only adopts improvements. As a result, there are 40 LLM calls for the Naive C++ strategy, together taking over five hours. Naive C++ + repair makes 49 LLM calls, spending a total of almost seven hours. Guided C++ makes 82 LLM calls in total, taking up slightly more than thirty six hours.

The optimizations attempted by Guided C++ are different than those attempted by Naive C++ + repair, but it is hard to say how much the compiler hints actually influence the LLM. Many of the hints sent to the model, are relatively low information hints like GVN LoadClobbered, LICM LoadWithLoopInvariantAddressInvalidated or Inlining NoDefinition. The LLM still mainly attempts higher-level algorithmic changes. The experiment does not isolate the contribution of compiler remarks from additional optimization attempts and performance-based selection. To really see the effect of the compiler hints, I should have added a controll that behaved like Guided C++, but without the compiler hints.

C++ candidates are compiled with -O3. IR candidates are tested with both backend -O0 and -O3, but the poster comparisons use backend -O3. The reported comparison therefore measures LLVM alone against LLM transformations followed by LLVM optimization. The reasoning here is that I am testing what an LLM can contribute when it is integrated as an extra optimization step in the compiler, not as a replacement for the existing optimization.

Conclusions

Andrei Alexandrescu spends much of his keynote talking about abstractions. My first conclusion will therefore not come as a surprise to anyone who watched it: In this experiment, the LLMs were much more effective at the higher-level C++ representation than at LLVM IR. Full LLVM IR produced only five valid candidates and extracted IR produced no successful modified candidate. Neither IR approach produced a result more than 2% faster than -O3. The C++ strategies did much better.

But also C++ candidates frequently failed to compile. Some failed the correctness tests. Some contained undefined behavior. LLM-generated code cannot be assumed to preserve functional behavior. Therefore, LLM-generated optimizations require strong validation.

The LLM optimizations observed in this experiment, are fundamentally different from traditional compiler optimizations. LLMs can substantially outperform -O3, but mainly by changing the program structure. These are transformations that conventional optimizing compilers generally don’t perform. The model may change algorithmic choices that the programmer deliberately made. An invisible compiler pass might not be the right place for these transformations.

Compiler-guided C++ gave the best results overall. It produced 19/40 candidates which passed the correctness validation and 14/40 candidates more than 2% faster than -O3. It is also the strategy with the highest inference times. From this experiment, we can’t really conclude how much of the improvements of the guided strategy are due to the compiler hints, and how much is due to the extra inference iterations.

Reliability depends strongly on the model. Qwen2.5-Coder was by far the most successful model in this experiment. Some models produced no valid candidates at all. Models with similar parameter counts, produced very different results. Testing seems to be the only way to find the best performing model.

Benchmarking is hard. A lot of things can go wrong. Repeating experiments can take a long time, and inevitably the deadline approaches.