Best LLMs for Coding: Tested Models, Task Results and API Costs

Today's best coding LLMs can take on tasks you'd normally assign to an experienced developer. But which ones are actually worth using? To find out, we tested five frontier models considered among the best LLMs for coding on the same codebases and measured how often each delivered a complete, working solution.

Here we compare their code, response times and API costs. We'll also look at local deployment and show how to evaluate the hosted models through an API.

How We Compared Coding Models

We asked each model to make 12 changes in a Python and JavaScript repository. Four requests described bugs, four asked for new features, and four required a fix with a regression test. The projects covered problems such as job retries, contact syncing and dependency resolution. Each contained about 500 to 800 lines across 7 to 13 files.

Every request included the task description and all the repository files. The model had one chance to submit a patch and could not run the code. Our runner applied the patch and checked it against visible and hidden tests. A change passed only if it met every requirement.

We checked the submitted regression tests separately. Each had to catch the original bug and pass after a correct fix. It also had to reject several deliberately incomplete fixes. This showed which tests really covered the defect and which ones simply passed along with the patch.

All requests went through AI/ML API with low reasoning effort, up to 16,000 output tokens and a 420-second deadline. GLM 5.3 received 32,000 tokens and 900 seconds because the route did not let us reduce its reasoning. After a network error, the request was sent again, up to three attempts in total. A model never got a second chance to fix wrong code.

Best Coding LLMs at a Glance

For a general coding assistant, GPT-6.1 Sol is the strongest choice. It completed more tasks than any other model, and each success cost about as much as one from DeepSeek. That makes it a good fit when you want working code without asking for corrections again and again.

Claude Opus 5.5 is worth a look when speed matters most, since it had the fastest median completion time. Gemini 4 Argon completed the same number of tasks for less money, although its answers took longer.

DeepSeek V4.1 Flash has the lowest token prices here, but it completed fewer tasks, because many requests used up their token limit before returning a patch. GLM 5.3 had even more trouble finishing before the deadline, so this API setup is hard to recommend for interactive coding.

The table shows how often each model met the full task requirements and what that work cost.

Costs include unsuccessful requests. GLM's total also includes estimates for interrupted requests. Cost per solved task is calculated before rounding.

5 Coding Models Compared

After running the measurements, here's how the best AI models for coding in our test compare.

GPT-6.1 Sol

Test results: Sol followed requirements that other models missed, which gave it an advantage. For example, our tasks asked the model to change code across several files and keep the existing behaviour intact. Sol was the only model that achieved this in 100% of runs across every given task.

In one task, for example, a good result had to pick compatible package versions and, when no combination worked, name every package involved in the conflict. Sol did both and passed all 81 checks in 40 seconds at a cost of $0.047.

Sol failed only one task, a tool that applies patches to text files line by line. Its code passed the 42 required checks, but its regression test was wrong. After patching an empty file, the test expected the Windows-style ending "a\r\n", while the project's README says that a text without line breaks gets \n, which failed the test.

Claude Opus 5.5

Test results: Opus was the fastest to deliver a finished answer, with a median of 26 seconds against 32 for Sol.

It also completed all four bugfix tasks. In one of them it fixed three defects in a JavaScript cache without breaking the way the cache merges simultaneous requests for the same data.

Its feature work shows that mostly correct code can still need a developer's attention. Its dependency resolver passed 77 of 81 checks but put the wrong packages in ResolutionError.packages. That field had to list the conflicting package and every package that depends on it.

Its regression tests had a similar gap. The account ledger task rebuilds each account from a sequence of events. One of our incomplete fixes counted a transaction twice when events arrived out of order, and the test Opus wrote did not catch it. In the patch tool task its test missed four of the seven incomplete fixes.

Gemini 4 Argon

Test results: Gemini completed as many tasks as Opus, eight, for about half the price. Its answers took longer, though, with a median of 38 seconds against 26 for Opus.

One of its successful fixes repaired contact syncing between devices and a server. That bug had four causes spread across three files, and Gemini's fix passed all 42 checks.

The query parser shows why its code still needs a careful review. The input title:"a\q contains two errors: an invalid escape at position 8 and an unclosed quote that opens at position 6. The specification asks for the leftmost error, so the right answer is 6. Gemini reported 8, and that one detail cost it 1 of the 45 checks. Opus made the same mistake.

Gemini also failed four checks in the dependency resolver, and its regression tests missed some of the incomplete fixes in the patch tool and account ledger tasks.

DeepSeek V4.1 Flash

Test results: DeepSeek's low prices pay off when it returns a complete answer. For example, it fixed a job queue bug that had six separate causes. The fix passed all 48 checks, and the request cost about $0.01.

It did not complete any of the four feature tasks. Even at the low reasoning setting it used 10,126 reasoning tokens per request on average. Five requests hit the 16,000-token limit before the model submitted a patch, and three of them were feature tasks.

The inventory task shows a different gap. The model had to fix how stock is reserved and add a regression test. DeepSeek fixed the code but did not write the test, so a developer would still have to finish the ticket.

In this setup the output limit held DeepSeek back. A larger limit might let more requests finish, but then the cost and the completion rate would have to be measured again.

GLM 5.3

Test results: GLM struggled to return a patch in a usable time. Nine requests hit the 900-second deadline, and one more used all 32,000 output tokens before submitting a patch. In those cases the developer got no code at all.

Its two successful fixes show that it can solve the problems themselves. It fixed invoice rounding in 870 seconds for $0.15 and inventory reservations in 548 seconds for $0.10.

The result depends heavily on the API configuration. In our preliminary checks, asking for none, minimal or low reasoning did not make its reasoning any shorter. We gave it a larger limit so that it could finish, but it still spent about 9 to 15 minutes on each task.

Waits like that are hard to accept in interactive coding. If you want GLM for that role, first check whether your deployment actually lets you control how long it reasons.

Response Time and Cost per Successful Task

Arguably the most useful time measurement you can make when it comes to interactive coding is how long you have to wait until the first token of the answer appears on the screen. We recorded the moment the first piece of the answer or of the tool call arrived. Reasoning that the model does before answering does not count.

A complete patch, however, can arrive much later. So we also measured the full time from sending the request to a checked result, which includes applying the patch and running the tests. Failed requests stay in the figures. All tasks ran at the same time, so these numbers show the API under that load.

Some requests never produced an answer, so they have no first-answer time.

Another limitation we encountered was that some models used a large number of tokens for reasoning. Since reasoning tokens count toward the output limit, this sometimes left little room for the actual response. DeepSeek was particularly prone to this, spending most of its output token budget on reasoning even at the lowest reasoning setting.

Note: The API did not report Gemini's reasoning separately, so we do not know how much of its output went to reasoning.

To get the cost of a successful task, we divided the spending on all requests by the number of completed tasks, which gave us the following numbers (including network retries and wrong answers):

Coding Failures and Workflow Limitations

These mistakes show why a coding assistant still needs checks outside the model. A patch can look right and still miss an error-handling rule or break part of the existing behaviour. Such gaps are easy to overlook when the main example works.

Generated tests need the same scrutiny. All five models failed the patch tool task, where the request included writing a regression test. Yet none of them sent an invalid tool call or an edit that could not be applied, so checking the tool call alone would not have caught a single one of these failures.

An agent that can run tests may catch a wrong assertion and ask the model to try again. That can improve the result, but it adds requests and waiting time. To compare models for such an agent, measure the whole session up to a verified result. Our test gave each model one attempt, so it does not show how well a model recovers after feedback.

The sample is also small. It has one complete run of 12 tasks, nine in Python and three in JavaScript, so a single changed result moves a model's score by 8.3 percentage points. Additional runs were incomplete and are not in the tables. The tasks were written with Claude's help, which may also bias the results.

Keep these limits in mind when you pick the best LLM for programming in your own projects. Sol's lead is a good reason to try it as a backend, but it does not predict how often it will succeed in your codebase. In larger projects the agent has to find the right files and keep track of a longer history. We did not test either, because every repository fit in one request.

Hosted APIs vs Local Coding Models

You can't run most of these models locally, except for GLM 5.3 and DeepSeek V4.1 Flash, whose weights you can download. With hosted access you pay the service to run the model. However, you don't have to maintain your own hardware.

Running even the best open source LLM for coding locally, on the other hand, requires expensive hardware. The FP8 release of GLM 5.3 takes 755 GB and can run on a server with eight NVIDIA H200 GPUs (141 GB each). The full 1M-token context needs eight B200 GPUs (180 GB each). DeepSeek V4.1 Flash needs at least 614 GB of GPU memory, also requiring four H200 GPUs with its Engram memory tables moved to system RAM, or one NVIDIA DGX Station GB300 workstation.

A complete eight-GPU H200 server, including CPUs, memory and the NVSwitch fabric, costs about $320,000 to $470,000 depending on the vendor. A DGX Station GB300 costs about $85,000 to $125,000. Power and maintenance come on top. Renting an eight-GPU H200 server costs about $29 to $42 per hour on RunPod and up to about $85 per hour at the large cloud providers. For comparison, all 60 requests of our test cost less than $5 through the API.

Reproduce Our Tests Through AI/ML API

To run a similar test, create a task.txt file with the coding task, relevant repository files and any public interfaces the model must preserve. Label each file by its path and leave hidden acceptance tests out of the prompt.

Our original task files and graders are needed to reproduce the exact scores. With your own tasks, you can follow the same procedure and submission format.

The example uses GPT-6.1 Sol and the patch format from our tests. The model submits changes through a submit_patch tool, specifying the file path, the exact text to find and its replacement. Each search string must appear exactly once in the target file. For new files, the search string is empty and the replacement contains the full file contents.

Save the following script as build_request.py:

import json
from pathlib import Path

request = {
    "model": "openai/gpt-6.1-sol",
    "reasoning_effort": "low",
    "max_completion_tokens": 16000,
    "stream": True,
    "stream_options": {"include_usage": True},
    "messages": [
        {"role": "system", "content": (
            "You are a senior software engineer working in an existing "
            "repository. You cannot run code or tests. Read the files, "
            "make the change, and submit it by calling the submit_patch "
            "tool exactly once."
        )},
        {"role": "user", "content": Path("task.txt").read_text()},
    ],
    "tools": [{"type": "function", "function": {
        "name": "submit_patch",
        "description": "Submit your final change with every edit.",
        "parameters": {
            "type": "object",
            "required": ["edits"],
            "properties": {"edits": {
                "type": "array",
                "items": {
                    "type": "object",
                    "required": ["path", "search", "replace"],
                    "properties": {
                        "path": {"type": "string"},
                        "search": {"type": "string"},
                        "replace": {"type": "string"},
                    },
                },
            }},
        },
    }}],
}
Path("request.json").write_text(json.dumps(request))

Then run the script and send the generated request to AI/ML API using your AIMLAPI_KEY:

python3 build_request.py
curl --no-buffer --max-time 420 \
  https://api.aimlapi.com/v1/chat/completions \
  -H "Authorization: Bearer $AIMLAPI_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @request.json

You can replace openai/gpt-6.1-sol with anthropic/claude-opus-5.5, google/gemini-4-argon or deepseek/deepseek-v4.1-flash, adjusting the parameters as needed. For GLM 5.3 (zhipu/glm-5.3), we used a 32,000-token output limit and a 900-second client timeout.

The response contains the proposed edits as streamed tool-call arguments. Apply them to a disposable copy of the repository and run your acceptance tests to check whether the change works.

For cost comparisons, record the API usage and billing data for every attempt, including failed ones. If an agent makes several requests or switches models to complete a task, add up all the charges rather than counting only the successful request.

See the GPT-6.1 Sol API documentation for the full list of request parameters.

Frequently Asked Questions

Which model is best for coding agents?

GPT-6.1 Sol performed best in our tests, making it the strongest starting point among the five models. However, our tests did not cover autonomous agents with access to tools and test feedback. For a comparison of coding applications, see our best AI coding assistants guide.

Can I use an open-weight coding model through an API?

Yes. Both DeepSeek V4.1 Flash and GLM 5.3 are available through AI/ML API, so you can use them without hosting the models yourself. You can also download their weights for local deployment. API usage is billed separately.

How do retries affect the cost of a coding task?

Every API request costs money, whether the model produces working code or not. To calculate the actual cost per task, include all attempts, revisions and requests to fallback models. Comparing token prices alone can be misleading.

Does higher reasoning effort improve coding performance?

We tested the models with low reasoning effort and default sampling settings, so we cannot say whether higher reasoning would improve their results. That would require another test measuring success rates, completion times and costs at different reasoning levels.