Cost-Conscious Coding Agents: When Lower-Cost Model Offloading Makes Sense

CyberShell Research Team

Technical report

A CyberShell Research technical report on lower-cost coding-agent offloading, model routing, verifier-backed execution, and practical cost-performance tradeoffs.

Alexandre

Subject matter expert

Context

Frontier coding models are becoming increasingly capable — and increasingly expensive. This experiment asks a practical question: how much can we reduce the cost of coding-agent work by using DeepSeek V4 Pro, and what do we get at that lower price?

We gave the same three Terminal-Bench tasks to Codex backed by DeepSeek V4 Pro and Codex backed by GPT-5.6 Sol.

What this experiment tests

This is a comparison of complete coding-agent systems, not a model-only leaderboard. Each system includes Codex, its tool loop, the benchmark harness, the inference route, the model, and the selected reasoning or agent configuration.

The experiment compares three conditions:

• DeepSeek V4 Pro x-high single-agent: Codex routes through Moon Bridge, OpenRouter, and Together to DeepSeek V4 Pro 0813, with multi-agent features disabled.
• DeepSeek V4 Pro Ultra: the same guarded route, using the highest available reasoning setting and Codex's multi-agent mode.
• GPT-5.6 Sol Ultra: Codex uses OpenAI's native API with the highest available reasoning setting and multi-agent mode.

This gave us a lower-cost model without routing our test data through a DeepSeek-operated endpoint.

Methodology

We used the same three tasks from Terminal-Bench 2.0 commit 69671fba, with Codex CLI 0.147.0 and Harbor 0.20.0. Containers, verifiers, limits, and timeouts were held constant.

Every task ran once, sequentially, in an isolated session with no retries. Web search and MCP were disabled. Before each task, we checked the requested model, reasoning setting, and — on the DeepSeek route — that Together was the provider.

The official task verifiers determined success. Times cover end-to-end execution; costs came from OpenRouter balance usage or GPT token usage.

The three benchmark tasks

Results by harness and agent

Each task ran in its own root-agent session. Subagent counts are suite-total spawn events, not the maximum number running concurrently.

Results and conclusions

DeepSeek x-high is a viable economic offload when latency is flexible

The cleanest comparison is between the two configurations that passed every task. DeepSeek x-high took 35m17s, versus 16m03s for GPT-5.6 Sol Ultra — approximately 2.2 times as long.

It cost $0.252382, versus $3.200433 for GPT — approximately 12.7 times cheaper, or a 92.1% cost reduction in this run.

That tradeoff is attractive when work is not urgent, the task is easy or medium in complexity, its inputs are appropriate for an external processor, and a deterministic test or verifier can catch failures.

Examples include bounded repository maintenance, data conversion, test generation, documentation transformations, and other queueable work. A practical production policy would route eligible jobs to guarded DeepSeek x-high first and escalate failures or latency-sensitive work to GPT.

The conclusion should remain narrow. Three single-attempt tasks do not establish a general quality rate, and fix-git was classified as easy. The experiment demonstrates feasibility and economics, not blanket equivalence between the systems.

GPT-5.6 Sol Ultra was fastest and achieved the only 3/3 Ultra result

GPT was the fastest condition and the only Ultra condition to pass all three tasks. It also made seven subagent spawns, while both DeepSeek configurations remained effectively single-agent.

In this experiment, GPT's additional $2.948051 over the successful DeepSeek run bought the shortest completion time and a 3/3 result. That premium may be justified for urgent work or workflows where a failed first attempt is expensive.

For a large queue of independently verifiable, non-urgent tasks, the cost difference strongly favours DeepSeek.

Why DeepSeek V4 Pro Ultra failed

DeepSeek Ultra never created the required /app/regex.txt file.

On regex-log, it entered a runaway tool-generation pattern and constructed extremely large inline shell and Python commands — approximately 104 KB and 58 KB in the original run. Two responses reported exactly 65,536 output tokens, and the resulting strings and heredocs were incomplete.

Codex then reported malformed function-call arguments — EOF while parsing a string — and its recovery request ended with HTTP 400, Server tool request failed.

The routing controls themselves passed: the trial used Together, the exact DeepSeek V4 Pro 0813 model, and wire effort max. The other two Ultra tasks passed, and no subagent was spawned.

A later diagnostic repeat, excluded from the comparison table, failed the same task through the same oversized-generation pattern, although it ended by timeout rather than the identical HTTP 400.

In contrast, x-high passed regex-log through the same Codex version, model, Together route, OpenRouter policy, Moon Bridge, and verifier while producing commands measured in kilobytes rather than tens of thousands of characters.

The evidence does not justify assigning fault to a single component:

• DeepSeek/provider behaviour is the likely trigger. The oversized repetitive content arrived in the upstream assistant tool-call output. However, no direct-provider control was run, so this experiment cannot separate DeepSeek itself from Together/OpenRouter's Responses implementation.
• Codex exposed the failure but was probably not its initial source. Codex correctly rejected incomplete function-call JSON. Its recovery path then received the HTTP 400.
• Moon Bridge is possible but comparatively unlikely. On this Responses route, Moon Bridge proxies the upstream response rather than transforming the function arguments. A bridge defect cannot be ruled out without a direct OpenRouter-to-Codex control.
• The combination matters. X-high and Ultra did not differ only in a standardized compute dial. Ultra also advertised multi-agent tooling, even though no subagent was spawned.

The next test should rerun regex-log through Codex directly to the same guarded OpenRouter/Together endpoint, removing Moon Bridge from the path. If it still fails, a controlled API test should compare streaming with non-streaming, then vary the reasoning level separately from multi-agent mode.

Future research

Before building a custom privacy codec, we should first use as much of OpenRouter's native protection as possible. That means testing and documenting the strictest useful model and provider allowlists, ZDR enforcement, disabled fallbacks, data-collection restrictions, and per-key spending controls.

Native controls are simpler to audit and maintain than a new transformation layer.

If we later test a lower-cost direct model endpoint, one research direction is a stateful privacy codec at the streaming boundary. Before outbound content reaches the endpoint, the codec could replace secrets, private URLs, internal hostnames, account identifiers, and other sensitive values with stable opaque tokens, then reverse the substitutions on the inbound stream.

It would need to handle values split across chunks, prevent placeholder collisions or token forgery, preserve enough syntax for coding tasks, keep mappings out of model-visible context, and fail closed when classification is uncertain.

That codec would reduce accidental disclosure, but it wouldn't make an untrusted provider safe for arbitrary proprietary code.

Sources

• Terminal-BenchTerminal-Bench 2.0 commit 69671fba
• OpenRouterProvider routing
• OpenRouterGuardrails
• OpenRouterZero Data Retention
• Together AIPrivacy and security