GPT-6.1 Sol: What the COBOL Migration Jump Means

COBOL is one of the oldest programming languages still in daily use, and it quietly runs a large share of the financial world. This week, OpenAI said its newest Sol model has become far better at translating it. On OpenAI’s own COBOL migration test, GPT-6.1 Sol reportedly scored 86.6% at low reasoning effort and 97.9% at high effort. The previous model, GPT-5.6 Sol, scored 5.4%.

Those are big numbers, and you may have seen them quoted without explanation. This article explains what the test measures, why two different scores are floating around, and why the result matters less, and more, than the headline suggests.

The Two Scores, Explained

Coverage of this announcement mixes up two figures, so here is how they fit together, according to the DEV Community analysis and other reports:

Setting	GPT-6.1 Sol	GPT-5.6 Sol
Low reasoning effort	86.6%	5.4%
High (maximum) reasoning effort	97.9%	Not reported in the coverage I read

So the jump from 5.4% to 86.6% compares the two models at the same low setting. The 97.9% figure is the new model with more reasoning time. One outlet describes a “16-fold” gain, which matches the low-effort comparison (86.6 divided by 5.4). Comparing 97.9% to 5.4% would be closer to 18-fold. When you see a headline using both numbers, check which setting it means.

What the Test Measures

According to the reports, OpenAI’s COBOL migration evaluation asks a model to translate COBOL programs into C++, Java, Kotlin, Python, or Rust. A migration counts as a pass only if the new program’s output matches the original program’s output exactly, byte for byte, across hidden tests. Those tests target tricky details such as rounding, missing records, and numeric conversion.

That standard is strict, and it is the right one for COBOL, because the language often handles money. A rewrite that runs without errors but rounds a cent differently is still wrong.

Why COBOL Matters

The Cryptonomist cites IBM’s estimate that COBOL underpins over 40% of online banking systems and handles about 95% of ATM transactions. These are estimates from a company with a stake in mainframes, so treat them as a rough indication of scale, not an audited figure.

The practical problem is people. COBOL programmers are retiring, formal COBOL teaching has declined, and many organizations are understaffed, according to TechTimes. At the same time, COBOL has far less public code online than a language like Python, which makes it harder for AI models to learn well. OpenAI says it specifically trained for languages with relatively little code and documentation.

A Second Benchmark: MultiLangBench

OpenAI also reported results on a separate test called MultiLangBench. In it, models must build a SQLite-compatible database engine from scratch in 20 programming languages. Reported results:

GPT-6 Astra at maximum reasoning averaged 87.4% across the 20 languages.
GPT-6 Luna averaged 40.3%.
GPT-5.6 Terra scored 37.9%.
GPT-5.6 Sol averaged 26.8%, according to Mezha.
On COBOL specifically, GPT-6 Astra scored 72.1%, up from 10.9%, according to TechTimes.

MultiLangBench is described as an open release, which means outsiders can inspect it. That is a notable difference from the COBOL migration test.

The Big Caveat: The Test Is Hidden

The DEV Community analysis makes the key point: the COBOL migration tests are designed by OpenAI and have not been published. Outside researchers cannot rerun the 86.6% or 97.9% figures. Only the MultiLangBench SQLite test is public. The numbers are OpenAI’s own measurements, and the mechanism behind the gains has not been fully disclosed.

That does not mean the results are wrong. It means they are a claim to verify, not a fact to rely on. Independent replication would change that.

What the Score Does Not Cover

TechTimes notes that the test covers single-file programs with clear specifications. Real mainframe estates are very different. The test does not cover:

Copybooks: shared data definitions that many programs depend on.
JCL job streams: the scripts that chain programs together.
CICS transaction environments: the systems that handle live transactions.
Undocumented business logic: rules that exist only in the code, or in people’s heads.

The article argues that for large estates, the main bottleneck is organizational and architectural: extracting business rules, mapping dependencies, and proving that the new system behaves like the old one. A model that translates one program accurately does not solve those problems.

OpenAI itself says the results do not mean critical software is ready for automatic migration, and that changes still need expert review and testing against each system’s business rules and dependencies.

The Silent Failure Problem

The most dangerous failure in a migration is not a crash. A crash is visible. The dangerous failure is a program that runs and produces a plausible, wrong answer. TechTimes cites Factory.ai’s Legacy-Bench, in which a payroll program passed most tests while computing wrong deductions.

This is why the DEV Community author suggests building your own byte-for-byte parity harness on real inputs. In plain terms: run the old and new programs on the same real data and compare every output, including the strange cases such as missing records and rounding.

A Sensible Pilot Plan for Teams

If your organization runs COBOL and wants to test AI migration, a cautious approach looks like this:

Pick a small, isolated program. Choose a batch job with clear inputs and outputs, not a program tangled with many others.
Collect real test data. Include ordinary records and the strange ones: missing fields, very large numbers, rounding edge cases.
Run old and new side by side. Compare every output, not only the final totals.
Have the original developers review the result. They know the unwritten rules.
Track what the AI got wrong. The pattern of failures tells you where the tool can be trusted.
Scale slowly. Add programs only after the first ones pass independent checks.

This is slower than the headline suggests, but it is how the risk of a silent wrong answer gets managed.

Key Terms

COBOL. A business-oriented programming language from the 1950s, still common in banking, insurance, and government.

Migration. Rewriting a program in a newer language while keeping its behavior the same.

Reasoning effort. A setting that controls how much thinking time a model uses. Higher effort usually costs more and takes longer.

Benchmark. A standardized test used to compare models. Its value depends on whether others can inspect and rerun it.

Parity testing. Running two versions of a program on the same input and checking that the outputs match.

Why Benchmarks Like This Keep Jumping

Large leaps between model versions are common in AI, especially on narrow tasks. A model that was trained with little exposure to a language can look nearly useless, then improve sharply once developers add targeted training data and evaluation. OpenAI says it specifically focused on languages with relatively little code and documentation.

That pattern has two sides. It shows that gaps in AI ability can close quickly. It also means a single benchmark can overstate general progress, because teams can tune models toward a test they know well. This is another reason independent, public tests matter.

What About Cost?

OpenAI estimated GPT-6.1 Sol at about 61% of GPT-5.6 Sol’s cost on this evaluation, which one outlet describes as 39% cheaper per API call. For the smaller GPT-6 Luna, OpenAI estimated $0.77 per run versus $3.48 for GPT-5.6 Sol, about 78% lower, according to Mezha.

Both figures cover the test, not the cost of migrating a real production system. A real project includes analysis, testing, parallel runs, and people. Do not read the per-sample figure as a project budget.

Who Else Is in This Space?

TechTimes lists several competitors working on parts of the same problem: IBM’s Watsonx Code Assistant for Z, which has access to mainframe metadata; AWS Transform; GitHub Copilot; Anthropic; and Cognition, maker of Devin. Cognition reportedly showed speed gains at a bank but said autonomous migration of transactional systems is out of reach for agents today.

Gartner is quoted as predicting that more than 70% of mainframe exit projects started in 2026 will miss their intended benefits, partly because teams overestimate what generative AI can do. That prediction is a useful counterweight to the headline numbers.

What It Means for Different Readers

For banks and insurers: Isolated, well-specified batch programs may now be worth a serious pilot. Large, interdependent systems still need heavy planning and human review.

For COBOL developers: The near-term effect looks like a tool that accelerates analysis, documentation, and test generation, not a replacement. The experts who know the business rules become more important to verify output.

For developers learning AI tools: The result is a good example of why you should read the setup behind a benchmark: which setting, which tests, and who can rerun them.

For general readers: It is another sign that AI’s coding ability is spreading beyond popular languages into the old code that keeps everyday services running.

Availability

The coverage I read does not say whether these COBOL capabilities are available to customers, since the results come from internal tests. Cryptonomist notes that it gives no availability, pricing, or release timing. TechTimes and Mezha describe GPT-6.1 Sol as introduced in late September as a more affordable alternative to Astra. Check OpenAI’s release notes for what your plan includes.

Readers following OpenAI’s recent moves may also want our coverage of why OpenAI cancelled GPT-6.1 Astra and our guide to whether ChatGPT Pro is worth it.

How to Evaluate an AI Migration Claim

Before you trust any benchmark number, run through these questions:

Which setting? Reasoning effort changes scores a lot.
Who can rerun it? A hidden test is a vendor claim.
What is covered? Single files are not whole systems.
What is the failure mode? Silent wrong answers are worse than crashes.
What is the cost basis? Per-sample costs are not project costs.
Who reviews the output? Require expert sign-off for anything that touches money.

Frequently Asked Questions

Did GPT-6.1 Sol score 97.9% or 86.6%? Both, depending on the setting. Reports say 86.6% at low reasoning effort and 97.9% at the highest effort, on OpenAI’s own COBOL migration test.

Can I check the result myself? Not for the COBOL migration test, which has not been published. MultiLangBench is public.

Can AI now migrate a whole bank’s COBOL? No. The test covers single programs with clear specifications. OpenAI and independent commentators both say expert review and system-level testing are still needed.

Why does COBOL matter? It runs critical financial systems, the workforce that understands it is shrinking, and there is little public COBOL code for AI models to learn from.

Is GPT-6.1 Sol the same as GPT-6.1 Astra? No. Astra is the model OpenAI decided not to release after safety concerns. Sol is a separate model described as a more affordable alternative.

The Bottom Line

The jump from 5.4% to 86.6% or 97.9% shows rapid progress on a hard, important problem. But the test is OpenAI’s own, the details are hidden, and real migrations involve far more than translating one program. The sensible reaction is neither hype nor dismissal: treat it as a reason to run careful pilots, with real data and expert review.