U.S.-based OpenAI announced on Oct. 19 (local time) that its latest models have made significant gains in their ability to develop and migrate code in programming languages such as COBOL, Fortran and Visual Basic, for which relatively little code or documentation is available for AI training. The company developed two proprietary evaluations to measure the progress. In an evaluation that tested the migration of COBOL programs to another language, it reported an accuracy rate of 97.9% with a higher-reasoning-effort setting.
Why legacy languages still matter—before looking at the numbers
COBOL was created in 1959. Some may be surprised that something so old is still running, but the language remains widely used in systems at financial institutions and government agencies. According to OpenAI, COBOL underpins more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions.
Fortran, another language included in the evaluation, is used in weather forecasting and nuclear reactor safety analysis. Compared with Python and JavaScript, there is less code and fewer materials for AI to learn from. Yet these languages still underpin society—a somewhat paradoxical situation.
The effort reportedly began after developers and enterprise customers said that coding models struggled with languages like these. It is worth noting that the work is rooted in problems people face on the ground.
A Migration Evaluation Where “It Runs” Isn’t Enough
The rigor of the second evaluation is worth noting. It involved migrating COBOL programs to one of five languages—C++, Java, Kotlin, Python or Rust—and checking whether they behaved exactly like the original. To count as a success, the output had to match byte for byte. The tests were also private, and a program was counted as successful only if it passed every one of them.
A specific example makes the point clear. In COBOL fixed-point arithmetic, extra decimal places are truncated unless rounding is explicitly specified. What if the migrated program instead rounds to the nearest value out of habit? No errors would appear, but the calculated amounts could still be off.
Imagine a one-yen discrepancy accumulating across millions of transactions in banking or insurance, and the importance of this evaluation becomes clear. Rather than simply replacing one language with another, it may be more accurate to describe the task as moving the “behavior” of code—with years of business rules embedded in it, often implicitly—intact into a different language.
How to Read the 97.9% Figure
Here are the results. In the migration evaluation, GPT-6.1 Sol achieved an accuracy rate of 86.6% with reasoning effort set low. GPT-5.6 Sol, using the same setting, scored 5.4—a wide gap. The API cost is said to be 61% of GPT-5.6 Sol’s. Increasing reasoning effort brought the accuracy rate to 97.9%.
The other evaluation, “MultiLangBench,” is different in nature. It asks models to build a SQLite-compatible database engine from scratch in each of 20 programming languages. GPT-6 Astra, with reasoning effort set to maximum, achieved an average test pass rate of 87.4% across the 20 languages, up from 26.8% for GPT-5.6 Sol.
Because the two evaluations are easy to confuse, here is a comparison.
【Evaluation Comparison】
・Migration evaluation: Migrate COBOL to one of five languages and check for a byte-for-byte match in output. GPT-6.1 Sol scored 86.6% with low reasoning effort and 97.9% with increased reasoning effort.
・MultiLangBench: Build a SQLite-compatible database engine in 20 languages. GPT-6 Astra’s average pass rate was 87.4%.
COBOL Remains the Most Challenging
On MultiLangBench, 19 of the 20 languages scored above 83%. COBOL came in at 72.1%. That was a major improvement from GPT-5.6 Sol’s 10.9%, but COBOL remains the most difficult language.
The lower-cost model GPT-6 Luna also outperformed GPT-5.6 Sol, with an average score of 40.3%. Its estimated API cost per sample was $0.77, said to be about 78% lower than GPT-5.6 Sol’s $3.48. The evaluation environment and test suite are scheduled to be released on GitHub.
AI Progress and the State of Legacy Modernization
This is not the first effort to use AI to migrate COBOL. In 2023, IBM announced “watsonx Code Assistant for Z,” which uses generative AI to convert COBOL to Java. This February, U.S.-based Anthropic made the case in a blog post for using Claude Code to help modernize COBOL. The post was seen as a factor behind IBM stock recording its biggest one-day percentage decline in about 25 years. The market’s interest in the field is evident in the fact that a single blog post could move the stock.
There is also a broader social context. In 2020, applications flooded state-run COBOL unemployment insurance systems in the U.S. amid the COVID-19 pandemic. Some may remember that a shortage of workers able to maintain the systems became a problem. Whether AI can help in areas facing a lack of skilled workers is not just a technical question; it has implications for the sustainability of public administration and finance.
Use AI without handing everything over to it
Despite the impressive figures, OpenAI itself says expert review and testing against a system’s business rules and dependencies remain necessary before applying the results to production software. Even if a system passes tests that check for byte-for-byte matches, real-world systems still have edge-case handling and connections to other systems that the evaluation tests may not account for.
I think it is more realistic to view AI not as “magic that replaces everything at once,” but as a “subcontractor that moves migration work forward in a way that is easy to verify.” When the original system’s specifications have been lost, the ability to design tests that confirm the original behavior has been reproduced will be more valuable than the speed of conversion. The 97.9% figure shows that, on the assumption that humans take responsibility for verification, the door to migration has opened dramatically.
Comments