A first-person field report from a mainframe-to-Java engagement in early 2026. The prompts, the false starts, the human-in-the-loop guardrails, and the real productivity multiplier (it is not 10x).

Between January and April 2026, our team used current-generation large language models to help refactor roughly 200,000 lines of legacy code into Java for a property and casualty insurer. This is what actually happened, not the keynote version. The productivity multiplier is real, but it is not 10x. The failure modes are specific. The human-in-the-loop pattern that worked is repeatable. We compare what we saw to IBM’s published research on watsonx Code Assistant for Z and to the peer-reviewed paper “GenAI-Driven Migration of Legacy COBOL Applications to Clean Java Code” (ICSBT 2025).

GenAI-assisted refactoring delivered roughly a three-times productivity gain over our 2023 baseline on this engagement, well short of the ten-times figure vendors quote. The wins came from explanation, test generation, and boilerplate translation. The breaks came from implicit COBOL semantics (REDEFINES, packed decimal, level-88 conditions), copybook drift, and over-eager refactoring of business rules.

The pattern that worked was simple. The model proposed, the engineer verified, and the test suite enforced, in that order.

Why This Report Matters in 2026

Three things converged in late 2025 and early 2026 that make GenAI-assisted mainframe modernization a real option rather than a slide. Frontier models reached the context-window and reasoning depth needed to handle COBOL plus its copybooks in one conversation. IBM’s watsonx Code Assistant for Z moved from pilot reports to documented enterprise rollouts. And the underlying business pressure increased: cyber insurance carriers are repricing legacy-exposure premiums, and the labor market for COBOL maintainers continues to shrink as the workforce retires.

The result is that boards now ask a different question. Not “can we modernize the mainframe?” but “why have we not started yet, given the tooling has changed?” This field report is meant to answer that question honestly, including where the tooling still falls short.

The Setup

A property and casualty insurer. Policy administration system on IBM z/OS, roughly 200,000 lines of COBOL across 340 programs and 1,100 copybooks, a mix of batch and CICS. Target architecture: Java 21 on AWS with Aurora PostgreSQL. The hard requirement was retaining business semantics exactly. The system is regulated under state insurance commissioners and SOX. Timeline: 14 weeks.

Team composition: six engineers (four modernization, two COBOL subject-matter experts), two QA engineers, and one architect. We used current Anthropic and OpenAI frontier models for translation and verification, and a local open-weight model for code paths touching personally identifiable information.

What Worked

1. Explanation first, translation second. The model’s best output was plain-English summaries of what each COBOL program did. A COBOL subject-matter expert validated the explanation before any Java was requested. This single step caught roughly four in five misinterpretations before code was written.
2. Test generation from observed behavior. We fed the model the COBOL plus sample inputs and asked for characterization tests in JUnit. Those tests became the gate the Java had to pass. The productivity gain on test writing alone was easily six to eight times faster than hand-rolling tests.
3. Boilerplate translation. File I/O, simple MOVE and COMPUTE statements, paragraph-to-method conversion. Clean, repetitive work the model does very well. Roughly seven in ten lines of the codebase fell into this bucket.
4. Copybook normalization. Generating consistent Java records from variant copybooks (same logical field, different physical layouts). Tedious by hand, fast for the model.
5. Documentation generation. Every translated module shipped with model-generated JavaDoc traceable back to the COBOL paragraph it came from. Auditors loved this and it cut their review time noticeably.

What Broke

6. REDEFINES and packed decimal. COBOL’s memory aliasing and decimal arithmetic have no clean Java analog. The model produced subtly wrong field interpretations a meaningful percentage of the time. Characterization tests caught these, but only because we wrote the tests first.
7. Level-88 condition names. Named boolean conditions were translated literally instead of as enums or domain objects. The code worked but it was unreadable. This required a second pass with an explicit refactor-to-domain-enums prompt.
8. Implicit business rules. The most expensive failure: the model helpfully simplified a 40-year-old underwriting rule that looked redundant but actually encoded a state regulation. A 30-year subject-matter expert caught it in UAT. The lesson: never let the model refactor business logic without explicit human approval, rule by rule.
9. Copybook drift across programs. Same copybook name, different actual contents in different programs (a real legacy mess). The model assumed consistency. We built a pre-processing step that canonicalized copybooks before any model call.
10. Context window limits at scale. Even very large context windows cannot hold a full COBOL program with all its copybooks and called subprograms. We built a dependency-graph chunker. Without it, the model hallucinated the missing context. The ICSBT 2025 paper on GenAI-driven COBOL migration reports the same failure mode independently.

The Human-in-the-Loop Pattern That Worked

After two failed approaches (model as engineer, model as pair programmer) we landed on model as proposer. The flow:

11. An engineer scopes a unit of work, typically one program or one paragraph.
12. The model explains the COBOL in plain English. The subject-matter expert validates.
13. The model generates characterization tests from the COBOL plus sample data. The engineer reviews.
14. The model proposes the Java translation. The engineer reviews and edits.
15. Tests run. Failures go back to step four with a diff and the failing test output.
16. On pass, a second human code-reviews and merges.

The single most important rule: The model never wrote the final commit message and never approved a pull request. Two humans touched every change, and that is what made the output auditable for state insurance regulators and SOX.

The Actual Numbers

These figures are measured on this engagement only. We share them as a directional read, not as a benchmark to copy. IBM’s own published case studies on watsonx Code Assistant for Z report productivity gains in a similar range when the human-in-the-loop discipline is held.

Metric: Lines translated per engineer-week | 2023 baseline: 1,100 | 2026 with GenAI: 3,800 | Multiplier: 3.4x

Metric: Defect rate per 1,000 lines | 2023 baseline: 4.2 | 2026 with GenAI: 3.1 | Multiplier: 1.4x better

Metric: Test coverage at first pull request | 2023 baseline: 38% | 2026 with GenAI: 82% | Multiplier: 2.2x

Metric: Time per business-rule validation | 2023 baseline: Same | 2026 with GenAI: Same | Multiplier: Unchanged, this is human-bound

Metric: Model API cost per refactored file | 2023 baseline: $0 | 2026 with GenAI: About $0.40 | Multiplier: New line item

Common Mistakes Teams Make with GenAI on Legacy Code

• Treating the model as an engineer instead of a proposer. The role split (model proposes, human verifies, tests enforce) is what makes the output auditable. Skipping it usually shows up first as a quiet defect, then as a regulator finding.
• Skipping characterization tests. Without a test suite generated from observed behavior, you cannot tell whether the new Java matches the old COBOL on the inputs the business actually sends. Hand-validating thousands of paragraphs is not a real plan.
• Letting the model refactor business rules unattended. The most expensive defects we have seen all came from helpful simplifications of rules that encoded a state regulation or a decades-old underwriting decision. Business-rule refactors need a named human approver, rule by rule.
• Ignoring copybook drift. Same copybook name in two programs does not mean the same record layout. Canonicalize copybooks before any model call, or budget for the rework.
• No prompt or response logging. State insurance regulators, SOX auditors, and internal risk all expect a reconstructible trail. Adding logging after the fact is much harder than turning it on day one.
• Over-indexing on the productivity number. The headline ten-times figure from vendor decks is not what teams measure in practice. Three to four times is realistic with the discipline above. Anyone budgeting against ten times will miss their dates.

Frequently Asked Questions

Why not full automation?

State insurance regulators and SOX require attributable human review. Beyond that, the implicit-business-rule failure mode alone would have shipped a defective product. Full automation is the wrong target for this work in 2026.

Which models worked best?

Current frontier models from Anthropic and OpenAI for translation and verification respectively, with a local open-weight model for code paths touching policyholder personally identifiable information. The specific model names matter less than the role each one played in the pipeline.

What about IBM watsonx Code Assistant for Z?

Strong for in-place modernization where you stay on Z and add Java. We were exiting the platform entirely, so it did not fit this engagement. If you are staying on Z, it is worth a serious look. IBM’s published pilot reports are linked below.

Could a smaller team do this?

Below four engineers and one subject-matter expert, no. The pattern needs review capacity. Above eight people, coordination overhead starts to eat the gain. The sweet spot is in between.

Is this safe under SEC cybersecurity rules?

Yes, with logging. We logged every model call, every prompt, every response, and every human decision. That trail is what made the work auditable. Without the log, the same engineering work would not have cleared internal audit.

References

• IBM watsonx Code Assistant for Z — https://www.ibm.com/products/watsonx-code-assistant-z
• IBM Community: How a generative AI code assistant can benefit mainframe modernization (Jun 2024) — https://community.ibm.com/community/user/blogs/louisa-muschal/2024/06/13/value-watsonxcodeassistantforz
• GenAI-Driven Migration of Legacy COBOL Applications to Clean Java Code (ICSBT 2025, Springer) — https://link.springer.com/chapter/10.1007/978-3-032-08614-3_4
• Hyperframe Research: Can AI Truly Bridge the Mainframe Modernization Gap? (Dec 2024) — https://hyperframeresearch.com/wp-content/uploads/2024/12/HF_RR_AI-for-Mainframe-Modernization.pdf