TL;DR
- An arXiv paper published on 30 July, arXiv:2607.28271, circulated under a headline about AI porting COBOL to Java and carrying the bugs across; its subject is how you establish that a migration is correct.
- The method runs the COBOL source and the generated Java in two instrumented environments off-mainframe, and a deterministic parity check between them decides right or wrong.
- The agent is never asked whether the Java is correct: it manufactures test inputs that would expose a difference, then names the branch conditions it could not get past.
- Two points stay open: 91.90% branch coverage came from the most production-like program and is the lowest of the three, and “in all accepted test cases” never says what the filter accepted or rejected.
In plain English
A deterministic oracle is a check that gives the same verdict every time and cannot be argued with, which is what makes it usable as a judge of a model’s output. Here the oracle is a parity check: feed identical inputs to the original COBOL and the generated Java, and compare what each one does. Both are instrumented with mocks, stand-in versions of the systems a program would normally call, so the whole thing runs on ordinary hardware instead of requiring booked mainframe time.
On top of that sits the Locksmith Loop. It performs what the authors call Witness Search, hunting for inputs that reach into program branches, followed by parity-preserving mutations of those inputs. Branch coverage measures how much of the program’s decision logic the tests actually exercise. When the loop hits a routing boundary it cannot get past, an analyser flags a Locked Paragraph: the specific condition blocking deeper exploration. The three case studies ran from 430 to 4,114 source lines, reaching nearly complete coverage on two open-source programs and 91.90% on an internal production-like one. The architecture worth borrowing is the split: the model generates candidates, and a deterministic check you own decides which survive.
A paper landed on arXiv on 30 July — arXiv:2607.28271, Agentic Method for Deterministic Validation of Legacy Code Migration, by Andras Ferenczi, Jordan Docherty, Mariya Bessonov, Matthew Findlay and Krishna Lingamneni. It went round the usual places under a headline about AI migrating legacy COBOL to Java and carrying the bugs across with it.
That headline misses the paper entirely. I know, because I put a version of it in my own notes before I read past the title. The subject here is validation — how you would ever establish that a migration was correct, once a model has done it.
The actual method
The setup is two runtime environments: the COBOL source and the generated Java target, each instrumented with mocks and executed off-mainframe on commodity hardware. That detail alone is doing real work — you can’t iterate on a problem you have to book mainframe time to observe.
On top of that sits what the authors call the Locksmith Loop. An iterative agentic loop performs “Witness Search” over the input mocks, hunting for inputs that penetrate program branches, followed by parity-preserving mutations. When it hits a routing boundary it can’t get past, an analyzer identifies what they name a Locked Paragraph — the specific condition blocking deeper exploration.
Notice what the agent is never asked to do. Nobody asks it whether the Java is right. Its job is to manufacture the test inputs that would expose a difference, and then to say plainly where it got stuck. The judgment of right-or-wrong is handed to a deterministic parity check between the two runtimes. The authors’ own framing of the contribution is “validating agentic coding output using a deterministic oracle”, and the oracle is the point.
The numbers, and what they don’t cover
Three case studies: two open-source COBOL programs and one internal production-like program, ranging from 430 to 4,114 source lines. Coverage improved past the plateau that plain input search reaches, hitting nearly complete coverage on the two open-source programs and 91.90% branch coverage on the internal one. The generated Java matched the COBOL reference under deterministic parity checks in all accepted test cases.
Two things about that I’d want answered before quoting it at anyone.
The first is that 91.90% is the number from the program most like real production work, and it’s the lowest of the three. The gap between “nearly complete” on open-source samples and 91.90% on something production-shaped is the gap where the interesting bugs live, because Locked Paragraphs are by definition the branches the method could not reach. A COBOL program that’s been in service for thirty years has accumulated exactly that kind of unreachable-looking branch, usually for a reason someone has since retired.
The second is “in all accepted test cases”. Accepted by what, and how many were not? An eleven-page paper with six figures is not going to carry that, and I’m not treating the parity result as stronger than its filter until I’ve seen it.
None of that is a knock on the work. It’s a well-shaped piece of engineering and the honest bit — naming the Locked Paragraph rather than papering over it — is the part I’d want in my own tooling.
Why this is the version worth reading
The reason the mistaken headline travels further than the paper is that “AI wrote buggy code” is a story people already have a slot for. “Researchers built a deterministic check so the AI’s output could be trusted at all” is a duller sentence and a far more useful one.
I’ve been chewing on a related idea for a while in a much smaller way: a check you have never seen fail tells you nothing when it passes. A test suite that goes green against code you already believe is correct hasn’t constrained anything. What the Locksmith Loop is doing is generating the inputs that would make the check fail if the migration were wrong, then reporting honestly about the branches where it couldn’t. That’s a different activity from testing, and it’s the activity that makes agentic output usable in a place where being wrong costs money.
If you take one thing from the paper into your own work, take the architecture rather than the tooling. The model generates candidates. A deterministic check, owned by you and incapable of being talked round, decides which ones survive.