AI code migrations: what Airbnb, Google and Amazon actually reported

Large code migrations are the kind of work every engineering organisation postpones: a test framework nobody maintains any more, a language version that is about to lose support, an ID type that is running out of room. They are repetitive, risky and boring, which makes them the obvious first job for LLMs and coding agents.

Three companies have published detailed accounts: Airbnb, Google and Amazon. Their headline numbers travel well on social media. This post looks at what each one actually reported, what kind of evidence it is, and what that means for planning your own migration.

The short answer: all three show that AI-assisted migration works at scale when every change can be checked automatically by tests, compilers or type checkers. None of them gives you a percentage you can safely put in your own plan: the numbers are self-reported estimates, they measure different things, and they come from companies with unusual tooling.

The three cases at a glance

Airbnb: a pipeline, retries and the right context

Airbnb's write-up is the most practical of the three. The team had nearly 3,500 React component test files written with Enzyme to move to React Testing Library. Deleting the tests was not an option because of the coverage they provided, and a find-and-replace could not bridge two frameworks that work differently.

They built a pipeline in which each file moves through fixed steps: refactor the test, fix Jest failures, fix lint and TypeScript errors, mark as done. A file only advances when the check for its current step passes. When a check fails, the model gets the error and the latest version of the file and tries again.

Three of their lessons are worth copying:

• Retries beat prompt polishing. Most files succeeded within 10 attempts; the hardest were retried 50 to 100 times. Brute-force retries with fresh error messages did more than prompt experiments.
• Context is the real prompt engineering. Prompts grew to 40,000 to 100,000 tokens, with up to 50 related files, sibling tests and hand-written examples. Choosing the right related files mattered more than wording.
• Breadth first. Their "sample, tune, sweep" loop picked 5 to 10 representative failing files, improved prompts and scripts on them, then re-ran everything, rather than getting stuck on the hardest file.

Google: deterministic tools first, the model second

Google published two papers. The first (Nikolov et al. 2025) is explicitly an experience report: the authors write that it is not a research study and makes no comparison with other approaches. The second (Ziftci et al. 2025) is a case study of 39 migrations carried out by three developers over twelve months.

The numbers are often quoted together, but each measures something different:

Google's lessons are less about prompts and more about engineering:

• LLMs alone are not enough. Simple prompting only works for the simplest migrations. What worked combined AST-based tools, heuristics and the model, with build and test checks and a repair loop.
• Deterministic tools as a safety net. AST tools are predictable and can catch unwanted edits by comparing syntax trees before and after a change. Smaller, well-defined prompts were more reliable than ambitious ones.
• Review is the bottleneck. For the JUnit migration, reviewers could not keep up, so the team deliberately limited how many changes it produced each week.
• Measure end to end. The authors argue for measuring total time saved, including review and landing, rather than the share of AI-generated code, and note that the long-term effect on code quality is unknown.

Amazon: the biggest numbers, the thinnest detail

In August 2024 Andy Jassy wrote that upgrading an application to Java 17 used to take about 50 developer-days and now took a few hours with Amazon Q Developer. The AWS DevOps blog adds the scale: tens of thousands of production applications moved from Java 8 or 11 to Java 17, an estimated 4,500 years of development work saved, and $260 million a year in savings.

Two details matter. First, the $260 million is not developer time: Amazon estimated it from the servers it could remove thanks to Java 17's better performance. It is a benefit of being on a newer runtime, which a manual upgrade would also have delivered, only later. Second, both figures are described by the post itself as estimates, published by the company that sells the tool.

How much weight does each case deserve?

• They are all self-reported. No control group, no independent replication. Companies publish the migrations that went well.
• Two of the three come from tool builders. Google builds its own migration tooling; Amazon sells the product it reports on. Airbnb has no product to sell, which makes its account the most neutral.
• They measure different things. Engineering effort against elapsed time (Airbnb), share of code written by the model (Google), estimated time saved (Google, Amazon). Do not put them on one chart or average them.
• The conditions were favourable. Each migration had a clear pattern and an automatic check for every change: tests, compilers, type checkers, syntax trees.

Put together, this is strong evidence that AI-assisted migration works at scale under those conditions, and weak evidence for any particular percentage in your codebase. Controlled studies of AI coding tools on everyday work are more mixed, from faster to slower; what the controlled studies say about AI developer productivity covers them.

What the three have in common

A decision rule for your next migration

1. Check that every change can be verified automatically. If the code under migration has weak tests, build that safety net first. Without it, the model's output cannot be trusted at scale, and neither can a human's.
2. Start with a sample, not the whole codebase. Run the pipeline on a few dozen files, fix what fails often, then sweep the rest. Repeat.
3. Budget for the tail. In Airbnb's case the last quarter of the files took most of the effort. Plan people for it from the start.
4. Plan reviewer capacity. Google throttled its own output to match reviewers. Agree with code owners how many changes a week they can take.
5. Measure end to end. Track total elapsed time and engineering hours including review, defects found after merge, and rollbacks. The share of code written by the model is not the outcome.

To turn this into numbers for your own migration, the same structure as A GraphRAG business case, done the evidence-based way applies: a cost model, a value range instead of a single figure, and a pilot with a go or no-go threshold agreed in advance.

All three cases describe pipelines around a model, or a managed upgrade tool. Coding agents that plan and edit across many files on their own change the picture again, in control, review and cost. That is a topic for a separate post, with numbers from our own agent-driven rewrites.

Whatever the pipeline, the controls are the ones any agent in production needs: scoped access, drafts by default, tested rollback and measured reliability. They are covered in Agentic AI from demo to production.

Sources

• Airbnb Engineering (2025). Accelerating Large-Scale Test Migration with LLMs.
• Nikolov et al. (2025). How is Google using AI for internal code migrations? arXiv:2501.06972.
• Ziftci et al. (2025). Migrating Code At Scale With LLMs At Google. arXiv:2504.09691.
• AWS DevOps Blog (August 2024). Amazon Q Developer just reached a $260 million dollar milestone.