Researchers at Microsoft studied the use of command-line AI coding tools and their effects on productivity among tens of thousands of the company’s software engineers. The tools were Anthropic’s Claude Code and GitHub’s Copilot CLI, and the study ran for about four months, from January 5 to April 29, 2026. Engineers who used a tool during its first week had an average of 24% more merged pull requests than those who did not, the researchers found.
Simply providing tools won’t make them catch on—the first step comes from “the desk next door”
Some executives and managers may imagine that if a company makes an AI tool available to everyone, they will all start using it the next day. But this study suggests that workplace relationships—not explanations of the tool’s features or announcements—played a greater role in whether people tried it in the first place.
People were more likely to start using a tool when coworkers they regularly shared code with or their direct manager used it. The figures are specific: when more than a quarter of colleagues who shared the same manager two levels up had used a tool in the previous two weeks, the odds of trying it rose by as much as 216%. The odds rose by 82% when a direct manager used it, and by 54% when more than a quarter of the people an engineer regularly exchanged code reviews with had used it.
An increase in odds is not the same as an “increase in probability”
It is important to be careful with the word “odds.” Odds express the likelihood of an event as a ratio of the chance it will happen to the chance it will not. A 216% increase may sound as though the number of people using a tool more than tripled, but it does not mean the share of users increased by 216%. When usage is low to begin with, differences can look large, and the actual growth in the number of users will be smaller than that figure.
Still, the overall pattern is clear: the actions of managers and coworkers can encourage people to try a tool. People often start using a new tool not by reading the manual, but by thinking, “If that person is using it, maybe I should too.” Personally, I think this is familiar in many workplaces, not just software development. A manager tries a tool first and shows how they use it during a review. That kind of low-key effort may be more effective than training materials.
Whether people keep using a tool depends on their everyday workflow
Whether someone who tries a tool goes on to use it regularly is another matter. The study found that continued use was linked to a person’s usual level of coding activity. Engineers who wrote a lot of code and frequently submitted pull requests were more likely to feel the benefits of AI and make the tool part of their routine, the researchers said.
A pull request is a request to have rewritten code incorporated into the main codebase. People who submit them frequently may also have more work that they can hand off to AI. The more hands-on work they do, the easier it is to experience the tool’s effects.
A familiar environment can make it harder to adopt a new tool
The study also found that people who had already been using other AI tools in a different development interface were willing to try the new tools but less likely to stick with them. One possibility is that many people simply feel more comfortable in an environment they already know.
It is a bit like switching smartphones. Even if a new phone has better features, you may want to go back to your old one if you are used to its app layout and quirks. Those introducing new AI tools may need to consider not just their performance, but also how they fit into the workflows users have already established.
How to interpret the 24% increase in pull requests
The impact on productivity was measured by the number of merged pull requests, a proxy for the amount of work completed. Engineers who used a tool during its first week had an average of 24% more merged pull requests than those who did not. The researchers reported that the effect was stable throughout the four-month study period, rather than being a temporary result of the initial excitement around the rollout.
However, productivity here is measured by just one metric: the number of pull requests. The figure does not show that code quality, the time spent on each pull request or the burden on teams improved by the same percentage. It is worth separating the fact that the number went up from the feeling that work as a whole became easier.
Claude Code: 11.4%; Copilot CLI: 24.9%
By tool, the increase in pull requests compared with weeks when the tool was not used was 11.4% for Claude Code users and 24.9% for Copilot CLI users. That is a roughly twofold difference, but it would be premature to read these figures as a direct ranking of the tools. Copilot CLI is offered by GitHub, which is part of Microsoft, and the two tools may not have been used in the same ways or by the same kinds of users within the company.
Even so, few studies have tracked both the spread of use and its outcomes over four months in real-world development settings involving tens of thousands of people. As discussions of AI coding tend to focus on “which model is smarter,” these findings offer a look at the realities of adoption: who starts using the tools and who keeps using them. We may be entering an era in which the ability to shape how tools spread in the workplace affects outcomes just as much as selecting the tools themselves.
Comments