GitHub Copilot

When I told my team of 15+ engineers that we were rolling out AI coding assistants, the reactions fell into three camps: enthusiastic adoption, cautious skepticism, and quiet resistance from people who’d seen too many tools come and go. Six months in, the results are mixed, interesting, and — honestly — not what I expected.

Here’s what actually changed, what didn’t, and what I’d do differently if I were starting the rollout today.

What the data says (and doesn’t say)

Before we rolled out, I wanted to ground our expectations in real numbers, not marketing claims. The research landscape is surprisingly rich right now:

• A randomized controlled trial across Microsoft, Accenture, and a Fortune 100 company (4,867 developers combined) found a 26% increase in completed tasks among developers using AI coding assistants, with less experienced developers seeing the biggest gains [Microsoft Research, 2025].
• A longitudinal study at a tech consulting firm (3 agile teams, 13 months) found a 59% increase in completed story points with the same number of commits — meaning developers delivered more value per line of code, not more lines [ACM 2026].
• But an enterprise field study at Okta found no immediate increase in pull requests or lines of code after Copilot adoption, despite improvements in engineer motivation and reduced working hours — what the authors call the “efficiency-throughput gap” [CACM 2025].

So the answer to “does AI make teams faster?” is: it depends on what you measure, and how long you wait. Individual efficiency goes up immediately. Team throughput takes longer — and only if you’ve fixed the bottlenecks around testing, code review, and deployment.

What actually changed on our team

Here are the concrete changes I observed after our rollout:

1. Junior engineers leveled up faster — and senior engineers had more to review

The RCT data matches what I saw: junior and mid-level engineers picked up the tools quickly and produced more output. That’s not surprising. What surprised me was the effect on senior engineers — they spent more time reviewing AI-generated code than writing it themselves. This created an unexpected bottleneck: our senior engineers, who were our capacity constraint, became the review bottleneck.

Lesson: If you roll out AI tools unevenly (some team members adopt, others don’t), you create coordination problems. One study of 556 teams found that exposure to an AI-adopting team member reduced merged PRs by 3.6 per month — because the rest of the team had to adapt to different contribution patterns [Asgari et al., SSRN 2026]. We mitigated this by rolling out to everyone at once, not in phases.

2. Code quality went up on simple tasks, down on complex ones

AI-generated code for boilerplate, CRUD operations, and straightforward utilities was good — often better than what junior devs would write. But for complex business logic, edge cases, and integration points, the generated code often introduced subtle bugs that took longer to find than writing the code from scratch would have taken.

A BCG survey of 100 CIOs and 300 engineers found that 5-10% cost savings and ~15% performance improvement are the early averages, but value is “spotty and limited to pockets” [BCG, April 2025]. This matched our experience exactly: some tasks got faster, others got messier.

3. Code review became harder, not easier

This was the biggest shock. Before AI, I knew what to look for in a code review — familiar patterns, known anti-patterns. With AI-generated code, the patterns are different. The code compiles and passes basic tests, but the architecture decisions embedded in the generated code are often wrong for our specific context.

One engineer put it to me: “It’s still faster to write correct code than to review LLM code and fix its problems.” That sentiment was common enough that we had to adjust our review process, not just our expectations [Orosz, Pragmatic Engineer 2025].

4. Our “toil paradox” was real

BCG identified this clearly: AI automates the “fun” coding tasks but leaves testing, security, and requirements gathering as bottlenecks — the parts of the job that actually take most of the time [BCG, April 2025]. On our team, we saw this immediately. Writing code got faster. Writing tests, thinking through edge cases, and understanding the requirements didn’t get faster at all. Those became our new constraints.

What I’d do differently

If I were starting this rollout over, here’s what I’d change:

1. Roll out to everyone at once. Phased rollouts create the coordination problems the research shows. If one person on the team is writing AI code and the rest aren’t, the review burden falls unevenly.
2. Invest in test generation and security scanning first. AI makes writing code faster. If your testing and security pipelines aren’t ready to handle 2x the output, you’ll just generate more technical debt faster.
3. Train senior engineers on reviewing AI code, not just writing it. This was the unexpected bottleneck. Our senior engineers needed a different skill set — reading generated code, spotting subtle architectural mismatches, and knowing when to reject AI output entirely.
4. Set expectations around the efficiency-throughput gap. Individual productivity goes up immediately. Team throughput takes months to materialize, and only if you fix the downstream bottlenecks. Communicate this to leadership.
5. Measure the right things. The SPACE framework (Performance, Efficiency, Activity, Compliance, Satisfaction) is worth using. Looking at just “lines of code” or “commits” misses the real story. One study showed a 59% increase in story points with no change in committed lines — the value density increased, not the volume [ACM 2026].

What I’d tell another manager considering AI adoption

It works. But it works differently than the marketing suggests. The gains are real, the data backs them up, but they’re not uniform and they’re not instant at the team level. The biggest challenge isn’t the tool — it’s the organizational change. Only 21% of companies report widespread adoption (>75% of developers using it regularly), and 50% of CIOs say unclear ROI is their top challenge [BCG, April 2025].

If you’re going to do it, do it at scale, fix your testing and review pipelines first, and be honest with leadership about the timeline from individual efficiency to team throughput.

Question for fellow managers: How are you measuring AI’s impact on your teams? Are you tracking individual productivity, team throughput, or something else? I’d love to compare notes.

---

Sources: Microsoft Research field experiments (4,867 developers, 2025); ACM 2026 longitudinal study (3 agile teams, 13 months); CACM 2025 Okta field study; BCG State of GenAI Across Engineering Survey (100 CIOs/CTOs + 300 engineers, January 2025); Asgari et al. SSRN (556 teams, 2026); Gergely Orosz, Pragmatic Engineer (2025). Full links inline throughout.