AI Productivity by Skill Level

Different skill levels respond differently to AI coding tools—test before assuming

Chip Huyen
AI Engineering with Chip Huyen

AI Productivity by Skill Level

"The group that gets the biggest performance boost... the senior engineer, the highest performing. So the highest performing engineers get the biggest boost out of it. And then the second group is the average performing." - Chip Huyen

What It Is

Observed patterns of how engineers at different skill levels respond to AI coding tools. Rather than assuming uniform productivity gains, this framework highlights that AI adoption benefits vary dramatically based on existing skill level and attitude—and the patterns differ across companies.

The key insight: you must test in your organization because the dynamics vary, and the reasons why are illuminating.

How It Works

One Company's Findings (Randomized Trial):

A company divided their engineering team into three buckets (unbeknownst to the engineers):

  • Highest performing
  • Average performing
  • Lowest performing

Half of each group received Cursor access. Over time, they observed:

  1. Highest performers gained the most - They already have strong problem-solving skills and good engineering practices; AI amplifies their existing capabilities
  2. Average performers gained second-most - They benefit from AI assistance but may lack the judgment to optimize its use
  3. Lowest performers gained least - Either they "don't care much about work" and go on autopilot with AI, or they lack the skills to evaluate and improve AI output

The Counter-Pattern (Different Companies):

Some companies report senior engineers are the most resistant to AI tools because:

  • They're highly opinionated and have high standards
  • They find AI-generated code quality unacceptable
  • They've built efficient workflows over years that AI disrupts

How to Apply It

For Leaders:

  1. Run your own test - Don't assume uniform adoption; measure actual productivity changes by skill level
  2. Watch for resistance patterns - Senior resistance might indicate quality concerns worth investigating
  3. Consider restructuring - Some companies have shifted senior engineers to review/guideline roles, with juniors producing AI-assisted code

For IC Engineers:

  1. Assess honestly - Your skill level affects how much you'll gain from AI tools
  2. Develop judgment - The highest performers excel because they can evaluate and improve AI output
  3. Stay curious - Resistance often comes from valid quality concerns—investigate, don't dismiss

The Emerging Model:

"They get more senior engineers to be more in the peer review... They write a lot of processes on how to work well. And then they have more junior engineers just produce code and submit PRs, but senior engineers more in the reviewing case."

When to Use It

  • When planning AI tool rollout to engineering teams
  • When measuring AI productivity impact
  • When structuring engineering teams for AI-assisted development
  • When senior engineers resist AI adoption
  • When deciding how to train different skill levels on AI tools

Source

  • Guest: Chip Huyen
  • Episode: "AI Engineering with Chip Huyen"
  • Key Discussion: (00:46:28) - Randomized trial showing different AI gains by skill level
  • YouTube: Watch on YouTube

Related Frameworks

Back to Team Building