What Actually Improves AI Apps

"A question that gets asked a lot is, 'How do we keep up to date with the latest AI news?' Why? Why do you need to keep up to date with the latest AI news? If you talk to the users, understand what they want or don't want, look into the feedback, then you can actually improve the application way, way, way more." - Chip Huyen

What It Is

A reality check on where AI teams should invest their time. While the AI ecosystem generates constant hype about new frameworks, models, and techniques, the most impactful improvements to AI applications come from fundamental product development practices—not from chasing the latest news.

This framework contrasts what people think improves AI apps (technology-focused activities) versus what actually improves them (user-focused activities).

How It Works

What People Think Improves AI Apps:

1. Staying up to date with the latest AI news
2. Adopting the newest agentic framework
3. Agonizing about what vector databases to use
4. Constantly evaluating which model is smarter
5. Fine-tuning a model

What Actually Improves AI Apps:

1. Talking to users
2. Building more reliable platforms
3. Preparing better data
4. Optimizing end-to-end workflows
5. Writing better prompts

The contrast reveals a common trap: teams get distracted by technological possibilities while ignoring the fundamentals that drive user value.

How to Apply It

1. Audit your team's time allocation - Track how much time goes to evaluating new technologies versus improving data quality and understanding users
2. Ask two questions before adopting new tech - "How much improvement could I get from optimal vs. non-optimal solutions?" and "If I adopt this, how hard would it be to switch later?"
3. Invest in data preparation - The biggest performance gains in RAG solutions come from better data preparation, not infrastructure choices
4. Talk to actual users - Understanding what users want or don't want has more impact than knowing which model is 2% better on benchmarks
5. Resist premature optimization - Don't over-commit to new, untested technologies that lock you in

When to Use It

• When planning AI product development priorities
• When your team is debating technology choices
• When feeling pressure to adopt the latest AI trends
• When deciding between building features vs. improving foundations
• When setting AI strategy and resource allocation

Source

• Guest: Chip Huyen
• Episode: "AI Engineering with Chip Huyen"
• Key Discussion: (00:05:30) - The viral LinkedIn post on what actually improves AI apps
• YouTube: Watch on YouTube

Related Frameworks

• Evals as PRD - Use evals to guide product development
• Problem-First Approach - Start with the problem, not the technology