Shopify just revealed something that should make every product manager pause.
In a detailed breakdown of Shopify's AI-first engineering playbook, Farhan Thawar, VP & Head of Engineering, dropped a concept that deserves far more attention than it's getting: comprehension debt.
The idea is simple but profound. As engineers increasingly rely on AI to write code, they risk losing deep understanding of their own systems. "The brain is a muscle that you can't let atrophy," Farhan warns. His guardrail? Engineers must understand systems 2-3 layers below where they're working, using AI to accelerate learning—not replace it.
Here's the thing: this same risk applies to product teams. And for product managers, the stakes might actually be higher.
What Is Comprehension Debt?
Think of comprehension debt like technical debt, but for understanding. Technical debt accumulates when you take shortcuts in code that you'll pay for later. Comprehension debt accumulates when you take shortcuts in learning that you'll pay for later.
For engineers, it means shipping code generated by AI without truly understanding how it works. The system runs fine—until it breaks. And when it breaks, no one knows why.
For product managers, comprehension debt looks different. It's when you rely on AI summaries of customer feedback without developing genuine intuition for your users. The product ships fine—until it doesn't resonate. And when it doesn't resonate, you have no idea where you went wrong.
The Customer Research Shortcut Trap
Let's be honest about what's happening in product teams right now.
AI tools can now summarize hundreds of support tickets in seconds. They can analyze NPS surveys, extract themes from user interviews, and generate insights reports that would have taken a researcher weeks to produce.
This is genuinely powerful. It's also genuinely dangerous.
Here's the trap: When you read an AI-generated summary that says "customers are frustrated with onboarding complexity," you learn a fact. But you don't develop feel. You don't hear the specific words customers use, the emotional undertones in their complaints, the subtle differences between a power user struggling and a casual user giving up.
That feel—that intuition—is what separates PMs who build things people love from PMs who build things that technically address requirements.
The "Code Is Cheap" Parallel
Farhan made another observation that translates perfectly to product management: "Code is cheap now. But I don't want code, I want solutions."
The same applies to customer insights. Data is cheap now. But you don't want data, you want understanding.
You can collect more customer feedback than ever. You can analyze it faster than ever. But analysis isn't understanding. A summary isn't insight. And a list of feature requests isn't a product strategy.
The PMs who thrive in the AI era won't be the ones who process the most data. They'll be the ones who develop the deepest understanding of their customers—using AI to accelerate that understanding, not shortcut it.
Three Ways Product Teams Accumulate Comprehension Debt
1. Reading Summaries Instead of Source Material
AI can summarize 500 support tickets into five themes. Extremely useful. But if you only read the summary, you miss:
- The exact language customers use (which shapes how you communicate value)
- Edge cases that don't fit neatly into themes (which often signal emerging needs)
- Emotional intensity differences between similar complaints (which helps you prioritize)
- The context around complaints (which reveals root causes)
The summary is a map. Maps are useful. But you still need to walk the territory sometimes.
2. Delegating Customer Calls to Transcription
"I'll just have the AI transcribe and summarize the customer calls."
Yes, you should transcribe calls—it makes them searchable and shareable. But if your only interaction with customers is reading AI summaries of conversations, you're missing everything that doesn't translate to text:
- The pause before an answer (indicating uncertainty)
- The enthusiasm spike when discussing a feature (indicating delight)
- The tangent they keep returning to (indicating what actually matters to them)
- Their questions about your questions (indicating their mental model)
The best PMs I know still get on calls. Not every call—that doesn't scale. But enough calls to stay calibrated.
3. Trusting Sentiment Analysis Without Validation
AI sentiment analysis is impressive. It can tell you that 73% of feedback about your new feature is positive.
But sentiment analysis is a blunt instrument. It can miss sarcasm. It can weight a one-line "looks cool" the same as a three-paragraph analysis. It can categorize frustration with your company as frustration with your product.
When you make decisions based on sentiment scores without ever reading the underlying feedback, you're flying blind with instruments that aren't calibrated to your specific context.
The Shopify Lesson: Understand the Layers Below
Farhan's solution for engineers is elegant: understand systems 2-3 layers below where you're working.
For product managers, the equivalent might be: understand customer needs 2-3 layers below the surface request.
- Layer 1: "I want a dark mode" (the feature request)
- Layer 2: "I work late at night and the bright screen hurts my eyes" (the use case)
- Layer 3: "I'm a freelance developer shipping code at 2 AM because that's when I have uninterrupted time, and I'm building my own startup while working a day job" (the life context)
AI can easily give you Layer 1. It can often surface Layer 2. But Layer 3—the deep context that shapes how people actually use your product—requires the kind of understanding that only comes from genuine engagement.
That understanding is what lets you build features that feel like magic instead of features that check boxes.
How to Use AI Without Accumulating Comprehension Debt
The solution isn't to avoid AI tools. That would be like Shopify engineers refusing to use GitHub Copilot. The productivity gains are too significant to ignore.
Instead, treat AI as what Shopify calls a way to "accelerate learning, not replace it."
Use AI for Coverage, Not Comprehension
Let AI process the full breadth of customer feedback. Use it to surface patterns you might miss, to quantify themes, to track sentiment over time. That's coverage—making sure nothing falls through the cracks.
But for comprehension, go deeper on fewer things. Pick representative examples from each theme and read them in full. Watch actual session recordings instead of just reading AI highlights. Get on calls with customers who represent important segments.
Make AI Reveal Complexity, Not Hide It
The default AI behavior is to simplify. "Here are the top 5 themes." That's useful, but it can create false confidence.
Better prompts reveal complexity: "What are the themes that appear frequently? What are the themes that appear rarely but with high emotional intensity? What contradictory things are customers saying? Where does the feedback cluster around multiple possible root causes?"
Use AI to surface the messiness, not smooth it over.
Build Calibration Rituals
Just like Shopify engineers should periodically work in lower-level code to stay sharp, product managers need rituals that keep them calibrated to customer reality.
- Weekly deep-dive: Pick one piece of feedback and trace it all the way through—the user's history, their full conversation, their product behavior.
- Monthly customer calls: Not discovery calls with an agenda, just conversations to hear how people talk about their work.
- Quarterly support shadowing: Sit with your support team and watch real-time customer interactions.
These rituals ensure that when you read AI summaries, you have the context to interpret them correctly.
The Competitive Advantage of Deep Understanding
Here's the counterintuitive reality: As AI makes surface-level customer analysis trivially easy, deep customer understanding becomes a bigger competitive advantage, not a smaller one.
Everyone will have AI summaries of their feedback. Everyone will have sentiment dashboards. Everyone will be able to identify the top feature requests.
What separates companies will be the ones whose product teams actually understand their customers. Who can spot the non-obvious need. Who can predict how a change will land before shipping it. Who can write marketing copy that sounds like how customers actually talk.
That understanding can be accelerated by AI. But it can't be replaced by AI.
The Real Question
The Shopify playbook makes clear that their 20% productivity gains come from engineers who use AI to amplify their skills, not replace their thinking.
The same opportunity exists for product teams. AI can make you dramatically more effective at understanding customers—if you use it right. Or it can create a comfortable illusion of understanding while your actual customer intuition atrophies.
The real question isn't whether to use AI for customer insights. It's whether you're using it to accelerate learning or replace it.
One path leads to products that resonate. The other leads to products that technically address requirements.
Your customers will know the difference—even if your metrics don't show it until it's too late.
Building a product and want to stay close to your customers without drowning in data? Pelin helps product teams synthesize customer feedback while maintaining the depth of understanding that AI summaries alone can't provide. See how it works.