High conversation volume means nothing without context. A chatbot handling 10,000 conversations where 25% end in frustration isn't performing well—it's creating problems at scale.
This guide covers the metrics that matter, how to review them effectively, and how to turn data into improvements.
The Five Metrics That Matter
1. Resolution Rate
What it measures: Percentage of conversations where the customer's problem was solved without human help.
Benchmarks:
| Rate | Assessment |
|---|---|
| 50-70% | Acceptable |
| 70-85% | Good |
| 85%+ | Excellent |
How to improve: Read failed conversations. Find patterns. Add content to fill gaps.
2. Customer Satisfaction (CSAT)
What it measures: How customers feel after chatting with the bot.
Benchmarks:
| Score | Assessment |
|---|---|
| 3.5/5 | Acceptable |
| 4.0/5 | Good |
| 4.5/5 | Excellent |
Warning sign: High resolution but low satisfaction means the bot is "technically" answering but in an unhelpful or frustrating way.
3. Deflection Rate
What it measures: Percentage of potential support tickets the bot handled instead.
Value formula:
Monthly Savings = Deflected Conversations × Cost Per Ticket
Example: Cost per ticket $18. Bot deflects 800 conversations/month. Savings = $14,400/month.
4. Top Failed Topics
What it measures: What questions is the bot failing to answer?
This tells you exactly what content to create.
5. Leads Captured
What it measures: How many leads the chatbot generates from conversations.
Track both volume and conversion rate from chat to lead.
The Weekly Analytics Ritual (25 Minutes)
Minutes 0-5: The Anomaly Check
Compare this week vs. last week:
- Conversation volume: Did it spike or drop significantly?
- Resolution rate: Did it change more than 5%?
- Satisfaction score: Did it change more than 0.3 points?
If anything's off, investigate. If not, continue.
Minutes 5-15: Failed Conversation Review
This is the most valuable part.
Filter conversations by "Unresolved" or "Escalated."
Read 5-10 failed conversations. For each one, note:
- What was the customer trying to do?
- Why did the bot fail?
- Is this fixable?
Example review:
| Conversation | Customer Need | Why Bot Failed | Action |
|---|---|---|---|
| #1 | Cancel subscription | Didn't know how | Add cancellation flow |
| #2 | Check order status | Needs real-time data | Appropriate escalation |
| #3 | Discount for annual | Info not on website | Add annual pricing |
| #4 | Integration question | Not documented | Add integrations page |
| #5 | Account locked | Security issue | Appropriate escalation |
Result: 3 actionable improvements, 2 appropriate escalations.
Minutes 15-20: Business Metrics Check
- Leads captured this week: Compare to last week
- Estimated ticket deflection: Bot-resolved conversations × cost per ticket
- Notable wins: Any particularly good conversations worth noting
Track in a simple spreadsheet:
| Week | Conversations | Resolution Rate | Leads | Deflection Value |
|---|---|---|---|---|
| Week 1 | 847 | 78% | 47 | $11,898 |
| Week 2 | 912 | 81% | 52 | $14,746 |
| Week 3 | 889 | 80% | 49 | $14,224 |
Trend lines matter more than any single week.
Minutes 20-25: Action Item Review
- Did last week's improvements work?
- What's still in progress?
- What are this week's top 3 actions?
Keep a running log:
| Date | Issue | Action Taken | Result |
|---|---|---|---|
| Week 1 | Shipping questions failing | Added shipping FAQ | +12% resolution on shipping |
| Week 2 | Annual discount unknown | Updated pricing page | Bot handles correctly now |
| Week 3 | Integration questions | In progress—writing docs | TBD |
Common Analytics Mistakes
Tracking Too Much
20+ metrics means nobody looks at most of them. Focus on 5 that drive decisions:
- Resolution rate
- CSAT
- Leads captured
- Failed topics
- Deflection value
No Baseline
"Resolution rate is 74%. Is that good?" Without a baseline, you can't know.
Establish baseline in week 1. Compare everything to that.
Monthly Reviews Only
Monthly is too slow. Problems compound for 30 days before you notice. Weekly catches issues before they become trends.
Vanity Metrics
"10,000 messages!" means nothing. "78% of those messages resulted in resolved issues" means something.
Focus on outcomes, not activity.
No Action Loop
Analytics without action is wasted time. Every metric should have an action threshold:
| Metric | Threshold | Action |
|---|---|---|
| Resolution drops 10%+ | Alert | Investigate immediately |
| New failed topic appears | Weekly | Add content for topic |
| CSAT drops 0.5+ | Alert | Review recent conversations |
| Leads drop 30%+ | Alert | Check widget/trigger issues |
The Improvement Cycle
Data → Insight → Action → Measurement
Week 1: Resolution rate 74%. Review failed conversations. Discover 15% of failures are about "enterprise pricing" (not on website).
Week 2: Add enterprise pricing page. Train bot on enterprise content.
Week 3: Check resolution rate. Now 79%. Enterprise pricing failures dropped 80%.
Week 4: Find next gap. Repeat.
This cycle, done consistently, compounds. 2% better each week = 100% better in a year.
Setting Up Alerts
Configure automatic notifications:
Critical Alerts (immediate):
- Conversations drop 50%+ (something's broken)
- Resolution rate drops below 60%
- CSAT drops below 3.5
Weekly Summary:
- Conversation volume
- Resolution rate
- Top failed topics
- Leads captured
Dashboard Configuration
Overview Tab:
- Conversations this week (vs. last week)
- Resolution rate trend (last 4 weeks)
- CSAT trend (last 4 weeks)
- Leads captured
Performance Tab:
- Top 10 topics (by volume)
- Bottom 10 topics (by resolution rate)
- Failed conversations list
Business Tab:
- Deflection value (calculated)
- Lead source breakdown
- Conversion rate from chat
Export CSV monthly for deeper analysis.
Getting Started
Start free with Kya to start measuring effectively:
- Week 1: Run normally. Establish baseline metrics.
- Week 2: Start Friday review ritual. Note top 3 failed topics.
- Week 3: Add content for those topics. Track improvement.
- Week 4+: Continue the cycle. Get better each week.
The compound effect of weekly improvements is significant. Small fixes, consistently applied, transform performance.
Numbers without action are just numbers.
A chatbot that gets 2% better every week will be unrecognizable in a year.
One that never reviews analytics will still be frustrating customers with the same gaps.
Twenty-five minutes. Every week.
That's the difference.
