Feedback Analysis Refinement (Phase 6)¶

Goal¶

Improve usability of feature request analysis by: 1. Reducing "Other" category - Currently 35% of requests fall into catch-all "Other" theme 2. Enabling drill-down - See individual requests under each topic with citations 3. Consolidating outputs - One doc per source instead of 4 separate docs

Problem Analysis¶

Current State¶

Source	Requests	Has `topic` field	"Other" in themes
Slack	144	✓ (141 unique)	51 (35%)
Intercom	708	✗ (undefined)	257 (36%)

Root cause of "Other" bloat: Summary scripts use regex theme patterns. Requests not matching any pattern → "Other".

Gap: Intercom extraction didn't include topic field. Slack has it (141 unique values like integration.jira.sync, story_map.keyboard_shortcuts), Intercom doesn't.

Why Regex Themes Failed¶

Regex patterns are rigid: - /jira/i catches "Jira sync", misses "issue tracker integration" - /export/i catches "export to PDF", misses "generate document" - Long tail of unique phrasings always slips through

Solution: Use LLM-assigned topic for grouping instead of regex themes.

Step 1: Add Topic to Intercom (LLM Post-Processing)¶

Goal: Assign topic field to all 708 Intercom requests to match Slack's granularity.

Method: Subagent batch processing (same pattern as original extraction)

Input: feature_requests.json (708 requests with product_area but no topic)

Process: 1. Create topic-enrichment-prompt.md with instructions 2. Split 708 requests into batches of 20 (~35 batches) 3. For each batch: subagent reads requests, assigns topics, writes to topic-batches/batch_NNN.json 4. Merge all batches → updated feature_requests.json

Topic format: {product_area}.{specific_topic} - Examples: integration.jira.sync, account.cancel.self_service, story_map.keyboard_shortcuts - Granular enough to group similar requests - Hierarchical for aggregation at different levels

Resumability: - Deterministic batch assignment: batch N = requests [(N-1)20 to N20-1] - Check existing batch files, continue from next - Progress log: topic-enrichment-log.txt

Step 2: Generate Consolidated Documents¶

Goal: Replace 4 separate docs per source with 1 consolidated doc containing all information.

Output structure:

# [Source] Feature Requests

## Summary
- Total requests: X
- By type: feature/bug/feedback breakdown
- By confidence: high/medium/low
- Date range: YYYY-MM to YYYY-MM
- Explicit vs implicit

## Topics (sorted by request count)

### 1. integration.jira.sync (24 requests)

| Date | Type | Description | Verbatim | Customer | Source |
|------|------|-------------|----------|----------|--------|
| 2025-03-04 | explicit | Two-way sync with Jira | "We need real-time sync..." | john@acme.com | [view](path/to/file.md) |
| 2025-03-12 | implicit | Sync delays frustrating | "Changes take hours to appear" | jane@corp.com | [view](path/to/file.md) |

### 2. story_map.keyboard_shortcuts (18 requests)
...

## Trends

### Monthly Volume
| Month | Total | Feature | Bug | Feedback |
|-------|-------|---------|-----|----------|
| 2025-01 | 50 | 42 | 6 | 2 |
...

### Topic Trends by Month
[Matrix of topics × months]

### Customer Segmentation
[By plan, by source channel]

Key features: - Topics sorted by request count (most requested first) - Each topic expands to show all individual requests - Each request has citation link to source file - Trends section preserved from current docs

Files¶

Inputs¶

feature_requests.json - Intercom requests (708, no topic)
slack_feature_requests.json - Slack requests (144, has topic)

Outputs (Step 1)¶

topic-enrichment-prompt.md - Instructions for topic assignment
topic-batches/batch_NNN.json - Batch results (35 files)
topic-enrichment-log.txt - Progress log
feature_requests.json - Updated with topic field

Outputs (Step 2)¶

intercom-feature-requests.md - Consolidated Intercom doc
slack-feature-requests.md - Consolidated Slack doc

Deleted (replaced by consolidated docs)¶

The following legacy files were deleted on 2026-01-20: - feature-request-catalog.md - feature-request-summary.md - feature-request-trends.md - slack-feature-request-catalog.md - slack-feature-request-summary.md - slack-feature-request-trends.md

Implementation Notes¶

Subagent Pattern (from Phase 3)¶

// Subagent receives:
// 1. Path to prompt file (instructions)
// 2. Batch of items to process
// 3. Output file path

// Subagent does:
// 1. Read prompt file
// 2. Process each item
// 3. Write results to output file

// Main context does:
// 1. Track which batches are complete
// 2. Resume from last incomplete batch
// 3. Merge all batches at end

Topic Assignment Guidelines¶

Topics should be: - Specific: integration.jira.field_mapping not just integration.jira - Consistent: Same feature → same topic across requests - Hierarchical: Enable aggregation at L1 (integration) or L2 (integration.jira)

Examples: | Description | Product Area | Topic | |-------------|--------------|-------| | Two-way sync with Jira | integration | integration.jira.sync | | Self-service cancellation | account | account.subscription.cancel | | Keyboard shortcuts for map | story_map | story_map.navigation.keyboard | | Export to PDF with images | export_import | export_import.pdf.images | | AI generates wrong stories | ai | ai.generation.quality |

Session Log¶

Analyzed current outputs and identified improvements needed:

Problems identified: 1. "Other" category too large (35% of requests in both sources) 2. Regex theme matching is rigid, misses synonyms 3. Intercom missing topic field that Slack has 4. 4 separate docs per source is fragmented

Solution designed: 1. Add topic to Intercom via LLM post-processing 2. Generate consolidated docs with drill-down capability 3. Drop regex themes, use LLM-assigned topics for grouping

Process documented following Phase 3 patterns: - Prompt in file - Batch processing with subagents - Deterministic batching for resumability - Progress logging

Next: Create topic enrichment prompt, run on Intercom data.

2026-01-20 08:06 UTC - Phase 6 Complete¶

Executed full refinement process:

Step 1: Topic Enrichment (Intercom) - Created topic-enrichment-prompt.md with guidelines and Slack topic examples for consistency - Prepared 36 input batches (20 requests each) - Ran 36 subagents in parallel to assign topics - Merged into updated feature_requests.json

Results: - 708 requests enriched with topics - 487 unique topics (vs 141 for Slack's 144 requests) - Top topic: account.subscription.cancel (30 requests)

Step 2: Consolidated Docs - Created generate-consolidated-doc.mjs script - Generated intercom-feature-requests.md (3,271 lines) - Generated slack-feature-requests.md (1,016 lines)

Doc structure: 1. Summary (counts, metrics, by type, by area) 2. Topics grouped by L1 category, sorted by count - Each topic has drill-down table with all requests - Each request has: type, description, verbatim, customer, source link 3. Trends (monthly volume, top topics by month)

Comparison - Topic distribution:

Source	Requests	Unique Topics	Top Topic
Intercom	708	487	account.subscription.cancel (30)
Slack	144	141	editor.formatting (2)

Note: Intercom has more topic clustering (708/487 = 1.45 requests/topic) because multiple customers asked for same features. Slack has almost 1:1 (144/141) because staff-curated posts are unique summaries.

Files created: - topic-enrichment-prompt.md - docs/feedback-research/scripts/prepare-topic-batches.mjs - docs/feedback-research/scripts/merge-topic-batches.mjs - docs/feedback-research/scripts/generate-consolidated-doc.mjs - topic-batches/input_*.json (36 input files) - topic-batches/batch_*.json (36 output files) - intercom-feature-requests.md (consolidated) - slack-feature-requests.md (consolidated)

Phase 6 refinement complete.

Added usability improvements for publishing:

1. Customer Segmentation (Intercom only) - Added "By Customer Plan" section showing request distribution across Trial/Standard/Plus/Pro/Basic - Shows top categories per plan (e.g., Trial users focus on account issues, Standard on integrations) - Slack doesn't have plan data in source files, so no segmentation there

2. Topic Index with Descriptions - Added summary table at top of Topics section with all topics - Each row: Category | Topic | Count | Description | Link - Descriptions use shortest request description as representative (truncated to 60 chars) - Anchor links for navigation ([→](#topic-anchor))

3. Removed Collapsible Approach - Initially tried pymdownx.details (collapsible sections) but didn't work well for Google Docs copy-paste - Reverted to plain markdown with anchor links - more portable

Script: docs/feedback-research/scripts/transform-feature-requests.mjs handles all transformations: - Loads topic batches and builds descriptions - Generates customer segmentation from contact data - Creates topic index with anchor links - Can be re-run to regenerate both files

4. Cleanup - Deleted 6 legacy files (old 4-file format per source) - Updated mkdocs.yml to only include new consolidated files

Final deliverables: - intercom-feature-requests.md - 708 requests, 487 topics, with segmentation - slack-feature-requests.md - 144 requests, 141 topics

How to regenerate:

# If topic assignments change:
node docs/feedback-research/scripts/merge-topic-batches.mjs

# To regenerate consolidated docs:
node docs/feedback-research/scripts/generate-consolidated-doc.mjs

# To add segmentation and topic index:
node docs/feedback-research/scripts/transform-feature-requests.mjs

See README.md → Publishing Documentation for deployment instructions.