Automate the Busywork, Not the Judgment

Overview

Customer chat transcripts are one of the richest sources of unmoderated customer voice: every conversation is a real person trying to do something on the site, in their own words. But the data was almost unusable at scale. A single export could be a hundred or more files, and pulling the chats for one page meant hours of manual filtering and lookups. I built a tool that turned that hours-long job into minutes, which changed what kind of research my team could even offer. The tool is the centerpiece; the story around it is a real impact it unlocked, a prototype for scaling it further, and a deliberate line I held about what AI should and should not do in research.

The problem

When my team exported chat data, we got two kinds of files: metadata about each conversation, including which page it happened on, and the transcripts themselves, one row per message. A single export could run to more than a hundred files. To pull the chats for a specific page, I had to filter the metadata, gather conversation IDs, and run lookups across every transcript file to find them. Two weeks of data, roughly fifteen or sixteen files, took me eight to ten hours of hands-on work. The consequence was quiet but real: when a stakeholder asked for chat research on a page, I had to weigh whether the answer was worth the effort, and on larger datasets the honest answer was often no, or a heavily sampled version that traded rigor for feasibility. That is the kind of constraint that shapes what a team can offer without anyone naming it.

Where it started

The tool grew out of a specific ask. A senior product partner needed chat data about the credit-check step in the purchase flow, where drop-off was significant and the team suspected confusion about how the check affected people’s credit scores. I ran the analysis by hand. It took time, but the finding was clear: customers did not know whether the credit check was a hard or a soft pull.

Those findings directly informed a messaging update across both the business and consumer sides, making the soft-credit-check language explicit at that step. Months later the stakeholder followed up, unprompted, to say the change had measurably improved customer confidence and reduced drop-off there, where people had been leaving to consider competitors who had already made this clear. That outcome was valuable, and it was expensive to produce. I remember thinking: if one round of chat research can do this, we need to be able to run it far more often than we can.

What I built

I approached it in two phases. First I wrote a script to automate the filtering and joining. It was faster than doing it by hand, but fragile: it only worked for one export shape and broke whenever a file naming convention changed. So I rebuilt it, with an AI coding assistant, into a reusable, general-purpose tool. It now scans a folder of any size, detects the two file types by pattern, and groups conversations by page, normalizing tracking parameters so that URL variants of the same page collapse into one entry. It lists the pages by conversation volume, then extracts every message for a chosen page into a clean spreadsheet, with the back-and-forth kept in order.

To test it at real scale, I ran it on a recent export of about ninety files, roughly two days of data, that would have taken forty-five to sixty hours by hand, more than a full work week. With the tool, the same job took about thirty minutes. That comparison is the clearest evidence I have of what shifts when the friction is removed.

What it made possible

The clearest documented impact is still the credit-check work. But the broader change is a capability shift, not just a time saving. Before the tool, page-level qualitative analysis on full datasets was infeasible, so we did not offer it as a routine option. Now it is something I can run the same day a stakeholder asks. A whole category of qualitative research moved from special project to normal work.

I am also honest about the limits of the data. Chat transcripts skew toward customers who had a problem or a question big enough to open a chat, and they under-represent people who read something confusing and simply left, who are often exactly the population a team most wants to understand. I flag this in every engagement, so findings from chat research are read for what they are: the voice of the customers who chose to engage, not a proxy for everyone.

Where I took it next

If I am the only person who can run the tool, the capability is limited to my capacity. So I designed a prototype for a self-serve insights portal with two roles. Stakeholders submit a request that asks not just for a page and a date range, but for the research question, any existing hypothesis, and the decision the findings will inform, which turns a data pull into a research consultation. Researchers work in a companion view that shows the extracted data alongside AI-surfaced candidate patterns, then move to a verification screen where they confirm or reject each pattern against the source transcripts and add their own interpretation. A delivery layer publishes a synthesized report with the evidence attached. I designed it to be accessible from the start, with semantic structure, keyboard navigation, sufficient contrast, and text alternatives, because a system meant to spread insight across an organization has to be usable by everyone in it.

How I practice

I built all of this because I am protective of research quality. It would be easy right now to hand qualitative work to a tool that generates themes from text and calls the summaries insights. That is not what this does. The tool automates the part that was never research work in the first place, cleaning files, joining tables, formatting data, and leaves the actual research to me: interpreting what a pattern means, verifying claims against the source, and framing recommendations so they land with the people who will use them. The tool made the research possible at scale. The researcher made it matter. That is the sentence I want the work to stand on.