Reading what customers write, without deciding it first

Who this piece is for

Anyone who has already tried. Anyone who exported the reviews into a spreadsheet, read them over two evenings, noted down a few recurring words and then let it go, because they had no way of knowing whether what they had found was true or only what they expected. The problem is not technical and it is almost always the same one: reading leaves no trace, and without a trace nothing can be checked, not even yourself.

It is not for anyone with twenty reviews. On twenty rows, any counting method is a complication that does not pay for itself. The threshold at which counting starts to say something the eye cannot see is higher than people think, and we come back to it at the end.

1. The case, in full

I was looking for a niche. In a sector of phone apps, I wanted to work out whether there was room to move in: which problems people report constantly, which of those nobody has solved, and which of the unsolved ones are worth enough to justify the work. The raw material was the public reviews of the products already on the market. Not my customers: other people’s customers, who write for free and in public and have no reason to be polite.

I thought I knew two things. The first was that the recurring problems in that sector were, roughly, the ones I would have listed from memory. The second was that a highly rated product is a product with few problems.

I started by building a program that does two things, one after the other. Taken separately they are both reasonable. It is the sequence that deceives.

First step: put together the reviews that resemble each other. This operation is called clustering and it worked: the groups came from the material, not from a list I had decided in advance. Among the ones that came out there was a cluster of 172 reports, the largest, and the program flagged it as the main opportunity.

Second step: give each group a value. Here the program looked at which problem category was the most frequent inside it — what statistics calls the mode — looked that category up in a table of weights, and assigned that weight to the whole group. I had written the table myself two days earlier, without having read a single review.

The flaw is in the jump between the two. Reviews that resemble each other in their wording are not reviews that report the same problem, and inside that group of 172 there were eleven different categories of problem.

Category	Reports
trouble using the app	79
price	23
other	23
feature that does not start	18
permissions and privacy	10
performance	9
content quality	3
sync reliability	2
missing feature	2
integration with other software	2
first launch	1

The most frequent category covered 79 reports out of 172. Those 79 picked the row in the weights table, and the value of the entire group came out of that row. The other 93 found themselves with the score of a category they did not belong to. It is not that they did not count: they counted towards the size of the group, and it was the size that carried it to the top. They did not count towards establishing what it was about.

Meanwhile, in the field that should have said what it was about, nothing had been written. That line is written by a language model, and on this group the call failed twice in a row: in place of the label there remained the fallback string the program uses when it does not have one, and the description stayed empty.

The point is not the failure, it is the two lines next to it. The step that was supposed to describe the group produced nothing, the one that was supposed to score it produced a number with a decimal, and the report put an unnamed entry at the top of the ranking. Neither step knew about the other.

The mean, instead of the mode, does not save the count. I checked afterwards. Giving each category the weight of its own row and taking into account how many reports it brings, the group drops to 0.57 and does not reach the 0.60 threshold needed to declare it the strong opportunity: it would have ended up among the ones to verify.

But then there is the row called “other”, 23 reports. It is not a category of problem, it is where whatever was not recognised ends up, and in the table it carries the lowest weight of all. Counting it means turning “I did not understand what they had written” into “it is worth little”. Take it out of the count and the group climbs back to 0.64, above the threshold, and is first again.

Three ways of doing the same sum, all defensible, and the verdict changes twice. None of the three read one review more than the others.

This is the mechanism to take away: taking the mode of a set and treating it as the description of the set. It happens with the tags in the management system, with ticket categories, with the “main reason” field in forms. The result is a label that describes less than half of what it claims to describe, and no way of noticing by looking at it.

The upshot is that the program did not discover what people were talking about. It measured how much people talked about the things I had thought of. Which is a legitimate question, if it is the one you want to ask. But it was not the one I thought I had asked, and the difference between the two cannot be seen by looking at the output: the output is identical, and it is a ranked list with decimals that looks like a result either way.

I got there by a side route. One of the groups identified as most promising had an empty label: the program had gathered a good hundred complaints declaring they were the same thing, and under what this is about there was nothing. That blank was the only honest part of the result. Everything else had a name, and I had written the names.

2. Summarising and counting are not the same operation

It is the most important distinction in the whole piece, and it is an operational one.

If you ask someone (or something) to summarise these five hundred reviews, what comes back is a text. Well written, plausible, and irretrievably unverifiable: there is no way to take a sentence from the summary and trace it back to the rows that produced it. If the summary says “many users complain about slowness”, many means nine or a hundred and ninety, and there is no way of knowing which without rereading everything, which is exactly what you were trying to avoid.

If instead you ask how many reviews mention slowness, and which ones, what comes back is a number and a list of rows. The number can be wrong, but it can be checked: you take ten of them, you read them, and you see whether they really are about slowness.

The practical difference: a summary is an opinion you cannot contest, a count is a claim you can disprove. Only the second is a basis for deciding, because only the second tells you when you got it wrong.

An unwelcome corollary: the count is far more boring to read, but that is a flaw only in appearance.

3. Grouping is a problem with no right answer, and that is not a figure of speech

There is no correct grouping of a set of reviews. It is not hard to find: it has been proved that there is none.

The result is from 2002 and it is Jon Kleinberg’s. We ask a grouping method for three things that all seem obvious: that the result should not change if we measure distances in centimetres rather than metres; that it should be able, in principle, to produce any grouping at all; and that if we move things that are already together closer and push things that are already apart further away, the result should stay put. These are three requirements nobody would question. Kleinberg proved that no method can satisfy all three at once.¹

Ten years later, three researchers tried to move the question from the method to its evaluation: granted that there is no right grouping, can we at least say which of two is better? The answer is that you cannot, unless you know why that person is grouping and what they will do with it afterwards. Grouping should not be treated as a mathematical problem independent of its use, but always inside its end use.²

The commercial consequence is immediate and worth being blunt about: anyone offering you “an objective segmentation of your customer base” is selling you an object that has been proved not to exist. Every grouping is the answer to a question, and you are the one who brings the question. The only honest choice is to state it before you start, instead of discovering it afterwards, dressed up as a result.

4. Two different operations that go by the same name

Grouping, in the technical sense, means assigning items to classes that are not defined in advance and that should reflect the structure of the material.³ Classes not defined in advance are not a detail of the procedure: they are the definition of the procedure.

Sorting material into known boxes is a different operation. It is just as legitimate, it has metrics of its own and it answers a sensible question. But it is a different question, and the two results are indistinguishable to look at: both come out as a ranked list with decimals.

Confusing the two is the most common mistake in this trade, and it takes a particularly insidious form. In the case I have described the groups were found by reading, so the grouping was genuine. What was already decided were not the groups: it was how much each category of problem was worth, set out in a list of categories with a weight beside each, written before reading anything at all.

The interesting part is what happens to the things that are not on that list. My list had no entry for advertising. Complaints about advertising therefore ended up under “other”, which carried the lowest weight in the table. When the themes were later allowed to emerge from the material, advertising turned out to be the entry with the most evidence behind it of them all.

The things you have no box for do not become small. They become invisible, and they do so in a way indistinguishable from being genuinely irrelevant. It is the only reason it is worth giving up the convenience of a grid decided in advance.

5. The number that ought to tell you how many groups there are, and why it often does not

There is a classic measure, the silhouette, which for each item compares how close it is to its group-mates against how close it is to the nearest other group. It is used to choose how many groups to make: you try different settings and keep the one with the highest score.

On short texts that resemble each other, that measure often fails to decide, and it is not the material’s fault. It rewards roundish, well-separated groups, and in spaces with very many dimensions distances tend to look alike, which flattens the scores and makes the solutions indistinguishable.⁴ It was also designed to choose the number of groups within a single representation, not to compare several.⁵ On short sentences the score improves by splitting: the maximum is reached by making one group per sentence, which as a grouping is the definition of failure.

The correction is not to look for a better measure. It is to take away that number’s power to decide and print it as a diagnosis: a flat curve, put in the final document, is the proof that the criterion had no standing to choose. It is worth noting that the procedure improves by removing, not by adding.

6. How it is done, in practice

If that number cannot decide, there is still a difference between a grouping that holds and one produced without noticing. The operational rule has three steps: read first, then name, then count. Never: name, count, read.

First step, no categories. For each review you do not ask “which category does this fall into”, but “what was this person trying to do, and what stopped them”. Two free-text fields, in the words of whoever wrote it. No list supplied. At this stage you are not classifying, you are reducing: a thousand rows of prose become a thousand goal-obstacle pairs, far more manageable and with nothing lost.

One review produces as many pairs as it needs, not one. It is the detail that looks like pedantry and is not: if every document produces a single entry, someone complaining about six things is counted on the first one they name, and the problems people tend to mention first come out as systematically more widespread. The count stays plausible and becomes false.

Second step, the groups emerge. You look at what resembles what. On a few hundred you do it by hand, sorting the obstacles alphabetically and scrolling: the duplicates jump out on their own. On thousands you need tools that measure how much two sentences resemble each other. Either way the rule does not change: the group’s name is written after looking at what is inside it, using the words that are inside it.

Third step, you count.

7. The few fixed columns

For each review, five fields, always the same:

• main theme (one only, mandatory)
• secondary themes (zero or more)
• tone (complaint, request, praise — and yes, one review can contain more than one)
• customer type, where it can be inferred: who pays and who does not, who is new and who has used the product for years
• anchor sentence: the piece of original text you derived the theme from

Why a few fixed fields beat a free description: because a free description does not add up. If for one review you write “complains about general slowness” and for another “finds loading heavy”, you have two different sentences for the same thing and no automatic way of noticing. Five columns with limited values can be counted, sorted, filtered, and above all checked.

The field everyone forgets is the third. An enthusiastic review containing a request is gold, because it comes from someone who uses the product and is sticking with it: if you do not distinguish the tone, that request ends up in the pile of complaints and you read it as a problem instead of as a question.

The field almost nobody collects is the fourth, and it is the one that changes decisions. Ten people asking for the same thing are not worth the same if eight of them already pay and two do not.

A warning about the third field: tone should be recorded, but not used for grouping. You group complaints and requests; praise is counted separately. Mix them and two sentences saying opposite things about the same feature end up in the same group, because they resemble each other in their wording.

8. The mandatory quotation

Every counted row must point to the original sentence it came from. If you cannot trace it back, the number does not exist.

It is a hard rule and it is worth applying without exceptions, because it is the only one that stops the procedure from degrading silently. A count with no anchor sentences goes on working, goes on producing credible tables, and there is no moment at which it visibly breaks: it is simply that, a few weeks later, nobody knows any more whether it is true.

Watch out for a detail that looks like pedantry and is not: the sentence must be copied, not reworded. A language model asked “which sentence proves this” tends to produce an elegant paraphrase that is not in the original text. It looks like the same thing. It is not: the anchoring is lost and nobody notices. The check is trivial and should be automated: the quoted sentence must appear, character for character, inside the original text. If it does not appear, it goes in the bin.

9. The spot check

This is where most guides say “check a sample” and move on. The details are everything.

How many rows. Enough that you can get them wrong informatively. On a corpus of a few hundred, a few dozen rows is a sample that says something; below twenty you are measuring chance more than method.

How they are chosen. Not the first twenty, which are almost always the most recent or the longest. Not the ones that intrigue you. Drawn at random, by a reproducible criterion, and ideally stratified: a few from each theme and each rating, or you end up checking the largest group and ignoring the ones where error is most likely.

Blind. You classify them, by hand, without looking at what the machine answered, and only then compare. It looks like bureaucratic formality and it is the decisive part: seeing someone else’s answer first does not make you copy it, it makes you find what you read reasonable. It is the same dynamic as the article itself, applied to you checking yourself.

What to do when three of them are wrong. It depends on how they are wrong, not on how many. Three scattered, different errors are noise: you note them and move on. Three errors of the same kind — say, all requests read as complaints — are not errors but a systematic flaw: that has to be fixed and then you start again from the beginning, because it has contaminated the whole corpus and not just the three rows you looked at.

When to throw everything away. When agreement is low and you cannot describe the pattern of the errors. A method that fails in a comprehensible way can be adjusted; a method that fails in an incomprehensible way is not adjusted, it is replaced.

The check nobody does, and that is worth more than all the others

Before blaming the tool: redo the same classification by hand, on the same rows, a few days later, and compare yourself with yourself.

It is inconvenient and it takes an afternoon. It is needed because it sets the ceiling. Agreement between you and the tool should not be compared against a hundred per cent, which nobody can reach: it should be compared against the agreement between you and yourself. If you are eighty per cent consistent, a tool that follows you eighty per cent of the time has got as far as you could have got, and continuing to file away at it is wasted time.

Without this check you end up, systematically, in one of the two opposite errors: throwing away a tool that was fine, or trusting one that was not. There is no way of knowing which of the two you are in without the yardstick.

10. The test that separates a real group from an idea of yours

If there is no right grouping, what distinguishes a real group from one produced by the way you happened to look? The answer from the literature is simple and almost nobody applies it: resistance to resampling.

The procedure, due to Christian Hennig, goes like this. You group as normal. Then you draw from the same data a new sample of the same size, with replacement, so that some items appear twice and others not at all. You group again. For each group in the original you look for the most similar one in the new run and measure how much they overlap. You repeat many times. A group that does not reappear under this treatment is considered dissolved.⁶ The thresholds in current use, which are practical conventions and not formal results: below 0.6 a group should be treated as unstable, above 0.7 as stable.

Strip away the mechanics: a group that survives only your selection of data is not a group, it is an opinion with rows inside it. The check is cheap, it is automated once, and it produces the one number worth putting next to each theme when you present the work to someone who has every right not to believe it.

It is also the check that, in my case, is not there yet. I say so because it is the most important missing piece of the whole procedure, and because anyone proposing an analysis of this kind to you should be able to tell you whether they have done it.

11. The leftovers bucket

Density-based methods have an explicit way of dealing with leftovers: they separate the groups supported by dense regions and can leave as noise the points that belong to none of them.⁷ The procedure used in this case was different: the agglomerative method assigned every row to a group. It could not produce a declared residue; it could only hide one inside a group.

The signal to recognise: the largest group, if it is also the hardest to name, is not necessarily the main theme. It may be the residue the method cannot leave out. And the residue says something different and useful: how much of the material the chosen way of looking cannot see.

In the case I have described the group of 172 was the largest and the only one left unnamed. I have already given the technical explanation for the missing name and it holds. But the composition also suggests a different possibility, one about the content rather than the label, and the two do not exclude each other: eleven categories, with the most frequent one under half, are consistent with a container of leftovers. I do not have the proof that it was one. I have the coincidence of the two signals that ought to raise the suspicion.

12. The tools, from the quickest to stand up to the slowest

None of these solves the problem. All of them solve a piece of the work, and the cost that matters is not the list price but how long it takes before the tool starts giving you something back.

The spreadsheet. It deserves to be named first out of honesty, and it solves more cases than people think. With a few hundred rows, five columns and a couple of pivot tables you get to a verifiable count and a ranking by frequency in an afternoon’s work. What it does well: counting, filtering, sorting, and keeping the number and the original sentence together in the same row, which is the most important thing. Where it stops: when the duplicates are worded differently, because the spreadsheet does not know that “slow” and “takes forever” are the same thing, and at that point you are the one doing the work by hand. Setup cost: near zero. It is the right choice far more often than anyone selling an alternative will admit.

NotebookLM (Google’s, with a free tier and paid plans). You upload the documents, you ask questions, and you get answers with a pointer back to the place in the document they came from. It is the use case closest to this article, because source citation is native rather than bolted on: the mandatory-quotation rule is enforced by the tool instead of by your own discipline. What it does well: interrogating a corpus without losing the anchoring. Where it stops: it does not count. It tells you a theme is there and shows you where, but if you ask how many times it appears you are asking for a summary again, with all the problems of a summary. Setup cost: half an hour, and it is the best ratio between what you get and what you have to learn.

Obsidian. For anyone working on their own texts who wants to keep everything local, with nothing leaving their computer. What it does well: linking, annotating, finding again, and building over time a structure that is yours and stays yours. Where it stops: it is not an analysis tool, it is a custody tool; the counting is something you have to organise yourself. Setup cost: high, and it grows with the urge to customise, which is its main trap.

Notion. For anyone whose material is already there and who just wants to give it a shape. What it does well: imposing the fixed columns from point 4 without your having to build them, and letting several people annotate together without overwriting each other. Where it stops: on volume, and on the fact that it becomes comfortable enough to make you stop asking whether the numbers are right. Setup cost: low if you already work in it, high if you adopt it specially.

When the documents run into thousands. The problem changes, not just the scale. At that point none of these is enough, because the bottleneck moves: it is no longer reading, it is grouping what resembles what without deciding it first, which requires measuring how much two sentences resemble each other and settling how close they have to be to belong together. That threshold is the most delicate decision in the whole procedure, there is no absolutely right value, and anyone who hides it behind a button has taken that decision for you. It is the point at which it is worth getting help from someone, or accepting that you will work on a smaller sample chosen well, which is almost always the better option and almost always the last one considered.

13. The mistakes you will make

There are five, they are repeatable, and none of them produces a result that looks wrong. They produce tidy, plausible, presentable results: which is why the only defence is checking, not the eye.

Fixing the value before knowing the facts. Assigning a weight to problem categories before having read the material is convenient and invisible: the grouping stays honest, the ranking does not. And whatever has no box on the list ends up under “other”, where the weight is by definition the lowest.

Imposing the shape of the result. “Give me at least twenty groups”, “what are the top three problems”, “no group too large”. These are reasonable as guardrails and disastrous as criteria: they produce exactly the number requested and make it look like a discovery.

Delegating a decision to a threshold written months earlier. A numeric rule is convenient because it removes a decision, and the decisions we get out of the way are the ones we never reread. Every threshold should be kept together with the date it was chosen and the reason, or it turns into a belief.

Comparing numbers that have changed meaning. Change the unit of counting and the new scores are not comparable with the old ones, because the denominator is a different thing. Two numbers with the same name and the same shape look like the same quantity, and nobody puts up a sign.

Printing a precision the number does not have. Three decimal places on a score that cannot tell eleventh place from twentieth is a promise you cannot keep. Three bands — high, medium, low — are less satisfying to look at and they are honest.

14. When it is not worth doing

This section exists because it is the one that saves you money, and it is the only part of the piece written against the interest of the person writing it.

Under a hundred documents, reading them all is faster than setting up any count, and on top of that you actually read them. Counting starts to pay off when the material exceeds what you are willing to reread a second time, and it is the second reading that counts.

With sources too different from each other (public reviews, support requests, notes from phone calls) the corpora do not add up: the same person writes differently depending on who is listening. You count them separately and compare afterwards.

When the people who write are not the people who use. It is the most serious limit and no tool solves it. People who leave reviews are a self-selected minority, driven by a strong emotion in one of the two directions. Someone with an average problem works around it and writes nothing, and that is almost always the largest group. A count on reviews measures the shape of expressed discontent very well. It does not measure satisfaction, and there is no way of making it say so.

When the question calls for a judgement and not a count. “Is it worth entering this market” is not a question a count answers. It says which problems exist and how often. It does not say whether anyone would pay to have them solved, which is another question with another method.

And the phenomena that get away regardless. Irony is read literally. What is implied is not there. A document that contradicts itself internally is classified on its most explicit part. Planted reviews add up alongside the rest as though they were people.

15. A note on privacy

If the documents uploaded to a third-party service contain customer material — names, support requests, contracts — the question does not close in a paragraph. It concerns three distinct things that often get confused: what goes into training the models, what the suppliers’ contracts say about it, and what remains the responsibility of whoever collected that data, who is the data controller even when someone else does the processing.

The warning is this: choosing the tool is also choosing where your customers’ documents end up, and it comes before the first upload. The case described here was simpler, because the material was public reviews and not a client’s documents. Those are still personal data, and accessible does not mean free.

16. Where the problem has moved to

Take away the list of themes and what is left standing is the number of themes. The threshold that decides how much two sentences have to resemble each other to end up together is searched for automatically, but with two constraints written into the configuration: a cap on the share of material any single group can take, and a minimum number of groups, which is twenty. These are requirements about the shape of the result, not measures of how well that partition is supported by the data. With those constraints, finding twenty groups is not a discovery: it is an exit condition.

The program no longer decides what people are talking about. It still decides how many things people are talking about, and I wrote that number.

Hence the question to ask anyone who hands you a segmentation, and there is only one: who chose how many groups to make, and on what basis? Nine times out of ten the answer is a parameter. Every so often it is a measurement. The difference is everything.

17. The shape of the result

At the end of the work, what is left is less spectacular than you would imagine, and that is a good thing.

What is left is a table with one row per review and the five fixed columns. A list of themes with, beside each, how many times they appear, from how many different people, and in what tone. For each theme, the list of original sentences it was derived from, so that anyone can trace the number back to the row. The result of the spot check, with the errors found and their kind. And the list of things the count is not able to say, written alongside the rest and not buried at the end.

What is not left is a recommendation, because that depends on things the reviews do not contain.

A note on charts, and why there are none here

In this field the standard chart is the cloud of coloured dots: thousands of dimensions squashed onto two, one colour per group. It is handsome, it goes into presentations, and it has to be read knowing that two things the eye does automatically are wrong.

The first: you cannot read the relative sizes of the groups. The most widely used algorithm expands dense groups and compresses sparse ones, by construction, so two blobs of the same size can contain very different numbers of items.⁸ The second: the distances between the blobs may mean nothing. These methods preserve who is near whom, not by how much.⁹ That the mistake is widespread is not an impression: a review of 114 scientific papers documents it as current practice.¹⁰

The charts that are worth something in this trade are different and more boring: how well each group resists resampling, where the machine and a human being disagreed, and how the measure that was supposed to decide the number of groups behaves. Three ugly figures that say whether the work holds. The coloured cloud says how the work looks in a photograph.

---

Bibliography

Footnotes

1. Kleinberg, Jon M. “An Impossibility Theorem for Clustering.” Neural Information Processing Systems (2002). https://proceedings.neurips.cc/paper/2002/hash/43e4e6a6f341e00671e123714de019a8-Abstract.html ↩

2. Guyon, Isabelle & Luxburg, Ulrike & Williamson, Robert. (2009). Clustering: Science or art. https://proceedings.mlr.press/v27/luxburg12a.html ↩

3. Rautenstrauch, P., & Ohler, U. (2025). Shortcomings of silhouette in single-cell integration benchmarking. Nature Biotechnology. https://doi.org/10.1038/s41587-025-02743-4 ↩

4. P. J. Rousseeuw, Silhouettes: a graphical aid to the interpretation and validation of cluster analysis, Journal of Computational and Applied Mathematics 20, 1987, for the original measure (https://doi.org/10.1016/0377-0427(87)90125-7). On the concentration of distances in high dimensionality: K. Beyer et al., When Is Nearest Neighbor Meaningful?, ICDT 1999 (https://doi.org/10.1007/3-540-49257-7_15). ↩

5. Rautenstrauch, P., & Ohler, U. (2025). Shortcomings of silhouette in single-cell integration benchmarking. Nature Biotechnology. https://doi.org/10.1038/s41587-025-02743-4 ↩

6. C. Hennig, Cluster-wise assessment of cluster stability, Computational Statistics & Data Analysis 52, 2007 (https://doi.org/10.1016/j.csda.2006.11.025). Reference implementation: clusterboot in the R package fpc (https://cran.r-project.org/package=fpc). See also C. Hennig, What are the true clusters?, Pattern Recognition Letters 64, 2015 (https://doi.org/10.1016/j.patrec.2015.04.009). ↩

7. R. J. G. B. Campello, D. Moulavi, J. Sander, Density-Based Clustering Based on Hierarchical Density Estimates, Advances in Knowledge Discovery and Data Mining, PAKDD 2013, pp. 160–172. https://doi.org/10.1007/978-3-642-37456-2_14 ↩

8. M. Wattenberg, F. Viégas, I. Johnson, How to Use t-SNE Effectively, Distill, 2016. https://doi.org/10.23915/distill.00002 ↩

9. L. McInnes, J. Healy, J. Melville, UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, arXiv:1802.03426, 2018. https://arxiv.org/abs/1802.03426 ↩

10. H. Jeon, J. Park, S. Shin, J. Seo, Stop Misusing t-SNE and UMAP for Visual Analytics, arXiv:2506.08725, 2025. https://arxiv.org/abs/2506.08725 ↩