How Alethman turns thousands of reports into one story

People ask me what the engine does between "reads the news" and "shows a story". This post is the answer in plain language, with the numbers we set and why we set them. None of it is a secret. Most of it is careful counting.

Reading, politely

Every fifteen minutes the engine polls a few hundred validated news feeds: India's first, then those of any other country a client workspace follows. It identifies itself with an honest user agent that links back to us, reads each site's robots.txt and stays out of what a site disallows, honours crawl delays, and sends conditional requests, so an unchanged feed costs the publisher almost nothing. A site that refuses us is recorded as refusing us. We don't route around it.

For each article it keeps the outlet, the headline, the time of publication and the text. It also notes where a wire agency is credited: in the byline, in a bracketed dateline, or in a closing "with inputs from" line.

Step one: collapse the copies

The most important step is the least glamorous. Before anything is grouped into stories, near-identical texts are collapsed into one duplicate group.

The method is standard and well understood. Each article's text is cut into overlapping five-word shingles, and a MinHash signature over those shingles finds candidate pairs quickly through locality-sensitive hashing. A second estimator, SimHash, checks each pair a different way. A candidate joins a group only if it matches the group's representative article directly, never by chaining through another member, so a group cannot drift from one text to a different one a step at a time. Very short items join only on an identical headline.

If any member of a group carries an agency credit, the whole group is marked as wire copy. One credited copy is enough, because outlets routinely strip the credit when they reprint.

The result: forty sites carrying one agency story become one duplicate group, and that group is one voice, not forty.

Step two: decide what is the same event

Next, each article gets an embedding from a multilingual language model: a list of 768 numbers that places texts with similar meaning close together. We use a multilingual model because a large share of Indian news is not in English, and we set how much text we feed it by Hindi, which uses more characters per token, rather than by English.

Then the clusterer runs, and two choices matter.

First, the unit being clustered is the duplicate group, not the article. Cluster articles, and forty copies of one wire story look like a dense, important region made entirely of one text. The clusterer would build stories around reprints.

Second, the thresholds are measured, not guessed. On Indian news, two random unrelated articles already score as fairly similar under this kind of model: in our measurements the median random pair sat around 0.83 and the 99th percentile around 0.92. A threshold that "feels right" at 0.8 merges the whole day into one story.

So membership has a two-part rule. Every story has a label article, the member closest to its centre. A new member joins if it scores at least 0.95 against that label article, or at least 0.92 and shares at least one content word with the label's headline.

We anchor on the label article, not the cluster's centre, because of a hand check of sixty memberships. The centre of a big cluster drifts towards "generic news from this outlet", and anchoring on it let 21 wrong members through. Against the label article with the word check, the same set kept 21 of 30 right members and none of the 21 wrong ones. We would rather miss a member than glue two events together.

A story needs at least three duplicate groups to exist at all. Once a night the engine re-solves the past week from scratch with HDBSCAN, a density-based clustering method. In between, every fifteen minutes, new groups attach to live stories at a stricter 0.94. A story that has been quiet for five days is over: a new article on the same subject starts a new story instead of reviving the old one.

Across languages

The word check cannot work across scripts, and the same event in two languages scores lower than the same event in one. We measured it. Pairs of stories in different languages whose centres scored 0.94 or above were the same event every time; just below that, related but distinct events begin. So a story in a language other than English or one of India's own languages merges with its counterpart only at 0.94 or above. India's English and Indian-language stories are not merged with each other this way.

Persian against English on the same event scored lower than unrelated English pairs. No threshold joins Persian safely, so we leave it out rather than pretend.

Step three: count the outlets

Now the number shown beside every story: independent outlets. It is the count of distinct outlets among the story's members that are not wire copy.

That definition has a history. An early review of the engine found the count was adding up articles, so "27 independent outlets" could mean six outlets and 28 articles. It now counts outlets, once each, and only those whose copy is their own.

Each story also gets a trajectory from its timestamps. It is new if its first article is under six hours old, rising if the last six hours carried clearly more than the six before, decaying if neither window has anything, and steady otherwise.

Step four: place it

Every article is tagged with places from a gazetteer we built for India: the 36 states and union territories, close to 800 districts from Wikidata with names in 22 languages, cities, regions such as Vidarbha, and contested places. A match in the headline weighs three times as much as one in the body, and a match in the first 200 characters twice as much. Names that are also ordinary words count only with a district word after them or the state named. A bare "India" never places a dot.

A state claims a story when its places carry at least a third of the weight of the story's strongest place. A story about Nagaon that mentions Delhi once belongs to Assam, not to Delhi.

What a model does, and what it doesn't

Language models do part of the reading: a short summary and the tone of each article, and, for stories with enough voices, a statement of the story's central question with each article's stance on it and how it frames it. Every one of those rows records which model and which prompt version produced it.

What a model never does is decide the shape of the coverage. Whether a story has split into blocs of outlets, and how strong a coordination finding is, are computed in code from those rows, with the evidence grade and the outlets attached. That is deliberate. The statement can only say what the numbers support.

If you want the short version for a non-engineer, it is on How it works. If you want to argue with a threshold, write to us at contact@alethman.com. I would like to hear it.