What we learned auditing 836 VC feeds

An index nobody audits turns into a graveyard quietly. Ours had been growing for years, mostly by addition, so we spent a day fetching all 836 feeds and comparing what came back against what we had stored.

The headline numbers

128 feeds published something in the last 30 days. 518 have published nothing in over a year. Twenty-five have never produced a single post since the day they were added, which means they were broken from the start and nobody noticed.

That last category stings. A feed that never worked looks identical to a feed from a publisher who doesn't post much, unless you go and check.

The mistakes we found in our own data

Six feeds were registered twice. Sam Altman's blog is in there as both the http and https version of the same address, which means his posts were split across two entries and each copy looked half as active as it was. Same story for a handful of others under slightly different names.

Five feeds were publishing normally while we quietly missed the posts. One was 1,146 days behind. These are the worst kind of bug because nothing errors: the pipeline runs, reports success, and silently skips a source.

668 feeds had no author stored. 306 had an empty byline, which is why some entries in the digest used to show up with a bare comma where a name should be.

What we fixed and what we didn't

We fixed the duplicates and the silent misses first, because those cost readers posts. Author names and titles we're now filling from two sources: the feeds themselves, which often carry a byline in the XML, and a weekly pass that reads the actual article page.

We did not delete the dormant feeds. There's a reasonable argument for cutting 500 entries and claiming a tighter index, but a dormant blog with a good archive is still useful to someone searching for it, and some of those authors will start writing again. What we changed is how we describe the number.

The part worth stealing

If you run anything that ingests other people's feeds, the single most useful check we added was comparing the feed's own newest item against the newest item you've stored. Not "did the fetch succeed", which is what most monitoring looks at. The gap between those two dates is where silent failure hides.

We also learned to distrust our own fetching when a result looks too bad to be true. Our first sweep reported 600 dead feeds. Almost all of those were rate limits caused by hitting the same host 24 times at once, not dead blogs. Slowing down turned 600 failures into 34 real ones.

The index is public if you want to see what's in it, and submissions are open if you know a blog we're missing.