Reading the SEC at scale.
Four years monitoring the SEC’s publication firehose. What shifted, what we built to absorb the next shift, and the metrics we now track religiously.
Volume, drift, and structural change
The SEC publishes more, more often, in more formats than it did three years ago. The shift is not linear: it comes in steps. A site redesign moves the press-release feed off RSS. A new sub-bureau starts publishing on a separate schedule. A class of filing migrates from PDF to structured XBRL. Each step broke a different piece of our pipeline, and each fix taught us something about what to build next.
The three breaks
The first break was the press-release feed redesign in 2024. The RSS endpoint was deprecated; the new feed was JavaScript-rendered. We had assumed the SEC would always serve a static feed. The fix was a headless renderer in front of GAWAIN, plus a structured fallback that scraped the JSON payload directly. The second break was the XBRL migration, which we had been preparing for, but which came with a six-week parallel-run that we had to absorb. The third break was the sub-bureau schedule shift, which we caught only because a customer flagged a delayed alert.
What we built
We added per-source freshness SLAs that pager-alert when the gap between expected and observed publication exceeds a threshold. We added structural-shift detectors that compare today’s parser output against a rolling baseline; a sustained drop in extraction success triggers a review. We added a manual override that lets a regulatory editor pin a source’s schedule when the regulator publishes erratically.
The metrics we now track
Four numbers live on the SEC dashboard. Median latency from regulator publication to RegAlytics ingestion. Daily extraction success rate, by source. Field-level completeness on the alert record. Customer-reported miss count, which is the only metric that matters when the others lie.
Building on SEC coverage?
Talk to the team about SEC-specific coverage detail and the alert categories we track.