A feed monitor that only checks for an HTTP 200 response will miss the failures that matter. A source can be reachable while serving an HTML error page, a valid feed can stop receiving new entries, a parser upgrade can discard identifiers, and a webhook destination can fail after ingestion succeeds. Useful monitoring follows an update through the entire pipeline and reports which stage is unhealthy.

Engineering blogs and changelogs make this especially important because the cadence is irregular. A quiet week may be normal, while a missing major release is costly. Monitoring therefore needs baselines, source-specific expectations, and evidence rather than a universal “stale after 24 hours” rule. The objective is to detect loss of visibility without creating alerts every time a team publishes less often.

Define four separate health dimensions

Reachability answers whether the source can be fetched under policy. Document validity answers whether the response is a supported, parseable feed. Content freshness answers whether the observed update pattern is plausible for this source. Delivery health answers whether newly discovered entries reached configured destinations. Combining these into one status hides the difference between publisher outages and your own processing failures.

Track each dimension independently and derive a customer-facing summary from them. A source might be reachable and valid but stale compared with its history. Another might be unreachable now while the last successful snapshot remains current enough for readers. A third might ingest perfectly but have a failing team webhook. The DevRSS monitoring model preserves these distinctions so operators can respond to the right owner.

Instrument the HTTP exchange

Record DNS duration, connection duration, TLS duration, time to first byte, total duration, status, redirect chain, response size, content type, cache validators, and whether the document changed. These measurements reveal patterns that a binary check cannot. Increasing connection latency can precede timeouts. Repeated redirects can signal a migration. A sudden size increase can indicate an error page or accidentally embedded media.

Use conditional requests whenever validators are available. HTTP semantics define validators and conditional request behavior; the HTTP Semantics specification is the authoritative reference for interpreting responses and headers. A not-modified response is a successful observation, not a monitoring failure. Store the validator sent and the response received so cache behavior can be audited.

Validate content type and document shape

Servers frequently use generic content types, so headers alone should not decide format. Inspect a bounded prefix and parse the document in a safe mode. The monitor should detect when a previously valid feed begins returning HTML, a login page, a bot challenge, or a hosting error. Report the observed document type and a sanitized snippet rather than simply saying “parse failed.” That evidence often makes the publisher-side fix obvious.

Schema validation is useful but should not be the only criterion. Many widely consumed feeds contain minor specification defects. Classify findings as fatal, warning, or informational. A fatal error prevents item extraction; a warning indicates nonconforming data that was handled with a documented fallback. Preserve warning counts over time. A sudden rise after a parser change can expose a regression even when items still appear.

Monitor entry identity and sequence behavior

Count total entries, newly observed entries, updated entries, duplicate identifiers, missing identifiers, invalid dates, and items dropped by policy. Track the newest publisher timestamp and newest first-seen timestamp separately. If the document changes but no entries are discovered, the publisher may be editing metadata—or your normalizer may be failing. If every poll creates “new” entries, identifier derivation is unstable.

Create invariants for each source. The feed URL should not oscillate between hosts. Stable entries should not receive new internal IDs. Item count should not jump from dozens to zero without an explicit explanation. Publication dates should not all reset to the current time. These checks catch subtle problems that remain invisible to uptime monitors. Keep a source timeline that shows fetches, document hashes, entry counts, warnings, and deliveries in one view.

Detect staleness from the source's own history

A fixed staleness threshold treats a daily security bulletin and a quarterly infrastructure blog the same. Instead, learn a simple baseline from historical intervals between new entries. Use robust statistics such as a median and a high percentile rather than an average that can be distorted by bursts. Require enough observations before declaring an expectation. For new sources, use a generous default or allow the owner to set an expected cadence.

Staleness should be a graded state. “Quiet” can mean the source has exceeded its usual interval but is still reachable and valid. “Suspect” can mean several expected intervals passed or the feed stopped changing while the website shows new content. “Broken” should require stronger evidence such as repeated fetch failures, an invalid replacement document, or a confirmed discrepancy. This reduces noisy alerts while still surfacing important changes.

Compare the feed with a secondary signal carefully

For high-value sources, a secondary check can detect a feed that is valid but abandoned. Examples include a release API, a changelog index page, a package registry, or a repository tag stream. Treat this as reconciliation, not scraping the whole site into another source of truth. Compare stable identifiers or canonical URLs and report specific missing items. The secondary signal should have its own rate policy and failure state.

Do not automatically publish content discovered only through an untrusted scrape. Instead, alert that the feed appears behind and show the evidence. A publisher may intentionally exclude certain updates or change URL structure. Human review can confirm whether the feed contract changed. Over-automated reconciliation can create false items and duplicate notifications—the very failures monitoring is supposed to prevent.

Monitor delivery as a separate service level

Once an entry is committed, measure queue delay, first delivery attempt, successful delivery time, retry count, destination response class, and dead-letter state. A source freshness dashboard should link to delivery incidents but not merge their status. This lets the ingestion team prove that an update was captured even when a customer endpoint was unavailable.

Define service levels around what you control. For example, measure time from first successful source observation to durable event creation, then from event creation to accepted delivery. Do not promise time from publisher creation when you cannot know when the publisher updated the feed. The delivery and syndication layer should expose replay controls so failed destinations can recover without refetching or rewriting source history.

Alert on transitions, not every failed poll

Internet sources fail transiently. Alerting on the first timeout creates noise and encourages operators to ignore the system. Use consecutive-failure thresholds, time windows, and severity transitions. A single timeout can be recorded; a source that moves from healthy to degraded after several failed attempts can notify an owner; a source that remains down beyond its usual cadence can escalate. Recoveries should close the same incident rather than creating a disconnected “up” alert.

Group related failures by host and network path. If fifty feeds on one domain fail together, create one incident with affected sources rather than fifty pages. Correlate parser failures by parser version and error signature. Correlate delivery failures by destination host. These groupings turn raw metrics into operational understanding and reduce duplicate investigation.

Dashboard and runbook checklist

  • Show last attempt, last success, last change, last new item, and last successful delivery as separate timestamps.
  • Preserve redirect history, validators, response metadata, and bounded diagnostic samples.
  • Graph latency, document size, item count, warning count, and discovery lag over time.
  • List identifier collisions, invalid dates, dropped items, and parser-version changes.
  • Calculate source-specific staleness only after enough history exists.
  • Group incidents by host, error signature, parser version, and destination.
  • Provide a replay path for parsing and delivery that does not modify source timestamps.
  • Link every alert to a runbook explaining likely owners and safe diagnostic steps.

Monitor the contract, not just the endpoint

Feed monitoring is an exercise in preserving visibility. The endpoint must be reachable, the document must remain interpretable, entry identity must remain stable, and discovered updates must reach their destinations. Each stage produces evidence that helps distinguish a quiet publication from a broken pipeline.

Start with clear health dimensions and a source timeline. Add adaptive staleness and secondary reconciliation only where the value justifies the complexity. A monitor earns trust when it can explain why it is concerned and what changed, not when it produces the most alerts.