Reference · built on requestOperator by FibricIT & reliability

Vendor Outage

Reads third-party status pages and your own integration error rates. Proposes a customer notice or a failover when a provider degrades.

About

A provider's outage reaches your customers through your product, and your status page says nothing until someone notices. Vendor Outage reads the public status page of each provider you depend on and the error rate of your own integration with it, and compares the two.

When a component the integration relies on goes to partial_outage or major_outage and your error monitor is firing, it proposes a notice: an incident on your own status page, and a problem ticket in Zendesk that arriving incidents are linked to. Where you keep a standby provider, it proposes the failover instead, as a DNS record change with the record to restore. The on-call engineer picks one in Slack. The choice is applied once and closed out when the provider posts resolved and your errors recover.

This is a reference listing. It documents what Fibric would read from Vendor Outage and what it could propose, based on the vendor's published interfaces. Fibric builds it under a managed deployment when you request it; selecting it here installs nothing.

Inputs

  • Each provider's public status API at <subdomain>.statuspage.io/api/v2: summary.json, incidents/unresolved.json, and components.json
  • The page indicator, none, minor, major, or critical, and each component's status: operational, degraded_performance, partial_outage, or major_outage
  • Subscriber webhooks from a provider's page: component_update with old_status and new_status, and incident with status and incident_updates
  • Your own integration error rate through Datadog monitors and SLOs, delivered by the Webhooks integration with $ALERT_TRANSITION and $EVENT_TITLE
  • Tickets arriving in Zendesk with the symptom the outage produces, matched to the provider incident
  • The DNS record in Cloudflare that points your integration at the provider, through GET /zones/{zone_id}/dns_records
  • The on-call engineer's choice on the message it posts in Slack

Proposed actions

  • Target capability: propose an incident on your own status page through POST /pages/{page_id}/incidents, with name, status investigating, body, component_ids, and deliver_notifications
  • Target capability: propose a problem ticket in Zendesk and linking each arriving incident ticket to it by problem_id, with one public comment for all
  • Target capability: propose a failover: a DNS record update in Cloudflare that points the integration at your standby provider, with the record to restore
  • Target capability: propose a message in Slack to the on-call channel with the provider incident, your error rate, and the choice between notice and failover
  • Target capability: propose setting the incident to resolved, and solving the Zendesk problem, once the provider posts resolved and your error monitor recovers

Proposed actions are target capabilities. Every action runs propose-first and needs a validated deployment and the appropriate permissions.

What you can build

  • Tell customers before they tell you

    A provider's component goes to major_outage and your Datadog error monitor fires. The operator drafts the incident for your status page with the affected components and posts it to Slack. The on-call engineer approves. Subscribers are notified once.

    With Statuspage, Datadog, Slack

  • One problem, many tickets

    Tickets about the same failure arrive in Zendesk. The operator proposes a problem ticket and links each new incident by problem_id. One public comment reaches every requester, and solving the problem solves the linked incidents.

    With Zendesk, Statuspage

  • Fail over to the standby provider

    Your transactional email provider degrades. The operator proposes the Cloudflare DNS record that points the integration at the standby, with the current record kept for the way back. Approval applies it once.

    With Cloudflare, Datadog

  • Page the on-call with the evidence

    The error monitor fires and no provider page explains it. The operator proposes a trigger event to PagerDuty with a dedup_key, the monitor, and the providers it checked, so the responder starts with what is known.

    With PagerDuty, Datadog

Requirements

  • A list of providers, each with its Statuspage subdomain and the component names your integration depends on
  • A Datadog monitor or SLO per integration that measures your own error rate, notifying through the Webhooks integration
  • A status page of your own on Statuspage, with an API key and the components customers see
  • A Zendesk agent able to create problem tickets and public comments
  • For failover, a standby provider already configured, and the Cloudflare zone that holds the record
Authentication
Runs on a Statuspage API key for your own page, a Datadog API and application key, a Zendesk agent, and, for failover, a Cloudflare API token scoped to DNS on the zone. Providers' public status APIs need no key.

Limits

  • The public status API covers providers hosted on Statuspage. A provider with another status host is read only through your own error rate
  • A provider post alone does not trigger a proposal. The operator waits for your error monitor to agree, and reports a disagreement instead
  • The Statuspage management API allows 1 request per second per token, measured over a 60 second rolling window
  • A DNS failover takes effect as resolvers expire the old record. The record's ttl sets how long that takes

Access and pricing

Reference listing. Fibric builds the operator under a managed deployment when you request it. Your quote covers the build, capabilities, usage, and support.

Request Vendor Outage ↗

Questions and answers

What does the on-call engineer approve?
One of two proposals per provider incident: a notice or a failover. The notice shows the status page text, the components, and the Zendesk problem ticket. The failover shows the DNS record before and after. Either is applied once, after the click. Nothing is posted or changed before it.
What record is left after a notice or a failover?
The incident on your status page with its updates, the Zendesk problem ticket with its linked incidents, and, for a failover, the DNS record's previous content. Each carries who approved, when, the provider incident it answered, and the error rate at the time.
Does it fail over or post a notice on its own?
No. It proposes, and a person chooses. If the provider page and your error rate disagree, it says so and proposes nothing. If nobody answers, the incident stays unposted, the record stays where it was, and the proposal is repeated at the next check.
Ask about Vendor Outage

Ask about the capabilities and requirements in this listing.

For project-specific requirements, contact Fibric.