Google Cloud Pub/Sub

What you can build

• Deliver a topic to an operator by push

  A push subscription POSTs each message to the connector's HTTPS endpoint with a signed JWT. A 2xx response acknowledges it; anything else is delivered again after the acknowledgement deadline, so nothing is lost while the operator is busy.

  With Integration Health

• Know when a pipeline stalls

  An export subscription writes the topic into a BigQuery table. The operator compares publishTime on the newest rows with the clock and proposes a freshness notice when the gap grows past what the table usually shows.

  With Data Freshness

• Reprocess after a bad deploy

  Before a release, the connector proposes a snapshot of the operator's subscription. If the release misreads messages, the operator proposes a seek back to that snapshot, and the retained messages arrive again.

  With Webhook Replay

• Pull at the operator's own pace

  The connector pulls batches of up to 1000 messages, acknowledges each one the operator has read, and extends the deadline with modifyAckDeadline when a batch takes longer than expected.

  With Anomaly Notice

Requirements

• A Google Cloud project with the topics the operator will read and a subscription the connector can own on each
• roles/pubsub.subscriber, which carries pubsub.subscriptions.consume, on every subscription the connector reads
• roles/pubsub.publisher, which carries pubsub.topics.publish, on any topic the connector writes to
• For push delivery, a public HTTPS endpoint that answers 102, 200, 201, 202, or 204 to acknowledge a message

Authentication

A Google Cloud service account granted roles/pubsub.subscriber or roles/pubsub.publisher at the project level or on the individual topic and subscription; push requests carry a JWT that Pub/Sub signs.

Limits

• A message's data field is at most 10 MB, with at most 100 attributes. A publish request carries at most 1,000 messages or 10 MB.
• Unacknowledged messages are kept 7 days by default and a topic can retain up to 31 days; an unused subscription expires after 31 days.
• After negative acknowledgements, push delivery backs off for between 100 milliseconds and 60 seconds and ramps back up with a slow-start algorithm.
• A project holds at most 10,000 topics and 10,000 subscriptions, and publisher throughput is capped per region by region size.

Questions and answers

Push or pull: which does this connector use?

Either. Push delivers to an HTTPS endpoint you own and treats 102, 200, 201, 202, or 204 as an acknowledgement; any other status means redelivery after the deadline. Pull requests batches of up to 1000 messages and acknowledges each one explicitly.

What happens to messages nobody acknowledges?

A subscription keeps them for 7 days by default and keeps delivering them. A dead-letter topic can be configured on the subscription for messages that keep failing, so they can be inspected instead of circling.

Can messages be reprocessed after a bug?

Yes. Create a snapshot of the subscription, or seek the subscription to a timestamp or to that snapshot, and the retained messages are delivered again. Retention on the topic or subscription decides how far back a seek can reach.