Why a failed exhibit import needs a separate queue
Imagine a museum receiving inventory-change events from regional collections teams. Most messages update a staging catalog, but one malformed record lacks an accession identifier. The importer repeatedly fails and the same message returns to the subscription, delaying unrelated work. Google Cloud Pub/Sub supports a dead-letter topic for undeliverable messages. By putting a reviewable boundary around repeated failures, staff can preserve the problematic event for investigation without declaring it safely imported. This guide uses a hypothetical event stream; no message was actually published or forwarded. The design focuses on keeping provenance intact, understanding retries and preventing a repair operation from creating an accidental second catalog record.
Configure the dead-letter policy on the subscription
The dead-letter topic is not a property of the source publishing topic. It is configured on the subscription whose delivery failures should be handled. Create a distinct dead-letter topic and a subscription to read from it, then point the source subscription’s deadLetterPolicy at that destination. Pub/Sub documentation allows a configurable maximum between five and one hundred delivery attempts, with five as the default. Choose a value based on how failures are investigated, not on a belief that the number is an exact cutoff. Give each subscription a name that reflects its dataset and environment, and keep staging and production policies separate. A missing dead-letter subscriber is a risk, because forwarded messages must remain observable.
Give the Pub/Sub service account the necessary roles
A dead-letter topic can exist yet fail operationally because the managed Pub/Sub service agent lacks permission to publish to the destination or acknowledge source messages as required. The documentation calls out grants after resource setup; use its IAM guidance for the actual project configuration. Check cross-project references carefully when the source subscription and dead-letter topic live in separate projects. Avoid giving the importer broad project-owner rights merely to resolve an access error. The service agent needs the documented roles, whereas a human reviewer needs only enough access to inspect relevant sanitized event attributes. Treat an authorization failure as a configuration problem to fix, not a reason to silently increase the retry limit.
Treat delivery-attempt counts as approximate
Pub/Sub describes forwarding as best effort. The count can reset in some circumstances, and the service may attempt delivery a different number of times from the configured maximum. For pull subscriptions, the client can receive a delivery_attempt field; push delivery uses deliveryAttempt. When Pub/Sub forwards a message, it wraps the original payload and adds source-subscription metadata. Staff should use these fields as troubleshooting clues, not as a precise invoice or a legal record of exactly how many processing attempts occurred. A single event might have been handled by multiple application attempts, and a worker might have committed a side effect before failing its acknowledgment. That risk is independent of the transport’s retry counter.
Distinguish a poison message from a temporary outage
Not all failures deserve the same recovery action. An accession number with an invalid format will probably fail again until a human corrects source data. A temporary database outage might succeed when the service is restored. A permission error could affect every message of a subscription and indicate a broader incident. Put a structured failure category in an internal incident record, along with a pseudonymous event identifier, relevant version and sanitized exception type. Do not place full donor correspondence in alerts. The diagram compares a successful acknowledgment, a retrying event and a dead-lettered event; these are teaching illustrations rather than measured Pub/Sub traffic. The correct route depends on the underlying fault, not on which shape looks most reassuring.
Make import operations idempotent before replay
A handler should check whether an event ID and payload revision have already produced the intended catalog mutation. Prefer conditional writes or an application-level processed-event ledger appropriate to the database, and make the order of operations explicit. Otherwise a worker can commit an update, crash before acknowledgment and commit it again on replay. For a museum record, repeated application might duplicate inventory movements or overwrite the newer approved caption. A replay script should first compare original event identifiers, current record revision and the approved source of truth. Automatic blind replay of every dead-lettered message is dangerous when the failure came from incorrect content rather than a transient system condition.
Inspect the wrapped message and decide its disposition
The dead-letter subscription receives a wrapper with source metadata, not necessarily the same envelope the normal subscriber originally saw. The review tool should extract the original message carefully and preserve attributes needed for routing, while stripping any unnecessary private values from the investigation view. Create dispositions such as repair upstream and replay, discard with documented reason, or escalate for rights review. When re-publishing a corrected event, attach an explicit lineage reference so it can be traced back to the original failure. Never simply edit the historical source message in place while claiming the replay is identical. The goal is a traceable correction process that staff can explain after the incident.
Monitor the destination as well as the source
An empty primary backlog does not prove the import succeeded. It can mean messages were acknowledged by the importer, filtered out for that subscription, or moved to the dead-letter route. Monitor the source’s delivery failures, the dead-letter subscription backlog, message age and resolution disposition together. Test with harmless synthetic failures and ensure operations staff can identify both misconfigured IAM and a reader that stopped pulling from the dead-letter subscription. Alerts should route to a responsible team and carry references, not raw message bodies. A collection manager should also receive a count of unresolved import events that affect a published catalog snapshot, so the snapshot is not labeled complete without evidence.
Define a supervised recovery handoff
Write down who can change the dead-letter policy, who may inspect messages, who can approve corrections and who runs a replay. Specify the maximum-attempt value with a note that actual counts are best effort, plus the target topic, linked subscription and IAM grants. Attach a recovery checklist that validates event identity, current record version and any duplicate side effects before republishing. A successful Pub/Sub acknowledgment confirms transport handling, not that the collection metadata is correct or cleared for public use. Preserve the distinction between resolving a technical failure and approving the revised archival description. The service gives a useful containment mechanism; human review and application safeguards complete the workflow.
How We Research
For this guide, we reviewed the official product documentation on November 8, 2026: Google Cloud Dead-Letter Topics Guide, Google Cloud Subscription Properties, Google Pub/Sub Subscriptions REST Reference. We checked the named setting or feature against its published behavior, mapped the documented constraints to the specific hypothetical scenario, and reviewed whether the topic duplicated titles in the supplied LabFin WordPress inventory or earlier LabFin draft article packages. No authenticated vendor service was run; we did not perform a benchmark, observe real customer data, verify third-party rights or claim operational outcomes. All illustrated records and comparisons are teaching examples. Confirm product documentation and security requirements again before implementing a workflow in a real organization.
Frequently Asked Questions
No. Pub/Sub configures the dead-letter policy on each subscription.
Google documents values from 5 through 100; the default is 5.
No. Forwarding is best effort, and delivery-attempt tracking may reset in some circumstances.
Yes. Create a reader subscription so forwarded messages can be reviewed and recovered.
Only after the handler is idempotent and the failure is understood; some messages require data correction or human approval.