Delivery and Reliability

Reliability model

Signal Client combines webhook rotation with optional fallback persistence so that burst traffic and temporary failures do not become lost bot-side updates. The design is intentionally layered: first try live delivery through the pool, then queue failed envelopes and retry them with exponential backoff. This gives bot teams a practical path to reliability without forcing every producer service to implement its own retry scheduler.

The FallbackWorker is the core reliability primitive. When the fallback system is enabled and a send call fails, the worker persists the envelope into a FallbackStore and retries it in background batches. After exhausting the maximum number of attempts, the envelope is moved to a dead-letter partition for manual inspection.

Fallback configuration reference

The FallbackConfig object controls every aspect of the retry system. All fields except enabled and store have sensible defaults, so a minimal configuration only needs to toggle the feature on and choose a persistence backend.

Choosing a fallback store

The FallbackStore interface defines a persistence contract with eight methods: enqueue, claimBatch, markProcessed, reschedule, moveToDeadLetter, has, pendingCount, and deadLetterCount. Two built-in implementations cover the most common deployment scenarios. You can also implement the interface yourself for custom backends.

import { SignalClient } from '@spraxium/signal-client';

const client = new SignalClient({
  webhookUrls: [process.env.SIGNAL_WEBHOOK_URL_1!],
  hmacSecret: process.env.SIGNAL_HMAC_SECRET!,
  rateLimitStrategy: 'wait',
  fallback: {
    enabled: true,
    store: 'file',
    filePath: '.spraxium-signal-fallback.json',
    retryIntervalMs: 5_000,
    maxAttempts: 10,
    batchSize: 20,
    events: ['config.update', 'billing.status.changed'],
  },
});

FileFallbackStore keeps pending, dead-letter, and processed entries in a single JSON file. It loads the file lazily on first access and flushes after every mutation. The processed list is automatically trimmed to the last 5000 entries when it exceeds 10000, preventing unbounded growth. This store works well for local development and small single-instance services.

RedisFallbackStore persists entries as individual Redis keys under the spraxium:signal:fallback:pending: prefix, with dead-letter entries under spraxium:signal:fallback:deadletter: and processed IDs in a set at spraxium:signal:fallback:processed. Redis is preferable for multi-instance emitters because fallback state is shared across replicas without file system coordination.

Exponential backoff and jitter

When a retry attempt fails, the worker reschedules the envelope using an exponential backoff formula with additive jitter. The formula prevents retry storms by spreading attempts over increasingly longer intervals while the jitter component prevents multiple envelopes from retrying at the same instant.

The exact formula is: retryIntervalMs * 2^min(attempt, 8) + random() * retryIntervalMs * 0.3. The exponent is capped at 8 to prevent absurdly long delays on high attempt counts. With the default retryIntervalMs of 5000 and maxAttempts of 10, the maximum backoff before jitter is roughly 1280 seconds (about 21 minutes) at the eighth attempt.

Dead-letter handling

When an envelope exhausts the configured maxAttempts, the worker moves it to the dead-letter partition via moveToDeadLetter. The original entry is removed from the pending queue and stored with the last error reason attached. Dead-letter envelopes are never retried automatically, so they require manual investigation.

In the file store, dead-letter entries accumulate in the deadLetter array of the JSON file. In the Redis store, they move to individual keys under spraxium:signal:fallback:deadletter:. Both stores track the count via deadLetterCount, which you should monitor as part of your operational health checks.

The stats method on the FallbackWorker returns both pending and deadLetter counts in a single call. Exposing this data to your monitoring system is the simplest way to detect persistent delivery failures before they affect user-facing bot behavior.

const { pending, deadLetter } = await client.fallback!.stats();

if (deadLetter > 0) {
  logger.warn(
    `Signal fallback: ${deadLetter} dead-letter envelopes require investigation`,
  );
}

if (pending > 100) {
  logger.warn(
    `Signal fallback: ${pending} pending envelopes, possible sustained delivery failure`,
  );
}

Fallback entry lifecycle

Reliability profiles by scenario

Graceful shutdown

Call destroy on the client during your application shutdown sequence to stop the FallbackWorker cleanly. This clears the internal setInterval timer and prevents partial retry loops during rolling deployments. The worker uses timer.unref() to avoid keeping the Node.js process alive, but explicitly destroying the client ensures deterministic cleanup.

process.on('SIGTERM', () => {
  client.destroy();
  process.exit(0);
});

If you skip the destroy call, pending retry cycles may execute partially during shutdown, potentially producing duplicate deliveries. In containerized environments where shutdowns happen frequently, this is especially important to handle correctly.

Operational runbook