$npx -y skills add marckohlbrugge/37signals-skills --skill rails-webhooksBuild and review Rails webhook systems with safe delivery, retries, observability, and tenant-aware security controls. Use when adding webhook endpoints, outbound deliveries, retry logic, or webhook admin tooling.
| 1 | # Rails Webhooks |
| 2 | |
| 3 | Use for outbound/inbound webhook architecture and reliability. Modeled on Fizzy's event-driven webhook pipeline (with Campfire's simpler bot webhooks as contrast). |
| 4 | |
| 5 | ## Delivery Model (outbox pattern) |
| 6 | |
| 7 | - Domain events enqueue one dispatch job; the dispatch job fans out, creating one persisted `Delivery` row per webhook, each with a state enum (`pending in_progress completed errored` — string-backed via `index_by(&:itself)`). |
| 8 | - Each delivery row auto-enqueues its own send via `after_create_commit :deliver_later` — persist first, deliver second, so state survives crashes and retries. |
| 9 | - Fan-out is crash-safe with `ActiveJob::Continuable`: cursor over matching webhooks (`find_each(start: step.cursor)` + `step.advance!`) so a mid-batch crash resumes, not restarts. |
| 10 | - Record request metadata (headers, payload) and response (status, body) on the delivery row for audit/debugging. Cap stored/streamed response bodies (~100KB) with a running byte count. |
| 11 | - Use a dedicated `webhooks` queue so slow destinations can't starve other work. |
| 12 | |
| 13 | ## Failure Classification (the key distinction) |
| 14 | |
| 15 | - **Expected destination failures** (timeout, TLS, DNS, connection refused, HTTP 4xx/5xx): rescue, mark delivery `completed` with a symbolic error (`response: { error: :connection_timeout }`). The delivery ran; the destination failed. Do not retry the job. |
| 16 | - **Unexpected exceptions** (our bug): mark `errored!`, re-raise so ActiveJob retries. |
| 17 | - This split keeps retry behavior, dashboards, and delinquency tracking honest. |
| 18 | |
| 19 | ## Delinquency Circuit Breaker |
| 20 | |
| 21 | - Track consecutive failures + `first_failure_at` per webhook; auto-deactivate after N failures spanning a minimum window (Fizzy: 10 failures over 1+ hour). |
| 22 | - Reset the counter on success. |
| 23 | - Surface inactive state in the UI with a manual reactivation endpoint (`resource :activation`). |
| 24 | |
| 25 | ## Security Baseline |
| 26 | |
| 27 | - Treat webhook URLs as untrusted input; apply full SSRF protections (resolve + validate IP, block private ranges, pin IP, re-validate per redirect — see rails-security-multitenancy). |
| 28 | - Revalidate destination at send time, not just on create. |
| 29 | - **Destination URL is immutable after create** — updates permit name/subscriptions only. Retargeting requires a new webhook (and new secret). |
| 30 | - Sign payloads: HMAC signature header + timestamp header. |
| 31 | - Whitelist subscribable events at the model layer: `normalizes :subscribed_actions, with: ->(v) { Array.wrap(v).map(&:to_s).uniq & PERMITTED_ACTIONS }`. |
| 32 | - Require admin-level auth for webhook management endpoints. |
| 33 | |
| 34 | ## Integration Adapters |
| 35 | |
| 36 | - One delivery pipeline can serve multiple destination types: detect by URL pattern (`for_slack?`, `for_campfire?`), then vary content type, payload format (JSON/form/HTML), and rendering template per destination. Don't fork the delivery code per integration. |
| 37 | - Render payload URLs with tenant-correct `script_name` so links in payloads work. |
| 38 | |
| 39 | ## Inbound Webhooks (receiving) |
| 40 | |
| 41 | - Verify the signature first (`construct_event`-style), then re-fetch canonical state from the provider's API instead of trusting payload content or ordering. |
| 42 | - For chat-bot style callbacks: gate what responses can do by content type (only `text/plain`/`text/html` become replies), and prevent loops — never trigger a bot from its own messages. |
| 43 | |
| 44 | ## Tenant Safety |
| 45 | |
| 46 | - Keep webhook records and delivery queries tenant-scoped. |
| 47 | - Ensure event fan-out cannot leak cross-tenant data. |
| 48 | |
| 49 | ## Operational Hygiene |
| 50 | |
| 51 | - Recurring cleanup of old delivery records by retention policy (e.g. every 4 hours, `delete_all` on a stale scope). |
| 52 | - Surface delivery status/history in the webhook admin UI. |
| 53 | - Emit useful logs/metrics for success rate, retries, and latency. |
| 54 | |
| 55 | ## Red Flags |
| 56 | |
| 57 | - Fire-and-forget delivery with no persisted audit trail. |
| 58 | - Retrying destination failures the same way as code errors. |
| 59 | - Mutable webhook destination URLs. |
| 60 | - No circuit breaker — hammering dead endpoints forever. |
| 61 | - Unbounded reads of destination responses. |
| 62 | - No tenant scoping in delivery creation/lookup. |
| 63 | - No backpressure or queue isolation for high-volume events. |