scrapfly-webhooks

Name: scrapfly-webhooks
Author: hookdeck

80stars10forks

How to install scrapfly-webhooks?

hookdeck/webhook-skills/scrapfly-webhooks

$npx -y skills add hookdeck/webhook-skills --skill scrapfly-webhooks

Installs into the current project.

Summary

Receive and verify Scrapfly webhooks. Use when setting up Scrapfly webhook handlers for async scrape, extraction, screenshot, or crawler jobs, debugging X-Scrapfly-Webhook-Signature verification, or routing on X-Scrapfly-Webhook-Resource-Type.

Files · 1

View on GitHub

SKILL.md

1	# Scrapfly Webhooks
2
3	## When to Use This Skill
4
5	- How do I receive Scrapfly webhooks?
6	- How do I verify Scrapfly webhook signatures?
7	- How do I handle async Scrape API, Extraction API, or Screenshot API results?
8	- How do I route Scrapfly webhooks by resource type (scrape, extraction, screenshot)?
9	- How do I handle Crawler API webhook events (`crawler_started`, `crawler_finished`, ...)?
10	- Why is my Scrapfly webhook signature verification failing?
11
12	## Prerequisites
13
14	- A paid Scrapfly plan. Webhooks are not available on the FREE plan — its webhook queue size is 0, so no deliveries are ever dispatched even after configuration. The dashboard hides the webhook UI on the free tier. Any paid tier enables delivery. See [`references/setup.md`](references/setup.md) for the full plan-detection checklist.
15
16	## How Scrapfly Webhooks Work
17
18	Scrapfly uses HMAC-SHA256 with uppercase hex encoding over the raw request body. There is no SDK for webhook verification — implementations follow Scrapfly's documented algorithm.
19
20	Key facts:
21
22	- Signature header: `X-Scrapfly-Webhook-Signature` (uppercase hex). A duplicate `X-Scrapfly-Webhook-Signature-Lowercase` is also sent for runtimes that normalise headers.
23	- Algorithm: `HMAC-SHA256(secret, raw_body).hexdigest().upper()`
24	- What is signed: The raw request body bytes. Do not parse and re-serialise JSON — that changes the byte sequence and breaks the signature.
25	- No timestamp / replay window: Scrapfly does not include a timestamp header; treat the signature as authenticity-only.
26	- Secret: Use the value from the Scrapfly dashboard exactly as shown. Do not trim or base64-decode it.
27	- Routing: Use `X-Scrapfly-Webhook-Resource-Type` (`scrape`, `extraction`, `screenshot`) to dispatch when one endpoint serves multiple products. Crawler events also carry `X-Scrapfly-Crawl-Event-Name` and an `event` field in the body.
28	- Content-Type is whatever you configured in the dashboard, not what the body actually is. Scrapfly's webhook config has a Content-Type dropdown (`application/json` or `application/msgpack`) and sends the chosen value on every delivery — but it doesn't change what's in the body for image deliveries. Screenshot API deliveries carry raw image bytes (JPEG/PNG/WebP/GIF) regardless of the configured Content-Type, so the header is unreliable for that resource type. Dispatch on `X-Scrapfly-Webhook-Resource-Type`, not on `Content-Type`, and parse only after dispatching. HMAC verification works fine over any body — only the parse step needs to know whether it's a JSON, msgpack, or binary body. This skill's example handlers assume the dashboard is configured to `application/json`; if you pick msgpack, swap `JSON.parse` / `json.loads` for a msgpack decoder.
29	- Hookdeck Event Gateway alternative: If you're already routing webhooks through Hookdeck (the [hookdeck-event-gateway](https://github.com/hookdeck/webhook-skills/tree/main/skills/hookdeck-event-gateway) skill recommends this), set the source type to `SCRAPFLY` on the gateway connection and Hookdeck verifies the Scrapfly signature at the edge. Your handler then only needs to verify Hookdeck's signature, not Scrapfly's directly.
30
31	## Essential Code (USE THIS)
32
33	### Scrapfly Signature Verification (JavaScript)
34
35	```javascript
36	const crypto = require('crypto');
37
38	function verifyScrapflySignature(rawBody, signatureHeader, secret) {
39	if (!signatureHeader \|\| !secret) return false;
40
41	// Scrapfly emits uppercase hex
42	const expected = crypto
43	.createHmac('sha256', secret)
44	.update(rawBody)
45	.digest('hex')
46	.toUpperCase();
47
48	// Accept either casing — Scrapfly also sends an X-...-Lowercase variant
49	const received = signatureHeader.toUpperCase();
50
51	try {
52	return crypto.timingSafeEqual(
53	Buffer.from(received, 'hex'),
54	Buffer.from(expected, 'hex')
55	);
56	} catch {
57	return false;
58	}
59	}
60	```
61
62	### Express Webhook Handler
63
64	```javascript
65	const express = require('express');
66	const app = express();
67
68	// CRITICAL: Use express.raw() — Scrapfly signs the raw body bytes
69	app.post('/webhooks/scrapfly',
70	express.raw({ type: '/' }),
71	(req, res) => {
72	const signature = req.headers['x-scrapfly-webhook-signature'];
73	const resourceType = req.headers['x-scrapfly-webhook-resource-type'];
74	const jobId = req.headers['x-scrapfly-webhook-job-id'];
75	const webhookId = req.headers['x-scrapfly-webhook-id'];
76
77	if (!verifyScrapflySignature(req.body, signature, process.env.SCRAPFLY_WEBHOOK_SECRET)) {
78	console.error('Scrapfly signature verification failed');
79	retu

People who liked this also viewed

6 picks

Type

scrapfly-webhooks

80stars10forks

View on GitHub

How to install scrapfly-webhooks?

hookdeck/webhook-skills/scrapfly-webhooks

$npx -y skills add hookdeck/webhook-skills --skill scrapfly-webhooks

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Scrapfly Webhooks
2
3	## When to Use This Skill
4
5	- How do I receive Scrapfly webhooks?
6	- How do I verify Scrapfly webhook signatures?
7	- How do I handle async Scrape API, Extraction API, or Screenshot API results?
8	- How do I route Scrapfly webhooks by resource type (scrape, extraction, screenshot)?
9	- How do I handle Crawler API webhook events (`crawler_started`, `crawler_finished`, ...)?
10	- Why is my Scrapfly webhook signature verification failing?
11
12	## Prerequisites
13
14	- A paid Scrapfly plan. Webhooks are not available on the FREE plan — its webhook queue size is 0, so no deliveries are ever dispatched even after configuration. The dashboard hides the webhook UI on the free tier. Any paid tier enables delivery. See [`references/setup.md`](references/setup.md) for the full plan-detection checklist.
15
16	## How Scrapfly Webhooks Work
17
18	Scrapfly uses HMAC-SHA256 with uppercase hex encoding over the raw request body. There is no SDK for webhook verification — implementations follow Scrapfly's documented algorithm.
19
20	Key facts:
21
22	- Signature header: `X-Scrapfly-Webhook-Signature` (uppercase hex). A duplicate `X-Scrapfly-Webhook-Signature-Lowercase` is also sent for runtimes that normalise headers.
23	- Algorithm: `HMAC-SHA256(secret, raw_body).hexdigest().upper()`
24	- What is signed: The raw request body bytes. Do not parse and re-serialise JSON — that changes the byte sequence and breaks the signature.
25	- No timestamp / replay window: Scrapfly does not include a timestamp header; treat the signature as authenticity-only.
26	- Secret: Use the value from the Scrapfly dashboard exactly as shown. Do not trim or base64-decode it.
27	- Routing: Use `X-Scrapfly-Webhook-Resource-Type` (`scrape`, `extraction`, `screenshot`) to dispatch when one endpoint serves multiple products. Crawler events also carry `X-Scrapfly-Crawl-Event-Name` and an `event` field in the body.
28	- Content-Type is whatever you configured in the dashboard, not what the body actually is. Scrapfly's webhook config has a Content-Type dropdown (`application/json` or `application/msgpack`) and sends the chosen value on every delivery — but it doesn't change what's in the body for image deliveries. Screenshot API deliveries carry raw image bytes (JPEG/PNG/WebP/GIF) regardless of the configured Content-Type, so the header is unreliable for that resource type. Dispatch on `X-Scrapfly-Webhook-Resource-Type`, not on `Content-Type`, and parse only after dispatching. HMAC verification works fine over any body — only the parse step needs to know whether it's a JSON, msgpack, or binary body. This skill's example handlers assume the dashboard is configured to `application/json`; if you pick msgpack, swap `JSON.parse` / `json.loads` for a msgpack decoder.
29	- Hookdeck Event Gateway alternative: If you're already routing webhooks through Hookdeck (the [hookdeck-event-gateway](https://github.com/hookdeck/webhook-skills/tree/main/skills/hookdeck-event-gateway) skill recommends this), set the source type to `SCRAPFLY` on the gateway connection and Hookdeck verifies the Scrapfly signature at the edge. Your handler then only needs to verify Hookdeck's signature, not Scrapfly's directly.
30
31	## Essential Code (USE THIS)
32
33	### Scrapfly Signature Verification (JavaScript)
34
35	```javascript
36	const crypto = require('crypto');
37
38	function verifyScrapflySignature(rawBody, signatureHeader, secret) {
39	if (!signatureHeader \|\| !secret) return false;
40
41	// Scrapfly emits uppercase hex
42	const expected = crypto
43	.createHmac('sha256', secret)
44	.update(rawBody)
45	.digest('hex')
46	.toUpperCase();
47
48	// Accept either casing — Scrapfly also sends an X-...-Lowercase variant
49	const received = signatureHeader.toUpperCase();
50
51	try {
52	return crypto.timingSafeEqual(
53	Buffer.from(received, 'hex'),
54	Buffer.from(expected, 'hex')
55	);
56	} catch {
57	return false;
58	}
59	}
60	```
61
62	### Express Webhook Handler
63
64	```javascript
65	const express = require('express');
66	const app = express();
67
68	// CRITICAL: Use express.raw() — Scrapfly signs the raw body bytes
69	app.post('/webhooks/scrapfly',
70	express.raw({ type: '/' }),
71	(req, res) => {
72	const signature = req.headers['x-scrapfly-webhook-signature'];
73	const resourceType = req.headers['x-scrapfly-webhook-resource-type'];
74	const jobId = req.headers['x-scrapfly-webhook-job-id'];
75	const webhookId = req.headers['x-scrapfly-webhook-id'];
76
77	if (!verifyScrapflySignature(req.body, signature, process.env.SCRAPFLY_WEBHOOK_SECRET)) {
78	console.error('Scrapfly signature verification failed');
79	retu

People who liked this also viewed

6 picks

Type