api-and-saas-ingestion-patterns

Name: api-and-saas-ingestion-patterns
Author: vaquarkhan

19stars4forks

How to install api-and-saas-ingestion-patterns?

vaquarkhan/data-engineering-agent-skills/api-and-saas-ingestion-patterns

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill api-and-saas-ingestion-patterns

Installs into the current project.

Summary

Guides agents through API and SaaS ingestion workflows. Use when extracting data from REST, GraphQL, or SaaS platforms with pagination, rate limits, auth rotation, backfills, or unstable source contracts.

Files · 1

View on GitHub

SKILL.md

1	# API And SaaS Ingestion Patterns
2
3	## Overview
4
5	Use this skill when the source system is an external API or SaaS platform rather than a database or file drop. It helps agents design reliable extraction, pagination, throttling, auth handling, and backfill-safe ingestion contracts.
6
7	## When to Use
8
9	- extracting from REST or GraphQL APIs
10	- ingesting SaaS platform data
11	- handling pagination, cursors, or sync tokens
12	- dealing with rate limits and source-side throttling
13	- backfilling historical API data safely
14
15	Do not treat APIs like static tables. They change behavior, availability, and limits over time.
16
17	## Workflow
18
19	1. Define the source contract.
20	Include:
21	- endpoint or object name
22	- auth method
23	- extraction window
24	- pagination style
25	- rate limits
26	- data freshness expectations
27
28	2. Design for extraction resilience.
29	Handle:
30	- retries
31	- backoff
32	- token refresh
33	- idempotent re-fetch behavior
34	- partial page failure
35
36	3. Make incremental behavior explicit.
37	Decide whether the source supports:
38	- updated timestamps
39	- cursors
40	- change tokens
41	- full snapshots only
42
43	4. Record raw source evidence where useful.
44	API sources often need raw response retention for incident diagnosis.
45
46	5. Validate contracts against source drift.
47
48	## Common Rationalizations
49
50	\| Rationalization \| Reality \|
51	\| --- \| --- \|
52	\| "We can just loop through pages." \| Pagination bugs and retry gaps often cause silent data loss. \|
53	\| "The vendor API is stable enough." \| SaaS APIs change rate limits, fields, and semantics more often than teams expect. \|
54	\| "If a request fails, we can rerun later." \| Without windowing and idempotency rules, reruns can miss or duplicate data. \|
55
56	## Red Flags
57
58	- no rate-limit strategy exists
59	- extraction windows depend on undocumented source behavior
60	- retries ignore duplicate or partial-page risks
61	- auth rotation and token expiry are not considered
62
63	## Verification
64
65	- [ ] The source contract covers pagination, limits, auth, and cadence
66	- [ ] Extraction retries and failures are operationally safe
67	- [ ] Incremental or snapshot behavior is explicit
68	- [ ] Source drift and raw evidence handling are considered

People who liked this also viewed

6 picks

Type

api-and-saas-ingestion-patterns

19stars4forks

View on GitHub

How to install api-and-saas-ingestion-patterns?

vaquarkhan/data-engineering-agent-skills/api-and-saas-ingestion-patterns

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill api-and-saas-ingestion-patterns

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# API And SaaS Ingestion Patterns
2
3	## Overview
4
5	Use this skill when the source system is an external API or SaaS platform rather than a database or file drop. It helps agents design reliable extraction, pagination, throttling, auth handling, and backfill-safe ingestion contracts.
6
7	## When to Use
8
9	- extracting from REST or GraphQL APIs
10	- ingesting SaaS platform data
11	- handling pagination, cursors, or sync tokens
12	- dealing with rate limits and source-side throttling
13	- backfilling historical API data safely
14
15	Do not treat APIs like static tables. They change behavior, availability, and limits over time.
16
17	## Workflow
18
19	1. Define the source contract.
20	Include:
21	- endpoint or object name
22	- auth method
23	- extraction window
24	- pagination style
25	- rate limits
26	- data freshness expectations
27
28	2. Design for extraction resilience.
29	Handle:
30	- retries
31	- backoff
32	- token refresh
33	- idempotent re-fetch behavior
34	- partial page failure
35
36	3. Make incremental behavior explicit.
37	Decide whether the source supports:
38	- updated timestamps
39	- cursors
40	- change tokens
41	- full snapshots only
42
43	4. Record raw source evidence where useful.
44	API sources often need raw response retention for incident diagnosis.
45
46	5. Validate contracts against source drift.
47
48	## Common Rationalizations
49
50	\| Rationalization \| Reality \|
51	\| --- \| --- \|
52	\| "We can just loop through pages." \| Pagination bugs and retry gaps often cause silent data loss. \|
53	\| "The vendor API is stable enough." \| SaaS APIs change rate limits, fields, and semantics more often than teams expect. \|
54	\| "If a request fails, we can rerun later." \| Without windowing and idempotency rules, reruns can miss or duplicate data. \|
55
56	## Red Flags
57
58	- no rate-limit strategy exists
59	- extraction windows depend on undocumented source behavior
60	- retries ignore duplicate or partial-page risks
61	- auth rotation and token expiry are not considered
62
63	## Verification
64
65	- [ ] The source contract covers pagination, limits, auth, and cadence
66	- [ ] Extraction retries and failures are operationally safe
67	- [ ] Incremental or snapshot behavior is explicit
68	- [ ] Source drift and raw evidence handling are considered

People who liked this also viewed

6 picks

Type