file-and-partner-feed-ingestion

Name: file-and-partner-feed-ingestion
Author: vaquarkhan

19stars4forks

How to install file-and-partner-feed-ingestion?

vaquarkhan/data-engineering-agent-skills/file-and-partner-feed-ingestion

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill file-and-partner-feed-ingestion

Installs into the current project.

Summary

Guides agents through file-based and partner-feed ingestion workflows. Use when landing data from SFTP, managed file transfer, shared buckets, recurring flat files, manifests, or externally supplied feeds that need validation, replay safety, and publish discipline.

Files · 1

View on GitHub

SKILL.md

1	# File And Partner Feed Ingestion
2
3	## Overview
4
5	Use this skill when ingestion depends on externally delivered files rather than APIs or CDC streams. It helps agents design landing, validation, late-file handling, checksum or manifest controls, replay safety, and publish boundaries for partner-managed feeds.
6
7	## When to Use
8
9	- onboarding `SFTP`, MFT, or shared-bucket file feeds
10	- ingesting recurring `CSV`, `JSON`, `XML`, or columnar extracts from external partners
11	- validating manifests, checksums, control totals, or arrival windows
12	- handling late, duplicate, partial, or corrected file deliveries
13	- defining replay-safe landing and publish behavior for externally supplied data
14
15	Do not assume file ingestion is simple just because the payload arrives in a batch.
16
17	## Workflow
18
19	1. Define the feed contract.
20	Clarify:
21	- file format and expected schema
22	- naming convention
23	- arrival schedule or SLA
24	- manifest, checksum, or control-total expectations
25	- ownership and contact path for the partner or source team
26
27	2. Design the landing boundary.
28	Decide:
29	- where raw files land
30	- what metadata is captured
31	- how duplicates are detected
32	- how partial or corrupt deliveries are quarantined
33
34	3. Validate before publish.
35	Include:
36	- schema and required-field checks
37	- checksum or manifest validation
38	- row-count or control-total validation
39	- late-file and missing-file detection
40
41	4. Make replay behavior explicit.
42	Decide how:
43	- corrected files are handled
44	- duplicate files are ignored or reconciled
45	- replay windows are bounded
46	- downstream publishes stay protected until validation passes
47
48	5. Define failure and escalation behavior.
49	Document:
50	- who gets alerted
51	- what blocks publish
52	- how the partner is contacted
53	- how recovery evidence is recorded
54
55	## Common Rationalizations
56
57	\| Rationalization \| Reality \|
58	\| --- \| --- \|
59	\| "It is just a file drop." \| File-based feeds often fail through late arrival, partial delivery, duplicate upload, or silent schema drift. \|
60	\| "We can validate after loading." \| Weak landing controls let corrupt or incomplete partner data flow too far downstream. \|
61	\| "If the partner resends the file, we are fine." \| Resends without replay-safe handling can create duplicate or conflicting publishes. \|
62
63	## Red Flags
64
65	- file naming and arrival expectations are undocumented
66	- manifest, checksum, or control totals are ignored
67	- duplicate and corrected file behavior is undefined
68	- late or missing files do not create actionable alerts
69	- downstream publish opens before the landing validations complete
70
71	## Verification
72
73	- [ ] The feed contract defines format, arrival, and ownership expectations
74	- [ ] Landing, quarantine, and duplicate-handling rules are explicit
75	- [ ] Validation covers schema, completeness, and control totals where relevant
76	- [ ] Replay and corrected-file behavior are documented before go-live
77	- [ ] Failure handling and partner escalation paths are operationally clear

People who liked this also viewed

6 picks

Type

file-and-partner-feed-ingestion

19stars4forks

View on GitHub

How to install file-and-partner-feed-ingestion?

vaquarkhan/data-engineering-agent-skills/file-and-partner-feed-ingestion

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill file-and-partner-feed-ingestion

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# File And Partner Feed Ingestion
2
3	## Overview
4
5	Use this skill when ingestion depends on externally delivered files rather than APIs or CDC streams. It helps agents design landing, validation, late-file handling, checksum or manifest controls, replay safety, and publish boundaries for partner-managed feeds.
6
7	## When to Use
8
9	- onboarding `SFTP`, MFT, or shared-bucket file feeds
10	- ingesting recurring `CSV`, `JSON`, `XML`, or columnar extracts from external partners
11	- validating manifests, checksums, control totals, or arrival windows
12	- handling late, duplicate, partial, or corrected file deliveries
13	- defining replay-safe landing and publish behavior for externally supplied data
14
15	Do not assume file ingestion is simple just because the payload arrives in a batch.
16
17	## Workflow
18
19	1. Define the feed contract.
20	Clarify:
21	- file format and expected schema
22	- naming convention
23	- arrival schedule or SLA
24	- manifest, checksum, or control-total expectations
25	- ownership and contact path for the partner or source team
26
27	2. Design the landing boundary.
28	Decide:
29	- where raw files land
30	- what metadata is captured
31	- how duplicates are detected
32	- how partial or corrupt deliveries are quarantined
33
34	3. Validate before publish.
35	Include:
36	- schema and required-field checks
37	- checksum or manifest validation
38	- row-count or control-total validation
39	- late-file and missing-file detection
40
41	4. Make replay behavior explicit.
42	Decide how:
43	- corrected files are handled
44	- duplicate files are ignored or reconciled
45	- replay windows are bounded
46	- downstream publishes stay protected until validation passes
47
48	5. Define failure and escalation behavior.
49	Document:
50	- who gets alerted
51	- what blocks publish
52	- how the partner is contacted
53	- how recovery evidence is recorded
54
55	## Common Rationalizations
56
57	\| Rationalization \| Reality \|
58	\| --- \| --- \|
59	\| "It is just a file drop." \| File-based feeds often fail through late arrival, partial delivery, duplicate upload, or silent schema drift. \|
60	\| "We can validate after loading." \| Weak landing controls let corrupt or incomplete partner data flow too far downstream. \|
61	\| "If the partner resends the file, we are fine." \| Resends without replay-safe handling can create duplicate or conflicting publishes. \|
62
63	## Red Flags
64
65	- file naming and arrival expectations are undocumented
66	- manifest, checksum, or control totals are ignored
67	- duplicate and corrected file behavior is undefined
68	- late or missing files do not create actionable alerts
69	- downstream publish opens before the landing validations complete
70
71	## Verification
72
73	- [ ] The feed contract defines format, arrival, and ownership expectations
74	- [ ] Landing, quarantine, and duplicate-handling rules are explicit
75	- [ ] Validation covers schema, completeness, and control totals where relevant
76	- [ ] Replay and corrected-file behavior are documented before go-live
77	- [ ] Failure handling and partner escalation paths are operationally clear

People who liked this also viewed

6 picks

Type