connecting-to-data-source

Name: connecting-to-data-source
Author: aws

2.5kinstalls190forks

How to install connecting-to-data-source?

aws/agent-toolkit-for-aws/connecting-to-data-source

$npx -y skills add aws/agent-toolkit-for-aws --skill connecting-to-data-source

Installs into the current project.

Summary

Create and troubleshoot AWS Glue connections to JDBC databases (Oracle, SQL Server, PostgreSQL, MySQL, RDS), Redshift, Snowflake, and BigQuery. Gathers connection hints from user, discovers existing connections and RDS/Redshift candidates, registers credentials in Secrets Manager

Files · 1

View on GitHub

SKILL.md

1	# Connect to Data Source
2
3	Register an external data source with AWS Glue so downstream skills (ingesting-into-data-lake) can move data from it. A Glue connection stores the network config, driver, and credential reference for one source. Create once per source, reuse across jobs.
4
5	## Philosophy
6
7	A connection is a named pipe, not a pipeline. This skill produces a tested, reusable Glue connection. It does not move data.
8
9	## Common Tasks
10
11	You MUST execute commands using AWS MCP server tools when connected -- they provide validation, sandboxed execution, and audit logging. Fall back to AWS CLI only if MCP is unavailable. You MUST explain each step before executing.
12
13	## Workflow
14
15	### 1. Verify Dependencies and Context
16
17	- You MUST check whether AWS MCP tools or AWS CLI are available and inform the user if missing
18	- You MUST confirm target AWS region and verify credentials with `aws sts get-caller-identity`
19
20	### 2. Classify the Source
21
22	Ask the user which source type they want to connect to, or infer from hints:
23
24	\| User says... \| Source type \| Connection type \| Reference \|
25	\|---\|---\|---\|---\|
26	\| "Oracle", "SQL Server", "Postgres", "MySQL", "RDS \<engine\>" \| JDBC database \| `JDBC` \| [jdbc-setup.md](references/jdbc-setup.md) \|
27	\| "Redshift", "my cluster", "my data warehouse on AWS" \| Redshift \| `JDBC` \| [jdbc-setup.md](references/jdbc-setup.md) (Redshift section) \|
28	\| "Snowflake" \| Snowflake \| `SNOWFLAKE` \| [snowflake-setup.md](references/snowflake-setup.md) \|
29	\| "BigQuery", "Google analytics warehouse" \| BigQuery \| `BIGQUERY` \| [bigquery-setup.md](references/bigquery-setup.md) \|
30
31	If the user names DynamoDB or a local file, stop and tell them: DynamoDB is read directly by Glue without a connection, and local files belong in the ingesting-into-data-lake skill's local-upload workflow.
32
33	### 3. Gather Connection Hints from the User
34
35	You MUST ask for hints the user can provide -- do not guess.
36
37	For all sources:
38
39	- Desired connection name (lowercase, hyphens: `oracle-prod-sales`, `snowflake-analytics`)
40	- Existing Secrets Manager secret, or create one
41	- Is source reachable from a Glue VPC (same, peered, VPN, Direct Connect)
42
43	JDBC: hostname/endpoint, port, database, whether RDS/Aurora/self-managed, IAM DB auth enabled (Aurora/RDS MySQL/Postgres), SSL required.
44
45	Snowflake: account identifier, warehouse, role, default database, auth (password, key-pair, OAuth).
46
47	BigQuery: GCP project ID, location, whether service account JSON is provisioned.
48
49	### 4. Discover Existing Connections and Candidate Sources
50
51	Check what exists before creating.
52
53	Existing Glue connections:
54
55	```bash
56	aws glue get-connections --filter ConnectionType=<TYPE> --region <REGION>
57	```
58
59	If a suitable one exists, confirm and skip to Step 7.
60
61	Candidate sources in account (JDBC/Redshift only):
62
63	- RDS: `aws rds describe-db-instances`
64	- Aurora: `aws rds describe-db-clusters`
65	- Redshift: `aws redshift describe-clusters`
66
67	Present candidates to user; let them pick. See [discovery.md](references/discovery.md).
68
69	### 5. Register Credentials
70
71	You MUST encourage AWS Secrets Manager over plaintext passwords. You SHOULD prefer IAM database authentication where supported (Aurora/RDS MySQL and PostgreSQL, Redshift). See [credential-security.md](references/credential-security.md).
72
73	- You MUST confirm with user before creating a new Secrets Manager secret
74	- You MUST NOT write plaintext credentials into chat or logs
75	- For IAM DB auth, no secret is needed
76
77	### 6. Create the Glue Connection
78
79	Follow the source-specific reference for connection properties:
80
81	```bash
82	aws glue create-connection --connection-input '<JSON>' --region <REGION>
83	```
84
85	Private sources require `PhysicalConnectionRequirements` (SubnetId, SecurityGroupIdList, AvailabilityZone). See [network-setup.md](references/network-setup.md).
86
87	### 7. Test the Connection
88
89	You MUST test before handing off. Testing is two-phase: a quick API check, then an engine-level verification.
90
91	#### Phase A: Glue TestConnection (network and credential sanity check)
92
93	```bash
94	aws glue test-connection --connection-name <NAME> --region <REGION>
95	```
96
97	This

People who liked this also viewed

6 picks

Type

connecting-to-data-source

2.5kinstalls190forks

View on GitHub

How to install connecting-to-data-source?

aws/agent-toolkit-for-aws/connecting-to-data-source

$npx -y skills add aws/agent-toolkit-for-aws --skill connecting-to-data-source

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Connect to Data Source
2
3	Register an external data source with AWS Glue so downstream skills (ingesting-into-data-lake) can move data from it. A Glue connection stores the network config, driver, and credential reference for one source. Create once per source, reuse across jobs.
4
5	## Philosophy
6
7	A connection is a named pipe, not a pipeline. This skill produces a tested, reusable Glue connection. It does not move data.
8
9	## Common Tasks
10
11	You MUST execute commands using AWS MCP server tools when connected -- they provide validation, sandboxed execution, and audit logging. Fall back to AWS CLI only if MCP is unavailable. You MUST explain each step before executing.
12
13	## Workflow
14
15	### 1. Verify Dependencies and Context
16
17	- You MUST check whether AWS MCP tools or AWS CLI are available and inform the user if missing
18	- You MUST confirm target AWS region and verify credentials with `aws sts get-caller-identity`
19
20	### 2. Classify the Source
21
22	Ask the user which source type they want to connect to, or infer from hints:
23
24	\| User says... \| Source type \| Connection type \| Reference \|
25	\|---\|---\|---\|---\|
26	\| "Oracle", "SQL Server", "Postgres", "MySQL", "RDS \<engine\>" \| JDBC database \| `JDBC` \| [jdbc-setup.md](references/jdbc-setup.md) \|
27	\| "Redshift", "my cluster", "my data warehouse on AWS" \| Redshift \| `JDBC` \| [jdbc-setup.md](references/jdbc-setup.md) (Redshift section) \|
28	\| "Snowflake" \| Snowflake \| `SNOWFLAKE` \| [snowflake-setup.md](references/snowflake-setup.md) \|
29	\| "BigQuery", "Google analytics warehouse" \| BigQuery \| `BIGQUERY` \| [bigquery-setup.md](references/bigquery-setup.md) \|
30
31	If the user names DynamoDB or a local file, stop and tell them: DynamoDB is read directly by Glue without a connection, and local files belong in the ingesting-into-data-lake skill's local-upload workflow.
32
33	### 3. Gather Connection Hints from the User
34
35	You MUST ask for hints the user can provide -- do not guess.
36
37	For all sources:
38
39	- Desired connection name (lowercase, hyphens: `oracle-prod-sales`, `snowflake-analytics`)
40	- Existing Secrets Manager secret, or create one
41	- Is source reachable from a Glue VPC (same, peered, VPN, Direct Connect)
42
43	JDBC: hostname/endpoint, port, database, whether RDS/Aurora/self-managed, IAM DB auth enabled (Aurora/RDS MySQL/Postgres), SSL required.
44
45	Snowflake: account identifier, warehouse, role, default database, auth (password, key-pair, OAuth).
46
47	BigQuery: GCP project ID, location, whether service account JSON is provisioned.
48
49	### 4. Discover Existing Connections and Candidate Sources
50
51	Check what exists before creating.
52
53	Existing Glue connections:
54
55	```bash
56	aws glue get-connections --filter ConnectionType=<TYPE> --region <REGION>
57	```
58
59	If a suitable one exists, confirm and skip to Step 7.
60
61	Candidate sources in account (JDBC/Redshift only):
62
63	- RDS: `aws rds describe-db-instances`
64	- Aurora: `aws rds describe-db-clusters`
65	- Redshift: `aws redshift describe-clusters`
66
67	Present candidates to user; let them pick. See [discovery.md](references/discovery.md).
68
69	### 5. Register Credentials
70
71	You MUST encourage AWS Secrets Manager over plaintext passwords. You SHOULD prefer IAM database authentication where supported (Aurora/RDS MySQL and PostgreSQL, Redshift). See [credential-security.md](references/credential-security.md).
72
73	- You MUST confirm with user before creating a new Secrets Manager secret
74	- You MUST NOT write plaintext credentials into chat or logs
75	- For IAM DB auth, no secret is needed
76
77	### 6. Create the Glue Connection
78
79	Follow the source-specific reference for connection properties:
80
81	```bash
82	aws glue create-connection --connection-input '<JSON>' --region <REGION>
83	```
84
85	Private sources require `PhysicalConnectionRequirements` (SubnetId, SecurityGroupIdList, AvailabilityZone). See [network-setup.md](references/network-setup.md).
86
87	### 7. Test the Connection
88
89	You MUST test before handing off. Testing is two-phase: a quick API check, then an engine-level verification.
90
91	#### Phase A: Glue TestConnection (network and credential sanity check)
92
93	```bash
94	aws glue test-connection --connection-name <NAME> --region <REGION>
95	```
96
97	This

People who liked this also viewed

6 picks

Type