creating-data-lake-table

Name: creating-data-lake-table
Author: aws

2.4kinstalls190forks

How to install creating-data-lake-table?

aws/agent-toolkit-for-aws/creating-data-lake-table

$npx -y skills add aws/agent-toolkit-for-aws --skill creating-data-lake-table

Installs into the current project.

Summary

Create managed Iceberg tables using Amazon S3 Tables (s3tables API namespace) with automatic compaction and snapshot management. Sets up table bucket, namespace, table, schema, Glue catalog registration, partitioning, IAM access control. Triggers on: create table, data lake table

Files · 1

View on GitHub

SKILL.md

1	# Create Data Lake Tables with Amazon S3 Tables
2
3	## Overview
4
5	Amazon S3 Tables provides managed Iceberg tables with automatic compaction and snapshot management. Queryable via Athena and Iceberg-compatible engines.
6
7	## Common Tasks
8
9	You MUST use AWS MCP server tools when connected, they provide command validation, sandboxed execution, and audit logging. Fall back to AWS CLI if MCP unavailable.
10
11	## Decision Guide
12
13	Before creating, You MUST check what exists:
14
15	You MUST run `aws glue get-tables --database-name <NAME>` when user mentions a database.
16
17	\| What you find \| Action \|
18	\|---------------\|--------\|
19	\| Fuzzy database name ("our analytics db") \| You MUST STOP. Delegate to `finding-data-lake-assets` to resolve. \|
20	\| Non-S3-Tables table with matching name \| You MUST STOP. Delegate to `finding-data-lake-assets`. You MUST NOT create until user confirms. \|
21	\| Existing S3 Tables table with matching name \| You MUST check schema match. Reuse if compatible, recreate only if user confirms. \|
22	\| No matching tables \| Proceed with creation (Steps 1-8). \|
23	\| User explicitly requests new S3 Tables table \| Skip checks, proceed with creation. \|
24
25	Creation paths:
26
27	- Existing data in S3: Create empty table (Steps 1-8), then use `ingesting-into-data-lake` skill.
28	- Glue ETL pipeline: Read `references/table-creation-glue-etl.md` first, then Steps 1-6.
29	- Lake Formation access control: Search AWS docs for `"S3 Tables integration with Lake Formation"`.
30
31	### 1. Verify Dependencies
32
33	Constraints:
34
35	- You MUST check whether AWS MCP server tools or AWS CLI are available and inform user if missing
36	- You MUST confirm target AWS region and verify credentials with `aws sts get-caller-identity`
37
38	### 2. Understand the Schema
39
40	- Explicit schema: Validate Iceberg types.
41	- Loose description: Ask columns, types, grain. Propose and confirm.
42	- Existing S3 data: Infer schema from file headers only. Create empty table first, then use `ingesting-into-data-lake` skill.
43
44	Constraints:
45
46	- You MUST read `references/best-practices.md` for Iceberg type mapping, partitions, and naming.
47	- You MUST ask for all required parameters upfront: table name, columns, types, partition strategy. For schema evolution, see `references/athena-ddl-path.md`.
48	- You MUST use all lowercase names -- Glue rejects mixed case with `GENERIC_INTERNAL_ERROR`. Namespace and table names MUST NOT contain hyphens.
49	- You SHOULD suggest partition columns based on access patterns.
50
51	### 3. Create Table Bucket
52
53	Names: 3-63 chars, lowercase, numbers, hyphens.
54
55	```bash
56	aws s3tables create-table-bucket --name <BUCKET_NAME> --region <REGION>
57	```
58
59	Capture `table-bucket-arn`. Encryption (SSE-S3 default, SSE-KMS) and storage class (STANDARD, INTELLIGENT_TIERING) set at creation. See `references/best-practices.md`.
60
61	Constraints:
62
63	- You MUST check existing buckets with `aws s3tables list-table-buckets` and ask user to select or create new.
64	- If using SSE-KMS, KMS key policy MUST allow S3 Tables maintenance service principal to read data. Search AWS docs for `"S3 Tables KMS key policy"` for required policy.
65	- If bucket creation fails, see `references/best-practices.md` for common errors.
66
67	### 4. Create Namespace
68
69	```bash
70	aws s3tables create-namespace --table-bucket-arn <ARN> --namespace <NAMESPACE>
71	```
72
73	Constraints:
74
75	- You MUST list existing namespaces first and suggest reusing if relevant
76	- You MUST use lowercase names with no hyphens
77
78	### 5. Create Glue Data Catalog Integration
79
80	Check if `s3tablescatalog` exists (create once per region per account):
81
82	```bash
83	aws glue get-catalog --catalog-id s3tablescatalog
84	```
85
86	If not found, create (requires `glue:CreateCatalog`, `glue:passConnection`):
87
88	```bash
89	aws glue create-catalog --name "s3tablescatalog" --catalog-input '{
90	"FederatedCatalog": {
91	"Identifier": "arn:aws:s3tables:<REGION>:<ACCOUNT_ID>:bucket/*",
92	"ConnectionName": "aws:s3tables"
93	},
94	"CreateDatabaseDefaultPermissions": [{"Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"]}],
95	"CreateTableDefaultPermissions": [{"Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"]}],
96	"AllowFullTableExternalDataAccess": "True"
97	}'
98	```
99
100	Ver

People who liked this also viewed

6 picks

Type

creating-data-lake-table

2.4kinstalls190forks

View on GitHub

How to install creating-data-lake-table?

aws/agent-toolkit-for-aws/creating-data-lake-table

$npx -y skills add aws/agent-toolkit-for-aws --skill creating-data-lake-table

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Create Data Lake Tables with Amazon S3 Tables
2
3	## Overview
4
5	Amazon S3 Tables provides managed Iceberg tables with automatic compaction and snapshot management. Queryable via Athena and Iceberg-compatible engines.
6
7	## Common Tasks
8
9	You MUST use AWS MCP server tools when connected, they provide command validation, sandboxed execution, and audit logging. Fall back to AWS CLI if MCP unavailable.
10
11	## Decision Guide
12
13	Before creating, You MUST check what exists:
14
15	You MUST run `aws glue get-tables --database-name <NAME>` when user mentions a database.
16
17	\| What you find \| Action \|
18	\|---------------\|--------\|
19	\| Fuzzy database name ("our analytics db") \| You MUST STOP. Delegate to `finding-data-lake-assets` to resolve. \|
20	\| Non-S3-Tables table with matching name \| You MUST STOP. Delegate to `finding-data-lake-assets`. You MUST NOT create until user confirms. \|
21	\| Existing S3 Tables table with matching name \| You MUST check schema match. Reuse if compatible, recreate only if user confirms. \|
22	\| No matching tables \| Proceed with creation (Steps 1-8). \|
23	\| User explicitly requests new S3 Tables table \| Skip checks, proceed with creation. \|
24
25	Creation paths:
26
27	- Existing data in S3: Create empty table (Steps 1-8), then use `ingesting-into-data-lake` skill.
28	- Glue ETL pipeline: Read `references/table-creation-glue-etl.md` first, then Steps 1-6.
29	- Lake Formation access control: Search AWS docs for `"S3 Tables integration with Lake Formation"`.
30
31	### 1. Verify Dependencies
32
33	Constraints:
34
35	- You MUST check whether AWS MCP server tools or AWS CLI are available and inform user if missing
36	- You MUST confirm target AWS region and verify credentials with `aws sts get-caller-identity`
37
38	### 2. Understand the Schema
39
40	- Explicit schema: Validate Iceberg types.
41	- Loose description: Ask columns, types, grain. Propose and confirm.
42	- Existing S3 data: Infer schema from file headers only. Create empty table first, then use `ingesting-into-data-lake` skill.
43
44	Constraints:
45
46	- You MUST read `references/best-practices.md` for Iceberg type mapping, partitions, and naming.
47	- You MUST ask for all required parameters upfront: table name, columns, types, partition strategy. For schema evolution, see `references/athena-ddl-path.md`.
48	- You MUST use all lowercase names -- Glue rejects mixed case with `GENERIC_INTERNAL_ERROR`. Namespace and table names MUST NOT contain hyphens.
49	- You SHOULD suggest partition columns based on access patterns.
50
51	### 3. Create Table Bucket
52
53	Names: 3-63 chars, lowercase, numbers, hyphens.
54
55	```bash
56	aws s3tables create-table-bucket --name <BUCKET_NAME> --region <REGION>
57	```
58
59	Capture `table-bucket-arn`. Encryption (SSE-S3 default, SSE-KMS) and storage class (STANDARD, INTELLIGENT_TIERING) set at creation. See `references/best-practices.md`.
60
61	Constraints:
62
63	- You MUST check existing buckets with `aws s3tables list-table-buckets` and ask user to select or create new.
64	- If using SSE-KMS, KMS key policy MUST allow S3 Tables maintenance service principal to read data. Search AWS docs for `"S3 Tables KMS key policy"` for required policy.
65	- If bucket creation fails, see `references/best-practices.md` for common errors.
66
67	### 4. Create Namespace
68
69	```bash
70	aws s3tables create-namespace --table-bucket-arn <ARN> --namespace <NAMESPACE>
71	```
72
73	Constraints:
74
75	- You MUST list existing namespaces first and suggest reusing if relevant
76	- You MUST use lowercase names with no hyphens
77
78	### 5. Create Glue Data Catalog Integration
79
80	Check if `s3tablescatalog` exists (create once per region per account):
81
82	```bash
83	aws glue get-catalog --catalog-id s3tablescatalog
84	```
85
86	If not found, create (requires `glue:CreateCatalog`, `glue:passConnection`):
87
88	```bash
89	aws glue create-catalog --name "s3tablescatalog" --catalog-input '{
90	"FederatedCatalog": {
91	"Identifier": "arn:aws:s3tables:<REGION>:<ACCOUNT_ID>:bucket/*",
92	"ConnectionName": "aws:s3tables"
93	},
94	"CreateDatabaseDefaultPermissions": [{"Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"]}],
95	"CreateTableDefaultPermissions": [{"Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"]}],
96	"AllowFullTableExternalDataAccess": "True"
97	}'
98	```
99
100	Ver

People who liked this also viewed

6 picks

Type