core-dimension-distribution-causal-reasoning

Name: core-dimension-distribution-causal-reasoning
Author: kaori-seasons

15stars3forks

How to install core-dimension-distribution-causal-reasoning?

kaori-seasons/data-skill-hub/core-dimension-distribution-causal-reasoning

$npx -y skills add kaori-seasons/data-skill-hub --skill core-dimension-distribution-causal-reasoning

Installs into the current project.

Summary

Use when you need to identify the real core dimensions of a table, compare competing base tables, analyze field distributions, evaluate mapping cardinality, explain gaps between image/video or source/result layers, or perform causal reasoning about why a dimension becomes '无', mi

Files · 1

View on GitHub

SKILL.md

1	# Core Dimension Distribution And Causal Reasoning
2
3	Use this skill when the main question is not merely "what are the top values", but:
4
5	- which dimensions are truly stable enough to drive analysis
6	- which key should be treated as the business anchor
7	- why a dimension is missing, skewed, or inconsistent across layers
8	- whether a value gap should be merged, added, or traced back upstream
9
10	## What This Skill Solves
11
12	- choose the real base table among multiple candidates
13	- choose the right key among `session_id`, `live_room_id`, `child_order_id`, `spu`, `sku`, `product_id`
14	- explain heavy default values like `无` or `其他`
15	- analyze image-vs-video dimension gaps
16	- analyze source-vs-result layer loss
17	- decide whether a dimension difference means new enum, merge candidate, or pipeline loss
18
19	## First Principles
20
21	A field is a core dimension only if it is:
22
23	1. identifiable in runtime schema
24	2. sufficiently covered
25	3. semantically interpretable
26	4. stably connectable to other important fields
27
28	Always analyze in this order:
29
30	1. business question
31	2. grain
32	3. field profile
33	4. value distribution
34	5. mapping cardinality
35	6. cross-layer gap
36	7. causal explanation
37
38	If grain is unclear, the rest of the analysis is likely misleading.
39
40	## Workflow
41
42	### 1. Define the business question first
43
44	Examples:
45
46	- live-commerce operations: `session_id`, `child_order_id`, `product_id`, `pay_amount`
47	- content attribution: `file_id`, `platform_source_id`, `spu`, `sku`
48	- product semantics: `big_cate`, `mid_cate`, `track`, `gender`, `scene`, `style`
49
50	Do not start from field names alone.
51
52	### 2. Lock the grain before selecting dimensions
53
54	Ask:
55
56	- what does one row represent?
57	- what does one key represent?
58	- can this key recur across date, platform, session, or product?
59
60	Typical anti-patterns seen in the bundled evidence:
61
62	- `live_room_id` assumed to equal one session
63	- `spu` used where `sku` is required
64	- a light result table treated as equivalent to a richer fact table
65
66	### 3. Profile candidate dimensions
67
68	For each candidate dimension, collect:
69
70	- physical-column hit
71	- total rows
72	- null rows
73	- null ratio
74	- distinct count
75	- obvious dirty/default values
76
77	Start with:
78
79	- `scripts/run_live_core_dimension_probe.py`
80	- `scripts/run_taobao_live_orders_core_dimension_report.py`
81	- `scripts/run_content_core_dimension_report.py`
82
83	### 4. Read the distribution correctly
84
85	Check at minimum:
86
87	- Top1 ratio
88	- Top10 cumulative ratio
89	- total distinct values
90	- ratio of `无` / `其他` / blanks
91
92	Interpretation rules:
93
94	- high `无` often implies missing upstream labels or wrong field sourcing
95	- very high Top1 may be real business concentration, or collapsed dirty defaults
96	- extreme long tail may indicate uncontrolled dictionary granularity or missing merge rules
97
98	### 5. Evaluate mapping cardinality
99
100	Distribution alone is not enough. Measure relationships between dimensions.
101
102	Key examples:
103
104	- `child_order_id -> session_id`
105	- `product_id -> spu`
106	- `spu -> sku_id`
107	- `live_room_id -> pay_date`
108	- `spu -> live_room_id`
109
110	Include:
111
112	- average targets per source
113	- p50 / p90 / max
114	- multi-mapping source ratio
115
116	Use:
117
118	- `scripts/run_order_live_room_spu_distribution_probe.py`
119	- `scripts/run_live_core_dimension_probe.py`
120
121	### 6. Compare layers and rule coverage
122
123	When a value appears in one layer but not another, split the cause:
124
125	1. missing already in source
126	2. lost in result layer
127	3. absent from reference enum set
128	4. uncovered by historical rule set
129
130	Use:
131
132	- `scripts/run_picture_mid_track_gap_report.py`
133	- `scripts/run_picture_video_midcate_rule_gap_report.py`
134	- `scripts/run_picture_video_midcate_action_plan_report.py`
135	- `scripts/run_sample_002_gender_gap_report.py`
136
137	## Causal Reasoning Templates
138
139	### A. Large `无` bucket
140
141	Test in order:
142
143	1. upstream field truly empty
144	2. wrong source field selected
145	3. join miss
146	4. rule collapses valid values into `无`
147
148	`scripts/run_sample_002_gender_gap_report.py` is the reference pattern for this kind of diagnosis.
149
150	### B. Image has it, video does not
151
152	Test in order:
153
154	1. video source layer has no such data
155	2. video result layer dropped it
156	3. historical rules never covered it
157	4. it should be a legitimate new enum rather than a merge target
158
159	### C. Strong one-to-many or many-to-many mapping
160
161	Interpretation:
162
163	- the key is probably not the business anchor
164	- grain may be wrong
165	- a bridge layer may be required
166	- a time-window/session key may be required
167
168	### D. Competing base tables
169
170	Prefer the table with stronger evidence on:
171
172	- row coverage
173	- core-field hit count
174	- typed time fields
175	- presence of key enhancement dimensions
176
177	## Bundled Scripts
178
179	- `scripts/run_live_core_dimension_probe.py`

People who liked this also viewed

6 picks

Type

core-dimension-distribution-causal-reasoning

15stars3forks

View on GitHub

How to install core-dimension-distribution-causal-reasoning?

kaori-seasons/data-skill-hub/core-dimension-distribution-causal-reasoning

$npx -y skills add kaori-seasons/data-skill-hub --skill core-dimension-distribution-causal-reasoning

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Core Dimension Distribution And Causal Reasoning
2
3	Use this skill when the main question is not merely "what are the top values", but:
4
5	- which dimensions are truly stable enough to drive analysis
6	- which key should be treated as the business anchor
7	- why a dimension is missing, skewed, or inconsistent across layers
8	- whether a value gap should be merged, added, or traced back upstream
9
10	## What This Skill Solves
11
12	- choose the real base table among multiple candidates
13	- choose the right key among `session_id`, `live_room_id`, `child_order_id`, `spu`, `sku`, `product_id`
14	- explain heavy default values like `无` or `其他`
15	- analyze image-vs-video dimension gaps
16	- analyze source-vs-result layer loss
17	- decide whether a dimension difference means new enum, merge candidate, or pipeline loss
18
19	## First Principles
20
21	A field is a core dimension only if it is:
22
23	1. identifiable in runtime schema
24	2. sufficiently covered
25	3. semantically interpretable
26	4. stably connectable to other important fields
27
28	Always analyze in this order:
29
30	1. business question
31	2. grain
32	3. field profile
33	4. value distribution
34	5. mapping cardinality
35	6. cross-layer gap
36	7. causal explanation
37
38	If grain is unclear, the rest of the analysis is likely misleading.
39
40	## Workflow
41
42	### 1. Define the business question first
43
44	Examples:
45
46	- live-commerce operations: `session_id`, `child_order_id`, `product_id`, `pay_amount`
47	- content attribution: `file_id`, `platform_source_id`, `spu`, `sku`
48	- product semantics: `big_cate`, `mid_cate`, `track`, `gender`, `scene`, `style`
49
50	Do not start from field names alone.
51
52	### 2. Lock the grain before selecting dimensions
53
54	Ask:
55
56	- what does one row represent?
57	- what does one key represent?
58	- can this key recur across date, platform, session, or product?
59
60	Typical anti-patterns seen in the bundled evidence:
61
62	- `live_room_id` assumed to equal one session
63	- `spu` used where `sku` is required
64	- a light result table treated as equivalent to a richer fact table
65
66	### 3. Profile candidate dimensions
67
68	For each candidate dimension, collect:
69
70	- physical-column hit
71	- total rows
72	- null rows
73	- null ratio
74	- distinct count
75	- obvious dirty/default values
76
77	Start with:
78
79	- `scripts/run_live_core_dimension_probe.py`
80	- `scripts/run_taobao_live_orders_core_dimension_report.py`
81	- `scripts/run_content_core_dimension_report.py`
82
83	### 4. Read the distribution correctly
84
85	Check at minimum:
86
87	- Top1 ratio
88	- Top10 cumulative ratio
89	- total distinct values
90	- ratio of `无` / `其他` / blanks
91
92	Interpretation rules:
93
94	- high `无` often implies missing upstream labels or wrong field sourcing
95	- very high Top1 may be real business concentration, or collapsed dirty defaults
96	- extreme long tail may indicate uncontrolled dictionary granularity or missing merge rules
97
98	### 5. Evaluate mapping cardinality
99
100	Distribution alone is not enough. Measure relationships between dimensions.
101
102	Key examples:
103
104	- `child_order_id -> session_id`
105	- `product_id -> spu`
106	- `spu -> sku_id`
107	- `live_room_id -> pay_date`
108	- `spu -> live_room_id`
109
110	Include:
111
112	- average targets per source
113	- p50 / p90 / max
114	- multi-mapping source ratio
115
116	Use:
117
118	- `scripts/run_order_live_room_spu_distribution_probe.py`
119	- `scripts/run_live_core_dimension_probe.py`
120
121	### 6. Compare layers and rule coverage
122
123	When a value appears in one layer but not another, split the cause:
124
125	1. missing already in source
126	2. lost in result layer
127	3. absent from reference enum set
128	4. uncovered by historical rule set
129
130	Use:
131
132	- `scripts/run_picture_mid_track_gap_report.py`
133	- `scripts/run_picture_video_midcate_rule_gap_report.py`
134	- `scripts/run_picture_video_midcate_action_plan_report.py`
135	- `scripts/run_sample_002_gender_gap_report.py`
136
137	## Causal Reasoning Templates
138
139	### A. Large `无` bucket
140
141	Test in order:
142
143	1. upstream field truly empty
144	2. wrong source field selected
145	3. join miss
146	4. rule collapses valid values into `无`
147
148	`scripts/run_sample_002_gender_gap_report.py` is the reference pattern for this kind of diagnosis.
149
150	### B. Image has it, video does not
151
152	Test in order:
153
154	1. video source layer has no such data
155	2. video result layer dropped it
156	3. historical rules never covered it
157	4. it should be a legitimate new enum rather than a merge target
158
159	### C. Strong one-to-many or many-to-many mapping
160
161	Interpretation:
162
163	- the key is probably not the business anchor
164	- grain may be wrong
165	- a bridge layer may be required
166	- a time-window/session key may be required
167
168	### D. Competing base tables
169
170	Prefer the table with stronger evidence on:
171
172	- row coverage
173	- core-field hit count
174	- typed time fields
175	- presence of key enhancement dimensions
176
177	## Bundled Scripts
178
179	- `scripts/run_live_core_dimension_probe.py`

People who liked this also viewed

6 picks

Type