datanalysis-credit-risk

Name: datanalysis-credit-risk
Author: github

7.2kinstalls4.6kforks

How to install datanalysis-credit-risk?

github/awesome-copilot/datanalysis-credit-risk

$npx -y skills add github/awesome-copilot --skill datanalysis-credit-risk

Installs into the current project.

Summary

Credit risk data cleaning and variable screening pipeline for pre-loan modeling. Use when working with raw credit data that needs quality assessment, missing value analysis, or variable selection before modeling. it covers data loading and formatting, abnormal period filtering,

Files · 1

View on GitHub

SKILL.md

1	# Data Cleaning and Variable Screening
2
3	## Quick Start
4
5	```bash
6	# Run the complete data cleaning pipeline
7	python ".github/skills/datanalysis-credit-risk/scripts/example.py"
8	```
9
10	## Complete Process Description
11
12	The data cleaning pipeline consists of the following 11 steps, each executed independently without deleting the original data:
13
14	1. Get Data - Load and format raw data
15	2. Organization Sample Analysis - Statistics of sample count and bad sample rate for each organization
16	3. Separate OOS Data - Separate out-of-sample (OOS) samples from modeling samples
17	4. Filter Abnormal Months - Remove months with insufficient bad sample count or total sample count
18	5. Calculate Missing Rate - Calculate overall and organization-level missing rates for each feature
19	6. Drop High Missing Rate Features - Remove features with overall missing rate exceeding threshold
20	7. Drop Low IV Features - Remove features with overall IV too low or IV too low in too many organizations
21	8. Drop High PSI Features - Remove features with unstable PSI
22	9. Null Importance Denoising - Remove noise features using label permutation method
23	10. Drop High Correlation Features - Remove high correlation features based on original gain
24	11. Export Report - Generate Excel report containing details and statistics of all steps
25
26	## Core Functions
27
28	\| Function \| Purpose \| Module \|
29	\|------\|------\|----------\|
30	\| `get_dataset()` \| Load and format data \| references.func \|
31	\| `org_analysis()` \| Organization sample analysis \| references.func \|
32	\| `missing_check()` \| Calculate missing rate \| references.func \|
33	\| `drop_abnormal_ym()` \| Filter abnormal months \| references.analysis \|
34	\| `drop_highmiss_features()` \| Drop high missing rate features \| references.analysis \|
35	\| `drop_lowiv_features()` \| Drop low IV features \| references.analysis \|
36	\| `drop_highpsi_features()` \| Drop high PSI features \| references.analysis \|
37	\| `drop_highnoise_features()` \| Null Importance denoising \| references.analysis \|
38	\| `drop_highcorr_features()` \| Drop high correlation features \| references.analysis \|
39	\| `iv_distribution_by_org()` \| IV distribution statistics \| references.analysis \|
40	\| `psi_distribution_by_org()` \| PSI distribution statistics \| references.analysis \|
41	\| `value_ratio_distribution_by_org()` \| Value ratio distribution statistics \| references.analysis \|
42	\| `export_cleaning_report()` \| Export cleaning report \| references.analysis \|
43
44	## Parameter Description
45
46	### Data Loading Parameters
47	- `DATA_PATH`: Data file path (best are parquet format)
48	- `DATE_COL`: Date column name
49	- `Y_COL`: Label column name
50	- `ORG_COL`: Organization column name
51	- `KEY_COLS`: Primary key column name list
52
53	### OOS Organization Configuration
54	- `OOS_ORGS`: Out-of-sample organization list
55
56	### Abnormal Month Filtering Parameters
57	- `min_ym_bad_sample`: Minimum bad sample count per month (default 10)
58	- `min_ym_sample`: Minimum total sample count per month (default 500)
59
60	### Missing Rate Parameters
61	- `missing_ratio`: Overall missing rate threshold (default 0.6)
62
63	### IV Parameters
64	- `overall_iv_threshold`: Overall IV threshold (default 0.1)
65	- `org_iv_threshold`: Single organization IV threshold (default 0.1)
66	- `max_org_threshold`: Maximum tolerated low IV organization count (default 2)
67
68	### PSI Parameters
69	- `psi_threshold`: PSI threshold (default 0.1)
70	- `max_months_ratio`: Maximum unstable month ratio (default 1/3)
71	- `max_orgs`: Maximum unstable organization count (default 6)
72
73	### Null Importance Parameters
74	- `n_estimators`: Number of trees (default 100)
75	- `max_depth`: Maximum tree depth (default 5)
76	- `gain_threshold`: Gain difference threshold (default 50)
77
78	### High Correlation Parameters
79	- `max_corr`: Correlation threshold (default 0.9)
80	- `top_n_keep`: Keep top N features by original gain ranking (default 20)
81
82	## Output Report
83
84	The generated Excel report contains the following sheets:
85
86	1. 汇总 - Summary information of all steps, including operation results and conditions
87	2. 机构样本统计 - Sample count and bad sample rate for each organization
88	3. 分离OOS数据 - OOS sample and modeling sample counts
89	4. Step4-异常月份处理 - Abnormal months that were removed
90	5. 缺失率明细 - Overall and organization-level missing rates for each feature
91	6. Step5-有值率分布统计 - Distribution of features in different value ratio ranges
92	7. Step6-高缺失率处理 - High missing rate featur

People who liked this also viewed

6 picks

Type

datanalysis-credit-risk

7.2kinstalls4.6kforks

View on GitHub

How to install datanalysis-credit-risk?

github/awesome-copilot/datanalysis-credit-risk

$npx -y skills add github/awesome-copilot --skill datanalysis-credit-risk

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Data Cleaning and Variable Screening
2
3	## Quick Start
4
5	```bash
6	# Run the complete data cleaning pipeline
7	python ".github/skills/datanalysis-credit-risk/scripts/example.py"
8	```
9
10	## Complete Process Description
11
12	The data cleaning pipeline consists of the following 11 steps, each executed independently without deleting the original data:
13
14	1. Get Data - Load and format raw data
15	2. Organization Sample Analysis - Statistics of sample count and bad sample rate for each organization
16	3. Separate OOS Data - Separate out-of-sample (OOS) samples from modeling samples
17	4. Filter Abnormal Months - Remove months with insufficient bad sample count or total sample count
18	5. Calculate Missing Rate - Calculate overall and organization-level missing rates for each feature
19	6. Drop High Missing Rate Features - Remove features with overall missing rate exceeding threshold
20	7. Drop Low IV Features - Remove features with overall IV too low or IV too low in too many organizations
21	8. Drop High PSI Features - Remove features with unstable PSI
22	9. Null Importance Denoising - Remove noise features using label permutation method
23	10. Drop High Correlation Features - Remove high correlation features based on original gain
24	11. Export Report - Generate Excel report containing details and statistics of all steps
25
26	## Core Functions
27
28	\| Function \| Purpose \| Module \|
29	\|------\|------\|----------\|
30	\| `get_dataset()` \| Load and format data \| references.func \|
31	\| `org_analysis()` \| Organization sample analysis \| references.func \|
32	\| `missing_check()` \| Calculate missing rate \| references.func \|
33	\| `drop_abnormal_ym()` \| Filter abnormal months \| references.analysis \|
34	\| `drop_highmiss_features()` \| Drop high missing rate features \| references.analysis \|
35	\| `drop_lowiv_features()` \| Drop low IV features \| references.analysis \|
36	\| `drop_highpsi_features()` \| Drop high PSI features \| references.analysis \|
37	\| `drop_highnoise_features()` \| Null Importance denoising \| references.analysis \|
38	\| `drop_highcorr_features()` \| Drop high correlation features \| references.analysis \|
39	\| `iv_distribution_by_org()` \| IV distribution statistics \| references.analysis \|
40	\| `psi_distribution_by_org()` \| PSI distribution statistics \| references.analysis \|
41	\| `value_ratio_distribution_by_org()` \| Value ratio distribution statistics \| references.analysis \|
42	\| `export_cleaning_report()` \| Export cleaning report \| references.analysis \|
43
44	## Parameter Description
45
46	### Data Loading Parameters
47	- `DATA_PATH`: Data file path (best are parquet format)
48	- `DATE_COL`: Date column name
49	- `Y_COL`: Label column name
50	- `ORG_COL`: Organization column name
51	- `KEY_COLS`: Primary key column name list
52
53	### OOS Organization Configuration
54	- `OOS_ORGS`: Out-of-sample organization list
55
56	### Abnormal Month Filtering Parameters
57	- `min_ym_bad_sample`: Minimum bad sample count per month (default 10)
58	- `min_ym_sample`: Minimum total sample count per month (default 500)
59
60	### Missing Rate Parameters
61	- `missing_ratio`: Overall missing rate threshold (default 0.6)
62
63	### IV Parameters
64	- `overall_iv_threshold`: Overall IV threshold (default 0.1)
65	- `org_iv_threshold`: Single organization IV threshold (default 0.1)
66	- `max_org_threshold`: Maximum tolerated low IV organization count (default 2)
67
68	### PSI Parameters
69	- `psi_threshold`: PSI threshold (default 0.1)
70	- `max_months_ratio`: Maximum unstable month ratio (default 1/3)
71	- `max_orgs`: Maximum unstable organization count (default 6)
72
73	### Null Importance Parameters
74	- `n_estimators`: Number of trees (default 100)
75	- `max_depth`: Maximum tree depth (default 5)
76	- `gain_threshold`: Gain difference threshold (default 50)
77
78	### High Correlation Parameters
79	- `max_corr`: Correlation threshold (default 0.9)
80	- `top_n_keep`: Keep top N features by original gain ranking (default 20)
81
82	## Output Report
83
84	The generated Excel report contains the following sheets:
85
86	1. 汇总 - Summary information of all steps, including operation results and conditions
87	2. 机构样本统计 - Sample count and bad sample rate for each organization
88	3. 分离OOS数据 - OOS sample and modeling sample counts
89	4. Step4-异常月份处理 - Abnormal months that were removed
90	5. 缺失率明细 - Overall and organization-level missing rates for each feature
91	6. Step5-有值率分布统计 - Distribution of features in different value ratio ranges
92	7. Step6-高缺失率处理 - High missing rate featur

People who liked this also viewed

6 picks

Type