tiledbvcf

Name: tiledbvcf
Author: LeonChaoX

698stars61forks

How to install tiledbvcf?

leonchaox/qinyan-academic-skills/tiledbvcf

$npx -y skills add LeonChaoX/qinyan-academic-skills --skill tiledbvcf

Installs into the current project.

Summary

Efficient storage and retrieval of genomic variant data using TileDB. Scalable VCF/BCF ingestion, incremental sample addition, compressed storage, parallel queries, and export capabilities for population genomics.

Files · 1

View on GitHub

SKILL.md

1	# TileDB-VCF
2
3	## Overview
4
5	TileDB-VCF is a high-performance C++ library with Python and CLI interfaces for efficient storage and retrieval of genomic variant-call data. Built on TileDB's sparse array technology, it enables scalable ingestion of VCF/BCF files, incremental sample addition without expensive merging operations, and efficient parallel queries of variant data stored locally or in the cloud.
6
7	## When to Use This Skill
8
9	This skill should be used when:
10	- Learning TileDB-VCF concepts and workflows
11	- Prototyping genomics analyses and pipelines
12	- Working with small-to-medium datasets (< 1000 samples)
13	- Need incremental addition of new samples to existing datasets
14	- Require efficient querying of specific genomic regions across many samples
15	- Working with cloud-stored variant data (S3, Azure, GCS)
16	- Need to export subsets of large VCF datasets
17	- Building variant databases for cohort studies
18	- Educational projects and method development
19	- Performance is critical for variant data operations
20
21	## Quick Start
22
23	### Installation
24
25	Preferred Method: Conda/Mamba
26	```bash
27	# Enter the following two lines if you are on a M1 Mac
28	CONDA_SUBDIR=osx-64
29	conda config --env --set subdir osx-64
30
31	# Create the conda environment
32	conda create -n tiledb-vcf "python<3.10"
33	conda activate tiledb-vcf
34
35	# Mamba is a faster and more reliable alternative to conda
36	conda install -c conda-forge mamba
37
38	# Install TileDB-Py and TileDB-VCF, align with other useful libraries
39	mamba install -y -c conda-forge -c bioconda -c tiledb tiledb-py tiledbvcf-py pandas pyarrow numpy
40	```
41
42	Alternative: Docker Images
43	```bash
44	docker pull tiledb/tiledbvcf-py # Python interface
45	docker pull tiledb/tiledbvcf-cli # Command-line interface
46	```
47
48	### Basic Examples
49
50	Create and populate a dataset:
51	```python
52	import tiledbvcf
53
54	# Create a new dataset
55	ds = tiledbvcf.Dataset(uri="my_dataset", mode="w",
56	cfg=tiledbvcf.ReadConfig(memory_budget=1024))
57
58	# Ingest VCF files (must be single-sample with indexes)
59	# Requirements:
60	# - VCFs must be single-sample (not multi-sample)
61	# - Must have indexes: .csi (bcftools) or .tbi (tabix)
62	ds.ingest_samples(["sample1.vcf.gz", "sample2.vcf.gz"])
63	```
64
65	Query variant data:
66	```python
67	# Open existing dataset for reading
68	ds = tiledbvcf.Dataset(uri="my_dataset", mode="r")
69
70	# Query specific regions and samples
71	df = ds.read(
72	attrs=["sample_name", "pos_start", "pos_end", "alleles", "fmt_GT"],
73	regions=["chr1:1000000-2000000", "chr2:500000-1500000"],
74	samples=["sample1", "sample2", "sample3"]
75	)
76	print(df.head())
77	```
78
79	Export to VCF:
80	```python
81	import os
82
83	# Export two VCF samples
84	ds.export(
85	regions=["chr21:8220186-8405573"],
86	samples=["HG00101", "HG00097"],
87	output_format="v",
88	output_dir=os.path.expanduser("~"),
89	)
90	```
91
92	## Core Capabilities
93
94	### 1. Dataset Creation and Ingestion
95
96	Create TileDB-VCF datasets and incrementally ingest variant data from multiple VCF/BCF files. This is appropriate for building population genomics databases and cohort studies.
97
98	Requirements:
99	- Single-sample VCFs only: Multi-sample VCFs are not supported
100	- Index files required: VCF/BCF files must have indexes (.csi or .tbi)
101
102	Common operations:
103	- Create new datasets with optimized array schemas
104	- Ingest single or multiple VCF/BCF files in parallel
105	- Add new samples incrementally without re-processing existing data
106	- Configure memory usage and compression settings
107	- Handle various VCF formats and INFO/FORMAT fields
108	- Resume interrupted ingestion processes
109	- Validate data integrity during ingestion
110
111
112	### 2. Efficient Querying and Filtering
113
114	Query variant data with high performance across genomic regions, samples, and variant attributes. This is appropriate for association studies, variant discovery, and population analysis.
115
116	Common operations:
117	- Query specific genomic regions (single or multiple)
118	- Filter by sample names or sample groups
119	- Extract specific variant attributes (position, alleles, genotypes, quality)
120	- Access INFO and FORMAT fields efficiently
121	- Combine spatial and attribute-based filtering
122	- Stream large query results
123	- Perform aggregations across samples or regions
124
125
126	### 3. Data Export and Interoperability
127
128	Export data in various formats for downstream analysis or integration with other genomics tools. This is appropriate for sharing datasets, creating analysis subsets, or feeding other pipelines.
129
130	Common operations:
131	- Export to standard VCF/BCF formats
132	- Generate TSV files with selected fields
133	- Create sample/region-specific subsets
134	- Maintain data provenance and metadata
135	- Lossless data export preserving all annotations
136	- Compressed output formats
137	- Streaming exports

People who liked this also viewed

6 picks

Type

tiledbvcf

698stars61forks

View on GitHub

How to install tiledbvcf?

leonchaox/qinyan-academic-skills/tiledbvcf

$npx -y skills add LeonChaoX/qinyan-academic-skills --skill tiledbvcf

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# TileDB-VCF
2
3	## Overview
4
5	TileDB-VCF is a high-performance C++ library with Python and CLI interfaces for efficient storage and retrieval of genomic variant-call data. Built on TileDB's sparse array technology, it enables scalable ingestion of VCF/BCF files, incremental sample addition without expensive merging operations, and efficient parallel queries of variant data stored locally or in the cloud.
6
7	## When to Use This Skill
8
9	This skill should be used when:
10	- Learning TileDB-VCF concepts and workflows
11	- Prototyping genomics analyses and pipelines
12	- Working with small-to-medium datasets (< 1000 samples)
13	- Need incremental addition of new samples to existing datasets
14	- Require efficient querying of specific genomic regions across many samples
15	- Working with cloud-stored variant data (S3, Azure, GCS)
16	- Need to export subsets of large VCF datasets
17	- Building variant databases for cohort studies
18	- Educational projects and method development
19	- Performance is critical for variant data operations
20
21	## Quick Start
22
23	### Installation
24
25	Preferred Method: Conda/Mamba
26	```bash
27	# Enter the following two lines if you are on a M1 Mac
28	CONDA_SUBDIR=osx-64
29	conda config --env --set subdir osx-64
30
31	# Create the conda environment
32	conda create -n tiledb-vcf "python<3.10"
33	conda activate tiledb-vcf
34
35	# Mamba is a faster and more reliable alternative to conda
36	conda install -c conda-forge mamba
37
38	# Install TileDB-Py and TileDB-VCF, align with other useful libraries
39	mamba install -y -c conda-forge -c bioconda -c tiledb tiledb-py tiledbvcf-py pandas pyarrow numpy
40	```
41
42	Alternative: Docker Images
43	```bash
44	docker pull tiledb/tiledbvcf-py # Python interface
45	docker pull tiledb/tiledbvcf-cli # Command-line interface
46	```
47
48	### Basic Examples
49
50	Create and populate a dataset:
51	```python
52	import tiledbvcf
53
54	# Create a new dataset
55	ds = tiledbvcf.Dataset(uri="my_dataset", mode="w",
56	cfg=tiledbvcf.ReadConfig(memory_budget=1024))
57
58	# Ingest VCF files (must be single-sample with indexes)
59	# Requirements:
60	# - VCFs must be single-sample (not multi-sample)
61	# - Must have indexes: .csi (bcftools) or .tbi (tabix)
62	ds.ingest_samples(["sample1.vcf.gz", "sample2.vcf.gz"])
63	```
64
65	Query variant data:
66	```python
67	# Open existing dataset for reading
68	ds = tiledbvcf.Dataset(uri="my_dataset", mode="r")
69
70	# Query specific regions and samples
71	df = ds.read(
72	attrs=["sample_name", "pos_start", "pos_end", "alleles", "fmt_GT"],
73	regions=["chr1:1000000-2000000", "chr2:500000-1500000"],
74	samples=["sample1", "sample2", "sample3"]
75	)
76	print(df.head())
77	```
78
79	Export to VCF:
80	```python
81	import os
82
83	# Export two VCF samples
84	ds.export(
85	regions=["chr21:8220186-8405573"],
86	samples=["HG00101", "HG00097"],
87	output_format="v",
88	output_dir=os.path.expanduser("~"),
89	)
90	```
91
92	## Core Capabilities
93
94	### 1. Dataset Creation and Ingestion
95
96	Create TileDB-VCF datasets and incrementally ingest variant data from multiple VCF/BCF files. This is appropriate for building population genomics databases and cohort studies.
97
98	Requirements:
99	- Single-sample VCFs only: Multi-sample VCFs are not supported
100	- Index files required: VCF/BCF files must have indexes (.csi or .tbi)
101
102	Common operations:
103	- Create new datasets with optimized array schemas
104	- Ingest single or multiple VCF/BCF files in parallel
105	- Add new samples incrementally without re-processing existing data
106	- Configure memory usage and compression settings
107	- Handle various VCF formats and INFO/FORMAT fields
108	- Resume interrupted ingestion processes
109	- Validate data integrity during ingestion
110
111
112	### 2. Efficient Querying and Filtering
113
114	Query variant data with high performance across genomic regions, samples, and variant attributes. This is appropriate for association studies, variant discovery, and population analysis.
115
116	Common operations:
117	- Query specific genomic regions (single or multiple)
118	- Filter by sample names or sample groups
119	- Extract specific variant attributes (position, alleles, genotypes, quality)
120	- Access INFO and FORMAT fields efficiently
121	- Combine spatial and attribute-based filtering
122	- Stream large query results
123	- Perform aggregations across samples or regions
124
125
126	### 3. Data Export and Interoperability
127
128	Export data in various formats for downstream analysis or integration with other genomics tools. This is appropriate for sharing datasets, creating analysis subsets, or feeding other pipelines.
129
130	Common operations:
131	- Export to standard VCF/BCF formats
132	- Generate TSV files with selected fields
133	- Create sample/region-specific subsets
134	- Maintain data provenance and metadata
135	- Lossless data export preserving all annotations
136	- Compressed output formats
137	- Streaming exports

People who liked this also viewed

6 picks

Type