lamindb

Name: lamindb
Author: K-Dense-AI

31kstars3.1kforks

How to install lamindb?

k-dense-ai/scientific-agent-skills/lamindb

$npx -y skills add K-Dense-AI/scientific-agent-skills --skill lamindb

Installs into the current project.

Summary

Use when working with LaminDB, the open-source lineage-native lakehouse for biological datasets and models. Covers setup, artifact registration, query/search, lineage tracking, validation, ontology-backed annotation with Bionty, collections, branches, storage, and workflow integr

Files · 1

View on GitHub

SKILL.md

1	# LaminDB
2
3	## Overview
4
5	LaminDB is an open-source, lineage-native lakehouse for biology. It makes datasets and models queryable, traceable, validated, reproducible, and FAIR (Findable, Accessible, Interoperable, Reusable) while storing data in open formats across local filesystems, S3, GCS, Hugging Face, SQLite, and Postgres.
6
7	Core Value Proposition:
8	- Queryability: Search and filter artifacts, records, runs, features, schemas, and collections
9	- Traceability: Track inputs, outputs, parameters, source code, and environments for notebooks, scripts, functions, and pipelines
10	- Validation: Curate DataFrame, AnnData, SpatialData, TileDB-SOMA, Parquet, Zarr, and other biological formats with schemas
11	- FAIR Compliance: Standardize annotations with Bionty-backed ontologies and custom registries
12	- Change management: Organize work with projects, branches, spaces, collections, and saved notes or plans
13
14	## When to Use This Skill
15
16	Use this skill when:
17
18	- Managing biological datasets: scRNA-seq, bulk RNA-seq, spatial transcriptomics, flow cytometry, multi-modal data, EHR data
19	- Tracking computational workflows: Notebooks, scripts, functions, shell scripts, and pipeline execution (Nextflow, Snakemake, Redun)
20	- Curating and validating data: Schema validation, standardization, ontology-based annotation
21	- Working with biological ontologies: Genes, proteins, cell types, tissues, diseases, pathways (via Bionty)
22	- Building data lakehouses: Unified query interface across multiple datasets
23	- Ensuring reproducibility: Automatic versioning, lineage tracking, environment capture
24	- Integrating ML pipelines: Connecting with Weights & Biases, MLflow, Hugging Face, Lightning, scVI-tools
25	- Deploying data infrastructure: Setting up local or cloud-based data management systems
26	- Collaborating on datasets: Sharing curated, annotated data with standardized metadata
27
28	## Core Capabilities
29
30	LaminDB provides six interconnected capability areas, each documented in detail in the references folder.
31
32	### 1. Core Concepts and Data Lineage
33
34	Core entities:
35	- Artifacts: Versioned datasets (DataFrame, AnnData, Parquet, Zarr, etc.)
36	- Records & ULabels: Experimental entities, typed records, and simple labels
37	- Collections: Versioned, immutable sets of artifacts
38	- Runs & Transforms: Computational lineage tracking (what code produced what data)
39	- Features: Typed metadata fields for annotation and querying
40	- Projects, Branches & Spaces: Project grouping, change management, and access boundaries
41
42	Key workflows:
43	- Create and version artifacts from files or Python objects
44	- Track notebook/script execution with `ln.track()` and `ln.finish()`
45	- Track function workflows with `@ln.flow()` and `@ln.step()`
46	- Annotate artifacts with records, ulabels, projects, and typed features
47	- Visualize data lineage graphs with `artifact.view_lineage()`
48	- Query by provenance (find all outputs from specific code/inputs)
49
50	Reference: `references/core-concepts.md` - Read this for detailed information on artifacts, records, runs, transforms, features, versioning, and lineage tracking.
51
52	### 2. Data Management and Querying
53
54	Query capabilities:
55	- Registry exploration and lookup with auto-complete
56	- Single record retrieval with `get()`, `one()`, `one_or_none()`
57	- Filtering with comparison operators (`__gt`, `__lte`, `__contains`, `__startswith`)
58	- Feature-based queries, including expression-style queries with `Feature` objects
59	- Cross-registry traversal with double-underscore syntax
60	- Full-text search across registries
61	- Advanced logical queries with `ln.Q` objects (AND, OR, NOT)
62	- Streaming large datasets without loading into memory
63
64	Key workflows:
65	- Browse artifacts with filters and ordering
66	- Query by features, creation date, creator, size, etc.
67	- Stream large files in chunks or with array slicing
68	- Organize data with hierarchical keys
69	- Group artifacts into collections
70
71	Reference: `references/data-management.md` - Read this for comprehensive query patterns, filtering examples, streaming strategies, and data organization best practices.
72
73	### 3. Annotation and Validation
74
75	Curation process:
76	1. Validation: Confirm datasets match desired schemas
77	2. Standardization: Fix typos, map synonyms to canonical terms
78	3. Annotation: Link datasets to metadata entities for queryability
79
80	Schema types:
81	- Flexible schemas: Validate only known columns, allow additional metadata
82	- Minimal required schemas: Specify essential columns, permit extras
83	- Strict schemas: Co

People who liked this also viewed

6 picks

Type

lamindb

31kstars3.1kforks

View on GitHub

How to install lamindb?

k-dense-ai/scientific-agent-skills/lamindb

$npx -y skills add K-Dense-AI/scientific-agent-skills --skill lamindb

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# LaminDB
2
3	## Overview
4
5	LaminDB is an open-source, lineage-native lakehouse for biology. It makes datasets and models queryable, traceable, validated, reproducible, and FAIR (Findable, Accessible, Interoperable, Reusable) while storing data in open formats across local filesystems, S3, GCS, Hugging Face, SQLite, and Postgres.
6
7	Core Value Proposition:
8	- Queryability: Search and filter artifacts, records, runs, features, schemas, and collections
9	- Traceability: Track inputs, outputs, parameters, source code, and environments for notebooks, scripts, functions, and pipelines
10	- Validation: Curate DataFrame, AnnData, SpatialData, TileDB-SOMA, Parquet, Zarr, and other biological formats with schemas
11	- FAIR Compliance: Standardize annotations with Bionty-backed ontologies and custom registries
12	- Change management: Organize work with projects, branches, spaces, collections, and saved notes or plans
13
14	## When to Use This Skill
15
16	Use this skill when:
17
18	- Managing biological datasets: scRNA-seq, bulk RNA-seq, spatial transcriptomics, flow cytometry, multi-modal data, EHR data
19	- Tracking computational workflows: Notebooks, scripts, functions, shell scripts, and pipeline execution (Nextflow, Snakemake, Redun)
20	- Curating and validating data: Schema validation, standardization, ontology-based annotation
21	- Working with biological ontologies: Genes, proteins, cell types, tissues, diseases, pathways (via Bionty)
22	- Building data lakehouses: Unified query interface across multiple datasets
23	- Ensuring reproducibility: Automatic versioning, lineage tracking, environment capture
24	- Integrating ML pipelines: Connecting with Weights & Biases, MLflow, Hugging Face, Lightning, scVI-tools
25	- Deploying data infrastructure: Setting up local or cloud-based data management systems
26	- Collaborating on datasets: Sharing curated, annotated data with standardized metadata
27
28	## Core Capabilities
29
30	LaminDB provides six interconnected capability areas, each documented in detail in the references folder.
31
32	### 1. Core Concepts and Data Lineage
33
34	Core entities:
35	- Artifacts: Versioned datasets (DataFrame, AnnData, Parquet, Zarr, etc.)
36	- Records & ULabels: Experimental entities, typed records, and simple labels
37	- Collections: Versioned, immutable sets of artifacts
38	- Runs & Transforms: Computational lineage tracking (what code produced what data)
39	- Features: Typed metadata fields for annotation and querying
40	- Projects, Branches & Spaces: Project grouping, change management, and access boundaries
41
42	Key workflows:
43	- Create and version artifacts from files or Python objects
44	- Track notebook/script execution with `ln.track()` and `ln.finish()`
45	- Track function workflows with `@ln.flow()` and `@ln.step()`
46	- Annotate artifacts with records, ulabels, projects, and typed features
47	- Visualize data lineage graphs with `artifact.view_lineage()`
48	- Query by provenance (find all outputs from specific code/inputs)
49
50	Reference: `references/core-concepts.md` - Read this for detailed information on artifacts, records, runs, transforms, features, versioning, and lineage tracking.
51
52	### 2. Data Management and Querying
53
54	Query capabilities:
55	- Registry exploration and lookup with auto-complete
56	- Single record retrieval with `get()`, `one()`, `one_or_none()`
57	- Filtering with comparison operators (`__gt`, `__lte`, `__contains`, `__startswith`)
58	- Feature-based queries, including expression-style queries with `Feature` objects
59	- Cross-registry traversal with double-underscore syntax
60	- Full-text search across registries
61	- Advanced logical queries with `ln.Q` objects (AND, OR, NOT)
62	- Streaming large datasets without loading into memory
63
64	Key workflows:
65	- Browse artifacts with filters and ordering
66	- Query by features, creation date, creator, size, etc.
67	- Stream large files in chunks or with array slicing
68	- Organize data with hierarchical keys
69	- Group artifacts into collections
70
71	Reference: `references/data-management.md` - Read this for comprehensive query patterns, filtering examples, streaming strategies, and data organization best practices.
72
73	### 3. Annotation and Validation
74
75	Curation process:
76	1. Validation: Confirm datasets match desired schemas
77	2. Standardization: Fix typos, map synonyms to canonical terms
78	3. Annotation: Link datasets to metadata entities for queryability
79
80	Schema types:
81	- Flexible schemas: Validate only known columns, allow additional metadata
82	- Minimal required schemas: Specify essential columns, permit extras
83	- Strict schemas: Co

People who liked this also viewed

6 picks

Type