clickhouse-js-node-rowbinary

Name: clickhouse-js-node-rowbinary
Author: ClickHouse

494stars32forks

How to install clickhouse-js-node-rowbinary?

clickhouse/agent-skills/clickhouse-js-node-rowbinary

$npx -y skills add ClickHouse/agent-skills --skill clickhouse-js-node-rowbinary

Installs into the current project.

Summary

Generate TypeScript/JavaScript code that reads/decodes AND writes/encodes ClickHouse RowBinary streams for the ClickHouse HTTP server. Use this skill whenever a user wants to parse or produce RowBinary, RowBinaryWithNames, or RowBinaryWithNamesAndTypes. Node.js only, doesn'

Files · 1

View on GitHub

SKILL.md

1	# ClickHouse JS RowBinary Codec Generator for Node.js
2
3	This skill generates both directions of the wire format: readers (decode
4	bytes → values) and writers (encode values → bytes, the mirror). A given
5	task normally needs only one side. This file is the shared entry point — the
6	format gate plus the principles common to both directions; the per-direction
7	decisions, guidance, and the per-type reference tables live in two sibling files.
8
9	Pick your side — read only the one you need:
10
11	- *Decoding a `RowBinary` response** from ClickHouse into JS values →
12	[reader.md](reader.md). Streaming vs whole-buffer, row-objects vs columnar,
13	fixed vs runtime schema, and the per-type reader reference.
14	- Encoding JS values into a `RowBinary` payload to send to ClickHouse →
15	[writer.md](writer.md). The `Sink`/`writeX` building blocks, `writeRows`
16	streaming, and the per-type writer reference.
17
18	The per-type code is real, split by direction under `src/readers/` and
19	`src/writers/`.
20
21	## First: is RowBinary even the right format?
22
23	RowBinary exists for throughput, but it is **not automatically the fastest
24	path** — match the format to the shape of the data before committing to a
25	bespoke parser.
26
27	*Prefer a `JSON` format (e.g. `JSONEachRow`) when** the result is mostly
28	strings / JSON-like values that you consume wholesale — randomly accessing
29	essentially every field, running string/regexp methods on them, treating values
30	as text. V8's native `JSON.parse` is heavily optimized C++ and builds JS strings
31	and objects faster than a JS-level RowBinary decoder can; pair it with HTTP
32	response compression (`gzip` / `zstd`, which crushes JSON's repetitive keys) and
33	the wire cost shrinks too.
34
35	RowBinary clearly wins when the result is dominated by:
36
37	- Wide numerics — `Int128`/`Int256`/`UInt128`/`UInt256`,
38	`Decimal128`/`Decimal256`.
39	- Binary / fixed-width blobs — `IPv4`, `IPv6`, `UUID`, `FixedString`.
40	- High-volume fixed-width numeric columns generally, where each value is a
41	single `DataView` read.
42
43	Prefer the `Native` format when columnar load and client-side analytics are
44	the main goal (fold/scan/filter columns, feed typed arrays to a Worker or WASM).
45	`Native` is column-major, so it loads straight into one typed array per column
46	with no transpose.
47
48	For help choosing and consuming a `JSON*` format (or CSV / TSV) instead, use the
49	`clickhouse-js-node-coding` skill.
50
51	## Core guidance (both directions)
52
53	These principles apply whether you are generating a reader or a writer; the
54	side-specific operational guidance is in [reader.md](reader.md) /
55	[writer.md](writer.md).
56
57	- Little-endian only. RowBinary is little-endian; target x86/ARM. Read and
58	write every multi-byte number with `DataView` accessors passing a literal
59	`true` for the `littleEndian` flag.
60
61	- Correct first, then optimize. First emit a correct codec built from the
62	plain per-type API. Only after it's correct (and tested) specialize it. Don't
63	bake performance assumptions in before correctness.
64
65	- Monomorphize generic/composite types. Emit specialized, inlined code per
66	type combination instead of passing functions as arguments where the type is
67	known ahead of time.
68
69	- Inline the leaf ops. The per-type `readX`/`writeX` functions are the
70	correct, composable reference; the generated codec should INLINE their bodies,
71	not call them, so the row loop is straight-line with no per-field indirection
72	(and so the fixed-width coalescing can fold the offset arithmetic together).
73
74	- Annotate the type per column. Inlining erases the type structure, so put a
75	short comment above each column's encode/decode block naming the ClickHouse
76	type it handles.
77
78	- Shared scratch is not reentrant. Some hot methods reuse a module-level
79	scratch buffer as a write-then-read pair — correct only because the access is
80	fully synchronous. An `async`/`yield` boundary between populating and reading
81	it corrupts the value.
82
83	- TypeScript by default. Generate TypeScript code and helpers unless the user
84	explicitly asks for plain JavaScript.
85
86	## Worked examples
87
88	Six end-to-end examples with real speedup are catalogued in [EXAMPLES.md](EXAMPLES.md).
89
90	## Out of scope
91
92	- JSON / CSV / TSV / Parquet parsing → use `clickhouse-js-node-coding`.
93	- Connection errors, hangs, type mismatches → use
94	`clickhouse-js-node-troubleshooting`.
95	- Browser / Web Worker / Edge → `@clickhouse/client-web`.
96
97	## Still Stuck?
98
99	- [ClickHouse RowBinary format](https://clickhouse.com/docs/interfaces/formats#rowbinary)
100	- [ClickHouse data types](https://clickhouse.com/docs/sql-referen

People who liked this also viewed

6 picks

Type

clickhouse-js-node-rowbinary

494stars32forks

View on GitHub

How to install clickhouse-js-node-rowbinary?

clickhouse/agent-skills/clickhouse-js-node-rowbinary

$npx -y skills add ClickHouse/agent-skills --skill clickhouse-js-node-rowbinary

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# ClickHouse JS RowBinary Codec Generator for Node.js
2
3	This skill generates both directions of the wire format: readers (decode
4	bytes → values) and writers (encode values → bytes, the mirror). A given
5	task normally needs only one side. This file is the shared entry point — the
6	format gate plus the principles common to both directions; the per-direction
7	decisions, guidance, and the per-type reference tables live in two sibling files.
8
9	Pick your side — read only the one you need:
10
11	- *Decoding a `RowBinary` response** from ClickHouse into JS values →
12	[reader.md](reader.md). Streaming vs whole-buffer, row-objects vs columnar,
13	fixed vs runtime schema, and the per-type reader reference.
14	- Encoding JS values into a `RowBinary` payload to send to ClickHouse →
15	[writer.md](writer.md). The `Sink`/`writeX` building blocks, `writeRows`
16	streaming, and the per-type writer reference.
17
18	The per-type code is real, split by direction under `src/readers/` and
19	`src/writers/`.
20
21	## First: is RowBinary even the right format?
22
23	RowBinary exists for throughput, but it is **not automatically the fastest
24	path** — match the format to the shape of the data before committing to a
25	bespoke parser.
26
27	*Prefer a `JSON` format (e.g. `JSONEachRow`) when** the result is mostly
28	strings / JSON-like values that you consume wholesale — randomly accessing
29	essentially every field, running string/regexp methods on them, treating values
30	as text. V8's native `JSON.parse` is heavily optimized C++ and builds JS strings
31	and objects faster than a JS-level RowBinary decoder can; pair it with HTTP
32	response compression (`gzip` / `zstd`, which crushes JSON's repetitive keys) and
33	the wire cost shrinks too.
34
35	RowBinary clearly wins when the result is dominated by:
36
37	- Wide numerics — `Int128`/`Int256`/`UInt128`/`UInt256`,
38	`Decimal128`/`Decimal256`.
39	- Binary / fixed-width blobs — `IPv4`, `IPv6`, `UUID`, `FixedString`.
40	- High-volume fixed-width numeric columns generally, where each value is a
41	single `DataView` read.
42
43	Prefer the `Native` format when columnar load and client-side analytics are
44	the main goal (fold/scan/filter columns, feed typed arrays to a Worker or WASM).
45	`Native` is column-major, so it loads straight into one typed array per column
46	with no transpose.
47
48	For help choosing and consuming a `JSON*` format (or CSV / TSV) instead, use the
49	`clickhouse-js-node-coding` skill.
50
51	## Core guidance (both directions)
52
53	These principles apply whether you are generating a reader or a writer; the
54	side-specific operational guidance is in [reader.md](reader.md) /
55	[writer.md](writer.md).
56
57	- Little-endian only. RowBinary is little-endian; target x86/ARM. Read and
58	write every multi-byte number with `DataView` accessors passing a literal
59	`true` for the `littleEndian` flag.
60
61	- Correct first, then optimize. First emit a correct codec built from the
62	plain per-type API. Only after it's correct (and tested) specialize it. Don't
63	bake performance assumptions in before correctness.
64
65	- Monomorphize generic/composite types. Emit specialized, inlined code per
66	type combination instead of passing functions as arguments where the type is
67	known ahead of time.
68
69	- Inline the leaf ops. The per-type `readX`/`writeX` functions are the
70	correct, composable reference; the generated codec should INLINE their bodies,
71	not call them, so the row loop is straight-line with no per-field indirection
72	(and so the fixed-width coalescing can fold the offset arithmetic together).
73
74	- Annotate the type per column. Inlining erases the type structure, so put a
75	short comment above each column's encode/decode block naming the ClickHouse
76	type it handles.
77
78	- Shared scratch is not reentrant. Some hot methods reuse a module-level
79	scratch buffer as a write-then-read pair — correct only because the access is
80	fully synchronous. An `async`/`yield` boundary between populating and reading
81	it corrupts the value.
82
83	- TypeScript by default. Generate TypeScript code and helpers unless the user
84	explicitly asks for plain JavaScript.
85
86	## Worked examples
87
88	Six end-to-end examples with real speedup are catalogued in [EXAMPLES.md](EXAMPLES.md).
89
90	## Out of scope
91
92	- JSON / CSV / TSV / Parquet parsing → use `clickhouse-js-node-coding`.
93	- Connection errors, hangs, type mismatches → use
94	`clickhouse-js-node-troubleshooting`.
95	- Browser / Web Worker / Edge → `@clickhouse/client-web`.
96
97	## Still Stuck?
98
99	- [ClickHouse RowBinary format](https://clickhouse.com/docs/interfaces/formats#rowbinary)
100	- [ClickHouse data types](https://clickhouse.com/docs/sql-referen

People who liked this also viewed

6 picks

Type