surreal-sync
starsforks
How to install surreal-sync?
24601/surreal-skills/surreal-sync
$npx -y skills add 24601/surreal-skills --skill surreal-syncData migration and synchronization to SurrealDB from MongoDB, PostgreSQL, MySQL, Neo4j, Kafka, and JSONL. Full and incremental CDC sync. Part of the surreal-skills collection.
SKILL.md
| 1 | # Surreal-Sync -- Data Migration and Synchronization |
| 2 | |
| 3 | Surreal-Sync is a CLI tool for migrating data from various database sources to |
| 4 | SurrealDB with full and incremental synchronization via Change Data Capture (CDC). |
| 5 | |
| 6 | ## Supported Sources |
| 7 | |
| 8 | | Source | Full Sync | Incremental CDC | Method | |
| 9 | |--------|-----------|----------------|--------| |
| 10 | | MongoDB | Yes | Yes | Change streams | |
| 11 | | MySQL | Yes | Yes | Trigger-based CDC + sequence checkpoints | |
| 12 | | PostgreSQL (triggers) | Yes | Yes | Trigger-based CDC + sequence checkpoints | |
| 13 | | PostgreSQL (wal2json) | Yes | Yes | Logical replication with wal2json plugin | |
| 14 | | Neo4j | Yes | Yes | Timestamp-based tracking | |
| 15 | | JSONL Files | Yes | N/A | Batch import from JSON Lines | |
| 16 | | Apache Kafka | Yes | Yes | Consumer subscriptions with deduplication | |
| 17 | |
| 18 | ## Quick Start |
| 19 | |
| 20 | ```bash |
| 21 | # Install surreal-sync (Rust binary) |
| 22 | cargo install surreal-sync |
| 23 | |
| 24 | # Full sync from PostgreSQL (trigger-based) |
| 25 | surreal-sync from postgres trigger-full \ |
| 26 | --connection-string "postgresql://user:pass@localhost/mydb" \ |
| 27 | --surreal-endpoint "http://localhost:8000" \ |
| 28 | --surreal-username root \ |
| 29 | --surreal-password root \ |
| 30 | --to-namespace prod \ |
| 31 | --to-database main |
| 32 | |
| 33 | # Incremental CDC from PostgreSQL (wal2json) |
| 34 | surreal-sync from postgres wal2json \ |
| 35 | --connection-string "postgresql://user:pass@localhost/mydb" \ |
| 36 | --surreal-endpoint "http://localhost:8000" \ |
| 37 | --surreal-username root \ |
| 38 | --surreal-password root \ |
| 39 | --to-namespace prod \ |
| 40 | --to-database main |
| 41 | |
| 42 | # Full sync from MongoDB |
| 43 | surreal-sync from mongo full \ |
| 44 | --connection-string "mongodb://localhost:27017/mydb" \ |
| 45 | --surreal-endpoint "http://localhost:8000" \ |
| 46 | --surreal-username root \ |
| 47 | --surreal-password root \ |
| 48 | --to-namespace prod \ |
| 49 | --to-database main |
| 50 | |
| 51 | # Batch import from JSONL |
| 52 | surreal-sync from jsonl import \ |
| 53 | --file data.jsonl \ |
| 54 | --surreal-endpoint "http://localhost:8000" \ |
| 55 | --surreal-username root \ |
| 56 | --surreal-password root \ |
| 57 | --to-namespace prod \ |
| 58 | --to-database main |
| 59 | |
| 60 | # Consume from Kafka |
| 61 | surreal-sync from kafka consume \ |
| 62 | --bootstrap-servers "localhost:9092" \ |
| 63 | --topic my-events \ |
| 64 | --surreal-endpoint "http://localhost:8000" \ |
| 65 | --surreal-username root \ |
| 66 | --surreal-password root \ |
| 67 | --to-namespace prod \ |
| 68 | --to-database main |
| 69 | ``` |
| 70 | |
| 71 | ## CLI Pattern |
| 72 | |
| 73 | ``` |
| 74 | surreal-sync from <SOURCE> <COMMAND> \ |
| 75 | --connection-string [CONNECTION STRING] \ |
| 76 | --surreal-endpoint [SURREAL ENDPOINT] \ |
| 77 | --surreal-username [SURREAL USERNAME] \ |
| 78 | --surreal-password [SURREAL PASSWORD] \ |
| 79 | --to-namespace <NS> \ |
| 80 | --to-database <DB> |
| 81 | ``` |
| 82 | |
| 83 | ## Key Features |
| 84 | |
| 85 | - Automatic schema inference and SurrealDB table creation |
| 86 | - Record ID mapping from source primary keys |
| 87 | - Relationship extraction and graph edge creation |
| 88 | - Configurable batch sizes and parallelism |
| 89 | - Resumable sync with checkpoint tracking |
| 90 | - Deduplication for Kafka consumers |
| 91 | |
| 92 | ## Full Documentation |
| 93 | |
| 94 | See the main skill's rule file for complete guidance: |
| 95 | - **[rules/surreal-sync.md](../../rules/surreal-sync.md)** -- source configuration, schema mapping, CDC setup, conflict resolution, and production deployment |
| 96 | - **[surrealdb/surreal-sync](https://github.com/surrealdb/surreal-sync)** -- upstream repository |
People who liked this also viewed
6 picksType
- azure-storageAzure Storage Services including Blob Storage, File Shares, Queue Storage, Table Storage, and Data Lake. Answers questions about storage access tiers (hot, cool, cold, archive), when to use each tier,SkillsJul 2026
- supabase-postgres-best-practicesPostgres performance optimization and best practices from Supabase. Use this skill when writing, reviewing, or optimizing Postgres queries, schema designs, or database configurations.SkillsJul 2026
- supabaseUse when doing ANY task involving Supabase. Triggers: Supabase products (Database, Auth, Edge Functions, Realtime, Storage, Vectors, Cron, Queues); client libraries and SSR integrations (supabase-js, SkillsJul 2026
- convex-quickstartCreates or adds Convex to an app. Use for new Convex projects, npm create convex@latest, frontend setup, env vars, or the first npx convex dev run.SkillsJul 2026
- convex-performance-auditAudits Convex performance for reads, subscriptions, write contention, and function limits. Use for slow features, insights findings, OCC conflicts, or read amplification.SkillsJul 2026
- convex-setup-authSets up Convex auth, identity mapping, and access control. Use for login, auth providers, users tables, protected functions, or roles in a Convex app.SkillsJul 2026
