Idempotent NebulaGraph bulk ingestion from .NET using deterministic VIDs

NebulaGraph has no unique indexes, so duplicates in a graph load usually come from unstable vertex IDs and edge ranks. The author shows how to derive deterministic VIDs from business keys with FNV-1a and pin edge rank, which makes repeated loads safe. A batched .NET ingestion pipeline with checkpoints and retries is included.

Key points

• Derive VIDs deterministically from business keys; a 16-character FNV-1a hash fits a FIXED_STRING(16) space
• Do not use string.GetHashCode() for VIDs, since it changes between processes
• Pin edge rank to 0 or derive it from an immutable attribute, never from a timestamp
• Use INSERT for bulk loads; UPSERT serializes per partition and becomes a bottleneck at volume
• After loading, run SUBMIT JOB STATS and compare counts with distinct business keys in the source
• Escape backslashes before quotes when building nGQL string literals by hand

NebulaGraph enforces uniqueness only through its vertex and edge keys, so a plain INSERT VERTEX silently overwrites an existing record with the same VID. Repeated loads are therefore safe only when every business entity always maps to the same VID. The author proposes hashing the business key with FNV-1a into a fixed 16-character string VID, and pinning edge rank to 0 or deriving it from an immutable attribute instead of a timestamp.

The .NET pipeline streams records, sends batches of 256 as one multi-value nGQL statement, records completed batch offsets, and retries transient failures with backoff. The piece also compares driver-side loading with the Importer and Exchange tools for larger volumes, and warns about bulk UPSERT throughput, super-nodes, dangling edges and changing natural keys.

Real-world use

• A nightly .NET ETL job that reloads customer and transaction data into NebulaGraph without creating duplicates
• A crash-resumable import of ERP customer and order relationships into a graph space
• A recommendation service storing ordered user-product edges where rank carries a version number

Why it matters

If your loader runs more than once, duplicate vertices and phantom edges will appear unless VIDs and ranks are designed deterministically from the start. Fixing this after the fact means a costly re-import.

Related stories

• TableSchema and database structure upgrades: compare, plan, executeDVDEV Community: #dotnet3 d ago
• Read your own writes with Marten: ProjectLatest and Wolverine's UpdatedAggregateJMJeremy D. Miller (The Shade Tree Developer)6 d ago
• Collection Return Types in C#: Interfaces, Views, and SnapshotsDLNick Cosentino (Dev Leader)2 d ago