Ingestion Pipeline

The ingestion pipeline receives telemetry over QUIC via the nerve_ingester crate. Each frame passes through a six-stage pipeline before reaching the event bus. Backpressure is enforced at the transport layer using a token-bucket algorithm.

Pipeline Stages

Text

QUIC Stream
  │
  ▼
ProtocolAdapter::decode()   ← trait: pluggable per wire format
  │
  ▼
Validate        ← schema check, required fields, range bounds
  │
  ▼
DedupeFilter    ← bloom + LRU, keyed on (entity_id, ts, hash)
  │
  ▼
Router          ← domain-based fan-out to partition writers
  │
  ▼
Partitioner     ← assigns ClickHouse partition key
  │
  ▼
WalWriter       ← fsync WAL segment, then ack upstream
  │
  ▼
EventBus        ← broadcast to subscribers (GPU, agents, MV)

QUIC Listener

Crate: nerve_ingester
Transport: QUIC (quinn) with TLS 1.3, 0-RTT enabled
Bidirectional: Commands flow back to SDKs over the same QUIC bidi stream
Concurrency: One tokio task per accepted stream

Protocol Adapter Trait

Rust

pub trait ProtocolAdapter: Send + Sync {
    fn decode(&self, raw: &[u8]) -> Result<TelemetryFrame>;
    fn encode_command(&self, cmd: &Command) -> Result<Vec<u8>>;
}

Backpressure

Token bucket: Per-source rate limit (default 10k frames/sec)
WAL pressure: If WAL segment exceeds 256 MB, new streams are paused
Poison pill queue: Frames that fail validation 3 times are routed to a dead-letter queue for manual inspection

Deduplication

DedupeFilter uses a two-layer strategy: a counting Bloom filter for fast rejection (0.1% false-positive rate) backed by an LRU cache of recent (entity_id, ts, payload_hash) tuples. Duplicates are silently dropped and counted in the ingestion metrics.

Questions?

Reach out for help with integration, deployment, or custom domain codecs.

Get Started Request a Demo