Ingestion Adapters
Beyond the primary QUIC SDK path, Tetrapus supports pluggable ingestion adapters for HTTP, Kafka, and S3 sources. All adapters feed into the same pipeline: validate, deduplicate, route to WAL, and publish to the EventBus.
Adapter Types
| Adapter | Configuration | Use Case |
|---|---|---|
| HttpAdapter | url: String | Pull from REST endpoints or receive webhooks |
| KafkaAdapter | brokers: String, topic: String | Consume from Kafka topics (enterprise integration) |
| S3Adapter | bucket: String, key: String | Batch import from S3 object storage |
All adapters implement the Adapter trait with a single start(&self, bus: &EventBus) -> Result<()> method that publishes IngestEnvelope messages.
Write-Ahead Log (WAL)
Every ingested envelope is persisted to a WAL before EventBus publication, ensuring durability across crashes. The WAL handles three entry types:
| Entry Kind | Description |
|---|---|
| Telemetry | Standard telemetry payload from the API |
| Envelope | Full IngestEnvelope with metadata |
| Unknown | Opaque bytes (forward compatibility during migrations) |
Deduplication
The DedupeStore maintains an in-memory set of seen envelope IDs. Each incoming envelope is checked: if the ID has been seen before, it is silently dropped. The check_and_insert(id) method returns true for new messages and false for duplicates.
Backpressure Control
The BackpressureController provides an atomic pause/resume mechanism. When downstream systems (ClickHouse, EventBus consumers) are overloaded, the controller sets paused = true, which causes ingestion adapters to hold new data until the flag is cleared. Thread-safe via AtomicBool with SeqCst ordering.
Pipeline Configuration
Adapters are configured in data_sources within the YAML config. See Schema Reference for the full data_sources spec.
- type: in_process_sim — Built-in domain simulator (HVAC, aviation, etc.)
- type: live_source — External SDK source over QUIC
- type: asset_source — Static asset data (city labels, terrain features)
Questions?
Reach out for help with integration, deployment, or custom domain codecs.