validating-clickhouse-kafka-pipelines

This skill provides production-ready patterns for implementing defense-in-depth validation in ClickHouse + Kafka data pipelines. It ensures data integrity at both producer and consumer boundaries, handles errors without blocking consumption, automatically deduplicates messages, and provides comprehensive monitoring and observability.

The skill is built on validating at both producer and consumer layers:

Consumer-side validation provides defense-in-depth and context isolation:

Implements defense-in-depth validation patterns for ClickHouse + Kafka data pipelines. Covers producer validation (msgspec schemas), consumer validation (anti-corruption layers), ClickHouse error streaming (kafka_handle_error_mode='stream'), idempotency with ReplacingMergeTree, schema evolution, and monitoring. Use when implementing data pipelines, handling Kafka errors, setting up validation layers, managing duplicates/idempotency, or monitoring Kafka consumption. Triggers on "validate kafka pipeline", "clickhouse error handling", "kafka deduplication", or "implement anti-corruption layer". Source: dawiddutoit/custom-claude.