data-engineering-quality

Data validation and testing frameworks for ensuring pipeline correctness and data quality: Great Expectations (enterprise) and Pandera (lightweight). Integrates with orchestration tools for automated validation.

| Approach | Declarative "expectations" | Schema definitions with checks | | DataFrame Support | Pandas, Spark, SQL, BigQuery | Pandas, Polars, PySpark, Dask | | Validation Output | JSON results with detailed diagnostics | Boolean or exception | | Best For | Enterprise data platforms, comprehensive profiling | Python-centric pipelines, lightweight |

| Learning Curve | Steeper (concepts: DataContext, Checkpoints) | Lower (Python decorators/classes) | | Integration | CI/CD, Airflow, Prefect, Dagster | pytest, FastAPI, any Python code |

اختبار جودة البيانات والتحقق من صحتها باستخدام Great Opportions وPandera. التحقق من صحة المخطط واختبارات جودة البيانات والتوصيف والتحقق الآلي في خطوط الأنابيب. المصدر: legout/data-platform-agent-skills.