data-engineering-storage-formats

Comprehensive guide to modern data serialization formats for analytics and machine learning: Parquet, Apache Arrow, Lance, Zarr, Avro, and ORC. Learn compression tradeoffs, partitioning strategies, and when to use each format.

| Format | Type | Best For | Compression | Schema Evolution | Random Access |

| Parquet | Columnar | Analytics, data lakes | ✅ (Snappy, Zstd, LZ4) | ✅ (add/drop) | ✅ (row groups) | | Arrow/Feather | Columnar | In-memory, IPC, ML | ✅ (LZ4, Zstd) | Limited | ✅ (record batches) | | Lance | Columnar | ML pipelines, vectors | ✅ (Zstd, LZ4) | ✅ | ✅ (multi-modal) |

تنسيقات تسلسل البيانات الحديثة: Parquet، وApache Arrow (Feather/IPC)، وLance (ML-native)، وZarr (مصفوفات مقسمة)، وAvro، وORC. يغطي الضغط والتقسيم واختيار التنسيق. المصدر: legout/data-platform-agent-skills.