vaex

SKILL.md

Vaex is a high-performance Python library designed for lazy, out-of-core DataFrames to process and visualize tabular datasets that are too large to fit into RAM. Vaex can process over a billion rows per second, enabling interactive data exploration and analysis on datasets with billions of rows.

Vaex provides six primary capability areas, each documented in detail in the references directory:

Load and create Vaex DataFrames from various sources including files (HDF5, CSV, Arrow, Parquet), pandas DataFrames, NumPy arrays, and dictionaries. Reference references/coredataframes.md for:

使用此技能来处理和分析超出可用 RAM 的大型表格数据集（数十亿行）。 Vaex 擅长核外 DataFrame 操作、惰性求值、快速聚合、大数据的高效可视化以及大型数据集上的机器学习。当用户需要处理大型 CSV/HDF5/Arrow/Parquet 文件、对海量数据集执行快速统计、创建大数据可视化或构建不适合内存的 ML 管道时适用。 来源：ovachiever/droid-tings。

查看原文