data-prep-kit / data-prep-kit

What it does

• The kit provides a growing set of modules/transforms targeting laptop-scale to datacenter-scale processing.
• The data modalities supported _today_ are. Natural Language, Code, and Image.
• The modules are built on common frameworks for Python and Ray runtimes for scaling up data processing.
• The kit provides a framework for developing custom transforms for processing Parquet files as well as ZIP, NDJSON, and JSONL file formats.
• The kit provides examples of how a single transform can be deployed on Kubernetes clusters as a Python or a Ray job. Additionally, when multiple transforms are deployed in a sequence, the kit uses Tekton pipelines.