duoan / mega-data-factory
🏭 Mega Scale Multimodal DataPipeline for SOTA Foundation Models
Why this job: topics: image-generation, image-editing
What it does
- Pipeline Parallelism. Ray ObjectRef chaining enables concurrent stage execution without blocking (details)
- Distributed Data Loading. Sharded file loading with checkpoint support for fault recovery
- Backpressure Control. Bounded in-flight batches prevent OOM on large datasets
- Bucketed Deduplication. Distributed state sharding scales to 100B+ keys (details)
- Rust Acceleration. 10-25x speedup for image quality, hashing, and HTML extraction
Get started
No single install command. Read the steps in the README (opens in new tab)
- Maintenance
- Active
- Last commit 6 months ago
- Stars
- 380
- On GitHub
- Licence
- MIT
- Allows business use
- Downloads for
- None listed
- No release files
- Latest version
- None yet
- No releases
- Built with
- Python
- 95% of the code
Is it looked after?
- Contributors
- 0
- Last 90 days
- Issues
- 17% closed
- 10 still open
- Pull requests
- 15 merged
- 1 waiting
- Releases
- None
- Never published
- No contributing guide
- No security policy
- No code of conduct
More: languages, topics and project record
Languages
- Python 95%
- Jinja 3%
- Rust 2%
- Shell <1%
Topics
- data-centric-ai
- data-curation
- data-quality
- datapipeline
- datapipelines
- deeplearning
- foundation-models
- image-editing
- image-generation
- llm
- machine-learning
- mllm
Project record
- JobImage generation
- StatusListed
- Commits69
- Forks46
- Watching29
- Stars in 28 days+1
- CreatedJan 17, 2026
- First seen by usNov 7, 2026
- CheckedNov 7, 2026