Chonkie

Chonkie is an open-source, ultra-light, and lightning-fast chunking library designed to make text AI-ready. It provides a feature-rich, easy-to-use, and efficient solution for processing text data, offering wide support for various tokenizers, embedding models, and APIs. Chonkie is multilingual, cloud-ready, and follows a modular pipeline approach (CHOMP) to transform raw documents into refined, usable chunks for RAG applications.