High-Performance Enterprise Data Ingestion, Federation & Security Framework for AI and Search.
OpenCrawling is the reference Java and Spring Framework implementation of the Open Ingestion Standard (OIS). It is an open-source, cloud-native platform designed to solve the hardest problem in enterprise AI and search: securely connecting siloed content repositories to modern RAG, vector search, and data lakehouse pipelines.
Going far beyond simple web scraping, OpenCrawling acts as an intelligent, security-aware bridge between enterprise content stores (ECM, DMS, BPM/workflow engines, relational databases, data lakehouses, web crawlers) and modern search engines, vector databases, and Large Language Model (LLM) agents.
When building Retrieval-Augmented Generation (RAG) systems or enterprise search, getting the data is only half the battle. You also need to respect document permissions, propagate deletions (DELETE tombstones), handle incremental updates, and scale without bottlenecking source systems.
OpenCrawling is built from the ground up to address these challenges with modern software architecture:
- 🔒 Enterprise Security First (Zero-Trust ACL Mapping): Built-in synchronization of native Access Control Lists (ACLs) with deny-by-default enforcement. We ensure that your vector databases, search engines, and MCP-enabled AI agents only return documents the querying user is explicitly authorized to see.
- ⚡ High-Performance & Non-Blocking: Powered by Java 25 and Spring Boot 4.1+, leveraging Virtual Threads and Structured Concurrency alongside decoupled Apache Kafka and gRPC streaming pipelines for massive I/O throughput.
- 🤖 AI, RAG & MCP Ready: Native support for Apache Tika text extraction, semantic chunking, stateless multi-provider embedding generation (Spring AI 2.0+ / Ollama / OpenAI), and a built-in Model Context Protocol (MCP) server with pre-filtered ACL security.
- 📊 AI-Powered Observability (AIOps) & Lakehouse Audit Ledger: Native OpenTelemetry (OTel) correlated traces, Micrometer metrics, automated root-cause analysis (RCA) in the Admin UI, and an immutable Apache Iceberg / Ozone ingestion audit ledger.
- 🔌 Pluggable SPI & Maven Archetypes: Clean, modular Java SPIs (
RepositoryConnector,TransformationConnector,OutputConnector) and official Maven Archetypes (opencrawling-connector-archetypes) to scaffold custom connectors in seconds.
| Repository | Description |
|---|---|
opencrawling |
The core OpenCrawling platform, runtime engine, decoupled embedding microservice, React Admin UI, CLI, and enterprise connector suite. |
open-ingestion-standard |
The Open Ingestion Standard (OIS) vendor-neutral specification and JSON schemas for secure document ingestion, ACL tokens, and tombstone lifecycles. |
homebrew-tap |
Official Homebrew tap for installing the OpenCrawling CLI (oc) on macOS and Linux. |
opencrawling.github.io |
Official documentation and project website (opencrawling.org). |
Inside opencrawling/opencrawling, the platform is organized into decoupled modules:
| Module | Description |
|---|---|
oc-core |
Core SPIs (RepositoryConnector, TransformationConnector, OutputConnector), OIS domain models, security ACL contracts, and Virtual Thread orchestration. |
oc-runtime |
Spring Boot 4 ingestion runtime engine, Kafka streaming consumers/producers, Tika extraction, semantic chunking, REST API, and secure MCP Server. |
oc-embedding-service |
Stateless, horizontally scalable embedding microservice supporting dynamic multi-dimension models (384 / 768 / 1024) via Spring AI. |
oc-admin-ui |
React / TypeScript administration console for managing connectors, jobs, security mappings, Apache ManifoldCF migrations, and AIOps diagnostics. |
oc-cli & oc-java-client-sdk |
Command-line interface (oc) and Java Client SDK for declarative pipeline automation and CI/CD integration. |
oc-grpc-api & oc-grpc-push-connector |
High-performance Protobuf / gRPC internal transport and external push ingestion gateway. |
oc-observability |
OpenTelemetry trace correlation, Micrometer metrics, and AI-driven root-cause analysis (RCA). |
oc-mcf-migrator |
Automated migration engine to convert legacy Apache ManifoldCF job/connection definitions into OpenCrawling pipelines. |
oc-connector-archetypes |
Published Maven Central archetypes for scaffolding custom Repository, Transformation, and Output connectors. |
- Enterprise Content Management (ECM / DMS / CSP):
- Alfresco Content Services (ACS) (
oc-alfresco-repository-connector) - OASIS CMIS 1.0 / 1.1 (
oc-cmis-repository-connector— native Java 25 Browser Binding client) - Doxis 4 (
oc-doxis-repository-connector— SER Group Doxis CSB REST API with e-file & ACL mapping) - M-Files (
oc-mfiles-repository-connector— M-Files Web Service MFWS REST API & VAF incremental change feed) - SharePoint Online (
oc-sharepoint-repository-connector— Microsoft Graph v1.0 delta & tree crawling with Entra ID OAuth2) - Hyland OpenArch (
oc-openarch-repository-connector— Content Innovation Cloud / OpenArch repository source)
- Alfresco Content Services (ACS) (
- Business Process Management (BPM / Workflow):
- Alfresco Process Services (APS 26.2) (
oc-aps-repository-connector— REST + APS MCP Server discovery) - Flowable BPMN (
oc-flowable-repository-connector) - Camunda BPM (
oc-camunda-repository-connector)
- Alfresco Process Services (APS 26.2) (
- Data Lakehouse, Databases, Web & Streaming:
- Apache Iceberg (
oc-iceberg-repository-connector) - Apache StormCrawler (
oc-stormcrawler-repository-connector&oc-stormcrawler-bolt) - JDBC Relational Databases (
oc-jdbc-repository-connector) - Local & Network Filesystems (
oc-filesystem-repository-connector) - gRPC Push Ingestion Gateway (
oc-grpc-push-connector)
- Apache Iceberg (
- Vector & Hybrid Search Engines: PostgreSQL (
pgvector), Apache Solr 10, OpenSearch 2.x / 3.x, Vespa 8, Milvus, Qdrant, Luxir - Object Storage, Archive & Multi-Sink Federation: Apache Ozone (with Claim Check pattern), Hyland OpenArch, Apache SeaTunnel (Zeta Engine fan-out to 100+ sinks)
- Core: Java 25 (Virtual Threads & Structured Concurrency), Spring Boot 4.1+, Spring AI 2.0+
- Streaming & RPC: Apache Kafka, gRPC / Protobuf, Model Context Protocol (MCP)
- Extraction & Lakehouse: Apache Tika 4.1, Apache Iceberg 1.11, Apache Ozone 2.2
- Build & Deployment: Maven Multi-Module, Docker Compose, Kubernetes, Homebrew
- License: Apache License 2.0
We are building a community of developers, architects, and data engineers passionate about open-source enterprise data integration and AI infrastructure.
- 🌐 Official Website: opencrawling.org
- 📖 Read the Documentation: GitHub Wiki & DeepWiki AI Docs
- 📐 Standard Specification: Open Ingestion Standard (OIS)
- 🐛 Report Issues: Issue Tracker
- 💬 Join the Discussion: Slack Community
- ✉️ Official Contact: Reach out to us at info@opencrawling.org for general inquiries or security reports.
- ⭐ Star our Repositories: If you find our work useful, give us a star!
Built with passion for the Open Source and AI community.