Build a self-hosted RAG stack with AI automation services using Langflow, Milvus, Meilisearch, and Gemini for AI assistants.
A self-hosted Retrieval-Augmented Generation (RAG) stack can combine Langflow, Milvus, Meilisearch, and Gemini to build an AI assistant. This setup supports both semantic search and keyword-based search.
The stack can be run through a manual installation or Docker Compose. The manual method helps identify common setup issues, while Docker Compose provides a simpler way to bring the services together.
An Overview
- What You Need for a Self-Hosted RAG Stack
- Minimum System Requirements
- Option 1: Install the RAG Stack Manually
- Option 2: Use Docker Compose
- Connect the Services in Langflow
- Build the RAG Workflow
- Why Use Both Milvus and Meilisearch?
- Using Ollama Instead of Gemini
- Troubleshooting the Self-Hosted RAG Stack
- Final Architecture
What You Need for a Self-Hosted RAG Stack
The setup uses four main components:
| Component | Purpose |
|---|---|
| Langflow | Builds AI flows through a visual interface |
| Milvus | Handles vector search |
| Meilisearch | Handles full-text and keyword search |
| Gemini | Provides the language model |
For a development setup, the document uses a VM with Ubuntu 24.04.3 LTS, 16 GB RAM, 100 GB storage, and a 4-core CPU. No GPU is required when using the Gemini API.
Minimum System Requirements
The stack needs enough memory and storage because Milvus and Langflow can use significant system resources.
| Resource | Minimum |
|---|---|
| CPU | 4 cores with AVX support |
| RAM | 12 GB |
| Storage | 40 GB SSD |
| OS | Ubuntu 22.04+ or WSL2 |
| GPU | Not required for Gemini API |
For heavier use, the document recommends 8 or more CPU cores, 24 to 32 GB RAM, and 100 GB or more of NVMe storage.
Build Your AI Stack Today.
Option 1: Install the RAG Stack Manually
The manual setup starts with the VM and then installs each service separately.
Install Python and Langflow
First, update the VM and install Python, pip, and the Python virtual environment package.
sudo apt update && sudo apt upgrade -y
sudo apt install python3 python3-pip python3-venv -yYou can then create a virtual environment and install Langflow from PyPI.
python3 -m venv langflow-env
source langflow-env/bin/activate
pip install langflowLangflow can run on port 7860. When using a cloud VM, the service can listen on the VM address.
langflow run --host 0.0.0.0 --port 7860You also need to set the required Langflow environment values, including authentication, host, port, database, secret key, username, and password.
Install Milvus
Milvus uses etcd for metadata and MinIO for object storage. The document uses Milvus 2.6.4, etcd 3.5.13, and MinIO.
The main components are:
- Milvus for vector search
- etcd for metadata
- MinIO for object storage
After installation, configure Milvus with the correct etcd endpoint, MinIO address, bucket name, storage path, and port.
The document also recommends changing the default MinIO password before using the setup for a customer environment.
Install Meilisearch
Meilisearch handles keyword and full-text search. The setup creates a dedicated user, storage directory, log directory, and systemd service.
You can check the service with:
systemctl status meilisearchThe document also shows how to generate a master key and store it in the environment file.
Meilisearch should not be exposed to the public internet. It can run locally and connect with Langflow through the Python SDK.
Option 2: Use Docker Compose
Docker Compose brings the services together in one configuration.
The stack contains:
- etcd
- MinIO
- Milvus
- Meilisearch
- Langflow
Create a .env file for values such as the Gemini API key, MinIO credentials, and Meilisearch master key. This keeps sensitive values out of the Docker Compose file.
Then start the services with:
docker compose up -dYou can access Langflow on port 7860 and Meilisearch on port 7700. Langflow uses Milvus through port 19530.
Connect the Services in Langflow
Once Langflow is running, create a flow that connects the services.
For Gemini, use the Google Generative AI node and provide the API key.
Milvus, use the Milvus Vector Store node with the Milvus host and port.
Meilisearch, use a custom component or Python function with the Meilisearch Python SDK.
This setup gives the RAG system two search methods. Milvus handles semantic search, while Meilisearch handles keyword and metadata searches.
Build the RAG Workflow
The RAG system uses two separate flows:
- Ingestion flow
- Retrieval flow
Phase 1: Ingest Your Data
The ingestion flow loads documents into the search systems.
The basic flow is:
File → Text Splitter → Embeddings → Milvus
The document also sends the same text metadata to Meilisearch for keyword searches.
For the text splitter, the document uses a chunk size of 1000 with an overlap of 200.
Phase 2: Create the Chatbot
The retrieval flow handles user questions.
The basic flow is:
Chat Input → Milvus Search → Prompt → Gemini → Chat Output
Milvus finds relevant text based on vector meaning. The prompt then passes the retrieved context to Gemini so it can generate the response.
A sample prompt from the setup asks the model to answer using only the provided context and state when it does not know the answer.
Why Use Both Milvus and Meilisearch?
Milvus works well when the user searches by meaning. Meilisearch helps when the search depends on an exact term, product name, or part number.
For example, a question about fixing a leaking pipe can use semantic search. A search for a specific part number such as PX-900 can use keyword search.
This combination gives the RAG flow both semantic and exact text search.
Using Ollama Instead of Gemini
The stack can also use Ollama instead of Gemini.
In Langflow, replace the Google Generative AI node with the Ollama component. Then provide the Ollama base URL and the model name.
For a fully local RAG setup, the document also suggests using a local embedding model such as nomic-embed-text through Ollama.
Local models need more system resources. The document notes that models such as Llama 3.1 8B or Gemma 3 4B need an NVIDIA GPU with at least 8 GB to 12 GB of VRAM for smooth operation.
Troubleshooting the Self-Hosted RAG Stack
The stack can require significant system resources, especially when running Milvus and Langflow together.
Keep these points in mind:
- Allocate enough RAM to Docker.
- Use SSD storage for Milvus.
- Make sure the CPU supports AVX.
- Keep RAG data in the configured persistent storage folders.
- Use strong passwords for MinIO.
- Store API keys and service credentials in environment files.
- Check systemd logs when a service fails.
- Keep Meilisearch away from public access.
Final Architecture
The complete stack has four main roles:
| Layer | Tool | Role |
|---|---|---|
| Flow management | Langflow | Builds and connects the workflows |
| Semantic search | Milvus | Searches content by meaning |
| Keyword search | Meilisearch | Searches exact terms and metadata |
| Language model | Gemini or Ollama | Generates responses |
Together, these components provide a modular base for a self-hosted RAG application. Langflow controls the workflows, Milvus handles semantic search, Meilisearch handles precise keyword searches, and Gemini or Ollama generates the final response.
Conclusion
A self-hosted RAG stack can bring together Langflow, Milvus, Meilisearch, and Gemini in one setup. You can install the components manually or use Docker Compose. Then, the two RAG flows handle data ingestion and user queries. With Milvus for semantic search and Meilisearch for keyword search, the stack provides a practical base for building a RAG application.
