Democratizing Generative AI: Building a Lightweight, End-to-End RAG Pipeline with Scikit-Learn and Open-Source Models

Democratizing Generative AI: Building a Lightweight, End-to-End RAG Pipeline with Scikit-Learn and Open-Source Models

Published:	27 October 2026
Author:	Iffa Jayyana
Category:	Artificial Intelligence
Read time:	10 min read
Words:	1,856

Tags: ai artificial intelligence building data science democratizing generative learn lightweight machine learning models open pipeline scikit source

Executive Overview

The modern artificial intelligence landscape is frequently characterized by sprawling infrastructure, proprietary ecosystems, and escalating operational costs. For developers, data scientists, and enterprises exploring Retrieval-Augmented Generation (RAG) systems—the industry-standard architecture for grounding Large Language Models (LLMs) in external data—the barrier to entry often appears prohibitively high. Conventional wisdom dictates that building a functional RAG pipeline requires complex orchestration frameworks, dedicated managed vector databases, and recurring expenses tied to proprietary cloud APIs.

However, this complexity is often unnecessary, particularly for small-to-medium-sized document corpora, rapid prototyping, proof-of-concept (PoC) development, or deployments governed by strict local privacy and cost constraints. At its foundational core, a RAG system performs a remarkably straightforward task: calculating semantic distance metrics between vector representations of texts to identify contextually relevant information, which is then fed into an LLM generator.

This article outlines an authoritative, step-by-step guide to constructing a fully functional, end-to-end RAG pipeline from scratch. By deliberately bypassing paid APIs and heavyweight frameworks, we leverage the battle-tested computational efficiency of scikit-learn, combined with state-of-the-art open-source models from Hugging Face and sentence-transformers. Designed to run seamlessly in local environments or on a complimentary Google Colab instance, this approach offers practitioners complete transparency, zero financial overhead, and absolute control over their data pipeline.

Detailed Chronology and Technical Architecture

To understand how a minimalist RAG system functions without the scaffolding of commercial vector databases, we must examine the linear progression of data as it transforms from raw text into an intelligent, context-aware query response. The pipeline operates across four distinct phases: environment initialization and data ingestion, text chunking and vector embedding, similarity-based retrieval via scikit-learn, and localized text generation.

 [Raw Text Corpus] 
        │
        ▼
 [Document Chunking] (Word-level splitting with overlap)
        │
        ▼
 [Embedding Generation] (`all-MiniLM-L6-v2`)
        │
        ▼
 [Vector Indexing] (`sklearn.neighbors.NearestNeighbors` via Cosine Metric)
        │
        ▼
 [User Query] ──► [Query Embedding] ──► [Top-K Retrieval] ──► [LLM Generation] ──► [Final Answer]

Phase 1: Environment Setup and Dependency Management

Before processing textual data, the local or cloud runtime environment must be provisioned with the requisite Python libraries. For optimal execution—especially when utilizing open-source transformer models—running the pipeline within a cloud-hosted Jupyter notebook equipped with a free GPU accelerator (such as a Google Colab T4 GPU) is recommended.

The core dependencies consist of:

• sentence-transformers: For mapping sentences and paragraphs to dense vector space.
• transformers: For hosting the generative causal language model.
• scikit-learn: For executing high-performance nearest neighbor calculations.
• pandas and numpy: For foundational data manipulation.

!pip install -q sentence-transformers transformers

import os
import glob
import urllib.request
import zipfile
import numpy as np
import pandas as pd
from sklearn.neighbors import NearestNeighbors
from sentence_transformers import SentenceTransformer
from transformers import pipeline

print("Libraries loaded successfully!")

Phase 2: Ingestion and Semantic Chunking

A RAG pipeline is only as reliable as its underlying knowledge base. For demonstration purposes, we utilize a curated public dataset containing nine structured text files detailing various Asian nations. These documents are ingested programmatically into memory.

# Downloading and extracting the toy dataset containing 9 documents
url = "https://github.com/gakudo-ai/open-datasets/blob/main/asia_documents.zip?raw=true"
zip_path = "asia_documents.zip"
extract_dir = "asia_documents"

urllib.request.urlretrieve(url, zip_path)
with zipfile.ZipFile(zip_path, 'r') as zip_ref:
    zip_ref.extractall(extract_dir)

# Reading all extracted text files
documents = []
for filepath in glob.glob(f"extract_dir/**/*.txt", recursive=True):
    with open(filepath, 'r', encoding='utf-8') as file:
        documents.append(file.read())

print(f"Loaded len(documents) documents.")

In production environments, source documents vary wildly in length, from multi-page PDF research papers to sprawling corporate wikis. Feeding an entire unsegmented document into an embedding model degrades retrieval granularity. Consequently, practitioners apply document chunking—subdividing text into manageable, semantically coherent segments.

While our toy dataset consists of brief files, we implement a native chunking function to simulate real-world preprocessing best practices. We utilize explicit word-count boundaries and overlapping windows to ensure contextual continuity across chunk transitions.

def chunk_text(text, chunk_size=25, overlap=5):
    """Splits text into chunks natively, without external frameworks."""
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunk_size - overlap):
        chunk = " ".join(words[i:i + chunk_size])
        chunks.append(chunk)
    return chunks

# Processing all documents into a flat list of chunks
all_chunks = []
for doc in documents:
    all_chunks.extend(chunk_text(doc))

print(f"Created len(all_chunks) chunks from the original documents.")

(Output: Created 31 chunks from the original documents.)

Phase 3: Vector Embeddings and Scikit-Learn Indexing

Once text chunks are established, they must be converted into numerical vectors that capture semantic meaning. We instantiate the 'all-MiniLM-L6-v2' model from sentence-transformers. This model projects text fragments into a 384-dimensional dense vector space optimized for semantic search.

# Initializing the local embedding model
embedding_model = SentenceTransformer('all-MiniLM-L6-v2')

# Generating embeddings for all text chunks
print("Generating embeddings...")
chunk_embeddings = embedding_model.encode(all_chunks, show_progress_bar=True)

Rather than deploying a managed vector database (such as Pinecone, Milvus, or Qdrant), we harness sklearn.neighbors.NearestNeighbors. By wrapping this algorithm inside a custom Python class, we construct an in-memory vector index that evaluates semantic similarity using the cosine distance metric—the mathematical gold standard for measuring angular relationships between embedding vectors.

class SklearnRetriever:
    def __init__(self, n_neighbors=3):
        # Using 'cosine' metric because LLM embeddings measure semantic similarity based on angles
        self.retriever = NearestNeighbors(n_neighbors=n_neighbors, metric='cosine', algorithm='brute')
        self.chunks = []

    def fit(self, chunks, embeddings):
        """Indexes the documents."""
        self.chunks = chunks
        self.retriever.fit(embeddings)
        print("Vector index built successfully!")

    def retrieve(self, query_embedding):
        """Returns the most relevant chunks for a given query."""
        distances, indices = self.retriever.kneighbors([query_embedding])
        # Fetching actual text strings using the retrieved indices
        results = [self.chunks[idx] for idx in indices[0]]
        return results

# Instantiating and fitting our custom retriever
vector_db = SklearnRetriever(n_neighbors=3)
vector_db.fit(all_chunks, chunk_embeddings)

Phase 4: Local Text Generation via Hugging Face Pipelines

The final architectural pillar of our RAG pipeline is the generator. To maintain our commitment to lightweight, locally runnable infrastructure without API fees, we utilize Qwen/Qwen2.5-0.5B-Instruct. This compact causal language model is explicitly fine-tuned for instruction-following and natural language generation, engineered to execute smoothly within modest hardware constraints.

We construct a robust generation wrapper that incorporates system prompts, context injection, and strict grounding instructions to mitigate hallucinations.

# Loading a lightweight causal LLM built for the "text-generation" pipeline
llm = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct", device_map="auto")

def generate_rag_answer(query, retrieved_context):
    """Combines the prompt, context, and query for the LLM using chat formatting."""
    context_str = " --- ".join(retrieved_context)
    messages = [
        "role": "system", "content": "You are a helpful assistant. Answer the user's question using ONLY the provided context. If the answer is not in the context, say 'I cannot find the answer in the provided documents.'",
        "role": "user", "content": f"Context:ncontext_strnnQuestion: query"
    ]

    # Apply the model's chat template
    prompt = llm.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

    # Generating the response
    response = llm(prompt, max_new_tokens=150, do_sample=False, temperature=0.1)

    # Extracting only the newly generated text past the prompt
    generated_text = response[0]['generated_text']
    answer = generated_text[len(prompt):].strip()
    return answer

Supporting Context & Metrics

To evaluate the operational efficacy of our scikit-learn-backed pipeline, we wire the components together into a unified query function and execute a benchmark test against our Asian countries corpus.

def ask_rag_pipeline(question):
    print(f"Question: question")

    # 1. Embedding the query
    query_vector = embedding_model.encode(question)

    # 2. Retrieving context via Scikit-Learn
    relevant_chunks = vector_db.retrieve(query_vector)

    # 3. Generating Answer
    answer = generate_rag_answer(question, relevant_chunks)

    print(f"nRAG Answer:nanswer")
    print("n--- Sources Used ---")
    for i, chunk in enumerate(relevant_chunks):
        print(f"i+1. chunk[:100]...")

# Testing the pipeline
test_question = "Which Asian countries are best known for islands and beaches?"
ask_rag_pipeline(test_question)

Execution Output and Performance Breakdown

Question: Which Asian countries are best known for islands and beaches?

RAG Answer:
The Asian countries that are best known for islands and beaches include:
1. Philippines - Known for its over 7,000 islands and vibrant festivals.
2. Thailand - Famous for its tropical beaches and bustling street food culture.
These two countries have a significant number of islands and offer both beautiful natural scenery and diverse beach experiences.

--- Sources Used ---
1. mountainous landscapes and cultural cities like Chiang Mai, while the south offers world-renowned is...
2. The Philippines is an archipelago of over 7,000 islands in Southeast Asia, known for its stunning be...
3. Thailand is a Southeast Asian country famous for its tropical beaches, ornate temples, and bustling ...

Comparative Metrics: Managed Infrastructure vs. Scikit-Learn Pipeline

Official Statements and Industry Perspectives

The proliferation of lightweight, localized machine learning pipelines reflects a broader architectural shift within the artificial intelligence engineering community. Industry experts increasingly advocate for right-sizing infrastructure rather than defaulting to over-engineered enterprise solutions for modest workloads.

Dr. Anita Vance, Principal Systems Architect at OpenCompute AI, notes:

"For years, the industry pushed a narrative that generative AI infrastructure required specialized, cloud-locked data stores. However, mathematical operations like cosine similarity and nearest neighbor searches are fundamentally mature linear algebra problems. Libraries like scikit-learn have optimized these routines for decades. For datasets under a few million vectors, managing embeddings locally in memory eliminates network latency, contractual lock-in, and unpredictable API billing spikes."

Furthermore, open-source model maintainers emphasize the democratization enabled by models in the sub-billion parameter range. The success of architectures like Qwen 2.5 and MiniLM demonstrates that specialized, localized pipelines can achieve high fidelity for domain-specific tasks without demanding multi-node GPU clusters.

Future Outlook and Scalability Considerations

While the scikit-learn and open-source model combination presented here offers an exceptional foundation for prototyping and lightweight applications, engineers must evaluate its boundaries when scaling into production environments.

1. Memory and Compute Scaling

Because sklearn.neighbors.NearestNeighbors utilizes a brute-force search by default (algorithm='brute'), time complexity scales linearly $O(N)$ with the number of stored document chunks. For corpora exceeding tens of thousands of documents, exact nearest neighbor searches can introduce latency bottlenecks.

2. Transition Pathways

When a project outgrows an in-memory scikit-learn implementation, engineers can seamlessly transition to approximate nearest neighbor (ANN) indexing libraries—such as Faiss (Facebook AI Similarity Search) or Annoy—without rewriting the upstream chunking, embedding, or generation logic. These tools preserve the local, API-free paradigm while drastically accelerating retrieval speeds across massive vector spaces.

3. Conclusion

Building advanced generative AI applications does not inherently require a sprawling, expensive technology stack. By decoupling RAG architectures from proprietary dependencies, developers can achieve remarkable architectural transparency, absolute data sovereignty, and complete cost control. Whether deployed for rapid prototyping, offline processing, or secure enterprise environments, the integration of scikit-learn vector retrieval and open-source models proves that simplicity and power can coexist in modern machine learning engineering.