Retriever

gllm-retrieval | Involves EM | Tutorial: Retriever | Use Case: Create the Retriever | API Reference

What is a Retriever?

A Retriever is a fundamental component in Gen AI applications that is responsible for finding and retrieving relevant information from various data sources based on user queries.

How Retrievers Work:

1. Query Input: Receives a user query (text, structured query, or natural language)

2. Query Processing: Transforms the query into a format suitable for the target data source

3. Data Retrieval: Searches the connected data store(s) for relevant information

4. Output: Returns structured results (typically as Chunk objects) with metadata

What You Can Do with a Retriever

GL SDK provides several retriever types, each designed for specific use cases and data storage requirements:

1. Fulltext Retriever: For keyword and fulltext search over a data store with fulltext capability.

2. Vector Retriever: For semantic search over vector embeddings.

3. Hybrid Retriever: For combining fulltext and vector search (coming soon).

4. Legacy: Vector Retriever (Legacy) and SQL Retriever.

5. Graph Retriever: For traversing and querying graph-structured knowledge.

GL SDK makes it easy to perform common data operations regardless of which retriever type you choose. One of the most powerful features of the GL SDK is its unified approach to data storage. All datastores support storing data in a structured format using the Chunk schema.

Think of chunks as standardized containers for your data - they provide a consistent way to represent information across different storage types, making it easy to switch between datastores or combine them in your application.

from gllm_core.schema import Chunk

# Create chunks with structured content
chunks = [
    Chunk(content="AI is the future of technology."),
    Chunk(content="Machine learning enables pattern recognition."),
    Chunk(content="Deep learning powers modern AI applications.")
]