Reference Formatter

gllm-generation | Involves EM | Involves LM | Tutorial: Reference Formatter | Use Case: Adding Document References | API Reference

Prerequisites

This example specifically requires completion of all setup steps listed on the Prerequisites page.

You should be familiar with these concepts:

1. Language Model (LM) Invoker

2. LM Request Processor (LMRP)

What’s a Reference Formatter?

The reference formatter is a utility module designed to filter and format the references of an RAG pipeline response in a clear, standardized format. In this tutorial, you'll learn how to use the SimilarityBasedReferenceFormatter in just a few lines of code. You can also explore other types of reference formatters, available here.

Supported Reference Formatter

Formatter	Filtering method	Best for
LMBasedReferenceFormatter	LM judgment	Rich semantic filtering; more accurate but slower
NoOpReferenceFormatter	None (pass-through)	Development, pre-filtered inputs, or always-include-all cases
SimilarityBasedReferenceFormatter	Embedding cosine similarity	Fast, embedding-based filtering with a score threshold

Installation

Linux, macOS, or Windows WSL

# you can use a Conda environment
pip install --extra-index-url https://oauth2accesstoken:$(gcloud auth print-access-token)@glsdk.gdplabs.id/gen-ai-internal/simple/ "gllm-generation"

Windows Powershell

Windows Command Prompt

Quickstart

Let’s jump into a basic example using SimilarityBasedReferenceFormatter. Since it utilizes an embedding model, we can simply pass an EM invoker to build one. We can also set a threshold to control how strictness of the candidate chunks filtering.

import asyncio
from gllm_core.schema import Chunk
from gllm_inference.em_invoker import build_em_invoker
from gllm_generation.reference_formatter import SimilarityBasedReferenceFormatter

candidate_chunks = [
    Chunk(content="Indonesia is a country in Southeast Asia.", metadata={"file_name": "indonesia.txt"}),
    Chunk(content="Malaysia is a country in Southeast Asia.", metadata={"file_name": "malaysia.txt"}),
    Chunk(content="Singapore is a country in Southeast Asia.", metadata={"file_name": "singapore.txt"}),
    Chunk(content="The capital of Indonesia is Jakarta.", metadata={"file_name": "indonesia.txt"}),
    Chunk(content="The capital of Malaysia is Kuala Lumpur.", metadata={"file_name": "malaysia.txt"}),
    Chunk(content="The capital of Singapore is Singapore.", metadata={"file_name": "singapore.txt"}),
]
response = "Indonesia is a country in Southeast Asia. The capital of Indonesia is Jakarta."

em_invoker = build_em_invoker(model_id="openai/text-embedding-3-small")
ref_formatter = SimilarityBasedReferenceFormatter(em_invoker, threshold=0.7)
references = asyncio.run(ref_formatter.format_reference(response=response, chunks=candidate_chunks))
print(references)

Expected Output

References:
1. indonesia.txt

That’s it! You've just successfully used the StuffResponseSynthesizer! You can try to play around with the threshold to find a value that works best for your use cases.

Format Customization

The SimilarityBasedReferenceFormatter also supports customizing both the pre chunk display format as well as the entire references display format. These can bet set through the format_chunk_func and format_references_func parameters, respectively.

import asyncio
from gllm_core.schema import Chunk
from gllm_inference.em_invoker import build_em_invoker
from gllm_generation.reference_formatter import SimilarityBasedReferenceFormatter

candidate_chunks = [
    Chunk(content="Indonesia is a country in Southeast Asia.", metadata={"file_name": "indonesia.txt"}),
    Chunk(content="Malaysia is a country in Southeast Asia.", metadata={"file_name": "malaysia.txt"}),
    Chunk(content="Singapore is a country in Southeast Asia.", metadata={"file_name": "singapore.txt"}),
    Chunk(content="The capital of Indonesia is Jakarta.", metadata={"file_name": "indonesia.txt"}),
    Chunk(content="The capital of Malaysia is Kuala Lumpur.", metadata={"file_name": "malaysia.txt"}),
    Chunk(content="The capital of Singapore is Singapore.", metadata={"file_name": "singapore.txt"}),
]
response = "Indonesia is a country in Southeast Asia. The capital of Indonesia is Jakarta."

def custom_format_chunk_func(chunk: Chunk) -> str:
    return f"{chunk.metadata['file_name']}: {chunk.content!r}"

def custom_format_references_func(formatted_chunks: list[str]) -> str:
    references = "=== REFERENCES ==="
    for idx, formatted_chunk in enumerate(formatted_chunks):
        references += f"\n[{idx + 1}] {formatted_chunk}"
    return references

em_invoker = build_em_invoker(model_id="openai/text-embedding-3-small")
ref_formatter = SimilarityBasedReferenceFormatter(
    em_invoker,
    threshold=0.7,
    format_chunk_func=custom_format_chunk_func,
    format_references_func=custom_format_references_func,
)
references = asyncio.run(ref_formatter.format_reference(response=response, chunks=candidate_chunks))
print(references)

Expected Output

=== REFERENCES ===
[1] indonesia.txt: 'The capital of Indonesia is Jakarta.'
[2] indonesia.txt: 'Indonesia is a country in Southeast Asia.'

Returning Raw Chunks

When desired, we can also skip the formatting part altogether by setting the stringify param to False. In this case, the SimilarityBasedReferenceFormatter will simply return the raw relevant chunks.

import asyncio
from gllm_core.schema import Chunk
from gllm_inference.em_invoker import build_em_invoker
from gllm_generation.reference_formatter import SimilarityBasedReferenceFormatter

candidate_chunks = [
    Chunk(content="Indonesia is a country in Southeast Asia.", metadata={"file_name": "indonesia.txt"}),
    Chunk(content="Malaysia is a country in Southeast Asia.", metadata={"file_name": "malaysia.txt"}),
    Chunk(content="Singapore is a country in Southeast Asia.", metadata={"file_name": "singapore.txt"}),
    Chunk(content="The capital of Indonesia is Jakarta.", metadata={"file_name": "indonesia.txt"}),
    Chunk(content="The capital of Malaysia is Kuala Lumpur.", metadata={"file_name": "malaysia.txt"}),
    Chunk(content="The capital of Singapore is Singapore.", metadata={"file_name": "singapore.txt"}),
]
response = "Indonesia is a country in Southeast Asia. The capital of Indonesia is Jakarta."

em_invoker = build_em_invoker(model_id="openai/text-embedding-3-small")
ref_formatter = SimilarityBasedReferenceFormatter(em_invoker, threshold=0.7, stringify=False)
references = asyncio.run(ref_formatter.format_reference(response=response, chunks=candidate_chunks))
print(references)

Expected Output

[
    Chunk(
        id='ef4eb6c6-16d0-4318-86c0-cd010b8afb08',
        content='The capital of Indonesia is Jakarta.',
        metadata={'file_name': 'indonesia.txt'},
    ),
    Chunk(
        id='13862359-42d7-4684-bb09-95d03ea74bff',
        content='Indonesia is a country in Southeast Asia.',
        metadata={'file_name': 'indonesia.txt'},
    ),
]

Congratulations! You've finished the tutorial to use the SimilarityBasedReferenceFormatter!