Routing

gllm-misc | Involves LM | Involves EM | Tutorial: Routing | Use Case: Implement Semantic Routing | API Reference

What’s a Router?

A router selects which downstream path (tool, retriever, prompt, agent) to use for a given input. This helps ensure that each query is handled by the most suitable component.

Prerequisites

This example specifically requires completion of all setup steps listed on the Prerequisites page.

You should be familiar with these concepts:

1. Language Model (LM) Invoker

2. Embedding Model (EM) Invoker

3. LM Request Processor (LMRP)

Installation

Linux, macOS, or Windows WSL

# you can use a Conda environment
pip install --extra-index-url https://oauth2accesstoken:$(gcloud auth print-access-token)@glsdk.gdplabs.id/gen-ai-internal/simple/ "gllm-misc"

Windows Powershell

Windows Command Prompt

Quickstart

We will use the Aurelio semantic router, which wraps the semantic_router library from Aurelio Labs. You can compose it by following these simple steps:

1

Configure your routes

Define the available routes and their example queries:

import asyncio
from gllm_misc.router.aurelio_semantic_router.aurelio_semantic_router import AurelioSemanticRouter
from semantic_router.encoders import OpenAIEncoder


valid_routes = {"faq", "billing", "tech_support", "fallback"}
default_route = "fallback"  # must be in valid_routes

routes = {
    "faq": [
        "What are your business hours?",
        "Where can I find the user guide?",
        "How do I reset my password?",
    ],
    "billing": [
        "How do I update my payment method?",
        "Invoice not received",
        "Why was I charged twice?",
    ],
    "tech_support": [
        "App crashes on launch",
        "Connection timeout when uploading",
        "Error code 504 when syncing files",
    ],
    # "fallback" doesn't need samples; it's used when nothing matches
}

This configuration defines the available routes and example queries for each route.

2

Set up the encoder

Configure the OpenAI encoder for semantic similarity:

openai_api_key = "<YOUR_OPENAI_API_KEY>"
encoder = OpenAIEncoder("text-embedding-3-small", openai_api_key=openai_api_key)

The encoder converts text into embeddings for semantic similarity matching.

3

Create the router instance

Initialize the semantic router with your configuration:

router = AurelioSemanticRouter(
    default_route=default_route,
    valid_routes=valid_routes,
    encoder=encoder,
    routes=routes,
)

This creates a router that can classify queries into the appropriate routes.

4

Test the router

Test the router with sample queries:

user_input = "My credit card expired and billing failed."
route = asyncio.run(router.route(user_input))
print(f"Selected route: {route}")  # e.g., "billing"

This will classify the input query and return the most appropriate route.

Use Route Filtering

Optionally, you can restrict candidates with a route_filter (the filtered routes must include default_route)

filtered_route = asyncio.run(router.route(user_input, route_filter={"billing", "fallback"}))
print(f"Filtered route: {filtered_route}")

Route filtering allows you to limit which routes the router can select from.

Use an EM invoker as Encoder

You can use an EM Invoker as an Aurelio semantic router encoder.

from gllm_inference.em_invoker.openai_em_invoker import OpenAIEMInvoker
from gllm_misc.router.aurelio_semantic_router.aurelio_semantic_router import AurelioSemanticRouter
from gllm_misc.router.aurelio_semantic_router.encoders import EMInvokerEncoder


openai_api_key = "<YOUR_OPENAI_API_KEY>"
em_invoker = OpenAIEMInvoker("text-embedding-3-small", api_key=openai_api_key)
encoder = EMInvokerEncoder(em_invoker)

# Using the previous route examples

router = AurelioSemanticRouter(
    default_route=default_route,
    valid_routes=valid_routes,
    encoder=encoder,
    routes=routes,
)