🚀Getting Started

Introduction

This tutorial will guide you step-by-step on how to install the GenAI Evaluator SDK and run your first evaluation.

Prerequisites

Before installing, make sure you have:

1. Python 3.11+

2. Pip

3. OpenAI API Key

4. gcloud CLI - required because gllm-evals is a private library hosted in a private Google Cloud repository

After installing, please run

gcloud auth login

to authorize gcloud to access the Cloud Platform with Google user credentials.

Our internal gllm-evals package is hosted in a secure Google Cloud Artifact Registry. You need to authenticate via gcloud CLI to access and download the package during installation.

Installation

Linux or Windows WSL

Run the following command to install

pip install --extra-index-url "https://oauth2accesstoken:$(gcloud auth print-access-token)@glsdk.gdplabs.id/gen-ai-internal/simple/" "gllm-evals[deepeval,langchain,ragas]"

Windows Powershell

Windows Command Prompt

Environment Setup

Set a valid language model credential as an environment variable.

• In this example, let's use an OpenAI API key.

Get an OpenAI API key from OpenAI Console.

Linux, macOS, or Windows WSL

export OPENAI_API_KEY="sk-..."

Windows Powershell

Windows Command Prompt

Running Your First Evaluation

In this tutorial, we will evaluate RAG pipeline output.

1

Create a script called eval.py

import asyncio
import os
from gllm_evals.evaluator.geval_generation_evaluator import GEvalGenerationEvaluator
from gllm_evals.types import RAGData

async def main():
    evaluator = GEvalGenerationEvaluator(
        model="openai/gpt-4.1",
        model_credentials=os.getenv("OPENAI_API_KEY")
    )

    data = RAGData(
        query="What is the capital of France?",
        expected_response="Paris",
        generated_response="New York",
        retrieved_context="Paris is the capital of France.",
    )

    result = await evaluator.evaluate(data)
    print(result)

if __name__ == "__main__":
    asyncio.run(main())

2

Run the script

python eval.py

3

The evaluator will generate a response for the given input, e.g.:

{
    "generation": {
        "global_explanation": "The following metrics failed to meet expectations:\n1. Completeness is 1 (should be 3)\n2. Groundedness is 1 (should be 3)",
        "relevancy_rating": "bad",
        "score": 0,
        "possible_issues": ["Retrieval Issue", "Generation Issue"],
        "completeness": {
            "score": 1,
            "explanation": "The answer incorrectly identifies New York instead of Paris as the capital of France, which is a critical factual contradiction. No key facts are correctly covered; the response directly contradicts the expected output.",
        },
        "groundedness": {
            "score": 1,
            "explanation": "The response 'New York' is a direct contradiction of the retrieval context, which states that Paris is the capital of France. None of the information in the response is supported by or consistent with the provided context.",
        },
        "redundancy": {
            "score": 1,
            "explanation": "The response only contains a single answer with no repetition, restatement, or redundant elaboration. No ideas or phrases are repeated, and the response is concise and direct, matching the rubric for minimal redundancy.",
        },
        "language_consistency": {
            "score": 1,
            "explanation": "The instruction is in English and the actual output ('New York') is also in English. Despite being factually incorrect, the response maintains language consistency as per the rubric requirements.",
            "success": True,
        },
        "refusal_alignment": {
            "score": 1,
            "explanation": "is_refusal was detected from the expected response, which directly answers the query ('Paris'), so expected is not refusal. The actual output ('New York') also directly provides an answer, despite being incorrect, and does not refuse. Both are not refusal, so alignment is correct.",
            "success": True,
        },
    }
}

Congratulations! You have successfully run your first evaluation

Recommendation

If you want to run an end-to-end evaluation, use the evaluate() convenience function instead of the step-by-step commands above.

It will automatically handle experiment tracking (via the Experiment Tracker) and integrates results into your existing Dataset, so you don’t have to wire these pieces together manually.

Next Steps

You're now ready to start using our evaluators. We offer several prebuilt evaluators to get you started:

1. GEvalGenerationEvaluator

2. AgentEvaluator

3. QueryTransformerEvaluator

4. ClassicalRetrievalEvaluator

Looking for something else? Build your own custom evaluator here.

^{*All fields are optional and can be adjusted depending on the chosen metric.}