# Evaluation Fundamentals

This page explains what **Evaluation (Evals)** is, what it is not, and the core components used in **gllm-evals** so teams can align on reuse and avoid common misunderstandings.

## **What is Evaluation**

**Evaluation** is a structured and repeatable way to **measure the quality of an AI system’s outputs** against **explicit criteria**, producing **machine-readable results** (scores/verdicts, plus optional explanations/evidence).

Evaluation answers questions like:

* Is the output **correct** (when ground truth exists)?
* Is it **complete**, **grounded**, and not hallucinating?
* Is retrieval **good enough** for a given query?
* Does the system behavior satisfy a rubric (e.g., clarity, coherence, bias)?

The key point: **evaluation measures** outputs; it does not create those outputs.

## **What Evaluation is Not**

Evaluation is commonly confused with the system being evaluated, or with the criteria being checked. The following are **not** “evals”:

### **1) The thing we want to evaluate is not the evaluator**

Example misconception:

* “We evaluate 5W1H and one-sidedness, therefore the whole pipeline is evals.”

Correct framing:

* 5W1H and one-sidedness are **evaluation criteria** (what we care about).
* The **evaluator** is the component that produces structured scores/verdicts for those criteria.
* Other parts of the system (crawler, ingestion, storage, orchestration, UI) are not evaluation.

### **2) Evaluation is not the AI system itself (it does not replace the Evaluand)**

In gllm-evals, the system being evaluated is called the **Evaluand**: any AI component or system that produces outputs, such as:

* QnA agents
* RAG pipelines
* summarizers
* classifiers
* retrievers
* reporting/analysis systems

**Evaluation does not replace the Evaluand.** The Evaluand generates outputs; evaluation measures their quality.

### **3) Evaluation is not ad-hoc “LLM analysis”**

If a workflow calls an LLM to “analyze something” but has no explicit criteria, no stable scoring format, and is not repeatable across versions/runs, that is closer to **ad-hoc analysis** than evaluation.

Evaluation requires:

* explicit criteria/rubric,
* a stable scoring format (scores/verdicts),
* repeatability (so results can be compared across versions).

## **Components of Evaluation**

Evaluation typically includes these components:

### **1) Dataset (what you test on)**

A **dataset** defines evaluation cases (inputs) and optionally ground truth (expected outputs, labels, references). It enables repeatable comparison across changes.

Typical fields depend on the Evaluand, for example:

* QnA: question, expected answer, reference docs
* Retrieval: query, relevant documents/chunks
* Summarization: source text, expected summary or rubric criteria

### **2) Evaluand (the system under test)**

The **Evaluand** is the component being measured. It is responsible for producing the output that will be evaluated (answer, summary, retrieved context, classification label, etc.).

Important: **evaluation measures the Evaluand’s outputs; it does not implement the Evaluand.**

### **3) Evaluator / Scorer (how you score)**

An **evaluator/scorer** consumes:

* dataset fields (inputs and optional ground truth), and
* the Evaluand outputs,\
  then produces structured results such as:
* overall score,
* per-dimension scores (e.g., completeness, groundedness),
* pass/fail verdict,
* optional explanations/evidence.

### **4) Metrics (the scoring signals)**

A **metric** is an individual scoring signal used by an evaluator (e.g., exact match, groundedness score, contextual precision/recall, rubric scores). Evaluators often combine multiple metrics into a final verdict.

### **5) Experiment Tracker (how you compare runs)**

An **experiment tracker** records evaluation results so you can compare performance across:

* model versions,
* prompts,
* retrieval configuration,
* logic changes,
* dataset revisions.

This makes evaluation useful as an engineering tool: you can detect regressions and quantify improvements.

## **Reference-based vs Reference-less Evaluation**

Evaluation can be performed with or without ground truth.

### **Reference-based evaluation (with ground truth)**

Use this when you have an expected output or labeled reference.\
Examples:

* QnA accuracy with expected answer
* Retrieval evaluation with known relevant documents/chunks
* Classification with gold labels

Typical outcomes:

* correctness score,
* precision/recall,
* pass/fail vs expected output.

### **Reference-less evaluation (without ground truth)**

Use this when ground truth is unavailable or too expensive to label.

Reference-less evaluation typically uses **rubric-based criteria** and can act as an **in-system critic** component:

* The evaluator judges the output against criteria like coherence, consistency, groundedness, safety, or bias.
* This can be used:
  * offline (to measure quality), and/or
  * inside a system loop (to critique an output and trigger revision or escalation).

Example reference-less criteria:

* “Does this summary contain contradictions?”
* “Is this answer grounded in the provided context?”
* “Is this article analysis one-sided? Provide evidence spans.”

## **Summary**

* **Evaluation** is a reusable measurement layer for output quality against explicit criteria.
* **The Evaluand** is the system being evaluated (any AI component that generates outputs).
* Evaluation **does not replace** the Evaluand; it measures the Evaluand’s outputs and behavior.
* Core components: **Dataset, Evaluand, Evaluator/Scorer, Metrics, Experiment Tracker**.
* Evaluation can be **reference-based** (with ground truth) or **reference-less**, where it can also serve as an **in-system critic**.
