🔢Metric

Metrics are the core evaluation components in the gllm-evals framework. They define specific ways to measure and assess the performance of language models for generation, retrieval systems, and agent behaviors.

Metrics work in conjunction with evaluators to provide comprehensive evaluation capabilities. Evaluators can run multiple metrics in parallel or sequentially, combining their results into a comprehensive evaluation report.

Example Usage

import asyncio
import os

from gllm_evals import load_simple_qa_dataset
from gllm_evals.metrics import GEvalCompletenessMetric

async def main():
    data = load_simple_qa_dataset()
    metric = GEvalCompletenessMetric(model_credentials=os.getenv("GOOGLE_API_KEY"))
    result = await metric.evaluate(data.dataset[0])
    print(result)

if __name__ == "__main__":
    asyncio.run(main())

---

Available Metrics

Below are several existing metrics example. To view the full metrics list, see the Metrics directory.

1. Generation Evaluation Metrics

   1. GEvalCompletenessMetric

   2. GEvalGroundednessMetric

   3. GEvalRedundancyMetric

   4. GEvalLanguageConsistencyMetric

   5. GEvalRefusalAlignmentMetric

   6. CompletenessMetric

   7. GroundednessMetric

   8. RedundancyMetric

   9. LanguageConsistencyMetric

   10. RefusalAlignmentMetric

   11. RagasFactualCorrectness

   12. DeepEvalAnswerRelevancyMetric

   13. DeepEvalFaithfulnessMetric

   14. LangChainConcisenessMetric

   15. LangChainCorrectnessMetric

   16. LangChainGroundednessMetric

   17. LangChainHallucinationMetric

   18. LangChainHelpfulnessMetric

2. Retrieval Evaluation Metrics

   1. PyTrecMetric

   2. TopKAccuracy

   3. RagasContextPrecisionWithoutReference

   4. RagasContextRecall

   5. DeepEvalContextualPrecisionMetric

   6. DeepEvalContextualRecallMetric

   7. DeepEvalContextualRelevancyMetric

3. Agent Evaluation Metrics

   1. LangChainAgentTrajectoryAccuracyMetric

   2. DeepEvalToolCorrectnessMetric