A Hands-On Guide to Testing Agents with RAGAs and G-Eval

In this article, you will learn how to evaluate large language model applications using RAGAs and G-Eval-based frameworks in a practical, hands-on workflow.

Topics we will cover include:

How to use RAGAs to measure faithfulness and answer relevancy in retrieval-augmented systems.
How to structure evaluation datasets and integrate them into a testing pipeline.
How to apply G-Eval via DeepEval to assess qualitative aspects like coherence.

Let’s get started.

A Hands-On Guide to Testing Agents with RAGAs and G-Eval
Image by Editor

Introduction

RAGAs (Retrieval-Augmented Generation Assessment) is an open-source evaluation framework that replaces subjective “vibe checks” with a systematic, LLM-driven “judge” to quantify the quality of RAG pipelines. It assesses a triad of desirable RAG properties, including contextual accuracy and answer relevance. RAGAs has also evolved to support not only RAG architectures but also agent-based applications, where methodologies like G-Eval play a role in defining custom, interpretable evaluation criteria.

This article presents a hands-on guide to understanding how to test large language model and agent-based applications using both RAGAs and frameworks based on G-Eval. Concretely, we will leverage DeepEval, which integrates multiple evaluation metrics into a unified testing sandbox.

If you are unfamiliar with evaluation frameworks like RAGAs, consider reviewing this related article first.

Step-by-Step Guide

This example is designed to work both in a standalone Python IDE and in a Google Colab notebook. You may need to pip install some libraries along the way to resolve potential ModuleNotFoundError issues, which occur when attempting to import modules that are not installed in your environment.

We begin by defining a function that takes a user query as input and interacts with an LLM API (such as OpenAI) to generate a response. This is a simplified agent that encapsulates a basic input-response workflow.

import openai def simple_agent(query): # NOTE: this is a ‘mock’ agent loop # In a real scenario, you would use a system prompt to define tool usage prompt = f”You are a helpful assistant. Answer the user query: {query}” # Example using OpenAI (this can be swapped for Gemini or another provider) response = openai.chat.completions.create( model=”gpt-3.5-turbo”, messages=[{“role”: “user”, “content”: prompt}] ) return response.choices[0].message.content

import openai

def simple_agent(query):

# NOTE: this is a ‘mock’ agent loop

# In a real scenario, you would use a system prompt to define tool usage

prompt = f“You are a helpful assistant. Answer the user query: {query}”

# Example using OpenAI (this can be swapped for Gemini or another provider)

response = openai.chat.completions.create(

model=“gpt-3.5-turbo”,

messages=[{“role”: “user”, “content”: prompt}]

)

return response.choices[0].message.content

In a more realistic production setting, the agent defined above would include additional capabilities such as reasoning, planning, and tool execution. However, since the focus here is on evaluation, we intentionally keep the implementation simple.

Next, we introduce RAGAs. The following code demonstrates how to evaluate a question-answering scenario using the faithfulness metric, which measures how well the generated answer aligns with the provided context.

from ragas import evaluate from ragas.metrics import faithfulness # Defining a simple testing dataset for a question-answering scenario data = { “question”: [“What is the capital of Japan?”], “answer”: [“Tokyo is the capital.”], “contexts”: [[“Japan is a country in Asia. Its capital is Tokyo.”]] } # Running RAGAs evaluation result = evaluate(data, metrics=[faithfulness])

from ragas import evaluate

from ragas.metrics import faithfulness

# Defining a simple testing dataset for a question-answering scenario

data = {

“question”: [“What is the capital of Japan?”],

“answer”: [“Tokyo is the capital.”],

“contexts”: [[“Japan is a country in Asia. Its capital is Tokyo.”]]

}

# Running RAGAs evaluation

result = evaluate(data, metrics=[faithfulness])

Note that you may need sufficient API quota (e.g., OpenAI or Gemini) to run these examples, which typically requires a paid account.

Below is a more elaborate example that incorporates an additional metric for answer relevancy and uses a structured dataset.

test_cases = [ { “question”: “How do I reset my password?”, “answer”: “Go to settings and click ‘forgot password’. An email will be sent.”, “contexts”: [“Users can reset passwords via the Settings > Security menu.”], “ground_truth”: “Navigate to Settings, then Security, and select Forgot Password.” } ]

test_cases = [

{

“question”: “How do I reset my password?”,

“answer”: “Go to settings and click ‘forgot password’. An email will be sent.”,

“contexts”: [“Users can reset passwords via the Settings > Security menu.”],

“ground_truth”: “Navigate to Settings, then Security, and select Forgot Password.”

}

]

Ensure that your API key is configured before proceeding. First, we demonstrate evaluation without wrapping the logic in an agent:

import os from ragas import evaluate from ragas.metrics import faithfulness, answer_relevancy from datasets import Dataset # IMPORTANT: Replace “YOUR_API_KEY” with your actual API key os.environ[“OPENAI_API_KEY”] = “YOUR_API_KEY” # Convert list to Hugging Face Dataset (required by RAGAs) dataset = Dataset.from_list(test_cases) # Run evaluation ragas_results = evaluate(dataset, metrics=[faithfulness, answer_relevancy]) print(f”RAGAs Faithfulness Score: {ragas_results[‘faithfulness’]}”)

import os

from ragas import evaluate

from ragas.metrics import faithfulness, answer_relevancy

from datasets import Dataset

# IMPORTANT: Replace “YOUR_API_KEY” with your actual API key

os.environ[“OPENAI_API_KEY”] = “YOUR_API_KEY”

# Convert list to Hugging Face Dataset (required by RAGAs)

dataset = Dataset.from_list(test_cases)

# Run evaluation

ragas_results = evaluate(dataset, metrics=[faithfulness, answer_relevancy])

print(f“RAGAs Faithfulness Score: {ragas_results[‘faithfulness’]}”)

To simulate an agent-based workflow, we can encapsulate the evaluation logic into a reusable function:

import os from ragas import evaluate from ragas.metrics import faithfulness, answer_relevancy from datasets import Dataset def evaluate_ragas_agent(test_cases, openai_api_key=”YOUR_API_KEY”): “””Simulates a simple AI agent that performs RAGAs evaluation.””” os.environ[“OPENAI_API_KEY”] = openai_api_key # Convert test cases into a Dataset object dataset = Dataset.from_list(test_cases) # Run evaluation ragas_results = evaluate(dataset, metrics=[faithfulness, answer_relevancy]) return ragas_results

import os

from ragas import evaluate

from ragas.metrics import faithfulness, answer_relevancy

from datasets import Dataset

def evaluate_ragas_agent(test_cases, openai_api_key=“YOUR_API_KEY”):

“”“Simulates a simple AI agent that performs RAGAs evaluation.”“”

os.environ[“OPENAI_API_KEY”] = openai_api_key

# Convert test cases into a Dataset object

dataset = Dataset.from_list(test_cases)

# Run evaluation

ragas_results = evaluate(dataset, metrics=[faithfulness, answer_relevancy])

return ragas_results

The Hugging Face Dataset object is designed to efficiently represent structured data for large language model evaluation and inference.

The following code demonstrates how to call the evaluation function:

my_openai_key = “YOUR_API_KEY” # Replace with your actual API key if ‘test_cases’ in globals(): evaluation_output = evaluate_ragas_agent(test_cases, openai_api_key=my_openai_key) print(“RAGAs Evaluation Results:”) print(evaluation_output) else: print(“Please define the ‘test_cases’ variable first. Example:”) print(“test_cases = [{ ‘question’: ‘…’, ‘answer’: ‘…’, ‘contexts’: […], ‘ground_truth’: ‘…’ }]”)

my_openai_key = “YOUR_API_KEY” # Replace with your actual API key

if ‘test_cases’ in globals():

evaluation_output = evaluate_ragas_agent(test_cases, openai_api_key=my_openai_key)

print(“RAGAs Evaluation Results:”)

print(evaluation_output)

else:

print(“Please define the ‘test_cases’ variable first. Example:”)

print(“test_cases = [{ ‘question’: ‘…’, ‘answer’: ‘…’, ‘contexts’: […], ‘ground_truth’: ‘…’ }]”)

We now introduce DeepEval, which acts as a qualitative evaluation layer using a reasoning-and-scoring approach. This is particularly useful for assessing attributes such as coherence, clarity, and professionalism.

from deepeval.metrics import GEval from deepeval.test_case import LLMTestCase, LLMTestCaseParams # STEP 1: Define a custom evaluation metric coherence_metric = GEval( name=”Coherence”, criteria=”Determine if the answer is easy to follow and logically structured.”, evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT], threshold=0.7 # Pass/fail threshold ) # STEP 2: Create a test case case = LLMTestCase( input=test_cases[0][“question”], actual_output=test_cases[0][“answer”] ) # STEP 3: Run evaluation coherence_metric.measure(case) print(f”G-Eval Score: {coherence_metric.score}”) print(f”Reasoning: {coherence_metric.reason}”)

from deepeval.metrics import GEval

from deepeval.test_case import LLMTestCase, LLMTestCaseParams

# STEP 1: Define a custom evaluation metric

coherence_metric = GEval(

name=“Coherence”,

criteria=“Determine if the answer is easy to follow and logically structured.”,

evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],

threshold=0.7 # Pass/fail threshold

)

# STEP 2: Create a test case

case = LLMTestCase(

input=test_cases[0][“question”],

actual_output=test_cases[0][“answer”]

)

# STEP 3: Run evaluation

coherence_metric.measure(case)

print(f“G-Eval Score: {coherence_metric.score}”)

print(f“Reasoning: {coherence_metric.reason}”)

A quick recap of the key steps:

Define a custom metric using natural language criteria and a threshold between 0 and 1.
Create an LLMTestCase using your test data.
Execute evaluation using the measure method.

Summary

This article demonstrated how to evaluate large language model and retrieval-augmented applications using RAGAs and G-Eval-based frameworks. By combining structured metrics (faithfulness and relevancy) with qualitative evaluation (coherence), you can build a more comprehensive and reliable evaluation pipeline for modern AI systems.

Source link