Vectara, a platform for enterprise Retrieval-Augmented Generation (RAG) and AI-powered agents and assistants, today announced the launch of Open RAG Eval, its open-source RAG evaluation framework.
The framework, developed in conjunction with researchers from the University of Waterloo, allows enterprise users to evaluate response quality for each component
and configuration of their RAG systems in order to quickly and consistently optimize the accuracy and reliability of their AI agents and other tools.
Vectara Founder and CEO Amr Awadallah said, “AI implementations – especially for agentic RAG systems – are growing more complex by the day. Sophisticated workflows, mounting security and observability concerns along with looming regulations are driving organizations to deploy bespoke RAG systems on the fly in increasingly ad hoc ways. To avoid putting their entire AI strategies at risk, these organizations need a consistent, rigorous way to evaluate
performance and quality. By collaborating with Professor Jimmy Lin and his exceptional team at the University of Waterloo, Vectara is proactively tackling this challenge with our Open RAG Eval.”
Professor Jimmy Lin is the David R. Cheriton Chair in the School of Computer Science at the University of Waterloo. He and members of his team are pioneers in creating world-class benchmarks and datasets for information retrieval evaluation.
Professor Lin said, “AI agents and other systems are becoming increasingly central to how enterprises operate today and how they plan to grow in the future. In order to capitalize on the promise these technologies offer, organizations need robust evaluation methodologies that combine scientific rigor and practical utility in order to continually assess and optimize their RAG systems. My team and I have been thrilled to work with Vectara to bring our research findings to the enterprise in a way that will advance the accuracy and reliability of AI systems around the world.”
Open RAG Eval is designed to determine the accuracy and usefulness of the responses provided to user prompts, depending on the components and configuration of an enterprise RAG stack. The framework assesses response quality according to two major metric categories: retrieval metrics and generation metrics.
Users of Open RAG Eval can utilize this first iteration of the platform to help inform developers of these systems how a RAG pipeline performs along selected metrics. By inspecting these metric categories, an evaluator can compare otherwise ‘black-box’ systems on separate or aggregate scores.
A low relevance score, for example, may indicate that the user should upgrade or reconfigure the system’s retrieval pipeline, or that there is no relevant information in the dataset. Lower-than-expected generation scores, meanwhile, may mean that the system should use a stronger LLM – in cases where, for example, the generated response includes hallucinations – or that the user should update their RAG prompts.
The new framework is designed to seamlessly evaluate any RAG pipeline, including Vectara’s own GenAI platform or any other custom RAG solution.
Open RAG Eval helps AI teams solve such real-world deployment and configuration challenges as:
● Whether to use fixed token chunking or semantic chunking;
● Whether to use hybrid or vector search, and what value to use for lambda in hybrid
search deployments;
● Which LLM to use and how to optimize RAG prompts;
● Which threshold to use for hallucination detection and correction, and more.
Vectara’s decision to launch Open RAG Eval as an open-source, Apache 2.0-licensed tool reflects the company’s track record of success in establishing other industry standards in hallucination mitigation with its open-source Hughes Hallucination Evaluation Model (HHEM), which has been downloaded over 3.5 million times on Hugging Face.
As AI systems continue to grow rapidly in complexity – especially with agentic on the rise – and as RAG techniques continue to evolve, organizations will need open and extendable AI evaluation frameworks to help them make the right choices. This will allow organizations to also leverage their own data, add their own metrics, and measure their existing systems against emerging alternative options. Vectara’s open-s ource and extendable approach will help Open RAG Eval stay ahead of these dynamics by enabling ongoing contributions from the AI community while also ensuring that the implementation of each suggested and contributed evaluation metric is well understood and open for review and improvement.
Source: https://insideainews.com/
