Use of Ranking in 'RAGStringQueryEngine'

[mraguth]

I am using 'RAGStringQueryEngine' with a recursive retriever, leveraging Azure OpenAI and embeddings. My question is about the use of a 'reranker' in my query engine. Is it necessary to include a 'reranker'? If so, can I use the Azure embedding model?

Can I use the same Azure "text-embedding-ada-002" model for reranking? If yes, could you share some sample code? I'd like to integrate it into my query engine.

[dosubot[bot]]

Yes, it is necessary to include a 'reranker' in your 'RAGStringQueryEngine' with a recursive retriever to improve the accuracy and relevance of the retrieved results. You can use the Azure "text-embedding-ada-002" model for reranking.

Here is a sample code for integrating the Azure "text-embedding-ada-002" model for reranking:

from llama_index.core import get_response_synthesizer
from llama_index.core.postprocessor import AzureRerank
from llama_index.core.query_engine import RetrieverQueryEngine

# Set up the query engine with Azure "text-embedding-ada-002" model as reranker.
response_synthesizer = get_response_synthesizer(
    response_mode="tree_summarize",
)

# Initialize the Azure reranker
azure_rerank = AzureRerank(
    model="text-embedding-ada-002",
    api_key="YOUR_AZURE_API_KEY",
    endpoint="YOUR_AZURE_ENDPOINT",
    top_n=5
)

# Assuming `google_index` is your initialized GoogleIndex
retriever = google_index.as_retriever(similarity_top_k=5)
query_engine = RetrieverQueryEngine.from_args(
    retriever=retriever,
    response_synthesizer=response_synthesizer,
    node_postprocessors=[azure_rerank],
)

# Query for better result!
response = query_engine.query("Which program did this author attend?")
print(response.response)

Replace "YOUR_AZURE_API_KEY" and "YOUR_AZURE_ENDPOINT" with your actual Azure API key and endpoint. This setup will use the Azure "text-embedding-ada-002" model to rerank the retrieved nodes and provide more accurate results ^[1].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu. Thank you and I added node_postprocessors=[azure_embedding] in my code. However, I didn't see much difference or how I see the impact of that line of code? is it better or worst in my query engine? Any thoughts?

[dosubot[bot]]

To evaluate the impact of adding node_postprocessors=[azure_embedding] for reranking in your RAGStringQueryEngine and determine if it is making the results better or worse, you can use UpTrain, an open-source platform designed to evaluate and improve GenAI applications. UpTrain provides a series of checks to evaluate the quality of generated responses, retrieved context, and all interim steps. Here are the steps:

1. Set Up the Environment: Ensure you have the necessary packages installed and your environment configured.

2. Load Documents: Load the documents you want to query against.

3. Build the Index: Create an index from the loaded documents.

4. Define the Reranker: Set up the reranker with the desired model.

5. Query with Reranking: Execute a query using the query engine with the reranker and measure the time taken.

6. Query without Reranking: Execute the same query using the query engine without the reranker and measure the time taken.

7. Use UpTrain for Evaluation: Utilize UpTrain to perform evaluations on the responses. UpTrain provides checks such as Context Relevance, Factual Accuracy, Response Completeness, Context Reranking, and Context Conciseness.

Additionally, you can use the RetrieverEvaluator from LlamaIndex to evaluate the quality of the embeddings using metrics like Hit-Rate and MRR (Mean-Reciprocal-Rank). This can help you compare the quality of retrieved results from the ground-truth context.

Here is the code to perform these steps:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.core import Settings
from llama_index.postprocessor.flag_embedding_reranker import FlagEmbeddingReranker
from uptrain import UpTrainCallbackHandler, CallbackManager
from time import time

# Load documents
documents = SimpleDirectoryReader("./data/paul_graham").load_data()

# Set up LLM and embedding model
Settings.llm = OpenAI(model="gpt-3.5-turbo")
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")

# Build index
index = VectorStoreIndex.from_documents(documents=documents)

# Define the reranker
rerank = FlagEmbeddingReranker(model="BAAI/bge-reranker-large", top_n=5)

# Set up UpTrain callback handler
callback_handler = UpTrainCallbackHandler(
    key_type="openai",
    api_key=os.environ["OPENAI_API_KEY"],
    project_name_prefix="llama",
)
Settings.callback_manager = CallbackManager([callback_handler])

# Query with reranking
query_engine_with_rerank = index.as_query_engine(similarity_top_k=10, node_postprocessors=[rerank])
now = time()
response_with_rerank = query_engine_with_rerank.query("Which grad schools did the author apply for and why?")
time_with_rerank = round(time() - now, 2)

# Query without reranking
query_engine_without_rerank = index.as_query_engine(similarity_top_k=10)
now = time()
response_without_rerank = query_engine_without_rerank.query("Which grad schools did the author apply for and why?")
time_without_rerank = round(time() - now, 2)

# Print results
print(f"Elapsed with rerank: {time_with_rerank}s")
print(response_with_rerank)
print(response_with_rerank.get_formatted_sources(length=200))

print(f"Elapsed without rerank: {time_without_rerank}s")
print(response_without_rerank)
print(response_without_rerank.get_formatted_sources(length=200))

# Use UpTrain to evaluate the responses
# UpTrain will automatically capture the query, context, and response and run evaluations

To further evaluate the impact, you can use the RetrieverEvaluator from LlamaIndex to measure metrics like Hit-Rate and MRR for different embedding types. This will help you understand the quality of the retrieved results and make an informed decision about the effectiveness of the reranking.

from llama_index.core.evaluation import RetrieverEvaluator, generate_question_context_pairs, EmbeddingQAFinetuneDataset

# Generate synthetic dataset for evaluation
qa_dataset = generate_question_context_pairs(nodes, llm=llm, num_questions_per_chunk=2)

# Define RetrieverEvaluator for different embedding types
metrics = ["mrr", "hit_rate"]
retriever_evaluator = RetrieverEvaluator.from_metric_names(metrics, retriever=query_engine_with_rerank)

# Evaluate on the entire dataset
eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)

# Display results
import pandas as pd

def display_results(name, eval_results):
    metric_dicts = []
    for eval_result in eval_results:
        metric_dict = eval_result.metric_vals_dict
        metric_dicts.append(metric_dict)

    full_df = pd.DataFrame(metric_dicts)
    hit_rate = full_df["hit_rate"].mean()
    mrr = full_df["mrr"].mean()
    columns = {"Embedding Type": [name], "hit_rate": [hit_rate], "mrr": [mrr]}
    metric_df = pd.DataFrame(columns)
    return metric_df

metrics_results = display_results("rerank", eval_results)
print(metrics_results)

By comparing the elapsed times, UpTrain's evaluations, and the metrics from RetrieverEvaluator, you can determine if adding node_postprocessors=[azure_embedding] for reranking improves the results ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[mraguth]

@as you know I am using Azure Embeddings. Can you accommodate that to Azure embeddings, please? Below is my snippet.

# Initialize the Azure embedding model
azure_embedding = AzureOpenAIEmbedding(
    model="text-embedding-ada-002",
    api_key="YOUR_AZURE_API_KEY",
    azure_endpoint="YOUR_AZURE_ENDPOINT",
    api_version="2023-07-01-preview"
)

[dosubot[bot]]

To accommodate the evaluation process for Azure embeddings, you can modify the provided code snippet to use the AzureOpenAIEmbedding class. Here is the updated code snippet:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.embeddings.azure_openai import AzureOpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.core import Settings
from llama_index.postprocessor.flag_embedding_reranker import FlagEmbeddingReranker
from uptrain import UpTrainCallbackHandler, CallbackManager
from time import time
import os

# Load documents
documents = SimpleDirectoryReader("./data/paul_graham").load_data()

# Set up LLM and embedding model
Settings.llm = OpenAI(model="gpt-3.5-turbo")
Settings.embed_model = AzureOpenAIEmbedding(
    model="text-embedding-ada-002",
    api_key="YOUR_AZURE_API_KEY",
    azure_endpoint="YOUR_AZURE_ENDPOINT",
    api_version="2023-07-01-preview"
)

# Build index
index = VectorStoreIndex.from_documents(documents=documents)

# Define the reranker
rerank = FlagEmbeddingReranker(model="text-embedding-ada-002", top_n=5)

# Set up UpTrain callback handler
callback_handler = UpTrainCallbackHandler(
    key_type="openai",
    api_key=os.environ["OPENAI_API_KEY"],
    project_name_prefix="llama",
)
Settings.callback_manager = CallbackManager([callback_handler])

# Query with reranking
query_engine_with_rerank = index.as_query_engine(similarity_top_k=10, node_postprocessors=[rerank])
now = time()
response_with_rerank = query_engine_with_rerank.query("Which grad schools did the author apply for and why?")
time_with_rerank = round(time() - now, 2)

# Query without reranking
query_engine_without_rerank = index.as_query_engine(similarity_top_k=10)
now = time()
response_without_rerank = query_engine_without_rerank.query("Which grad schools did the author apply for and why?")
time_without_rerank = round(time() - now, 2)

# Print results
print(f"Elapsed with rerank: {time_with_rerank}s")
print(response_with_rerank)
print(response_with_rerank.get_formatted_sources(length=200))

print(f"Elapsed without rerank: {time_without_rerank}s")
print(response_without_rerank)
print(response_without_rerank.get_formatted_sources(length=200))

# Use UpTrain to evaluate the responses
# UpTrain will automatically capture the query, context, and response and run evaluations

Replace "YOUR_AZURE_API_KEY" and "YOUR_AZURE_ENDPOINT" with your actual Azure API key and endpoint. This setup will use the Azure "text-embedding-ada-002" model for reranking and evaluate the impact using UpTrain and other metrics ^[1][2].

To continue talking to Dosu, mention @dosu.