LangChain-SynData-RAG-Eval

Name: LangChain-SynData-RAG-Eval
Author: mddunlap924

40stars8forks

How to install LangChain-SynData-RAG-Eval?

mddunlap924/langchain-syndata-rag-eval

$git clone https://github.com/mddunlap924/LangChain-SynData-RAG-Eval

Installs into the current project.

Summary

The synthetic data generated is a query and answer for a given context. An example of a synthetically generated context-query-answer is shown below: ``` Provided Context (usually split from documents / text sources): Pure TalkUSA is an American mobile virtual network operator he

Files · 1

View on GitHub

README.md

1	<h1 align="center">
2	<!-- <a href="https://github.com/mddunlap924/VHSpy">
3	<img src="https://raw.githubusercontent.com/mddunlap924/PyVHS/main/doc/imgs/pyvhs.png" width="512" height="256" alt="pyvhs">
4	</a> -->
5	Synthetic Data Generation using LangChain for IR and RAG Evaluation
6	</h1>
7
8	<p align="center">This repository demonstrates LangChain, Llama2-Chat, and zero- and few-shot prompt engineering to enable synthetic data generation for Information Retrieval (IR) and Retrieval Augmented Generation (RAG) evaluation.
9	</p>
10
11	<p align="center">
12	<a href="#introduction">Introduction</a>  •
13	<a href="#highlights">Highlights</a>  •
14	<a href="#example-notebooks">Example Notebooks</a>  •
15	<a href="#background">Background</a>  •
16	<a href="#metrics">Metrics</a>  •
17	<a href="#benefits">Benefits</a>  •
18	<a href="#prompt-templates">Prompt Templates</a>  •
19	<a href="#issues">Issues</a>  •
20	<a href="#todos">TODOs</a>
21	</p>
22
23	<p align="center">
24	<a target="_blank" href="https://www.linkedin.com/in/myles-dunlap/"><img height="20" src="https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white" />
25	</a>
26	<a target="_blank" href="https://www.kaggle.com/dunlap0924"><img height="20" src="https://img.shields.io/badge/-Kaggle-5DB0DB?style=flat&logo=Kaggle&logoColor=white&" />
27	</a>
28	<a target="_blank" href="https://scholar.google.com/citations?user=ZpHuEy4AAAAJ&hl=en"><img height="20" src="https://img.shields.io/badge/-Google_Scholar-676767?style=flat&logo=google-scholar&logoColor=white&" />
29	</a>
30	</p>
31
32	# Introduction
33	Large language models (LLMs) have transformed Information Retrieval (IR) and search by comprehending complex queries. This repository showcases concepts and packages that can be used to generate sophisticated synthetic datasets for IR and Retrieval Augmented Generation (RAG) evaluation.
34
35	The synthetic data generated is a query and answer for a given context. An example of a synthetically generated context-query-answer is shown below:
36	```
37	Provided Context (usually split from documents / text sources):
38	Pure TalkUSA is an American mobile virtual network operator headquartered in Covington, Georgia, United States.
39	It is most notable for an industry-first offering of rollover data in their data add-on packages, which has since been discontinued.
40	Pure TalkUSA is a subsidiary of Telrite Corporation. Bring Your Own Phone!
41
42	Synthetically Generated Query:
43	What was the outstanding service offered by Pure TalkUSA?
44
45	Synthetically Generated Answer:
46	The outstanding service from Pure TalkUSA was its industry-first offering of rollover data.
47	```
48
49	When building an IR or RAG system, a dataset of context, queries, and answers is vital for evaluating the system's performance. Human-annotated datasets offer excellent ground truths but can be expensive and challenging to obtain; therefore, synthetic datasets generated using LLMs is an attractive solution and supplement.
50
51	By employing LLM prompt engineering, a diverse range of synthetic queries and answers can be generated to form a robust validation dataset. This repository showcases a process to generate synthetic data while emphasizing [zero- and few-shot prompting](https://openreview.net/pdf?id=gmL46YMpu2J#:~:text=Importantly%2C%20the%20few%2Dshot%20examples,highly%20efficient%20dual%20encoder%20models.) for creating highly customizable synthetic datasets. Figure 1 outlines the synthetic dataset generation process demonstrated in this repository.
52
53	<p align="center">
54	<img src="./imgs/syn-gen-process.png">
55	<br>
56	Figure 1: Synthetic Data Generation for IR and RAG Evaluation
57	</p>
58
59	NOTE: Refer to the [Background](#background) and [Metrics](#metrics) sections for a deeper dive on IR, RAG, and how to evaluate these systems.
60
61
62	# Highlights
63
64	A few of the key highlights in repository are:
65	- *Local LLM models on consumer grade hardwa

People who liked this also viewed

6 picks

Type

LangChain-SynData-RAG-Eval

40stars8forks

View on GitHub

How to install LangChain-SynData-RAG-Eval?

mddunlap924/langchain-syndata-rag-eval

$git clone https://github.com/mddunlap924/LangChain-SynData-RAG-Eval

Installs into the current project.

Summary

Files · 1

View on GitHub

README.md

1	<h1 align="center">
2	<!-- <a href="https://github.com/mddunlap924/VHSpy">
3	<img src="https://raw.githubusercontent.com/mddunlap924/PyVHS/main/doc/imgs/pyvhs.png" width="512" height="256" alt="pyvhs">
4	</a> -->
5	Synthetic Data Generation using LangChain for IR and RAG Evaluation
6	</h1>
7
8	<p align="center">This repository demonstrates LangChain, Llama2-Chat, and zero- and few-shot prompt engineering to enable synthetic data generation for Information Retrieval (IR) and Retrieval Augmented Generation (RAG) evaluation.
9	</p>
10
11	<p align="center">
12	<a href="#introduction">Introduction</a>  •
13	<a href="#highlights">Highlights</a>  •
14	<a href="#example-notebooks">Example Notebooks</a>  •
15	<a href="#background">Background</a>  •
16	<a href="#metrics">Metrics</a>  •
17	<a href="#benefits">Benefits</a>  •
18	<a href="#prompt-templates">Prompt Templates</a>  •
19	<a href="#issues">Issues</a>  •
20	<a href="#todos">TODOs</a>
21	</p>
22
23	<p align="center">
24	<a target="_blank" href="https://www.linkedin.com/in/myles-dunlap/"><img height="20" src="https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white" />
25	</a>
26	<a target="_blank" href="https://www.kaggle.com/dunlap0924"><img height="20" src="https://img.shields.io/badge/-Kaggle-5DB0DB?style=flat&logo=Kaggle&logoColor=white&" />
27	</a>
28	<a target="_blank" href="https://scholar.google.com/citations?user=ZpHuEy4AAAAJ&hl=en"><img height="20" src="https://img.shields.io/badge/-Google_Scholar-676767?style=flat&logo=google-scholar&logoColor=white&" />
29	</a>
30	</p>
31
32	# Introduction
33	Large language models (LLMs) have transformed Information Retrieval (IR) and search by comprehending complex queries. This repository showcases concepts and packages that can be used to generate sophisticated synthetic datasets for IR and Retrieval Augmented Generation (RAG) evaluation.
34
35	The synthetic data generated is a query and answer for a given context. An example of a synthetically generated context-query-answer is shown below:
36	```
37	Provided Context (usually split from documents / text sources):
38	Pure TalkUSA is an American mobile virtual network operator headquartered in Covington, Georgia, United States.
39	It is most notable for an industry-first offering of rollover data in their data add-on packages, which has since been discontinued.
40	Pure TalkUSA is a subsidiary of Telrite Corporation. Bring Your Own Phone!
41
42	Synthetically Generated Query:
43	What was the outstanding service offered by Pure TalkUSA?
44
45	Synthetically Generated Answer:
46	The outstanding service from Pure TalkUSA was its industry-first offering of rollover data.
47	```
48
49	When building an IR or RAG system, a dataset of context, queries, and answers is vital for evaluating the system's performance. Human-annotated datasets offer excellent ground truths but can be expensive and challenging to obtain; therefore, synthetic datasets generated using LLMs is an attractive solution and supplement.
50
51	By employing LLM prompt engineering, a diverse range of synthetic queries and answers can be generated to form a robust validation dataset. This repository showcases a process to generate synthetic data while emphasizing [zero- and few-shot prompting](https://openreview.net/pdf?id=gmL46YMpu2J#:~:text=Importantly%2C%20the%20few%2Dshot%20examples,highly%20efficient%20dual%20encoder%20models.) for creating highly customizable synthetic datasets. Figure 1 outlines the synthetic dataset generation process demonstrated in this repository.
52
53	<p align="center">
54	<img src="./imgs/syn-gen-process.png">
55	<br>
56	Figure 1: Synthetic Data Generation for IR and RAG Evaluation
57	</p>
58
59	NOTE: Refer to the [Background](#background) and [Metrics](#metrics) sections for a deeper dive on IR, RAG, and how to evaluate these systems.
60
61
62	# Highlights
63
64	A few of the key highlights in repository are:
65	- *Local LLM models on consumer grade hardwa

People who liked this also viewed

6 picks

Type