Retrieval Augmented Generation(RAG) is a technique that enhances the capabilities of LLMs by combining information retrieval with text generation. Instead of relying on pre-traned knowledge, RAG fetch relevant data from external sources and use it to generate more accurate responses.
streamlit python-dotenv pypdf2 google-generativeai
langchain # core framework langchain-huggingface # connect huggingface models faiss-cpu # Fast vector database to store embedded data langchain-communitiy # extra integration langchain-text-splitters # to split large text into smaller chunks sentence-transformers # pre-trained embedding models to convert text into vectors langchain-core # document,chain etc
Text -> split text -> convert vector -> store in database -> search similar content -> send to LLM -> Get answer for questions
'all-Mini-L6-v2' -> Simple hugging face embedding model which splits the text and convert the text into vectors