Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing Large Language Models (LLMs) with external knowledge [4]. At the heart of any effective RAG system lies well-crafted prompts that guide the retrieval and generation processes. As RAG systems move from development to production, managing these prompts becomes increasingly complex.
Prompt engineering for RAG systems presents unique challenges:
- Context-sensitivity: RAG prompts must effectively incorporate retrieved information
- Multi-step processes: Many RAG systems involve multiple prompts for different stages (query analysis, retrieval, generation)
- Iterative refinement: Prompts require continuous optimization based on performance metrics
- Team collaboration: Multiple stakeholders often contribute to prompt development
- Environment management: Different prompts may be needed for development, testing, and production
LangSmith, developed by LangChain, offers robust capabilities for prompt versioning and management that address these challenges, enabling more systematic development and deployment of RAG systems.
LangSmith's prompt management system is built around several key concepts [1]:
- Prompts: Templates that can contain variables and instructions for LLMs
- Commits: Saved versions of prompts that create an auditable history
- Tags: Human-readable labels (like "dev" or "production") that can be assigned to specific commits
- Playground: An environment for testing and iterating on prompts
According to the official documentation, "Every saved update to a prompt creates a new commit. You can view previous commits, making it easy to review earlier prompt versions or revert to a previous state if needed" [3].
LangSmith supports both major types of prompts used in modern LLM applications [1]:
- Chat Prompts: A list of messages with roles (system, user, assistant), which is the preferred format for most modern LLMs
- Completion Prompts: Single-string prompts, primarily used for legacy models
For RAG systems, chat prompts are typically preferred as they provide more flexibility in structuring the context and instructions [4].
LangSmith supports two main formatting styles for prompt variables [1]:
- F-string format: Using
{variable_name}syntax - Mustache format: Using
{{variable_name}}syntax, which offers more flexibility for conditional variables, loops, and nested keys
To get started with LangSmith, first install the required packages:
# Install LangSmith and related packages
pip install -U langsmith langchain langchainhubNext, set up your environment variables:
NOTE: the LANGCHAIN_ environment varialble previously used for LangSmith appear to be deprecated in favor of similar LANGSMITH_ ones (there are some slight variances in namming, etc.)
import os
# LangSmith API configuration
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "your_langsmith_api_key"
os.environ["LANGCHAIN_ENDPOINT"] = "https://api.smith.langchain.com"
# For accessing LangChain Hub (optional)
os.environ["LANGCHAIN_HUB_API_URL"] = "https://api.hub.langchain.com"
os.environ["LANGCHAIN_HUB_API_KEY"] = "your_hub_api_key"Initialize a LangSmith client to interact with the API:
from langsmith import Client
client = Client()Let's examine a typical RAG prompt structure:
from langchain.prompts import ChatPromptTemplate
# Basic RAG prompt
rag_prompt = ChatPromptTemplate.from_template("""
Given a provided context and question, you must answer the question based only on context.
If you cannot answer the question based on the context - you must say "I don't know".
Context: {context}
Question: {question}
""")The key components of this prompt include:
- Clear instructions on how to use the provided context
- Guidelines for handling cases where the context doesn't contain the answer
- Variables for context (
{context}) and the user's question ({question})
To create a versioned prompt in LangSmith:
# Push the prompt to LangSmith
original_prompt_url = client.push_prompt(
"standard-rag-prompt",
object=rag_prompt,
description="Basic RAG prompt that answers based only on context"
)
print(f"Prompt available at: {original_prompt_url}")When you want to create a new version, use the same method with the same name but a modified prompt:
# Enhanced "dope" RAG prompt
dope_rag_prompt = ChatPromptTemplate.from_template("""
Given a provided context and question, you must answer the question based only on context.
If you cannot answer the question based on the context - you must say "I don't know".
You must answer the questions in a dope way, be cool!
Context: {context}
Question: {question}
""")
# Push the updated prompt - creates a new version
dope_prompt_url = client.push_prompt(
"standard-rag-prompt", # Using the same name creates a new version
object=dope_rag_prompt,
description="RAG prompt with instructions to answer in a 'dope' style"
)Tags provide human-readable labels for specific prompt versions, making them easier to reference:
NOTE: this didn't work when I ran it on 4/21/2025, but something similar to this tag versioning approach may still work
# Tag the dope prompt for development
dev_tag_url = client.add_prompt_tag(
"standard-rag-prompt",
tag="development"
)
# Later, if the prompt proves successful, we can tag it for production
prod_tag_url = client.add_prompt_tag(
"standard-rag-prompt",
tag="production"
)According to a recent LangChain announcement, "Prompt Tagging allows users to label individual commits with version tags (e.g., 'dev', 'staging', 'v2') directly in the LangSmith UI. Users can then pull a prompt using the tag as a commit identifier in code" [3].
To incorporate versioned prompts into your RAG pipeline, you can pull them using various identifiers:
# Pull the latest version
latest_prompt = client.pull_prompt("standard-rag-prompt")
# Pull a specific version by its commit hash
specific_version = client.pull_prompt("standard-rag-prompt:a1b2c3d4")
# Pull the production version (using the tag)
production_prompt = client.pull_prompt("standard-rag-prompt@production")
# Pull the development version
dev_prompt = client.pull_prompt("standard-rag-prompt@development")Here's how to use these versioned prompts in a RAG pipeline using LangChain's LCEL (LangChain Expression Language):
from operator import itemgetter
from langchain_core.runnables import RunnablePassthrough
from langchain.schema import StrOutputParser
from langchain_openai import ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
# Setup components for the RAG pipeline
embeddings = OpenAIEmbeddings()
vectorstore = Chroma(embedding_function=embeddings)
retriever = vectorstore.as_retriever()
llm = ChatOpenAI(model="gpt-4.1-mini")
# Use the production prompt in the RAG chain
rag_chain = (
{"context": itemgetter("question") | retriever, "question": itemgetter("question")}
| production_prompt | llm | StrOutputParser()
)
# Alternatively, use the development prompt for testing
dev_rag_chain = (
{"context": itemgetter("question") | retriever, "question": itemgetter("question")}
| dev_prompt | llm | StrOutputParser()
)LangChain Hub offers an alternative approach for managing prompt versions:
from langchain import hub
# Pull a specific prompt version from the hub
versioned_prompt = hub.pull("your_username/prompt_name:commit_hash")
# Using a tag instead of a commit hash
tagged_prompt = hub.pull("your_username/prompt_name@tag_name")
# Create a RAG chain with the versioned prompt
hub_rag_chain = (
{"context": itemgetter("question") | retriever, "question": itemgetter("question")}
| versioned_prompt | llm | StrOutputParser()
)LangSmith provides several built-in evaluators that are particularly relevant for RAG systems:
from langsmith.evaluation import LangChainStringEvaluator, evaluate
# Basic QA evaluator
qa_evaluator = LangChainStringEvaluator("qa", config={"llm": eval_llm})
# Relevance evaluator
relevance_evaluator = LangChainStringEvaluator(
"labeled_criteria",
config={
"criteria": {
"relevance": (
"Does the answer directly address the question asked?"
)
},
"llm": eval_llm
}
)
# Groundedness evaluator
groundedness_evaluator = LangChainStringEvaluator(
"labeled_criteria",
config={
"criteria": {
"groundedness": (
"Is the answer fully supported by the retrieved context?"
)
},
"llm": eval_llm
}
)With evaluators defined, you can compare different prompt versions:
# Define a dataset for testing
dataset_name = "rag_evaluation_dataset"
# Evaluate the standard prompt
evaluate(
rag_chain.invoke,
data=dataset_name,
evaluators=[qa_evaluator, relevance_evaluator, groundedness_evaluator],
metadata={"prompt_version": "standard"}
)
# Evaluate the "dope" prompt
evaluate(
dev_rag_chain.invoke,
data=dataset_name,
evaluators=[qa_evaluator, relevance_evaluator, groundedness_evaluator],
metadata={"prompt_version": "dope"}
)This creates two experiments in LangSmith that can be compared to understand the impact of the prompt changes.
For RAG systems, you may want to implement specialized evaluators that assess [8]:
- Context Relevance: How relevant are the retrieved documents to the question?
- Context Precision: How much of the retrieved content is actually relevant?
- Context Recall: Does the retrieved context contain all the information needed?
- Answer Faithfulness: Is the generated answer faithful to the retrieved context?
LangSmith can be integrated with frameworks like RAGAS that provide these specialized metrics:
from ragas.langchain.evalchain import RagasEvaluatorChain
# Create RAGAS evaluator chains
context_precision_chain = RagasEvaluatorChain(metric="context_precision")
faithfulness_chain = RagasEvaluatorChain(metric="faithfulness")
context_recall_chain = RagasEvaluatorChain(metric="context_recall")
# Use these chains in LangSmith evaluation
evaluate(
rag_chain.invoke,
data=dataset_name,
evaluators=[
context_precision_chain,
faithfulness_chain,
context_recall_chain
],
metadata={"prompt_version": "standard", "ragas_metrics": True}
)Based on industry experience and the LangSmith documentation [7], here are recommended workflow patterns for managing RAG prompts:
-
Development Cycle:
- Create initial prompt based on RAG best practices
- Test on representative sample queries
- Push to LangSmith with "dev" tag
- Iterate based on feedback and evaluations
- When satisfied, tag with "staging" for broader testing
- After validation, tag with "production" for deployment
-
Branch-Based Development:
- Maintain separate prompt repositories for different RAG components
- Use descriptive names that indicate the prompt's purpose
- Create "branches" by forking prompts for experimental changes
- Merge successful changes back to the main prompt
-
Continuous Improvement:
- Monitor production performance metrics
- Identify problematic query patterns
- Create targeted improvements to address issues
- A/B test new versions against current production prompts
- Gradually roll out improvements using tags
Consistent naming helps organize prompts and versions:
- Prompt Names: Use descriptive names indicating purpose (
query-analysis-prompt,rag-answer-generation, etc.) - Tags: Use consistent environment tags (
dev,staging,prod) - Version Tags: For major versions, consider numbered tags (
v1,v2) - Feature Tags: For specialized variants (
concise,detailed,technical)
Each prompt version should include:
- Clear description of purpose
- Expected input variables
- Required context format
- Known limitations
- Performance characteristics
- Change log from previous versions
Let's examine a real-world example of how prompt versioning can improve a RAG system:
A technical documentation RAG system is experiencing issues:
- Users report that answers are technically correct but difficult to understand
- The system often provides too much irrelevant information
- Context retrieval seems accurate, but answer generation is problematic
Initial Production Prompt:
Given the retrieved documentation sections, provide a comprehensive answer to the user's question. Include all relevant technical details.
Context: {context}
Question: {question}
Version 2 (Clarity Focus):
Given the retrieved documentation sections, answer the user's question clearly and concisely. Focus on explaining concepts in simple terms first, then add technical details if necessary.
Context: {context}
Question: {question}
Version 3 (Structure Improvement):
Given the retrieved documentation sections, answer the user's question following these guidelines:
1. Start with a 1-2 sentence direct answer
2. Explain key concepts in simple terms
3. Add technical details if relevant
4. If code examples are available in the context, include a brief example
Context: {context}
Question: {question}
After testing these prompts through LangSmith evaluation:
| Metric | Original | Version 2 | Version 3 |
|---|---|---|---|
| Correctness | 94% | 92% | 93% |
| Relevance | 83% | 91% | 95% |
| Conciseness | 62% | 88% | 91% |
| User Satisfaction | 71% | 85% | 92% |
Based on these results, Version 3 was tagged as "production" and deployed, while retaining the ability to roll back if issues arose.
Modern RAG systems often use multiple prompts for different stages. LangSmith can manage the entire suite:
# Query analysis prompt
query_analysis_prompt = ChatPromptTemplate.from_template("""
Analyze the user's question and extract key search terms and concepts.
Question: {question}
Key search terms:
""")
# Retrieval prompt
retrieval_prompt = ChatPromptTemplate.from_template("""
You are a retrieval system. Based on these search terms, what information should be retrieved?
Search terms: {search_terms}
Retrieval strategy:
""")
# Answer generation prompt
answer_prompt = ChatPromptTemplate.from_template("""
Given this context and question, provide a helpful answer.
Context: {context}
Question: {question}
Answer:
""")
# Push all prompts to LangSmith
client.push_prompt("rag-query-analysis", object=query_analysis_prompt)
client.push_prompt("rag-retrieval", object=retrieval_prompt)
client.push_prompt("rag-answer-generation", object=answer_prompt)Advanced RAG systems might select different prompts based on query characteristics:
def select_appropriate_prompt(query):
"""Select the appropriate prompt based on query characteristics."""
# Analyze query complexity
if contains_technical_terms(query):
return client.pull_prompt("rag-answer@technical")
elif is_simple_question(query):
return client.pull_prompt("rag-answer@concise")
else:
return client.pull_prompt("rag-answer@detailed")
# Use in RAG pipeline
dynamic_rag_chain = (
{"context": retriever, "question": RunnablePassthrough()}
| (lambda x: {"context": x["context"], "question": x["question"],
"prompt": select_appropriate_prompt(x["question"])})
| (lambda x: x["prompt"].format(context=x["context"], question=x["question"]))
| llm
| StrOutputParser()
)LangSmith can be used in systems that automatically optimize prompts:
def evaluate_prompt_performance(prompt_version, test_dataset):
"""Evaluate a prompt version against test data."""
rag_chain = build_rag_chain_with_prompt(prompt_version)
results = evaluate(
rag_chain.invoke,
data=test_dataset,
evaluators=[qa_evaluator, relevance_evaluator]
)
return calculate_overall_score(results)
def auto_optimize_prompt(base_prompt, iterations=5):
"""Automatically optimize a prompt through iterations."""
current_best = base_prompt
best_score = evaluate_prompt_performance(current_best, "test_dataset")
for i in range(iterations):
# Generate variations of the prompt
variations = generate_prompt_variations(current_best)
# Evaluate each variation
for var in variations:
var_prompt = client.push_prompt(
f"auto-rag-{i}",
object=var,
description=f"Auto-generated variation {i}"
)
score = evaluate_prompt_performance(var_prompt, "test_dataset")
if score > best_score:
current_best = var_prompt
best_score = score
# Tag the best performing prompt
client.add_prompt_tag(current_best, "auto-optimized-best")
return current_bestEffective prompt management is critical for RAG systems, especially as they move from development to production. LangSmith provides a comprehensive framework for versioning, testing, and evaluating prompts [1][5] that enables teams to:
- Maintain a complete history of prompt evolution
- Deploy different prompt versions to different environments
- Systematically test and compare prompt performance
- Collaborate effectively across technical and non-technical team members
- Ensure stability and reliability in production systems
By implementing proper prompt versioning practices, organizations can build more robust, effective, and maintainable RAG systems that deliver consistent value to users [7][8].
[1] LangSmith Documentation, "Prompt Engineering Concepts." https://docs.smith.langchain.com/prompt_engineering/concepts, accessed April 2025.
[2] LangChain, "Evaluating RAG pipelines with Ragas + LangSmith." https://blog.langchain.dev/evaluating-rag-pipelines-with-ragas-langsmith/, August 2023.
[3] LangSmith, "Prompt Tagging in LangSmith for Version Control." https://changelog.langchain.com/announcements/prompt-tags-in-langsmith-for-version-control, October 2024.
[4] LangChain Documentation, "Build a Retrieval Augmented Generation (RAG) App." https://python.langchain.com/docs/tutorials/rag/, accessed April 2025.
[5] LangSmith, "Evaluation Quick Start." https://docs.smith.langchain.com/evaluation, accessed April 2025.
[6] PromptLayer, "Best Prompt Versioning Tools for LLM Optimization." https://blog.promptlayer.com/5-best-tools-for-prompt-versioning/, January 2025.
[7] LangChain, "LangSmith Cookbook." https://github.com/langchain-ai/langsmith-cookbook/, accessed April 2025.
[8] LangSmith Documentation, "Evaluate a RAG application." https://docs.smith.langchain.com/evaluation/tutorials/rag, accessed April 2025.