Black Friday50% off your first month on any monthly plan with EARLY50 See plans
All 29 lessons
  1. Foundations: what RAG is and a working pipeline

    1. 1. Getting started: what RAG is and when to use it
    2. 2. Types of RAG: a map from naive to agentic
    3. 3. Lab: build the smallest RAG pipeline end to end
    4. 4. How embeddings and vector search find meaning
    5. 5. Document types: what each one breaks, and how to prepare it
  2. Better retrieval: chunking and search

    1. 6. Chunking documents so answers survive
    2. 7. Chunking strategies compared: which to use when
    3. 8. Keyword search vs vector search, and fusing both
    4. 9. Retrieval algorithms compared: which to use when
    5. 10. Reranking the top results before the model reads them
    6. 11. Rewriting the question before you search
  3. Answers you can trust, measure and debug

    1. 12. Lab: grounded answers with citations and refusals
    2. 13. Measuring a RAG system with an evaluation set
    3. 14. Lab: evaluation methods and frameworks: Ragas and DeepEval
    4. 15. Troubleshooting a RAG system, symptom by symptom
  4. Production in a company

    1. 16. Access control: only retrieve what the user may see
    2. 17. Defending against prompt injection in retrieved text
    3. 18. Running RAG in production: freshness, cost and monitoring
    4. 19. Real-world RAG: what worked, what failed, and why
  5. Agentic RAG

    1. 20. Lab: from a fixed pipeline to an agent with a search tool
    2. 21. Self-correcting retrieval: grade, retry, fall back
    3. 22. GraphRAG and routing across many sources
    4. 23. Lab: numbers, tables and financial documents
  6. Frameworks: LangChain, LangGraph and LangSmith

    1. 24. The RAG toolbox: what each library does and why you import it
    2. 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
    3. 26. Lab: Ask Brightline in LangChain
    4. 27. Lab: agentic RAG as a LangGraph graph
    5. 28. Lab: tracing, evaluating and monitoring with LangSmith
    6. 29. Wrap-up: your production agentic RAG checklist
See the leaderboard

20 min

Lab: tracing, evaluating and monitoring with LangSmith

Trace every request, run evaluations on a dataset, and set up production monitoring and feedback in LangSmith.

By the end of this lesson you can

  • Trace plain Python and LangChain code in LangSmith, and read a trace
  • Run experiments on a dataset and compare two versions
  • Set up online evaluators, user feedback, dashboards and alerts for production

From earlier lessons · 1 of 2

In the LangGraph agent, where does the loop cap live?

Pick one answer.

From earlier lessons · 2 of 2

Which Ragas metrics can run on live traffic with no reference answer?

Pick one answer.

It's Monday. Twelve employees clicked thumbs-down on Ask Brightline over the weekend. You have no idea which questions they asked, which chunks came back, or which version of the prompt was live. LangSmith exists so that Monday never looks like this.

Lab: trace, evaluate, monitor

LangSmith is LangChain's service for tracing, evaluation and monitoring. It works with plain Python, LangChain and LangGraph. You'll trace Ask Brightline, run two versions against a dataset, then set up production monitoring. Checked against langsmith 0.14.17 in November 2026. Its docs moved to docs.langchain.com/langsmith.

Steps 1 and 2 run locally without an account. The rest needs a LangSmith API key. At the time of writing, the free Developer plan includes one seat and up to 5,000 base traces a month; pricing changes often, so check the current page.

[39][46]

Step 1: trace plain Python

The @traceable decorator records a function's inputs, outputs, timing and errors as a run. Nested calls become child runs, so one request becomes one trace: a tree of every step.

python
from langsmith import traceable
@traceable(run_type="retriever")
def retrieve_docs(question: str) -> list[str]:
return [d["text"] for d, _ in rerank(question, retrieve(question, k=10), keep=3)]
@traceable(run_type="llm")
def generate(question: str, docs: list[str]) -> str:
return llm(f"Context:\n{chr(10).join(docs)}\n\nQuestion: {question}\nAnswer from the context only.")
@traceable(name="ask_brightline")
def assistant(question: str) -> dict:
docs = retrieve_docs(question)
return {"answer": generate(question, docs), "contexts": docs}
retrieve() is lab 1's, rerank() is lesson 10's, llm() is from rag.py. LangChain and LangGraph code is traced automatically once tracing is on.

Step 2: turn tracing on, and the flag that silently fails

bash
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=... # from your LangSmith settings
export LANGSMITH_PROJECT=ask-brightline
Terminal. Newer agent-based workspaces use LANGSMITH_AGENT_ID and LANGSMITH_AGENT_ENVIRONMENT instead of LANGSMITH_PROJECT; non-US regions also set LANGSMITH_ENDPOINT.

Issue: no traces appear. The flag must be exactly true. With True or 1, the installed SDK leaves tracing off, without an error. Old tutorials also use LANGCHAIN_TRACING_V2 and LANGCHAIN_PROJECT. These still work as fallbacks, but the docs say to replace them.

Once traces flow, open one. You'll see the question, the three chunks the retriever returned, the exact prompt, the answer, and each step's latency and tokens. That's lesson 18's trace, without writing the logging yourself. Traces store full prompts and documents, so restrict project access like the source documents, and set retention to match your data policy.

[39]

Step 3: a dataset and an evaluator

python
from langsmith import Client
client = Client()
ds = client.create_dataset(dataset_name="brightline-policy-qa")
client.create_examples(dataset_id=ds.id, examples=[
{"inputs": {"question": "How much can I spend on a hotel in Chicago?"},
"outputs": {"answer": "$250 per night, since Chicago is a Tier 1 city."}},
{"inputs": {"question": "How much can I spend on a hotel in Denver?"},
"outputs": {"answer": "$175 per night, since Denver is not a Tier 1 city."}},
])
def amount_correct(inputs: dict, outputs: dict, reference_outputs: dict) -> dict:
expected = reference_outputs["answer"].split()[0] # "$250" or "$175"
return {"key": "amount_correct", "score": expected in outputs["answer"]}
In practice, load lesson 13's full evaluation set. An evaluator takes the inputs, your app's outputs and the reference, and returns a named score.

Step 4: run two experiments and compare

python
exp_a = client.evaluate(lambda x: assistant(x["question"]), data="brightline-policy-qa",
evaluators=[amount_correct], experiment_prefix="keep3")
exp_b = client.evaluate(lambda x: assistant_keep5(x["question"]), data="brightline-policy-qa",
evaluators=[amount_correct], experiment_prefix="keep5")
assistant_keep5 is assistant() with keep=5 in retrieve_docs. Each experiment runs every example and records the scores.
text
How much can I spend on a hotel in Chicago? -> Up to $250 per night. | amount_correct = True
How much can I spend on a hotel in Denver? -> Up to $250 per night. | amount_correct = False
Real output from a local run with a stub answer, using evaluate(..., upload_results=False), which sends nothing.

In the LangSmith UI, open the dataset, select both experiments and click Compare. Per question, it marks regressions in red and improvements in green against the experiment you pick as the baseline. This is lesson 13's per-question view, kept for every run. For model-judged scores, the docs pair evaluate with the separate openevals package of ready-made judges.

[40][41]

Step 5: collect user feedback

python
from langsmith import trace
def ask_with_feedback(question):
with trace(name="ask_brightline", inputs={"question": question}) as root:
root.outputs = assistant(question)
return root.outputs, root.id # show the answer, keep the trace id
def record_thumbs(trace_id, project_id, thumbs_up, comment=""):
client.create_feedback(key="user_thumbs", score=1 if thumbs_up else 0,
trace_id=trace_id, session_id=project_id, comment=comment)
Wire record_thumbs to your thumbs buttons. project_id comes from client.read_project(project_name=...).id.

Now each thumbs-down is attached to its exact trace. Monday's twelve complaints become twelve traces you can open, read and add to the dataset.

[45]

Step 6: monitor production

Four features turn traces into monitoring. These are set up in the UI:

  1. Online evaluators run an LLM judge or a Python or JS code check on a sample of live traces, such as 10% of traffic. Use a faithfulness judge from lesson 14 with no reference needed. Each evaluator has a weekly spend cap.
  2. Automation rules match traces and act on them: send thumbs-down traces to an annotation queue or dataset, call a webhook, or extend retention.
  3. Dashboards chart traces, model calls, cost and tokens, tool usage and feedback scores per project, and you can build custom ones grouped by metadata or tags.
  4. Alerts fire on run count, cost, errors, feedback score or latency, over 5- or 15-minute windows, and go to Slack, PagerDuty or a webhook.

Start with lesson 18's three numbers: refusal rate, negative feedback rate, and p95 latency and cost.

[42][43][44]

Match

Match each need to the LangSmith feature

Pick an item on the left, then its match on the right.

Worth keeping

  1. Tracing turns every request into an inspectable tree: query, chunks, prompt, answer, latency, tokens.
  2. LANGSMITH_TRACING must be exactly true, or nothing is traced.
  3. Experiments on a dataset make every change a measured comparison.
  4. Feedback, online evaluators, rules and alerts turn traces into production monitoring.

You've built Ask Brightline in plain Python and in LangChain and LangGraph, and wired it to LangSmith. In the last lesson you'll put it all together and test your judgement on new company scenarios.

Leaderboard

Loading…

50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.

Sources for this lesson (8)

Each fact in the lesson is marked with the number of the page it comes from.

  1. [39]LangSmith docs: Tracing quickstart
  2. [40]LangSmith docs: Evaluation quickstart
  3. [41]LangSmith docs: Compare experiment results
  4. [42]LangSmith docs: Online evaluations
  5. [43]LangSmith docs: Alerts
  6. [44]LangSmith docs: Dashboards
  7. [45]LangSmith docs: Attach user feedback
  8. [46]LangChain pricing (LangSmith plans)