All 29 lessons
Foundations: what RAG is and a working pipeline
Better retrieval: chunking and search
Answers you can trust, measure and debug
Production in a company
Agentic RAG
Frameworks: LangChain, LangGraph and LangSmith
- 24. The RAG toolbox: what each library does and why you import it
- 25. The framework landscape: LangChain, LangGraph, LlamaIndex, Haystack
- 26. Lab: Ask Brightline in LangChain
- 27. Lab: agentic RAG as a LangGraph graph
- 28. Lab: tracing, evaluating and monitoring with LangSmith
- 29. Wrap-up: your production agentic RAG checklist
20 min
Lab: tracing, evaluating and monitoring with LangSmith
Trace every request, run evaluations on a dataset, and set up production monitoring and feedback in LangSmith.
By the end of this lesson you can
- Trace plain Python and LangChain code in LangSmith, and read a trace
- Run experiments on a dataset and compare two versions
- Set up online evaluators, user feedback, dashboards and alerts for production
From earlier lessons · 1 of 2
In the LangGraph agent, where does the loop cap live?
Pick one answer.
From earlier lessons · 2 of 2
Which Ragas metrics can run on live traffic with no reference answer?
Pick one answer.
It's Monday. Twelve employees clicked thumbs-down on Ask Brightline over the weekend. You have no idea which questions they asked, which chunks came back, or which version of the prompt was live. LangSmith exists so that Monday never looks like this.
Lab: trace, evaluate, monitor
LangSmith is LangChain's service for tracing, evaluation and monitoring. It works with plain Python, LangChain and LangGraph. You'll trace Ask Brightline, run two versions against a dataset, then set up production monitoring. Checked against langsmith 0.14.17 in November 2026. Its docs moved to docs.langchain.com/langsmith.
Steps 1 and 2 run locally without an account. The rest needs a LangSmith API key. At the time of writing, the free Developer plan includes one seat and up to 5,000 base traces a month; pricing changes often, so check the current page.
Step 1: trace plain Python
The @traceable decorator records a function's inputs, outputs, timing and errors as a run. Nested calls become child runs, so one request becomes one trace: a tree of every step.
from langsmith import traceable@traceable(run_type="retriever")def retrieve_docs(question: str) -> list[str]:return [d["text"] for d, _ in rerank(question, retrieve(question, k=10), keep=3)]@traceable(run_type="llm")def generate(question: str, docs: list[str]) -> str:return llm(f"Context:\n{chr(10).join(docs)}\n\nQuestion: {question}\nAnswer from the context only.")@traceable(name="ask_brightline")def assistant(question: str) -> dict:docs = retrieve_docs(question)return {"answer": generate(question, docs), "contexts": docs}
Step 2: turn tracing on, and the flag that silently fails
export LANGSMITH_TRACING=trueexport LANGSMITH_API_KEY=... # from your LangSmith settingsexport LANGSMITH_PROJECT=ask-brightline
Issue: no traces appear. The flag must be exactly true. With True or 1, the installed SDK leaves tracing off, without an error. Old tutorials also use LANGCHAIN_TRACING_V2 and LANGCHAIN_PROJECT. These still work as fallbacks, but the docs say to replace them.
Once traces flow, open one. You'll see the question, the three chunks the retriever returned, the exact prompt, the answer, and each step's latency and tokens. That's lesson 18's trace, without writing the logging yourself. Traces store full prompts and documents, so restrict project access like the source documents, and set retention to match your data policy.
Step 3: a dataset and an evaluator
from langsmith import Clientclient = Client()ds = client.create_dataset(dataset_name="brightline-policy-qa")client.create_examples(dataset_id=ds.id, examples=[{"inputs": {"question": "How much can I spend on a hotel in Chicago?"},"outputs": {"answer": "$250 per night, since Chicago is a Tier 1 city."}},{"inputs": {"question": "How much can I spend on a hotel in Denver?"},"outputs": {"answer": "$175 per night, since Denver is not a Tier 1 city."}},])def amount_correct(inputs: dict, outputs: dict, reference_outputs: dict) -> dict:expected = reference_outputs["answer"].split()[0] # "$250" or "$175"return {"key": "amount_correct", "score": expected in outputs["answer"]}
Step 4: run two experiments and compare
exp_a = client.evaluate(lambda x: assistant(x["question"]), data="brightline-policy-qa",evaluators=[amount_correct], experiment_prefix="keep3")exp_b = client.evaluate(lambda x: assistant_keep5(x["question"]), data="brightline-policy-qa",evaluators=[amount_correct], experiment_prefix="keep5")
How much can I spend on a hotel in Chicago? -> Up to $250 per night. | amount_correct = TrueHow much can I spend on a hotel in Denver? -> Up to $250 per night. | amount_correct = False
In the LangSmith UI, open the dataset, select both experiments and click Compare. Per question, it marks regressions in red and improvements in green against the experiment you pick as the baseline. This is lesson 13's per-question view, kept for every run. For model-judged scores, the docs pair evaluate with the separate openevals package of ready-made judges.
Step 5: collect user feedback
from langsmith import tracedef ask_with_feedback(question):with trace(name="ask_brightline", inputs={"question": question}) as root:root.outputs = assistant(question)return root.outputs, root.id # show the answer, keep the trace iddef record_thumbs(trace_id, project_id, thumbs_up, comment=""):client.create_feedback(key="user_thumbs", score=1 if thumbs_up else 0,trace_id=trace_id, session_id=project_id, comment=comment)
Now each thumbs-down is attached to its exact trace. Monday's twelve complaints become twelve traces you can open, read and add to the dataset.
Step 6: monitor production
Four features turn traces into monitoring. These are set up in the UI:
- Online evaluators run an LLM judge or a Python or JS code check on a sample of live traces, such as 10% of traffic. Use a faithfulness judge from lesson 14 with no reference needed. Each evaluator has a weekly spend cap.
- Automation rules match traces and act on them: send thumbs-down traces to an annotation queue or dataset, call a webhook, or extend retention.
- Dashboards chart traces, model calls, cost and tokens, tool usage and feedback scores per project, and you can build custom ones grouped by metadata or tags.
- Alerts fire on run count, cost, errors, feedback score or latency, over 5- or 15-minute windows, and go to Slack, PagerDuty or a webhook.
Start with lesson 18's three numbers: refusal rate, negative feedback rate, and p95 latency and cost.
Match
Match each need to the LangSmith feature
Pick an item on the left, then its match on the right.
Worth keeping
- Tracing turns every request into an inspectable tree: query, chunks, prompt, answer, latency, tokens.
- LANGSMITH_TRACING must be exactly true, or nothing is traced.
- Experiments on a dataset make every change a measured comparison.
- Feedback, online evaluators, rules and alerts turn traces into production monitoring.
You've built Ask Brightline in plain Python and in LangChain and LangGraph, and wired it to LangSmith. In the last lesson you'll put it all together and test your judgement on new company scenarios.
Leaderboard
Loading…
50 XP for each lesson you finish, 10 for each right answer. Names show only for learners who turn on Show me on leaderboards.
Sources for this lesson (8)
Each fact in the lesson is marked with the number of the page it comes from.