LangSmith Trace Fetcher

# LangSmith Fetch - Agent Debugging Skill

Debug LangChain and LangGraph agents by fetching execution traces directly from LangSmith Studio in your terminal.

When to Use This Skill

• Automatically activate when user mentions:
• 🐛 "Debug my agent" or "What went wrong?"
• 🔍 "Show me recent traces" or "What happened?"
• ❌ "Check for errors" or "Why did it fail?"
• 💾 "Analyze memory operations" or "Check LTM"
• 📊 "Review agent performance" or "Check token usage"
• 🔧 "What tools were called?" or "Show execution flow"

Prerequisites

1. Install langsmith-fetch ```bash pip install langsmith-fetch ```

2. Set Environment Variables ```bash export LANGSMITH_API_KEY="your_langsmith_api_key" export LANGSMITH_PROJECT="your_project_name" ```

Verify setup: ```bash echo $LANGSMITH_API_KEY echo $LANGSMITH_PROJECT ```

Core Workflows

Workflow 1: Quick Debug Recent Activity

When user asks: "What just happened?" or "Debug my agent"

Execute: ```bash langsmith-fetch traces --last-n-minutes 5 --limit 5 --format pretty ```

1. Analyze and report:
2. ✅ Number of traces found
3. ⚠️ Any errors or failures
4. 🛠️ Tools that were called
5. ⏱️ Execution times
6. 💰 Token usage

Example response format: ``` Found 3 traces in the last 5 minutes:

• Trace 1: ✅ Success
• Agent: memento
• Tools: recall_memories, create_entities
• Duration: 2.3s
• Tokens: 1,245

• Trace 2: ❌ Error
• Agent: cypher
• Error: "Neo4j connection timeout"
• Duration: 15.1s
• Failed at: search_nodes tool

• Trace 3: ✅ Success
• Agent: memento
• Tools: store_memory
• Duration: 1.8s
• Tokens: 892

💡 Issue found: Trace 2 failed due to Neo4j timeout. Recommend checking database connection. ```

---

Workflow 2: Deep Dive Specific Trace

When user provides: Trace ID or says "investigate that error"

Execute: ```bash langsmith-fetch trace <trace-id> --format json ```

1. Analyze JSON and report:
2. 🎯 What the agent was trying to do
3. 🛠️ Which tools were called (in order)
4. ✅ Tool results (success/failure)
5. ❌ Error messages (if any)
6. 💡 Root cause analysis
7. 🔧 Suggested fix

Example response format: ``` Deep Dive Analysis - Trace abc123

Goal: User asked "Find all projects in Neo4j"

1. Execution Flow:
2. ✅ search_nodes(query: "projects")
3. → Found 24 nodes

1. ❌ get_node_details(node_id: "proj_123")
2. → Error: "Node not found"
3. → This is the failure point

1. ⏹️ Execution stopped

Root Cause: The search_nodes tool returned node IDs that no longer exist in the database, possibly due to recent deletions.

1. Suggested Fix:
2. Add error handling in get_node_details tool
3. Filter deleted nodes in search results
4. Update cache invalidation strategy

Token Usage: 1,842 tokens ($0.0276) Execution Time: 8.7 seconds ```

---

Workflow 3: Export Debug Session

When user says: "Save this session" or "Export traces"

Execute: ```bash # Create session folder with timestamp SESSION_DIR="langsmith-debug/session-$(date +%Y%m%d-%H%M%S)" mkdir -p "$SESSION_DIR"

# Export traces langsmith-fetch traces "$SESSION_DIR/traces" --last-n-minutes 30 --limit 50 --include-metadata

# Export threads (conversations) langsmith-fetch threads "$SESSION_DIR/threads" --limit 20 ```

Report: ``` ✅ Session exported successfully!

• Location: langsmith-debug/session-20251224-143022/
• Traces: 42 files
• Threads: 8 files

1. You can now:
2. Review individual trace files
3. Share folder with team
4. Analyze with external tools
5. Archive for future reference

Session size: 2.3 MB ```

---

Workflow 4: Error Detection

When user asks: "Show me errors" or "What's failing?"

Execute: ```bash # Fetch recent traces langsmith-fetch traces --last-n-minutes 30 --limit 50 --format json > recent-traces.json

# Search for errors grep -i "error\|failed\|exception" recent-traces.json ```

1. Analyze and report:
2. 📊 Total errors found
3. ❌ Error types and frequency
4. 🕐 When errors occurred
5. 🎯 Which agents/tools failed
6. 💡 Common patterns

Example response format: ``` Error Analysis - Last 30 Minutes

Total Traces: 50 Failed Traces: 7 (14% failure rate)

1. Error Breakdown:
2. Neo4j Connection Timeout (4 occurrences)
3. - Agent: cypher
4. - Tool: search_nodes
5. - First occurred: 14:32
6. - Last occurred: 14:45
7. - Pattern: Happens during peak load

1. Memory Store Failed (2 occurrences)
2. - Agent: memento
3. - Tool: store_memory
4. - Error: "Pinecone rate limit exceeded"
5. - Occurred: 14:38, 14:41

1. Tool Not Found (1 occurrence)
2. - Agent: sqlcrm
3. - Attempted tool: "export_report" (doesn't exist)
4. - Occurred: 14:35

1. 💡 Recommendations:
2. Add retry logic for Neo4j timeouts
3. Implement rate limiting for Pinecone
4. Fix sqlcrm tool configuration
5. ```

---

Common Use Cases

Use Case 1: "Agent Not Responding"

User says: "My agent isn't doing anything"

1. Steps:
2. Check if traces exist:
3. ```bash
4. langsmith-fetch traces --last-n-minutes 5 --limit 5
5. ```

1. If NO traces found:
2. - Tracing might be disabled
3. - Check: `LANGCHAIN_TRACING_V2=true` in environment
4. - Check: `LANGCHAIN_API_KEY` is set
5. - Verify agent actually ran

1. If traces found:
2. - Review for errors
3. - Check execution time (hanging?)
4. - Verify tool calls completed

---

Use Case 2: "Wrong Tool Called"

User says: "Why did it use the wrong tool?"

1. Steps:
2. Get the specific trace
3. Review available tools at execution time
4. Check agent's reasoning for tool selection
5. Examine tool descriptions/instructions
6. Suggest prompt or tool config improvements

---

Use Case 3: "Memory Not Working"

User says: "Agent doesn't remember things"

1. Steps:
2. Search for memory operations:
3. ```bash
4. langsmith-fetch traces --last-n-minutes 10 --limit 20 --format raw | grep -i "memory\|recall\|store"
5. ```

1. Check:
2. - Were memory tools called?
3. - Did recall return results?
4. - Were memories actually stored?
5. - Are retrieved memories being used?

---

Use Case 4: "Performance Issues"

User says: "Agent is too slow"

1. Steps:
2. Export with metadata:
3. ```bash
4. langsmith-fetch traces ./perf-analysis --last-n-minutes 30 --limit 50 --include-metadata
5. ```

1. Analyze:
2. - Execution time per trace
3. - Tool call latencies
4. - Token usage (context size)
5. - Number of iterations
6. - Slowest operations

1. Identify bottlenecks and suggest optimizations

---

Output Format Guide

Pretty Format (Default) ```bash langsmith-fetch traces --limit 5 --format pretty ``` **Use for:** Quick visual inspection, showing to users

JSON Format ```bash langsmith-fetch traces --limit 5 --format json ``` **Use for:** Detailed analysis, syntax-highlighted review

Raw Format ```bash langsmith-fetch traces --limit 5 --format raw ``` **Use for:** Piping to other commands, automation

---

Advanced Features

Time-Based Filtering ```bash # After specific timestamp langsmith-fetch traces --after "2025-12-24T13:00:00Z" --limit 20

# Last N minutes (most common) langsmith-fetch traces --last-n-minutes 60 --limit 100 ```

Include Metadata ```bash # Get extra context langsmith-fetch traces --limit 10 --include-metadata

# Metadata includes: agent type, model, tags, environment ```

Concurrent Fetching (Faster) ```bash # Speed up large exports langsmith-fetch traces ./output --limit 100 --concurrent 10 ```

---

Troubleshooting

"No traces found matching criteria"

1. Possible causes:
2. No agent activity in the timeframe
3. Tracing is disabled
4. Wrong project name
5. API key issues

Solutions: ```bash # 1. Try longer timeframe langsmith-fetch traces --last-n-minutes 1440 --limit 50

# 2. Check environment echo $LANGSMITH_API_KEY echo $LANGSMITH_PROJECT

# 3. Try fetching threads instead langsmith-fetch threads --limit 10

# 4. Verify tracing is enabled in your code # Check for: LANGCHAIN_TRACING_V2=true ```

"Project not found"

Solution: ```bash # View current config langsmith-fetch config show

# Set correct project export LANGSMITH_PROJECT="correct-project-name"

# Or configure permanently langsmith-fetch config set project "your-project-name" ```

Environment variables not persisting

Solution: ```bash # Add to shell config file (~/.bashrc or ~/.zshrc) echo 'export LANGSMITH_API_KEY="your_key"' >> ~/.bashrc echo 'export LANGSMITH_PROJECT="your_project"' >> ~/.bashrc

# Reload shell config source ~/.bashrc ```

---

Best Practices

1. Regular Health Checks ```bash # Quick check after making changes langsmith-fetch traces --last-n-minutes 5 --limit 5 ```

2. Organized Storage ``` langsmith-debug/ ├── sessions/ │ ├── 2025-12-24/ │ └── 2025-12-25/ ├── error-cases/ └── performance-tests/ ```

3. Document Findings When you find bugs: 1. Export the problematic trace 2. Save to `error-cases/` folder 3. Note what went wrong in a README 4. Share trace ID with team

4. Integration with Development ```bash # Before committing code langsmith-fetch traces --last-n-minutes 10 --limit 5

# If errors found langsmith-fetch trace <error-id> --format json > pre-commit-error.json ```

---

Quick Reference

```bash # Most common commands

# Quick debug langsmith-fetch traces --last-n-minutes 5 --limit 5 --format pretty

# Specific trace langsmith-fetch trace <trace-id> --format pretty

# Export session langsmith-fetch traces ./debug-session --last-n-minutes 30 --limit 50

# Find errors langsmith-fetch traces --last-n-minutes 30 --limit 50 --format raw | grep -i error

# With metadata langsmith-fetch traces --limit 10 --include-metadata ```

---

Resources

• LangSmith Fetch CLI: https://github.com/langchain-ai/langsmith-fetch
• LangSmith Studio: https://smith.langchain.com/
• LangChain Docs: https://docs.langchain.com/
• This Skill Repo: https://github.com/OthmanAdi/langsmith-fetch-skill

---

Notes for Claude

• Always check if `langsmith-fetch` is installed before running commands
• Verify environment variables are set
• Use `--format pretty` for human-readable output
• Use `--format json` when you need to parse and analyze data
• When exporting sessions, create organized folder structures
• Always provide clear analysis and actionable insights
• If commands fail, help troubleshoot configuration issues

---

Version: 0.1.0 Author: Ahmad Othman Ammar Adi License: MIT Repository: https://github.com/OthmanAdi/langsmith-fetch-skill