How to Add LLM Observability in 5 Minutes

You shipped an LLM feature. Users are hitting it. Something feels off about the responses, but you can't tell what's going wrong because you have no logs, no traces, no visibility.

This walkthrough gets you from zero to full observability in five minutes. By the end, you'll have every LLM call traced with inputs, outputs, token counts, and costs.

What You'll Get

After following these steps:

• Every LLM call is captured with full request/response data
• Token usage and cost are tracked per call and in aggregate
• You can search and filter traces by model, status, latency, and cost
• Prompts are versioned so you can see what changed and when

Path A: Managed Prompts (Fastest)

This is the fastest path. Your prompts live in the Tracia dashboard, and prompts.run() handles execution and tracing in one call.

Step 1: Install the SDK

npm install tracia

Step 2: Create a Prompt in the Dashboard

1. Open your Tracia dashboard and create a new prompt
2. Give it a name (e.g., customer-support)
3. Write your prompt text with {{variables}} for dynamic content
4. Choose a model (GPT-4o, Claude Sonnet, etc.)
5. Test it in the playground

Step 3: Call It from Your Code

import { Tracia } from 'tracia';

const tracia = new Tracia({ apiKey: 'tr_xxx' });

const response = await tracia.prompts.run('welcome-email', {
  name: 'Alice',
  product: 'Tracia'
});
// ✓ Prompt fetched, rendered, executed, and traced automatically

from tracia import Tracia

tracia = Tracia(api_key="tr_xxx")

response = await tracia.prompts.run("welcome-email", {
  "name": "Alice",
  "product": "Tracia"
})
# ✓ Prompt fetched, rendered, executed, and traced automatically

That's it. Tracia fetches the prompt, renders your variables, calls the model, and traces everything automatically. Every trace links back to the prompt version that produced it.

Step 4: Check Your Dashboard

Open your Tracia dashboard. You'll see every call listed with:

• Model and provider used
• Input and output content
• Token count (prompt + completion)
• Cost calculated from up-to-date pricing for 100+ models
• Latency in milliseconds
• Prompt version that generated the response

Want to change the prompt? Edit it in the dashboard. It's live instantly, no code change, no redeploy.

Path B: Local Execution (Keep Your Setup)

If you'd rather keep your existing provider setup and just add tracing, use runLocal().

Step 1: Install the SDK

npm install tracia

Step 2: Initialize and Call

import { Tracia } from 'tracia';

const tracia = new Tracia({ apiKey: 'tr_xxx' });

const response = await tracia.runLocal({
  model: 'gpt-4o', // or claude-sonnet-4, gemini-2.0-flash, +100 more
  messages: [{ role: 'user', content: 'Summarize this document...' }]
});
// ✓ Your prompt, your infrastructure, traced automatically

from tracia import Tracia

tracia = Tracia(api_key="tr_xxx")

response = await tracia.run_local(
  model="gpt-4o", # or claude-sonnet-4, gemini-2.0-flash, +100 more
  messages=[{"role": "user", "content": "Summarize this document..."}]
)
# ✓ Your prompt, your infrastructure, traced automatically

Your API key, your provider, your infrastructure. Tracia submits traces in the background with zero added latency to your LLM calls. It works across 100+ models from OpenAI, Anthropic, Google, and Amazon Bedrock.

Step 3: Check Your Dashboard

Same dashboard, same traces. You get full visibility into every call (inputs, outputs, tokens, costs, and latency) regardless of which execution mode you use.

Both Paths: What Comes Next

Adding Evaluations

Tracing tells you what happened. Evaluations tell you if the output was any good.

Tracia's evaluator system lets you define quality checks that run on your traces:

• Contains / regex match: Does the output include required content?
• JSON validation: Is the response valid JSON?
• Length limits / word count: Is the response the right size?
• LLM-as-judge: Use another model to score quality
• Custom checks: Define your own scoring rules

You can run evaluations manually on specific traces or set them up to run automatically.

Test Cases

Define expected inputs and outputs for your prompts, run them in batch, and validate changes before deploying to production. This catches quality regressions before your users do.

Cost Monitoring

Your analytics dashboard shows cost breakdowns by prompt, model, and time period. Use it to:

• Identify your most expensive prompts
• Compare model costs for the same task
• Spot unexpected cost spikes early

Next Steps

You now have full observability over your LLM calls. From here:

• Explore the prompt library for production-ready templates
• Set up evaluations to measure output quality over time
• Review your cost dashboard to identify optimization opportunities

The free tier includes 10,000 traces per month, enough to get started and see real value before committing to a paid plan.