Live Observability & Telemetry for watsonx Orchestrate: Monitoring Agent Health, Latency, and Token Spend

WXO Labs Series Icon By Markus van Kempen | Series: WXO Labs & Tutorial Guide (Part 13 of 15)
Interactive Lab Portal: https://markusvankempen.github.io/wxo-labs/
GitHub Repository: https://github.com/markusvankempen/wxo-labs

Live Observability & Telemetry for watsonx Orchestrate: Monitoring Agent Health, Latency, and Token Spend

Why This Pattern Matters

Operating 50+ enterprise agents without observability is running blind. Who is consuming all the Granite tokens? Why is the CRM tool taking 8 seconds to respond? We build an end-to-end telemetry collector and interactive live dashboard.

πŸ“‹ Prerequisites & Setup

Ensure your environment has the required dependencies:

pip install ibm-watsonx-orchestrate

Verify active environment:

orchestrate env list

πŸ› οΈ Step 1: Implement the Logic (telemetry_collector.py)

import time
from collections import defaultdict
from datetime import datetime

class AgentTelemetry:
    def __init__(self):
        self.metrics = defaultdict(lambda: {"invocations": 0, "total_ms": 0, "errors": 0, "tokens": 0})

    def record(self, agent_name: str, duration_ms: float, tokens: int, error: bool = False):
        m = self.metrics[agent_name]
        m["invocations"] += 1
        m["total_ms"] += duration_ms
        m["tokens"] += tokens
        if error:
            m["errors"] += 1

    def summary(self):
        print(f"πŸ“Š LIVE TELEMETRY SNAPSHOT [{datetime.utcnow().strftime('%H:%M:%S')}]")
        print("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━")
        for agent, m in self.metrics.items():
            avg_lat = m["total_ms"] / m["invocations"] if m["invocations"] > 0 else 0
            err_rate = (m["errors"] / m["invocations"] * 100) if m["invocations"] > 0 else 0
            print(f"Agent: {agent:25} | Invocations: {m['invocations']:5} | Avg Latency: {avg_lat:6.1f}ms | Tokens: {m['tokens']:7} | Errors: {err_rate:4.1f}%")

collector = AgentTelemetry()
collector.record("hello_world_greeter", 320, 185)
collector.record("finance_approval_agent", 840, 420)
collector.record("async_job_manager", 1450, 610)
collector.summary()

πŸ“¦ Step 2: Declare Dependencies (requirements.txt)

ibm-watsonx-orchestrate>=2.9.0

πŸ€– Step 3: Configure the AI Agent (agent.yaml)

spec_version: v1
kind: native
name: telemetry_enabled_agent
title: "Monitored Agent"
model: ibm/granite-3-8b-instruct

πŸš€ Step 4: Import & Test Live in Terminal

Import tool and test conversationally:

python3 telemetry_collector.py

Live Response:

πŸ“Š LIVE TELEMETRY SNAPSHOT [20:25:00]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Agent: hello_world_greeter       | Invocations:     1 | Avg Latency:  320.0ms | Tokens:     185 | Errors:  0.0%
Agent: finance_approval_agent    | Invocations:     1 | Avg Latency:  840.0ms | Tokens:     420 | Errors:  0.0%
Agent: async_job_manager         | Invocations:     1 | Avg Latency: 1450.0ms | Tokens:     610 | Errors:  0.0%

πŸ’‘ Pro Tips & Gotchas

🏷️ Recommended Community Tags

When publishing this blog post, select or enter the following tags in the community editor:

watsonx Orchestrate, Observability, Telemetry, Token Tracking, Grafana, Performance Monitoring, DevOps

πŸ—ΊοΈ Series Roadmap (15 Hands-On Labs)

This tutorial is Part 13 of 15 in the complete enterprise hands-on series:

Explore the interactive curriculum and code repository: