Designing Observability for AI Systems: Apps, Infra, CI & LLMs Monitoring

Observability has become essential for operating AI systems. By monitoring four layers—applications, infrastructure, CI/CD pipelines, and LLMs—in an integrated manner, teams can detect issues early and maintain stable operations.

📑Table of Contents

1. Observability Fundamentals and the Need in the AI Era
2. Monitoring Design Points for the Application Layer
3. Infrastructure Monitoring and Kubernetes/OpenTelemetry Integration
4. Applying Observability to CI/CD Pipelines
5. LLM-Specific Metrics and Design Patterns
6. Comparison Table: Traditional vs AI-Extended Observability
7. Precautions, Limitations, and Recommended Next Actions for Readers
8. Frequently Asked Questions (FAQ)
9. Summary

What to understand first

---

Observability Fundamentals and the Need in the AI Era

Observability refers to the ability to infer a system’s internal state from the outside. Unlike traditional monitoring that tracks known metrics, observability supports unknown problems as well. In the AI era, LLM calls and agent behaviors add complexity that conventional tools alone cannot handle.

OpenTelemetry serves as a unifying standard for multi-layer monitoring. Official documentation shows that semantic conventions for generative AI have moved from experimental to practical use. This enables analysis of token usage and latency alongside metrics from other layers.

Practical check

Readers should first identify which layer in their own systems lacks sufficient observability. Reviewing the official OpenTelemetry documentation at https://opentelemetry.io/docs/specs/semconv/gen-ai/ and comparing it with the current setup is a good starting point.

---

Monitoring Design Points for the Application Layer

At the application layer, HTTP and gRPC calls are measured together with LLM API call spans. Introducing OpenTelemetry instrumentation allows automatic recording of input/output token counts and tool invocation details.

For example, when using the Python OpenTelemetry SDK, simply attaching attributes that follow the GenAI semantic conventions makes LLM call traces collectable. This makes it easier to pinpoint why latency spikes for specific prompts.

Practical check

The key is choosing libraries that require minimal code changes to existing application frameworks. Routing data through the OpenTelemetry Collector is recommended to avoid vendor lock-in.

---

Infrastructure Monitoring and Kubernetes/OpenTelemetry Integration

At the infrastructure layer, Kubernetes cluster metrics and logs are collected via the OpenTelemetry Collector. Resource usage of LLM inference containers can be handled in the same pipeline alongside node and Pod status.

According to the official blog at https://opentelemetry.io/blog/2025/ai-agent-observability/, combining Kubernetes with OTel allows AI agent behavior to be analyzed in relation to infrastructure metrics. This helps quickly identify cases where GPU memory shortages cause increased LLM error rates.

Practical check

When implementing, readers can start by adding GenAI-related receivers to the OTel Collector configuration file. Parallel operation with existing Prometheus exporters is also possible.

---

Applying Observability to CI/CD Pipelines

In CI/CD pipelines, traces are propagated through each build and deployment step. OpenTelemetry context propagation makes it possible to track how code changes affect LLM response quality across the entire pipeline.

For instance, token consumption can be measured at the test stage, and builds can be failed if thresholds are exceeded. This prevents unexpected cost increases in production.

Practical check

A design consideration is ensuring trace IDs are consistently passed between pipeline jobs. Integration examples with tools such as GitHub Actions or Jenkins are available in official resources.

---

LLM-Specific Metrics and Design Patterns

LLM-specific monitoring focuses on model invocation counts, input/output token counts, tool usage counts, and error rates. The GenAI semantic conventions in OpenTelemetry allow these values to be recorded as standardized attributes.

One design pattern is to represent Vector DB operations and agent workflows as separate spans. This allows separate analysis of retrieval latency and generation latency in RAG setups.

Practical check

The official semantic-conventions repository (referenced at https://github.com/open-telemetry/semantic-conventions/issues/1732) shows active discussion of these experimental conventions, with ongoing updates as of 2026. Readers are encouraged to first map these attributes to their own LLM call logs.

---

Comparison Table: Traditional vs AI-Extended Observability

What changes in the signal set

Item	Traditional Observability	AI-Extended Observability
Primary targets	App and infra metrics	Apps, infra, LLM calls, token usage
Example tools	Prometheus, Grafana	OpenTelemetry + GenAI semantic conventions
Insights gained	CPU/memory usage, error rates	Token consumption trends, RAG latency, agent behavior
Ease of integration	Sufficient with existing stack	Requires additional Collector configuration
Cost impact	Relatively low	Varies significantly with token volume

---

Read the comparison as a migration step

The table shows that the AI-extended version requires handling LLM-specific signals in addition to traditional ones. Using OpenTelemetry as the foundation allows existing visualization tools to be reused with minimal changes.

---

Precautions, Limitations, and Recommended Next Actions for Readers

When introducing observability, first verify compatibility with the existing monitoring stack. While OpenTelemetry is vendor-neutral, it is necessary to check in advance whether LLM provider APIs support the semantic conventions.

A limitation is that experimental conventions may change in the future. Measuring token usage is effective for cost management, but environments with strict privacy requirements should combine it with log masking.

Practical check

Recommended next actions for readers: 1. Deploy the OpenTelemetry Collector to Kubernetes 2. Add GenAI semantic convention attributes to LLM invocation code 3. Display token usage and infrastructure metrics side-by-side in a Grafana dashboard 4. Set cost threshold alerts based on one week of operational data

Following these steps allows gradual strengthening of observability across the entire AI system.

---

Frequently Asked Questions (FAQ)

Q: How can token usage of LLMs be monitored with OpenTelemetry?

A: Use attributes from the GenAI semantic conventions (such as gen_ai.usage.prompt_tokens) to record token counts in LLM call spans. After aggregation by the OTel Collector, visualization is possible with Prometheus or Grafana. Refer to the official documentation examples.

Q: What is the minimum configuration for introducing AI Observability?

A: Start by embedding the OpenTelemetry SDK in the application and running a single OTel Collector instance. In Kubernetes environments, deploying the Collector as a DaemonSet is common. GenAI conventions can be enabled optionally.

Q: What is the compatibility with existing Prometheus/Grafana environments?

A: Compatibility is high. The OTel Collector can output metrics in Prometheus exporter format, so existing Grafana dashboards can be used almost as-is. The advantage is that only the new LLM-related metrics need to be added.

Summary

Designing observability for AI systems is effective when centering on OpenTelemetry to monitor applications through LLMs consistently. A design based on independent official sources enables practical monitoring while avoiding vendor lock-in.

As a next action, readers should apply OpenTelemetry instrumentation to at least one of their LLM calls and begin by visualizing token usage. Gradually extending coverage to CI/CD and infrastructure layers while consulting the official documentation is recommended.