WAF++ WAF++

WAF-AGN-070 – Agent Observability

Pillar: Agentic | Severity: Medium | Category: Observability | Automatable: High

Description

Agent activities must be visible and measurable. Telemetry must be collected for all agents including tokens, latency, GPU utilization, and decision metrics. Metrics must be ingested into a monitoring system. Distributed tracing must link agent operations end-to-end.

Rationale

Without observability, agent operations become a black box:

Performance issues: Cannot detect latency spikes, token waste, or model degradation.

Cost overruns: No visibility into token consumption leads to unexpected costs.

Failed debugging: Cannot identify root cause of agent failures without logs.

Compliance gaps: Many frameworks require monitoring of automated systems.

Missed optimization opportunities: Cannot identify improvement areas without metrics.

Requirements

  • Token usage metrics (input, output, total) must be collected

  • Latency metrics must be tracked and alert on anomalies

  • GPU/CPU utilization must be monitored for compute-heavy agents

  • Error rates and failure types must be tracked

  • Distributed tracing must link operations end-to-end

Implementation Guidance

  1. Set up metrics collection: Configure CloudWatch, Prometheus, or third-party monitoring

  2. Create dashboards: Build agent health dashboards for monitoring teams

  3. Configure alerts: Set alarms for unusual patterns (high latency, high errors)

  4. Implement tracing: Use X-Ray or OpenTelemetry for request tracking

  5. Document baseline metrics: Establish normal ranges for comparison

Maturity Levels

Level Name Criteria

1

No Observability

No metrics. No logging. No visibility into agent operations.

2

Basic Logging

Basic request/response logging. Minimal metrics.

3

Full Telemetry

Token metrics, latency, errors all collected. Dashboards configured.

4

Proactive Monitoring

Alerts on anomalies. Automated scaling based on load.

5

Predictive Observability

AI models predict issues before they occur.

Terraform Checks

waf-agn-070.tf.aws.metrics-configured

Checks: Agent metrics are configured (tokens, latency, GPU utilization).

Compliant Non-Compliant 
resource "aws_cloudwatch_metric_alarm" "agent_latency" {
  alarm_name          = "agent-latency-alarm"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "Latency"
  namespace           = "AWS/Bedrock"
  period              = 60
  statistic           = "Average"
  threshold           = 5000
  unit                = "Milliseconds"
}

resource "aws_cloudwatch_metric_alarm" "agent_tokens" {
  alarm_name          = "agent-token-alarm"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "TokenCount"
  namespace           = "AWS/Bedrock"
  period              = 300
  statistic           = "Sum"
  threshold           = 100000
  unit                = "Count"
}

No metrics alarms configured

waf-agn-070.tf.aws.distributed-tracing-enabled

Checks: Distributed tracing (X-Ray) is enabled for agent operations.

# Compliant
resource "aws_lambda_function" "agent_traced" {
  function_name = "agent-traced"
  environment {
    variables = {
      AWS_XRAY_TRACING_NAME = "agent-service"
    }
  }
}

Remediation: Configure metrics for tokens and latency. Enable X-Ray tracing. Set up log integration with monitoring system.

Evidence

Type Required Description

IaC

✅ Required

CloudWatch alarms, X-Ray configuration, metric definitions.

Config

✅ Required

Monitoring dashboards, alert thresholds, sampling rates.

Process

✅ Required

Metric analysis reports, alert response runbooks, performance baselines.

Governance

✅ Required

Observability policy defining requirements and monitoring standards.

Related Controls

Regulatory Mapping

Framework Controls

ISO 27001:2022

A.8.1 – Information security roles and responsibilities; A.8.2 – Privileged access rights

NIST SP 800-53

AU-2 – Event logging; AU-3 – Event content; SI-4 – Information system monitoring

NIST CSF 2.0

DE.CM – Continuous monitoring; DE.DP – Data protection

GDPR

Art. 32 – Security of processing; Art. 5(1)(f) – Integrity and confidentiality

PCI DSS v4.0

Req 10 – Monitor and detect threats

TISAX

Information security – Monitoring and logging

ANSSI SecNumCloud

Domain – Logging and monitoring

BIO

BIO – Logboekhouding en monitoring

ENS High

op.exp.6 – Gestión de cambios

UK NCSC CAF

A4 – Policy and assurance

FedRAMP

AU-2, AU-3, SI-4 (Moderate baseline)

CMMC 2.0

AU.L2-3.8.1 – Flow audit logs to central log management

IT-Grundschutz

ORP.1 – Informationstechnisch-related Angriffsschutz

IRAP

ISM – Logging and monitoring

CCCS PBMM

AU-2, AU-3 – Logging and audit

MAS TRM

Ch.9 – Change management

ISMAP

Monitoring and logging controls

FISC

Technical measures – Logging and monitoring

AWS Well-Architected Framework

Security Pillar – Logging and monitoring; Reliability Pillar – Monitoring

Google Cloud Architecture Framework

Reliability – Monitoring; Security – Monitoring

BSI C5:2020

OPS-01 – Operational monitoring

SOC 2 Type II

CC4.1 – Monitoring activities; CC7.1 – Infrastructure and software monitoring

CSA STAR

CCM Logging and Monitoring