Building a Multi-Tier AI Agent System

The Challenge

Modern DevOps and SRE teams need AI assistance that goes beyond simple chatbots. We needed a system that could:

• Help developers with infrastructure queries, deployments, and debugging
• Provide read-only public access for service status and documentation
• Enable authenticated team members to perform infrastructure operations
• Maintain strict security boundaries to prevent prompt injection and privilege escalation
• Scale across multiple communication channels (web, Slack, Discord)

Architecture Overview

The system uses a tiered approach where each bot has specific capabilities and access levels. All bots share a common knowledge base through PostgreSQL with pgvector for semantic search.

The Four Bots

Bot Interaction & Escalation Flow

Requests flow through the bot hierarchy with human-in-the-loop approval for destructive actions:

Security Model

Security is enforced at multiple layers:

Database mutations follow a strict flow: Developer creates PR → Human reviews → Merge to main → GitHub Actions with temporary Vault credentials → Database migration.

Web Chat Integration

The web chat provides two access levels for DevOps/SRE workflows:

End-User Use Cases

Beyond internal DevOps workflows, this architecture can be adapted for customer-facing scenarios:

Kubernetes RBAC

Each bot runs with a dedicated service account with minimal permissions:

Shared Memory with pgvector

All bots share context through a PostgreSQL database with pgvector extension for semantic search:

-- Documents table for context files
CREATE TABLE documents (
  id UUID PRIMARY KEY,
  name TEXT NOT NULL,
  content TEXT NOT NULL,
  embedding vector(1536),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Shared memory for cross-bot knowledge
CREATE TABLE shared_memory (
  id UUID PRIMARY KEY,
  key TEXT NOT NULL,
  value JSONB NOT NULL,
  embedding vector(1536),
  created_by TEXT NOT NULL,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Immutable audit log
CREATE TABLE audit_log (
  id UUID PRIMARY KEY,
  bot_id TEXT NOT NULL,
  action TEXT NOT NULL,
  details JSONB,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

Deployment with Helm

Each bot is deployed using a universal Helm chart with per-bot value overrides:

# Deploy Communicator
helm upgrade --install communicator ./chart \
  --namespace ai-agents \
  --create-namespace \
  -f values-communicator.yaml

# Deploy Operator
helm upgrade --install operator ./chart \
  --namespace ai-agents \
  -f values-operator.yaml

# Deploy Developer
helm upgrade --install developer ./chart \
  --namespace ai-agents \
  -f values-developer.yaml

Security: Defending Against Prompt Injection

AI agents with infrastructure access are high-value targets. We implement defense-in-depth against prompt injection and other attacks:

Key Takeaways