Webhooks

Outbound webhook notifications push security events to external systems in real time. Webhooks solve the “nobody knows an approval is pending” problem and enable SIEM/ChatOps integration without polling or log scraping.

Why webhooks

Without webhooks, pending approvals rely on the operator polling latchgate approvals list or using latchgate tui. In practice, approvals time out (default 5 minutes) before anyone notices. Webhooks close the feedback loop — the operator gets a push notification within seconds.

Beyond approvals, webhooks enable real-time alerting for denials, revocations, execution failures, and budget exhaustion without requiring Prometheus alerting rules or log forwarding pipelines.

Event types

Each webhook endpoint subscribes to specific event types. Events map to enforcement pipeline outcomes:

Event	Fires when	Urgency	Use case
approval.pending	Policy returns PendingApproval	High	Notify operator to review
approval.granted	Operator approves and execution succeeds	Medium	Audit trail, close ticket
approval.denied	Operator denies	Medium	Audit trail, notify agent owner
approval.expired	TTL expires without resolution	Medium	Alert on missed approvals
action.denied	Policy denies an action	Low–High	Security alerting
action.executed	Action completes successfully	Low	Activity feed, SIEM
action.failed	Provider execution or post-dispatch verification fails	Medium	Ops alerting
revocation	Kill-switch activated	Critical	Incident response
budget.exhausted	Session budget depleted	Low	Usage monitoring
budget.rollback_failed	Budget rollback fails after a post-debit error	High	Budget discrepancy alerting

The budget.rollback_failed event indicates a charged debit that could not be refunded (typically due to Redis unavailability). This creates a budget discrepancy that operators must reconcile.

Configuration

Via the operator TUI (recommended)

Launch the TUI and switch to the Setup screen, Webhooks sub-tab (6 → 4):

latchgate tui

• a — add an endpoint. The form prompts for name, URL, and events in sequence; press Enter to advance each field and Esc to cancel.
• r — remove the selected endpoint (confirm with y).
• t — send a test event to the selected endpoint through the full delivery pipeline (HMAC signing, SSRF check, HTTP POST) without retries, reporting delivery status.
• ↑/↓ (or k/j) — move the selection.

The form validates URL, events, and secret before writing to latchgate.toml, preserving comments and formatting. HTTPS is required in production. If no secret is provided, a cryptographically random one is generated.

CLI equivalent

latchgate config add-webhook --name slack-approvals \
  --url https://hooks.slack.com/services/T.../B.../xxx \
  --secret whsec_your-signing-secret \
  --events approval.pending,approval.expired \
  --format slack

latchgate config add-webhook --name security-siem \
  --url https://siem.corp.internal/api/v1/events \
  --secret whsec_siem-secret \
  --events action.denied,revocation,action.executed

Manage existing webhooks:

latchgate config list-webhooks                   # show all configured
latchgate config remove-webhook --name old-hook  # remove by name
latchgate config test-webhook --name slack-alerts  # send test event to one endpoint
latchgate config test-webhook                      # test all endpoints

The test-webhook command sends a synthetic test event through the full delivery pipeline (HMAC signing, SSRF check, HTTP POST) without retries, reporting delivery status for verification.

The CLI validates URL, events, format, and secret before writing. HTTPS is required in production. If --secret is omitted, a cryptographically random secret is auto-generated.

Manual TOML (alternative)

Add [[webhooks]] sections to latchgate.toml. Each section defines one endpoint with its own URL, signing secret, event filter, payload format, and retry policy.

[[webhooks]]
name = "slack-approvals"
url = "https://hooks.slack.com/services/T.../B.../xxx"
secret = "whsec_your-signing-secret-here"
events = ["approval.pending", "approval.expired"]
format = "slack"

[[webhooks]]
name = "security-siem"
url = "https://siem.corp.internal/api/v1/events"
secret = "whsec_siem-secret"
events = ["action.denied", "revocation", "action.executed"]
headers = { "Authorization" = "Bearer ${SIEM_TOKEN}" }

[[webhooks]]
name = "pagerduty-critical"
url = "https://events.pagerduty.com/v2/enqueue"
secret = "whsec_pd-secret"
events = ["revocation"]
format = "pagerduty"
headers = { "X-Routing-Key" = "${PAGERDUTY_ROUTING_KEY}" }

Field reference

Field	Required	Default	Description
name	yes	—	Human-readable identifier, unique across endpoints. Used in logs and dead-letter audit.
url	yes	—	HTTPS endpoint URL. HTTP rejected except localhost / 127.0.0.1 / [::1] in dev mode.
secret	yes	—	HMAC-SHA256 signing secret. Generate with openssl rand -hex 32. Convention: prefix whsec_.
events	yes	—	Event types to subscribe to.
format	no	generic	Payload format: generic, slack, discord, or pagerduty. See Payload formats.
headers	no	{}	Extra HTTP headers. Supports ${ENV_VAR} expansion from environment variables.
timeout_seconds	no	5	Per-request HTTP timeout. latchgate config add-webhook defaults to 10 for first-time headroom; edit latchgate.toml directly to change after creation.
max_retries	no	3	Retry attempts on 5xx/timeout. 0 = fire once, no retry.
retry_backoff_seconds	no	[1, 5, 30]	Backoff delay per retry attempt. Padded or truncated to match max_retries.
disable	no	false	Temporarily disable without removing config. Endpoint is still validated at startup.

Secrets and environment variables

Signing secrets and header values containing ${ENV_VAR} are resolved from environment variables at startup. Secrets can also reference SOPS-encrypted values using the same mechanism as action secrets.

[[webhooks]]
name = "siem"
url = "https://siem.corp.internal/v1/events"
secret = "${WEBHOOK_SIEM_SECRET}"
events = ["action.denied", "revocation"]
headers = { "Authorization" = "Bearer ${SIEM_TOKEN}" }

Payload formats

Generic (default)

Every webhook delivers a standard JSON envelope:

{
  "id": "evt_01JAxxxxxxxxxxxxxxxxxx",
  "type": "approval.pending",
  "timestamp": "2025-03-28T14:30:00Z",
  "gate_version": "0.1.0",
  "data": {
    "approval_id": "apr_01JAxxxxxxxxxxxxxxxxxx",
    "action_id": "github_create_issue",
    "principal": "agent-ops",
    "owner": "alice@company.com",
    "risk_level": "high",
    "request_hash": "sha256:...",
    "expires_at": "2025-03-28T14:35:00Z",
    "request_summary": {
      "owner": "acme",
      "repo": "app",
      "title": "Deploy hotfix"
    },
    "trace_id": "trc_01JA..."
  }
}

The type field is the event type string. The data object varies by event type. Two distinct owner values may appear: the top-level data.owner is the principal’s configured owner contact from [identity.peercred.principals] (the person responsible for the agent — useful for routing alerts), while data.request_summary.owner reflects whatever the agent passed in the request body (e.g. a GitHub repo owner). Do not conflate them. The request_summary in approval events contains request parameters with sensitive values redacted — fields matching declared secret names or common sensitive patterns (password, token, key, credential, auth, bearer, etc.) are replaced with ***REDACTED***.

Slack

When format = "slack", the payload is transformed into Slack Block Kit format ({ text, blocks }) compatible with Incoming Webhooks and chat.postMessage. No thin proxy or Slack Workflow needed for formatted messages.

Discord

When format = "discord", the payload is transformed into Discord embed format ({ content, embeds }) compatible with channel webhooks. No proxy needed.

PagerDuty

When format = "pagerduty", the payload is transformed into PagerDuty Events API v2 format ({ routing_key, event_action, payload }). Set the X-Routing-Key header to the PagerDuty integration key.

Per-event data fields

All events that carry a principal also carry an owner field (the principal’s configured owner contact, from [identity.peercred.principals]). owner is null when the principal has no owner configured. SIEM and routing rules can use owner to fan out per-team alerts without maintaining a principal => team lookup table.

approval.pending: approval_id, action_id, principal, owner, risk_level, request_hash, expires_at, request_summary, trace_id. Conditionally includes unresolved_domains (domains in the request not in the manifest allowlist) and unresolved_paths (paths not in the manifest allowlist) when non-empty.

approval.granted: approval_id, action_id, approved_by, receipt_id, trace_id

approval.denied: approval_id, action_id, denied_by, reason, trace_id

approval.expired: approval_id, action_id, principal, owner, created_at, expired_at

action.denied: action_id, principal, owner, deny_reason, trace_id

action.executed: action_id, principal, owner, receipt_id, verification_outcome, trace_id

action.failed: action_id, principal, owner, error_class, trace_id

revocation: old_epoch, new_epoch, operator_id

budget.exhausted: action_id, principal, owner, session_id

budget.rollback_failed: session_id, error, trace_id, label (identifies which post-debit error path triggered the rollback, e.g. "build_run_task_error", "dispatch_error")

Signing and verification

Every webhook request carries two headers:

X-LatchGate-Signature: sha256=<hex-encoded HMAC-SHA256>
X-LatchGate-Timestamp: 1711633800

The signature is computed over {timestamp}.{raw_body} using the endpoint’s signing secret. This is the same pattern used by Stripe, GitHub, and Slack. The timestamp prevents replay attacks.

Verification examples

Python:

import hmac, hashlib, time

def verify_latchgate_webhook(
    payload: bytes,
    timestamp: str,
    signature: str,
    secret: str
) -> bool:
    # 1. Reject stale timestamps (5-minute tolerance).
    if abs(time.time() - int(timestamp)) > 300:
        return False

    # 2. Compute expected signature.
    message = f"{timestamp}.".encode() + payload
    expected = "sha256=" + hmac.new(
        secret.encode(),
        message,
        hashlib.sha256
    ).hexdigest()

    # 3. Constant-time comparison.
    return hmac.compare_digest(expected, signature)

TypeScript:

import { createHmac, timingSafeEqual } from "crypto";

function verifyLatchGateWebhook(
  payload: Buffer,
  timestamp: string,
  signature: string,
  secret: string,
): boolean {
  if (Math.abs(Date.now() / 1000 - parseInt(timestamp)) > 300) return false;

  const message = Buffer.concat([Buffer.from(`${timestamp}.`), payload]);

  const expected =
    "sha256=" + createHmac("sha256", secret).update(message).digest("hex");

  return timingSafeEqual(Buffer.from(expected), Buffer.from(signature));
}

Go:

func verifyLatchGateWebhook(
    payload []byte,
    timestamp,
    signature,
    secret string,
) bool {
    ts, err := strconv.ParseInt(timestamp, 10, 64)
    if err != nil || abs(time.Now().Unix()-ts) > 300 {
        return false
    }
    mac := hmac.New(sha256.New, []byte(secret))
    mac.Write([]byte(fmt.Sprintf("%s.", timestamp)))
    mac.Write(payload)
    expected := "sha256=" + hex.EncodeToString(mac.Sum(nil))
    return hmac.Equal([]byte(expected), []byte(signature))
}

Delivery mechanism

LatchGate supports two webhook delivery modes, configured via webhook_mode in latchgate.toml:

Outbox mode (default, production)

webhook_mode = "outbox"

Events are persisted to a SQLite transactional outbox table before any delivery attempt. A background poller reads pending entries every 2 seconds and delivers them with retry and dead-letter handling. This guarantees zero event loss — even if the process crashes between event generation and delivery, the event is recovered on restart.

Failed deliveries are retried with configurable backoff. After max_retries exhausted, the entry is moved to a dead-letter table for operator review.

Async mode (dev only, no endpoints)

webhook_mode = "async"

Events are sent to a bounded in-process channel (1024 capacity) and delivered by background tasks. If the channel is full, events are dropped with a warning. If the process crashes, undelivered events are lost.

Important: webhook_mode = "async" is rejected at startup when any [[webhooks]] endpoints are configured. This validation prevents accidental event loss in production. Use outbox mode when webhook endpoints are present.

Retry behavior

• 2xx — delivered successfully.
• 4xx — dead-lettered immediately (client error, not retried).
• 5xx / timeout / network error — retried with configurable backoff (default: 1s, 5s, 30s).
• Retries exhausted — dead-lettered. In outbox mode, dead-lettered entries are queryable and retriable.

SSRF protection

Webhook URLs go through the same SSRF protection as provider HTTP requests: private IP blocking (loopback, RFC-1918, link-local, CGNAT, IPv6 loopback/ULA), DNS pinning (resolve-then-connect to close the DNS rebinding TOCTOU window), and redirect blocking. HTTPS is required — HTTP URLs are rejected at config validation.

Exception: localhost / 127.0.0.1 / [::1] is allowed in dev mode only (unsafe dev mode via latchgate up) for testing with local receivers.

Delivery guarantees

In outbox mode (default), webhooks are durable. Events survive process crashes, slow endpoints, and full channels. The evidence ledger remains the authoritative security record, but webhook events are no longer best-effort — approval.pending and revocation notifications are guaranteed to be delivered or dead-lettered.

In async mode, webhooks are best-effort with bounded retry. If the process crashes between event generation and delivery, the webhook is lost. The channel has a bounded capacity (1024 events); overflow drops events with a warning.

On graceful shutdown (SIGTERM/SIGINT), both modes drain queued events and wait for in-flight deliveries to complete (up to 10 seconds).

Integration patterns

Slack

Create an Incoming Webhook in Slack and use the slack format for native Block Kit messages:

[[webhooks]]
name = "slack-approvals"
url = "https://hooks.slack.com/services/T.../B.../xxx"
secret = "whsec_slack-secret"
events = ["approval.pending", "approval.expired"]
format = "slack"

With format = "slack", messages render with formatted blocks (bold, structured fields, timestamps) without needing a Slack Workflow. Use format = "generic" (default) if you prefer the raw JSON code block or have a custom Workflow handling the payload.

Microsoft Teams

Create an Incoming Webhook in a Teams channel or use a Power Automate flow triggered by HTTP request:

[[webhooks]]
name = "teams-security"
url = "https://outlook.office.com/webhook/..."
secret = "whsec_teams-secret"
events = ["approval.pending", "action.denied", "revocation"]

Teams renders the generic envelope as a code block. For rich Adaptive Cards, use Power Automate to parse the JSON and build a card.

Discord

Create a webhook in Discord (Channel Settings => Integrations => Webhooks) and use the discord format for native embeds:

[[webhooks]]
name = "discord-alerts"
url = "https://discord.com/api/webhooks/1234.../abcd..."
secret = "whsec_discord-secret"
events = ["approval.pending", "approval.granted", "action.denied"]
format = "discord"

With format = "discord", messages render as structured embeds with color coding by event type.

Telegram

Telegram’s Bot API expects {"chat_id": ..., "text": "..."}, which differs from LatchGate’s envelope. Use a thin proxy (Cloud Function, Lambda — approximately 20 lines of code) that receives the LatchGate envelope, extracts relevant fields, and calls the Telegram sendMessage API:

[[webhooks]]
name = "telegram-ops"
url = "https://your-proxy.example.com/latchgate-to-telegram"
secret = "whsec_telegram-secret"
events = ["approval.pending", "revocation"]

PagerDuty

Use the pagerduty format for native PagerDuty Events API v2 payloads:

[[webhooks]]
name = "pagerduty-critical"
url = "https://events.pagerduty.com/v2/enqueue"
secret = "whsec_pd-secret"
events = ["revocation"]
format = "pagerduty"
headers = { "X-Routing-Key" = "${PAGERDUTY_ROUTING_KEY}" }

Opsgenie

Similar to PagerDuty — use an Opsgenie Incoming Webhook integration that accepts arbitrary JSON, or a proxy to transform to Opsgenie’s alert format:

[[webhooks]]
name = "opsgenie-incidents"
url = "https://api.opsgenie.com/v2/alerts"
secret = "whsec_opsgenie-secret"
events = ["revocation", "action.denied"]
headers = { "Authorization" = "GenieKey ${OPSGENIE_API_KEY}" }

Generic SIEM (Splunk, Elastic, Datadog, Sentinel)

Configure the SIEM’s HTTP ingestion endpoint as the webhook URL with appropriate auth headers. The standard envelope is SIEM-friendly by design — consistent type field for indexing, ISO 8601 timestamps, flat data object. No transformation needed.

Splunk HEC:

[[webhooks]]
name = "splunk-siem"
url = "https://splunk.corp.internal:8088/services/collector/event"
secret = "whsec_splunk-secret"
events = [
    "action.denied",
    "action.executed",
    "revocation",
    "approval.granted",
    "approval.denied"
]
headers = { "Authorization" = "Splunk ${SPLUNK_HEC_TOKEN}" }

Elastic:

[[webhooks]]
name = "elastic-siem"
url = "https://elastic.corp.internal:9200/latchgate-events/_doc"
secret = "whsec_elastic-secret"
events = ["action.denied", "action.executed", "revocation"]
headers = {
    "Authorization" = "ApiKey ${ELASTIC_API_KEY}",
    "Content-Type" = "application/json"
}

Datadog:

[[webhooks]]
name = "datadog-logs"
url = "https://http-intake.logs.datadoghq.com/api/v2/logs"
secret = "whsec_datadog-secret"
events = ["action.denied", "revocation"]
headers = {
    "DD-API-KEY" = "${DATADOG_API_KEY}",
    "Content-Type" = "application/json"
}

Custom HTTPS endpoint

Any system that accepts HTTPS POST with a JSON body. Verify authenticity using the X-LatchGate-Signature and X-LatchGate-Timestamp headers with the signing verification code above.

Approval flow with webhooks

With webhooks configured, the approval flow becomes push-based:

1. Agent calls POST /execute => policy returns pending_approval.
2. Kernel creates the pending approval, returns 202 to the agent.
3. Webhook fires immediately — approval.pending arrives in Slack/Teams/PagerDuty within seconds.
4. Operator reviews and approves (via CLI latchgate approvals approve, TUI, or API).
5. Kernel executes through the hardened path => approval.granted + action.executed webhooks fire.

Without webhooks, step 3 is missing — the operator must poll. The 5-minute approval TTL becomes viable because the operator is alerted within seconds.

For terminal-first operators, latchgate tui provides a real-time dashboard with inline approve/deny. See Operator TUI for details.

Security considerations

Signing secrets must be strong. Generate with openssl rand -hex 32. Prefix with whsec_ by convention.

HTTPS required. HTTP endpoints rejected at config validation. Prevents credential leakage over plaintext.

No secret values in payloads. The request_summary field in approval events shows request parameters but redacts values matching declared secret names or common sensitive patterns. Uses both manifest-declared secret names and heuristic pattern matching.

Webhook URLs are operator-controlled. Only operators with access to latchgate.toml can configure endpoints. Agents cannot influence where webhooks are sent.

Delivery failures are operational, not security events. They are logged as structured warnings. The evidence ledger is the authoritative security record.

Graceful shutdown. On SIGTERM/SIGINT, the dispatcher drains queued events and waits up to 10 seconds for in-flight deliveries to complete before aborting remaining tasks.

Troubleshooting

Webhooks not firing

Check the startup banner — it should show Webhooks: active. If it shows disabled, no [[webhooks]] sections are configured. Verify latchgate.toml has at least one [[webhooks]] entry and run latchgate doctor to validate.

Webhook delivery failures in logs

Look for webhook delivery failed (dead-letter) warnings. The log includes the endpoint name, event type, event ID, and error. Common causes:

• DNS resolution failed — the webhook URL hostname cannot be resolved.
• SSRF blocked — the URL resolved to a private/reserved IP address.
• HTTP 4xx — the receiving endpoint rejected the payload (not retried). Check endpoint configuration on the receiver side.
• HTTP 5xx / timeout — the receiver is unhealthy (retried with backoff). Check receiver health.

Testing webhooks

In the TUI (Setup → Webhooks, 6 → 4), select an endpoint and press t to send a synthetic test event through the full delivery pipeline.

CLI equivalent

latchgate config test-webhook --name slack-alerts  # test one endpoint
latchgate config test-webhook                      # test all endpoints

The test sends a synthetic event with HMAC signing and SSRF checks, reporting delivery status without retries. Useful for verifying endpoint connectivity and signature validation after initial setup.

For local development, use a local HTTP receiver:

# Terminal 1: start a simple receiver
python3 -c "
from http.server import HTTPServer, BaseHTTPRequestHandler
import json

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers['Content-Length'])
        body = json.loads(self.rfile.read(length))
        print(json.dumps(body, indent=2))
        self.send_response(200)
        self.end_headers()

HTTPServer(('127.0.0.1', 9999), Handler).serve_forever()
"

# latchgate.toml (dev mode)
[[webhooks]]
name = "local-test"
url = "http://localhost:9999/webhook"
secret = "whsec_test-secret"
events = ["approval.pending", "action.denied", "revocation"]

Verifying signatures

If your receiver rejects signatures, check:

1. The secret in latchgate.toml matches the secret your receiver uses for verification.
2. Verification uses the raw request body bytes (not a re-serialized version).
3. The timestamp tolerance is sufficient (LatchGate uses the current time; allow at least 5 minutes for clock skew).

Startup fails with webhook_mode error

If you see an error about webhook_mode = "async" with configured endpoints: async mode is rejected when any [[webhooks]] entries are present because it drops events under load. Change to webhook_mode = "outbox" (the default) or remove webhook endpoint configuration.