Cloud Ops Runbook (Kanbus Console v1)

This runbook covers deploy, smoke checks, rollback, and incident triage for the cloud console stack (infra/cloud).

Preconditions

• Python commands run from conda env: conda run -n py311 ...
• AWS profile: anthus
• Local Docker daemon running (required for Lambda container asset builds)

Deploy

cd infra/cloud
AWS_PROFILE=anthus npx cdk synth -a "/opt/anaconda3/bin/conda run -n py311 python app.py"
AWS_PROFILE=anthus npx cdk deploy KanbusCloudFoundation --require-approval never -a "/opt/anaconda3/bin/conda run -n py311 python app.py"

Capture outputs:

• ApiBaseUrl
• UserPoolId
• UserPoolClientId
• IdentityPoolId
• IotDataEndpointAddress
• TenantEfsFileSystemId
• SyncQueueUrl
• GithubWebhookSecretArn

Smoke Checks

1. API auth + route check:

curl -i "${API_BASE_URL}${STAGE_PATH}/health"

2. Realtime bootstrap contract:

curl -s "${API_BASE_URL}${STAGE_PATH}/{account}/{project}/api/realtime/bootstrap" | jq .

3. Queue wiring:

AWS_PROFILE=anthus aws sqs get-queue-attributes --queue-url "$SYNC_QUEUE_URL" --attribute-names ApproximateNumberOfMessages ApproximateAgeOfOldestMessage
AWS_PROFILE=anthus aws sqs get-queue-attributes --queue-url "$SYNC_DLQ_URL" --attribute-names ApproximateNumberOfMessages

4. Lambda error baseline:

AWS_PROFILE=anthus aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Errors --dimensions Name=FunctionName,Value=<function-name> --start-time "$(date -u -v-15M +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '15 minutes ago' +%Y-%m-%dT%H:%M:%SZ)" --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --period 300 --statistics Sum

Rollback

1. Re-deploy previously known-good commit:

git checkout <known-good-commit>
cd infra/cloud
AWS_PROFILE=anthus npx cdk deploy KanbusCloudFoundation --require-approval never -a "/opt/anaconda3/bin/conda run -n py311 python app.py"

2. If webhook ingress is noisy, disable ingress route temporarily by adding a WAF or by rotating GithubWebhookSecretArn.

3. Confirm alarms and queue depth return to baseline.

Alarm Map

• kanbus-sync-dlq-visible-<env>: DLQ has messages.
• kanbus-sync-queue-age-<env>: sync queue backlog age high.
• kanbus-console-lambda-errors-<env>: API Lambda errors > 0.
• kanbus-webhook-lambda-errors-<env>: webhook ingress errors > 0.
• kanbus-git-sync-errors-<env>: git sync lambda errors > 0.
• kanbus-efs-writer-errors-<env>: EFS writer lambda errors > 0.
• kanbus-console-api-4xx-<env>: elevated client/auth failures and other 4xx responses.

Incident Triage

1. DLQ alarm firing:

• Inspect DLQ message bodies and attributes.
• Reproduce failure against worker logs.
• Redrive once fix is deployed.

2. Webhook ingress errors:

• Confirm signature header and secret retrieval.
• Confirm path tenant segments (account, project) are valid.

3. Console API 4xx spike:

• Confirm Cognito token issuer/audience values match deployed outputs.
• Confirm tenant route path and IoT topic scope claims.

4. Realtime path degraded:

• Verify /api/realtime/bootstrap returns expected endpoint/topic.
• Browser should auto-fallback to SSE when MQTT-over-WSS connect fails.