Idempotent Deployments - Safe to Run Multiple Times

Overview

All deployment scripts in Loan Defenders are idempotent - they can be run multiple times safely without recreating existing resources or causing downtime.

How It Works

Azure's Incremental Deployment Mode

Default Behavior:

az deployment group create \
  --name "deployment-name" \
  --resource-group "my-rg" \
  --template-file "template.bicep" \
  --parameters "@parameters.json"

Azure uses incremental mode by default, which means:

✅ Existing resources are preserved - Resources already deployed are not deleted - Only new or changed resources are created/updated - Resource IDs remain the same

✅ Only changes are applied - Azure compares desired state (Bicep) with current state (Azure) - Only differences are deployed - Unchanged resources are skipped

✅ Zero-downtime updates - Container Apps use rolling updates - New revision created before old one is removed - Traffic switches over seamlessly

✅ Safe to run anytime - After pushing new images - After changing configuration - After Bicep changes - Multiple times per day

Deployment Mode Comparison

Mode	Behavior	Use Case
Incremental (default)	Creates/updates only changed resources	✅ Use always (safe, recommended)
Complete	Deletes resources not in template	❌ Never use (destructive, dangerous)

Our scripts use: Incremental mode (Azure default)

What Happens on Re-run

Scenario 1: No Changes

# First run
./deploy-mcp-servers.sh dev
# ✓ Deploys 3 MCP servers

# Second run (no changes)
./deploy-mcp-servers.sh dev
# ✓ Validates resources exist
# ✓ Skips deployment (no changes)
# ⚡ Completes in ~30 seconds

Result: Near-instant validation, no changes made

Scenario 2: New Docker Image

# Push new image
./build-and-push-mcp-images.sh dev v1.2.0

# Deploy with new image
./deploy-mcp-servers.sh dev
# ✓ Detects image tag changed
# ✓ Creates new revision
# ✓ Rolling update (zero downtime)
# ✓ Old revision removed after new one is healthy
# ⏱️ ~2-3 minutes

Result: Zero-downtime rolling update to new image

Scenario 3: Configuration Change

# Edit dev-mcp-servers.parameters.json
# Change: cpu: "0.5" → "1.0"

# Deploy with new config
./deploy-mcp-servers.sh dev
# ✓ Detects CPU allocation changed
# ✓ Updates container app configuration
# ✓ Creates new revision with new CPU
# ✓ Rolling update (zero downtime)
# ⏱️ ~2-3 minutes

Result: Configuration updated, no recreation

Scenario 4: Bicep Module Change

# Edit container-app-mcp-server.bicep
# Add new environment variable

# Deploy with updated Bicep
./deploy-mcp-servers.sh dev
# ✓ Detects environment variable added
# ✓ Updates container app
# ✓ Creates new revision
# ✓ Rolling update (zero downtime)
# ⏱️ ~2-3 minutes

Result: Module changes applied safely

Common Patterns

Pattern 1: Frequent MCP Updates (Daily/Multiple Times Per Day)

# Make code changes in apps/mcp_servers/
# Build and push new images
./build-and-push-mcp-images.sh dev latest

# Deploy immediately (safe!)
./deploy-mcp-servers.sh dev
# ✓ Zero downtime
# ✓ Rolling update
# ✓ Safe to run multiple times

Frequency: As often as needed (multiple times per day)

Pattern 2: UI/API Updates (Daily/Multiple Times Per Day)

# Make code changes in apps/ui/ or apps/api/
# Build and push new images
./build-and-push-app-images.sh dev latest

# Deploy immediately (safe!)
./deploy-ui-api.sh dev
# ✓ Zero downtime for API
# ✓ Zero downtime for UI
# ✓ Safe to run multiple times

Frequency: As often as needed (multiple times per day)

Pattern 3: Infrastructure Updates (Rare)

# Make infrastructure changes (networking, AI services)
./deploy.sh dev
# ✓ Only changed resources updated
# ✓ Existing resources preserved
# ✓ Safe to run multiple times

Frequency: On infrastructure changes/upgrades (monthly/quarterly)

What Gets Created vs Updated

First Deployment

./deploy-mcp-servers.sh dev

Creates: - ✨ Container App: ldfdev-mcp-verification - ✨ Container App: ldfdev-mcp-documents - ✨ Container App: ldfdev-mcp-financial - ✨ Managed Identities (system-assigned) - ✨ Initial revision for each app

Duration: ~5-8 minutes

Second Deployment (Same Configuration)

./deploy-mcp-servers.sh dev

Checks: - ✓ Resources exist - ✓ Configuration matches - ✓ No changes needed

Creates/Updates: Nothing

Duration: ~30 seconds

Third Deployment (New Image)

# After: ./build-and-push-mcp-images.sh dev v1.2.0
./deploy-mcp-servers.sh dev

Creates: - ✨ New revision (with updated image)

Updates: - 🔄 Traffic routing (switch to new revision) - 🔄 Active revision marker

Preserves: - ✓ Container App resource (same ID) - ✓ Managed Identity (same ID) - ✓ RBAC assignments - ✓ Ingress configuration

Duration: ~2-3 minutes

Verification

How to Check if Deployment Changed Anything

# Run deployment
./deploy-mcp-servers.sh dev

# Check deployment status
az deployment group show \
  --name "mcp-servers-dev-20250115-123456" \
  --resource-group "ldfdev-rg" \
  --query "properties.provisioningState"

# Outputs:
# "Succeeded" - Deployment completed

Check What Changed

# Show deployment changes
az deployment group show \
  --name "mcp-servers-dev-20250115-123456" \
  --resource-group "ldfdev-rg" \
  --query "properties.changes"

Check Container App Revisions

# List revisions (shows history)
az containerapp revision list \
  --name "ldfdev-mcp-verification" \
  --resource-group "ldfdev-rg" \
  --query "[].{Name:name, Active:properties.active, Created:properties.createdTime, Traffic:properties.trafficWeight}" \
  --output table

Output shows: - Active revisions (receiving traffic) - Inactive revisions (previous versions) - Traffic distribution - Creation timestamps

Rollback Support

Automatic Rollback

If new revision fails health checks:

# Deploy new version
./deploy-mcp-servers.sh dev
# New revision created
# Health checks fail
# ✓ Traffic stays on old (healthy) revision
# ✓ Old revision preserved for 24 hours
# ✓ No downtime!

Manual Rollback

# List revisions
az containerapp revision list \
  --name "ldfdev-mcp-verification" \
  --resource-group "ldfdev-rg"

# Activate previous revision
az containerapp revision activate \
  --name "ldfdev-mcp-verification" \
  --resource-group "ldfdev-rg" \
  --revision "ldfdev-mcp-verification--abc123"

# Or redeploy previous image tag
./build-and-push-mcp-images.sh dev v1.1.0
./deploy-mcp-servers.sh dev

Best Practices

✅ DO

Run deployments frequently
After every image push
After configuration changes
Multiple times per day if needed

Use version tags for production

./build-and-push-mcp-images.sh prod v1.2.3

Test in dev first

./deploy-mcp-servers.sh dev
# Verify works
./deploy-mcp-servers.sh prod

Check deployment output
Review what changed
Verify no errors
Check health status

❌ DON'T

Don't manually delete resources
Let Bicep manage everything
Don't delete via Portal/CLI
Bicep tracks state
Don't worry about re-running
Scripts are safe to run multiple times
No harm in running again
Quick validation if no changes
Don't use --mode Complete
Never use complete mode
It deletes resources
Incremental is always safe

Performance

Deployment Times

Scenario	First Time	Re-run (no changes)	Re-run (new image)
Infrastructure	15-30 min	~30 sec	5-10 min
Container Platform	5-10 min	~30 sec	2-3 min
MCP Servers	5-8 min	~30 sec	2-3 min
UI + API	5-8 min	~30 sec	2-3 min

Key Insight: Re-runs are fast when nothing changed!

Troubleshooting

Issue: Deployment says "no changes" but image updated

Cause: Image tag same as before (both latest)

Solution: Use explicit version tags

./build-and-push-mcp-images.sh dev v1.2.3
# Edit parameters.json: imageTag: "v1.2.3"
./deploy-mcp-servers.sh dev

Issue: Want to force recreation

Solution: Delete and redeploy

# Delete container app
az containerapp delete \
  --name "ldfdev-mcp-verification" \
  --resource-group "ldfdev-rg"

# Redeploy
./deploy-mcp-servers.sh dev

Issue: Deployment stuck

Solution: Check deployment status

az deployment group show \
  --name "deployment-name" \
  --resource-group "ldfdev-rg"

Summary

✅ All scripts are idempotent - Safe to run multiple times

✅ Azure handles state - Compares desired vs current state

✅ Zero downtime - Rolling updates for container apps

✅ Fast re-runs - ~30 seconds if no changes

✅ Automatic rollback - Failed revisions don't get traffic

✅ Run frequently - After every image push if needed

Bottom line: Don't hesitate to run deployment scripts multiple times. They're designed for it!