Skip to content

ADR-041: 4-Layer Deployment Architecture with Unified PowerShell Execution

Status: Accepted
Date: 2025-10-17
Decision Makers: Development Team
Related ADRs: - ADR-030: Three-Workflow Deployment Architecture - ADR-037: Environment-Based Configuration - ADR-040: Cross-Platform Shell Scripting

Context

We identified three critical maintainability and efficiency issues in our Azure deployment approach:

Problem 1: Dual Deployment Paths (Architectural Divergence)

Current State: - Local deployments: Use deploy.sh (Bash → PowerShell) calling main-avm.bicep (single Bicep orchestrator) - GitHub Actions: Use inline PowerShell calling modular Bicep files separately - Result: Two completely different execution paths deploying the same infrastructure

Impact: - Behavioral divergence risk between local and CI/CD - Double maintenance burden - changes need to be made in two places - Different Bicep orchestration strategies (Bicep modules in main-avm.bicep vs manual orchestration in GitHub Actions) - Testing requires validating both paths

Problem 2: Monolithic Deployment Strategy

Current State: - All infrastructure deployed as single unit (~20 minutes) - Application code changes trigger full infrastructure redeployment - No ability to deploy individual applications selectively - First-time deployment = incremental deployment (no flexibility)

Impact: - Slow feedback loops for developers (20 min for simple API change) - Higher risk deployments (more surface area per deployment) - Resource waste (redeploying unchanged infrastructure) - Poor developer experience

Problem 3: IaC Principle Compliance Risk

Concern: - Moving to shared PowerShell modules could move orchestration logic from Bicep to scripts - This would violate Infrastructure-as-Code principles (scripts should deploy, not define) - Need to ensure Bicep remains the source of truth

Decision

We will implement a 4-Layer Deployment Architecture with Unified PowerShell Execution that addresses all three problems while maintaining IaC principles.

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│ Bicep Orchestrators (SOURCE OF TRUTH for Infrastructure)        │
├─────────────────────────────────────────────────────────────────┤
│ layer1-foundation.bicep    → VNet, Security, AI Services        │
│ layer2-platform.bicep      → ACR, Container Apps Environment    │
│ layer3-apps.bicep          → UI, API, MCP Server apps           │
│ all-in-one.bicep           → Sequential 1 → 2 → 3               │
└─────────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│ PowerShell Modules (DEPLOYMENT UTILITIES ONLY)                  │
├─────────────────────────────────────────────────────────────────┤
│ Common.psm1               → Auth, params, progress monitoring   │
│ Layer1-Foundation.psm1    → Deploys layer1-foundation.bicep     │
│ Layer2-Platform.psm1      → Deploys layer2-platform.bicep       │
│ Layer3-Apps.psm1          → Deploys layer3-apps.bicep           │
│ AllInOne.psm1             → Deploys all-in-one.bicep            │
└─────────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│ Entry Points (SAME for Local and CI/CD)                         │
├─────────────────────────────────────────────────────────────────┤
│ Local:         pwsh infrastructure/scripts/deploy-layer1.ps1    │
│ GitHub Actions: ./infrastructure/scripts/deploy-layer1.ps1      │
│ Result:        IDENTICAL BEHAVIOR                               │
└─────────────────────────────────────────────────────────────────┘

The 4 Layers

Layer 1: Foundation (Core Infrastructure)

Contents: VNet, NSGs, Subnets, Key Vault, Storage, Managed Identity, AI Services, AI Foundry, Log Analytics, (Optional) VPN Gateway

Deployment Time: ~10-15 min (+ 30-45 min if VPN)
Frequency: Rarely (network topology, security baseline changes)
Dependencies: None
Bicep: layer1-foundation.bicep

Layer 2: Platform (Container Platform)

Contents: Azure Container Registry (ACR), Container Apps Environment

Deployment Time: ~5 min
Frequency: Occasionally (scaling strategy, platform updates)
Dependencies: Layer 1 outputs (VNet, Log Analytics)
Bicep: layer2-platform.bicep

Layer 3: Applications (Container Apps)

Contents: UI Container App, API Container App, MCP Server Container Apps

Deployment Time: ~2-3 min per app
Frequency: Frequently (every code change)
Dependencies: Layer 1 + 2 outputs (Key Vault, ACR, Container Apps Env)
Bicep: layer3-apps.bicep

Key Feature: Selective deployment (can deploy just UI, just API, or just MCP servers)

Layer 4: All-In-One (Complete Deployment)

Contents: Sequential deployment of Layer 1 → 2 → 3

Deployment Time: ~20 min total
Frequency: First-time setup, disaster recovery
Dependencies: None (orchestrates all layers)
Bicep: all-in-one.bicep

Rationale

Solves Problem 1: Dual Deployment Paths

Before: 

Local:  deploy.sh → main-avm.bicep (Bicep orchestrates)
CI/CD:  Inline PowerShell → modular Bicep (manual orchestration)
Result: DIVERGENCE

After: 

Local:  deploy-layer1.ps1 → layer1-foundation.bicep
CI/CD:  deploy-layer1.ps1 → layer1-foundation.bicep
Result: IDENTICAL (single source of truth)

Benefits: - ✅ Single execution path for local and CI/CD - ✅ Changes made once, tested once - ✅ No behavioral divergence possible - ✅ Reduced maintenance burden (48% less code)

Solves Problem 2: Monolithic Deployment

Before: - API code change → Full 20 min deployment

After: - API code change → Layer 3 deployment → 2-3 min (85% faster!) - Infrastructure change → Layer 1 only → 10-15 min - Platform scaling → Layer 2 only → 5 min

Benefits: - ✅ Fast incremental deployments - ✅ Selective application deployment (just API, just MCP, etc.) - ✅ Lower risk (smaller deployment surface area) - ✅ Better developer experience

Solves Problem 3: IaC Compliance

Key Principle: PowerShell deploys, Bicep orchestrates

PowerShell Responsibilities (Deployment Utilities): - ✅ Prerequisites checking - ✅ Authentication (Connect-AzAccount, OIDC) - ✅ Parameter loading (from JSON files) - ✅ Progress monitoring (deployment status polling) - ✅ Error handling and user feedback - ❌ NOT: Infrastructure definition - ❌ NOT: Module orchestration - ❌ NOT: Dependency management

Bicep Responsibilities (Infrastructure Definition): - ✅ Infrastructure resource definition - ✅ Module orchestration (module x 'path' = { ... }) - ✅ Dependency management (dependsOn: [...]) - ✅ Output passing between modules - ✅ Parameter validation

Example (Correct Approach):

# PowerShell: Deploy Layer 1 (just calls Bicep)
function Deploy-Layer1Foundation {
    param([string]$Environment)

    # Load parameters (utility function)
    $config = Get-DeploymentConfig -Environment $Environment

    # Deploy Bicep orchestrator (Bicep handles orchestration!)
    New-AzResourceGroupDeployment `
        -TemplateFile "bicep/layer1-foundation.bicep" `
        -TemplateParameterFile $config.ParameterFile
}

// Bicep: Orchestrates modules (infrastructure logic!)
module networking 'modules/networking.bicep' = { ... }
module security 'modules/security.bicep' = {
  dependsOn: [networking]
  params: {
    vnetId: networking.outputs.vnetId  // Bicep passes outputs
  }
}

Benefits: - ✅ IaC principles fully maintained - ✅ Bicep is source of truth - ✅ Testable (can test Bicep independently) - ✅ Infrastructure logic in version control

Consequences

Positive

  1. Single Source of Truth
  2. One Bicep orchestrator per layer
  3. Same deployment path locally and in CI/CD

  4. Changes tested once, work everywhere

  5. Dramatic Performance Improvement

  6. Application deployments: 20 min → 2-3 min (85% reduction)
  7. Foundation updates: isolated, no app redeployment

  8. Platform updates: isolated, no foundation redeployment

  9. Maximum Flexibility

  10. Can deploy all layers or selective layers
  11. Can deploy all apps or selective apps (just API, just MCP)
  12. First-time deployment via all-in-one

  13. Incremental updates via layer-specific deployments

  14. Reduced Maintenance

  15. ~1,950 lines of code → ~1,000 lines (48% reduction)
  16. Single path to test and maintain

  17. Clear separation of concerns (Bicep defines, PowerShell deploys)

  18. Better Developer Experience

  19. Fast feedback loops (2-3 min for app changes)
  20. Clear deployment model (4 layers, obvious dependencies)
  21. Works identically in dev and prod

  22. Cross-platform (PowerShell 7+ on Windows, macOS, Linux)

  23. IaC Principles Maintained

  24. Bicep orchestrates infrastructure
  25. PowerShell only handles deployment mechanics
  26. Infrastructure definition in code (Bicep)
  27. Testable and version-controlled

Negative

  1. Initial Migration Effort
  2. Estimated 8-12 hours to implement
  3. Need to create 4 Bicep orchestrators
  4. Need to create 5 PowerShell modules
  5. Need to update 4 GitHub Actions workflows

  6. Mitigation: Phased migration, keep old scripts during transition

  7. Learning Curve

  8. Team needs to understand 4-layer model
  9. PowerShell module approach may be new
  10. Layer dependencies need to be understood

  11. Mitigation: Comprehensive documentation, training session

  12. Slightly Less Granular GitHub Actions Visualization

  13. Current: Each Bicep module as separate workflow step
  14. New: Layer deployment as single step (but can structure with multiple steps calling functions)

  15. Mitigation: Structure workflows with multiple steps, still get granular logs

  16. Output Dependency Management

  17. Layer 2/3 need to retrieve Layer 1/2 outputs
  18. Requires PowerShell to query previous deployments
  19. Mitigation: PowerShell modules handle this transparently

Neutral

  1. Bicep Incremental Mode Still Applies
  2. Deploying Layer 1 when nothing changed = fast (Bicep skips unchanged resources)
  3. Layer separation is for logical organization and selective deployment

  4. Not about reducing Bicep deployment scope (Bicep already handles that)

  5. GitHub Actions Features Still Available

  6. Secrets management: Pass to PowerShell as parameters
  7. OIDC authentication: Works with PowerShell (enable-AzPSSession)
  8. Caching, artifacts: Workflow-level features, still usable
  9. Parallel jobs: Can still parallelize if needed

Implementation

File Structure

infrastructure/
├── bicep/
│   ├── layer1-foundation.bicep          # Layer 1 orchestrator
│   ├── layer2-platform.bicep            # Layer 2 orchestrator
│   ├── layer3-apps.bicep                # Layer 3 orchestrator
│   ├── all-in-one.bicep                 # All-in-one orchestrator
│   ├── modules/                         # Reusable Bicep modules
│   │   ├── networking.bicep
│   │   ├── security.bicep
│   │   ├── ai-services.bicep
│   │   ├── container-platform.bicep
│   │   ├── container-app-ui.bicep
│   │   ├── container-app-api.bicep
│   │   └── container-apps-mcp-servers.bicep
│   └── environments/
│       └── dev.parameters.json
│
├── scripts/
│   ├── modules/                         # PowerShell modules
│   │   ├── Common.psm1                  # Shared utilities
│   │   ├── Layer1-Foundation.psm1       # Layer 1 deployment
│   │   ├── Layer2-Platform.psm1         # Layer 2 deployment
│   │   ├── Layer3-Apps.psm1             # Layer 3 deployment
│   │   └── AllInOne.psm1                # All-in-one deployment
│   ├── deploy-layer1.ps1                # Layer 1 entry point
│   ├── deploy-layer2.ps1                # Layer 2 entry point
│   ├── deploy-layer3.ps1                # Layer 3 entry point
│   └── deploy-all.ps1                   # All-in-one entry point
│
└── .github/workflows/
    ├── deploy-layer1-foundation.yml     # Layer 1 workflow
    ├── deploy-layer2-platform.yml       # Layer 2 workflow
    ├── deploy-layer3-apps.yml           # Layer 3 workflow
    └── deploy-all-layers.yml            # All-in-one workflow

Usage Examples

First-Time Deployment (All Layers)

# Local
pwsh infrastructure/scripts/deploy-all.ps1 -Environment dev

# GitHub Actions
# Workflow: "Deploy All Layers (All-In-One)"
# Environment: dev

Update API Code Only (Most Common!)

# Local
pwsh infrastructure/scripts/deploy-layer3.ps1 \
  -Environment dev \
  -APIImageTag "v1.2.3" \
  -DeployAPI \
  -DeployUI:$false \
  -DeployMCP:$false

# Time: 2-3 minutes (vs 20 min before!)

Infrastructure Change Only

# Local
pwsh infrastructure/scripts/deploy-layer1.ps1 -Environment dev

# Time: 10-15 minutes (no app redeployment)

Migration Strategy

Phase 1: Create Bicep Orchestrators (Week 1) - Create layer1-foundation.bicep (refactor main-avm.bicep) - Create layer2-platform.bicep (new) - Create layer3-apps.bicep (new) - Create all-in-one.bicep (new) - Test with validation mode

Phase 2: Create PowerShell Modules (Week 2) - Create Common.psm1 with shared utilities - Create layer-specific modules - Create entry point scripts - Test locally in dev environment

Phase 3: Update GitHub Actions (Week 3) - Create layer-specific workflows - Test in dev environment via workflow dispatch - Validate identical behavior to local

Phase 4: Documentation & Deprecation (Week 4) - Update deployment documentation - Create usage guides - Add deprecation warnings to old scripts - Team training session

Rollback Plan

If issues discovered: 1. Revert workflow YAML to previous version (git revert) 2. Old scripts (deploy.sh, deploy-platform.sh) remain functional 3. Keep new Bicep/PowerShell for future retry 4. Document issues and address before retry

Alternatives Considered

Alternative 1: Keep Current Dual-Path Architecture

Approach: Keep deploy.sh and GitHub Actions separate, improve each independently

Pros: - No migration effort - Familiar to team - Each path optimized for its environment

Cons: - ❌ Maintains dual maintenance burden - ❌ Behavioral divergence risk remains - ❌ Different Bicep orchestration strategies - ❌ Testing complexity (two paths to validate) - ❌ No solution to monolithic deployment problem

Decision: Rejected - doesn't solve core problems

Alternative 2: Bash-First Approach (Keep Bash Scripts)

Approach: Improve deploy.sh, make GitHub Actions call bash scripts

Pros: - Familiar (current approach) - Works on Linux/macOS natively - Less PowerShell knowledge needed

Cons: - ❌ Bash calling PowerShell anyway (Azure requires PowerShell) - ❌ Cross-platform issues (GNU vs BSD tools) - ❌ PowerShell is Azure's native language - ❌ Harder to test and maintain - ❌ GitHub Actions on Linux would still call bash calling PowerShell (double wrapper)

Decision: Rejected - adds unnecessary complexity, PowerShell is native to Azure

Alternative 3: PowerShell Orchestrates Bicep Modules (Wrong!)

Approach: PowerShell modules directly deploy individual Bicep modules, handle dependencies

Pros: - Granular control in PowerShell - Can optimize deployment order - Easy to retry individual modules

Cons: - ❌ Violates IaC principles (orchestration in scripts, not Bicep) - ❌ Duplicates Bicep's dependency management - ❌ Infrastructure logic split between Bicep and PowerShell - ❌ Harder to test (logic in two places) - ❌ Loses Bicep's dependency graph capabilities

Decision: Rejected - violates core IaC principles

Alternative 4: Composite GitHub Actions (GitHub-Native)

Approach: Create composite GitHub Actions, call with act CLI locally

Pros: - GitHub-native reusability - Version controlled - Can call locally via act

Cons: - ❌ Requires act CLI for local testing - ❌ More GitHub-specific (less portable) - ❌ Composite Actions harder to test locally - ❌ Team less familiar with composite actions

Decision: Rejected - PowerShell modules simpler and more portable

Alternative 5: Container-Based Deployment (Advanced)

Approach: Docker container with all tools, identical locally and CI/CD

Pros: - Guaranteed environment consistency - All tools pre-installed - Works identically everywhere

Cons: - ❌ Container overhead - ❌ More complex setup - ❌ Authentication complexity (mounting credentials) - ❌ Slower than native execution - ❌ Overkill for our needs

Decision: Rejected - unnecessary complexity

Validation

Success Criteria

  • ✅ All 4 layer Bicep files deploy successfully
  • ✅ PowerShell modules work locally on Windows, macOS, Linux
  • ✅ GitHub Actions workflows work identically to local
  • ✅ Layer 3 deployment completes in <5 minutes
  • ✅ All-in-one deployment completes in ~20 minutes
  • ✅ Can deploy apps selectively (just API, just MCP, etc.)
  • ✅ Bicep linter passes for all layer files
  • ✅ PowerShell script analyzer passes for all modules
  • ✅ Documentation complete and accurate
  • ✅ Team successfully deploys using new approach

Testing Plan

Phase 1: Local Validation Mode Testing - Test layer1 validation mode (no actual deployment) - Test layer2 validation mode - Test layer3 validation mode - Test all-in-one validation mode

Phase 2: Dev Environment Testing - Deploy Layer 1 to dev - Deploy Layer 2 to dev - Deploy Layer 3 to dev - Test selective app deployment (just API) - Test all-in-one deployment to new environment

Phase 3: GitHub Actions Testing - Test all workflows in dev environment - Verify identical behavior to local - Test failure scenarios and error handling

Notes

  • This ADR supersedes ADR-030 (Three-Workflow Architecture) with the refined 4-layer approach
  • Bicep incremental mode still applies - deploying unchanged layer is fast
  • Layer separation is logical (separation of concerns) AND performance (selective deployment)
  • GitHub Actions visualization can be maintained by structuring workflows with multiple steps that call PowerShell functions
  • Old scripts (deploy.sh, deploy-platform.sh) will be deprecated but kept functional during transition
  • Migration is low-risk - old approach continues working during parallel implementation

Status: Accepted
Next Actions: 1. Create layer1-foundation.bicep ✅ (Completed 2025-10-17) 2. Create layer2-platform.bicep 3. Create layer3-apps.bicep 4. Create all-in-one.bicep 5. Create PowerShell modules 6. Update GitHub Actions workflows 7. Update documentation 8. Team training

Last Updated: 2025-10-17