ADR-054: Application Gateway in Substrate Layer (Not Foundation)

Status: Accepted

Date: 2025-01-28

Context: Application Gateway deployment layer decision during CORS fix implementation

Context

The Problem

While implementing Application Gateway for public HTTPS access to the UI and API, an architectural question arose: Which deployment layer should Application Gateway belong to?

Initial implementation placed Application Gateway in the Foundation layer alongside VNet, subnets, and Azure Bastion. However, this created a problematic coupling between core networking infrastructure and platform ingress services.

The Challenge Question

User feedback that triggered architectural correction:

"Wait - I don't understand why is application gateway deployed as part of foundation. Suppose tomorrow I move to container apps then you will want me to deploy foundation again? Should it not be part of higher layers, I think more like substrate?"

This question exposed a fundamental architectural flaw: Foundation layer should contain immutable core infrastructure, not platform services that may change based on deployment strategy.

4-Layer Architecture Context

The infrastructure uses a frequency-based layering strategy:

Layer	Purpose	Change Frequency	Resources
Foundation	Core infrastructure	Once per environment (rarely)	VNet, Subnets, NSG, Bastion, Identity, Key Vault
Substrate	Platform services	Occasional (platform upgrades)	ACR, AI Foundry Hub/Project
AI Models	Model deployments	Weekly/monthly (model testing)	GPT-4o, GPT-4o-mini
Apps	Application containers	Daily/hourly (code changes)	ACI containers (API, UI, MCP)

Key Principle: Lower layers should not depend on upper layer decisions. Foundation should not need redeployment if application deployment strategy changes (ACI → Container Apps → AKS).

Decision

Application Gateway belongs in the Substrate layer, not Foundation.

Rationale

1. Separation of Concerns

Foundation = Core Networking - VNet and subnet definitions - Network Security Groups (NSGs) - Azure Bastion for secure access - Managed Identity - Key Vault

These resources never change regardless of application deployment strategy.

Substrate = Platform Services - Azure Container Registry (ACR) - platform for container images - AI Foundry Hub/Project - platform for AI models - Application Gateway - platform for public HTTPS ingress

These resources are platform-level decisions that may change as deployment strategies evolve.

2. Change Frequency Alignment

Scenario	Foundation Impact	Substrate Impact
Switch ACI → Container Apps	✅ No change	⚠️ May need ingress reconfiguration
Switch Container Apps → AKS	✅ No change	⚠️ May need ingress reconfiguration
Remove Application Gateway	✅ No change	⚠️ Update Substrate deployment
Change WAF rules	✅ No change	⚠️ Update App Gateway config

Conclusion: Application Gateway changes align with Substrate layer frequency, not Foundation.

3. Dependency Direction

Correct (Substrate depends on Foundation):

Foundation (VNet, Subnets)
    ↓ provides Gateway subnet
Substrate (App Gateway)
    ↓ provides public endpoint
Apps (ACI containers)

Incorrect (Foundation depends on App deployment strategy):

Foundation (VNet + App Gateway) ← Would need redeployment if app strategy changes
    ↓
Substrate (ACR, AI Foundry)
    ↓
Apps (ACI containers)

4. Gateway Subnet Remains in Foundation

Important distinction: - Gateway subnet = Foundation (part of VNet definition) - Application Gateway resource = Substrate (uses the subnet)

This is correct because: - VNet and subnet structure is core networking (Foundation) - Whether Application Gateway is deployed is a platform decision (Substrate) - Subnet exists regardless of whether App Gateway is deployed - Other ingress options (Azure Front Door, APIM) could use the same subnet

Implementation

Changes Made

1. Remove from Foundation Layer

File: infrastructure/bicep/foundation.bicep - ❌ Removed deployAppGateway parameter - ✅ Kept gatewaySubnetPrefix parameter (subnet is core networking) - ❌ Removed Application Gateway Public IP resource - ❌ Removed Application Gateway module deployment - ❌ Removed Application Gateway outputs - ✅ Added appGatewaySubnetId output (for Substrate to consume)

File: infrastructure/bicep/environments/dev/foundation.bicepparam - ❌ Removed deployAppGateway parameter - ✅ Kept gatewaySubnetPrefix parameter

2. Add to Substrate Layer

File: infrastructure/bicep/substrate.bicep - ✅ Added deployAppGateway parameter (default: false) - ✅ Added gatewaySubnetId parameter (from Foundation outputs) - ✅ Added appGatewaySku, appGatewayCapacity, enableWaf parameters - ✅ Added Application Gateway Public IP resource (conditional) - ✅ Added Application Gateway module deployment (conditional) - ✅ Added Application Gateway outputs (conditional)

File: infrastructure/bicep/environments/dev/substrate.bicepparam - ✅ Added deployAppGateway = false (disabled by default for dev) - ✅ Added gatewaySubnetId = 'placeholder' (passed by deployment script) - ✅ Added appGatewaySku = 'Standard_v2' (dev uses cheaper SKU) - ✅ Added appGatewayCapacity = 1 (dev uses single instance) - ✅ Added enableWaf = false (dev doesn't need WAF)

3. Networking Module (No Changes Needed)

File: infrastructure/bicep/modules/networking.bicep - ✅ Gateway subnet creation already implemented - ✅ Output appGatewaySubnetId already exists - ✅ Foundation layer passes deployAppGateway: true to always create Gateway subnet

Rationale: Gateway subnet is part of core networking design (Foundation), even if Application Gateway isn't deployed yet (Substrate decision).

4. Application Gateway Module (Already Created)

File: infrastructure/bicep/modules/app-gateway.bicep - ✅ Module created with path-based routing (/ → UI, /api/* → API) - ✅ Empty backend pools initially (populated by Apps layer) - ✅ Health probes for UI and API endpoints - ✅ Conditional WAF configuration

Deployment Flow

Foundation Deployment:

./infrastructure/scripts/deploy-foundation.sh dev ldfdev-rg

Creates VNet with all subnets (including Gateway subnet)
Outputs appGatewaySubnetId
Stores subnet ID in Key Vault for Substrate to retrieve

Substrate Deployment:

./infrastructure/scripts/deploy-substrate.sh dev ldfdev-rg

Retrieves appGatewaySubnetId from Foundation outputs (Key Vault)
If deployAppGateway = true:
- Deploys Public IP
- Deploys Application Gateway with empty backend pools
Outputs Application Gateway ID and endpoint

Apps Deployment:

./infrastructure/scripts/deploy-apps.sh dev ldfdev-rg

Deploys ACI containers (API, UI, MCP servers)
Gets ACI private IP address
Updates Application Gateway backend pools with ACI IP
Backend pool update happens AFTER ACI deployment

Consequences

Positive

✅ Architectural Clarity - Clear separation: Foundation = core infrastructure, Substrate = platform services - Layer responsibilities align with change frequency - No circular dependencies between layers

✅ Deployment Flexibility - Can change application deployment strategy (ACI → Container Apps) without redeploying Foundation - Can enable/disable Application Gateway without touching core networking - Foundation deployments are truly "deploy once" infrastructure

✅ Cost Optimization - Dev environments can disable Application Gateway by default (saves ~$125-250/month) - Only enable when testing public access scenarios - No wasted resources in Foundation layer

✅ Correct Dependency Direction - Foundation provides VNet + subnets (no dependencies on upper layers) - Substrate consumes Foundation outputs (correct: lower → upper) - Apps consume Substrate outputs (correct: lower → upper)

✅ Gateway Subnet Preserved - Gateway subnet always exists in Foundation (part of core network design) - Can be used by Application Gateway, Azure Front Door, or APIM - No need to modify Foundation if ingress strategy changes

Negative

⚠️ Deployment Script Complexity - deploy-substrate.sh must retrieve appGatewaySubnetId from Foundation outputs - deploy-apps.sh must update Application Gateway backend pools after ACI deployment - Two-pass deployment: (1) deploy App Gateway, (2) update backend pools

⚠️ Parameter Passing - gatewaySubnetId must be passed from Foundation → Substrate - Requires Key Vault storage of Foundation outputs (already implemented via ADR-048)

Neutral

🔄 Gateway Subnet Location - Gateway subnet defined in Foundation layer (part of VNet) - Application Gateway resource in Substrate layer (uses the subnet) - This is correct architecture but requires understanding the distinction

🔄 Migration for Existing Deployments - Existing deployments with Application Gateway in Foundation must: 1. Redeploy Foundation (removes App Gateway) 2. Deploy Substrate with deployAppGateway = true 3. Update backend pools via Apps deployment - One-time migration cost for correct architecture

Validation

Bicep Compilation

Both bicep files compile successfully:

$ az bicep build --file infrastructure/bicep/foundation.bicep
✅ SUCCESS - Warnings only (unused parameters, null references)

$ az bicep build --file infrastructure/bicep/substrate.bicep
✅ SUCCESS - Warnings only (unused parameters, null references)

No errors - architectural changes are syntactically correct.

Layer Responsibilities

Layer	Networking	Platform Services	Applications
Foundation	✅ VNet, Subnets, NSG, Bastion	❌	❌
Substrate	❌ (consumes Foundation)	✅ ACR, AI Foundry, App Gateway	❌
Apps	❌ (consumes Foundation)	❌ (consumes Substrate)	✅ ACI containers

Result: Clear layer boundaries with no architectural violations.

ADR-049: Container Deployment Strategy (ACI vs Container Apps) - Why ACI was chosen
ADR-050: Azure Bastion Replaces VPN Gateway - Foundation layer access strategy
ADR-051: Infrastructure Simplification (KISS) - Bash-only deployment scripts
ADR-052: Layer Renaming and AI Foundry Reorganization - Semantic layer names
ADR-053: Infrastructure Code Consolidation - DRY principles for deployment scripts

Implementation Status

Completed: - ✅ Removed Application Gateway from foundation.bicep - ✅ Removed Application Gateway params from foundation.bicepparam - ✅ Added Gateway subnet output to Foundation layer - ✅ Added Application Gateway to substrate.bicep - ✅ Added Application Gateway params to substrate.bicepparam - ✅ Both bicep files compile successfully

Pending: - ⏳ Update deploy-foundation.sh to store appGatewaySubnetId in Key Vault - ⏳ Update deploy-substrate.sh to retrieve appGatewaySubnetId and pass to deployment - ⏳ Update deploy-apps.sh to update Application Gateway backend pools after ACI deployment - ⏳ Test end-to-end deployment with Application Gateway enabled - ⏳ Document Application Gateway backend pool update process

Decision Makers

Architect: User (identified architectural flaw in initial placement)
Implementer: Claude Code (moved Application Gateway to Substrate)
Validator: Azure Bicep compiler (syntax validation)

Lessons Learned

1. User Feedback is Architectural Gold

The user's question "suppose tomorrow I move to container apps then you will want me to deploy foundation again?" immediately exposed the flaw in placing Application Gateway in Foundation layer.

Key Insight: When a user asks "what if I change X?", test whether the current architecture supports that change gracefully. If not, the layer boundaries are wrong.

2. Foundation Should Be Immutable

Foundation layer should contain only resources that: - Never change regardless of application deployment strategy - Are required by all possible deployment scenarios - Represent "deploy once" infrastructure

Application Gateway fails all three criteria.

3. Subnet ≠ Service That Uses Subnet

Important distinction: - Subnet definition = Core networking (Foundation) - Service using subnet = Platform service (Substrate) or application (Apps)

Always create subnet in Foundation, even if service isn't deployed yet. This preserves flexibility.

4. Change Frequency Drives Layer Assignment

When in doubt about layer assignment, ask: - "How often will this resource change?" - "What causes this resource to be redeployed?" - "Does this resource depend on application deployment strategy?"

Application Gateway changes for: - WAF rule updates (platform configuration) - SKU changes (platform optimization) - Ingress strategy changes (ACI → Container Apps → AKS)

All of these are Substrate-level concerns, not Foundation.

5. Architectural Mistakes Are Caught Early

User caught this architectural flaw before any deployment, during code review.

Cost of fixing: - During design (now): 30 minutes to move resources, zero infrastructure impact - After deployment: Would require Foundation redeployment, potential downtime, migration complexity - In production: Would require careful migration planning, risk analysis, change windows

Conclusion: Architecture reviews and "what if?" questions are invaluable.

Future Considerations

1. Alternative Ingress Options

Application Gateway placement in Substrate makes it easy to switch ingress strategies:

Strategy	Substrate Changes	Foundation Changes
Application Gateway	✅ Current implementation	✅ None
Azure Front Door	⚠️ Replace App Gateway module	✅ None
API Management	⚠️ Add APIM module	✅ None (APIM subnet exists)
Nginx Ingress (AKS)	⚠️ Remove App Gateway	✅ None

Benefit: Foundation layer remains stable regardless of ingress choice.

2. Multi-Region Deployment

For multi-region deployments: - Each region has its own Foundation (VNet, subnets) - Substrate can deploy Application Gateway per region - Global load balancing via Azure Front Door (separate Substrate module)

Application Gateway in Substrate supports this pattern naturally.

3. Environment-Specific Configuration

Environment	App Gateway Config	Cost
Dev	Disabled by default	$0/month
Test	Standard_v2, 1 instance	~$125/month
Staging	WAF_v2, 2 instances	~$500/month
Prod	WAF_v2, 3+ instances + autoscale	~$750+/month

Substrate layer allows per-environment Application Gateway configuration without touching Foundation.

References

Code Locations

Foundation layer: infrastructure/bicep/foundation.bicep
Substrate layer: infrastructure/bicep/substrate.bicep
App Gateway module: infrastructure/bicep/modules/app-gateway.bicep
Networking module: infrastructure/bicep/modules/networking.bicep
Parameter files: infrastructure/bicep/environments/{env}/

Documentation

4-Layer Architecture: CLAUDE.md (search "4-Layer Infrastructure Architecture")
Deployment guide: docs/deployment/direct-azure-deployment.md
ADR index: docs/architecture/decisions/README.md

End of ADR-054