Key Vault for Cross-Layer Deployment Outputs

✅ Implemented (ADR-048): Key Vault stores deployment outputs for fast cross-layer dependency retrieval.

Quick Summary: Each deployment layer stores its outputs in Azure Key Vault, allowing subsequent layers to retrieve dependencies in <5 seconds instead of 30-60 seconds via deployment queries.

Overview

The 4-layer deployment architecture requires passing outputs from one layer to the next: - Foundation → outputs VNet ID, subnet IDs, Key Vault name - Substrate → outputs ACR name, AI Foundry project name
- AI Models → outputs model deployment names - Apps → final deployment

Problem: Querying Azure deployment outputs is slow (30-60 seconds per query).

Solution (ADR-048): Store outputs as Key Vault secrets for instant retrieval.

How It Works

1. Foundation Layer (Stores Outputs)

./deploy-foundation.sh dev

# Deployment completes, script stores:
az keyvault secret set --vault-name kv-ldfdev \
  --name "foundation-vnetId" \
  --value "/subscriptions/.../vnet/ldfdev-vnet"

az keyvault secret set --vault-name kv-ldfdev \
  --name "foundation-subnetIds-compute" \
  --value "/subscriptions/.../subnets/compute-subnet"

# ... more outputs

Stored secrets: - foundation-vnetId - foundation-subnetIds-compute - foundation-subnetIds-data - foundation-subnetIds-bastion - foundation-keyVaultName - foundation-managedIdentityId - foundation-appInsightsName - foundation-logAnalyticsWorkspaceId

2. Substrate Layer (Retrieves + Stores)

./deploy-substrate.sh dev

# Script retrieves Foundation outputs from Key Vault:
vnetId=$(az keyvault secret show --vault-name kv-ldfdev \
  --name "foundation-vnetId" --query value -o tsv)

# Deployment completes, stores Substrate outputs:
az keyvault secret set --vault-name kv-ldfdev \
  --name "substrate-acrName" \
  --value "acrldfdev"

az keyvault secret set --vault-name kv-ldfdev \
  --name "substrate-aiProjectName" \
  --value "ldfdev-ai-foundry-project"

Stored secrets: - substrate-acrName - substrate-acrLoginServer - substrate-aiProjectName

3. AI Models Layer (Retrieves)

./deploy-ai-models.sh dev

# Retrieves Substrate outputs:
aiProjectName=$(az keyvault secret show --vault-name kv-ldfdev \
  --name "substrate-aiProjectName" --query value -o tsv)

# Uses aiProjectName to deploy models

4. Apps Layer (Retrieves)

./deploy-apps.sh dev

# Retrieves outputs from all previous layers:
vnetId=$(get from Key Vault)
acrName=$(get from Key Vault)
aiProjectName=$(get from Key Vault)

# Deploys ACI container group with all dependencies

Key Naming Convention

Format: {layer}-{outputName}

Examples:

foundation-vnetId
foundation-subnetIds-compute
foundation-keyVaultName
substrate-acrName
substrate-acrLoginServer
substrate-aiProjectName
ai-models-gpt4oDeploymentName
ai-models-gpt4oMiniDeploymentName

Rules: - Lowercase with hyphens - Layer prefix matches script name (foundation, substrate, ai-models, apps) - Output name describes the value (vnetId, acrName, etc.)

Fallback Mechanism

All scripts implement a fallback to deployment queries if Key Vault retrieval fails:

# Try Key Vault first (fast: <5 seconds)
vnetId=$(az keyvault secret show --vault-name $keyVaultName \
  --name "foundation-vnetId" --query value -o tsv 2>/dev/null)

# Fallback to deployment query if Key Vault failed (slow: 30-60 seconds)
if [ -z "$vnetId" ]; then
  echo "⚠️  Key Vault retrieval failed, querying deployment outputs..."
  vnetId=$(az deployment group show \
    --resource-group $resourceGroup \
    --name foundation-deployment \
    --query properties.outputs.vnetId.value -o tsv)
fi

Why fallback exists: - First deployment (Key Vault not populated yet) - Key Vault access issues - Key Vault accidentally deleted - Manual deployment without Key Vault

Deployment Method Comparison

Bash Scripts (✅ Using Key Vault)

Location: infrastructure/scripts/deploy-*.sh

Key Vault Integration: - ✅ Stores outputs after deployment - ✅ Retrieves outputs before deployment - ✅ Automatic fallback to deployment queries - ✅ <5 second retrieval time

Example:

cd infrastructure/scripts
./deploy-foundation.sh dev    # Stores outputs
./deploy-substrate.sh dev     # Retrieves from KV, stores outputs
./deploy-ai-models.sh dev     # Retrieves from KV
./deploy-apps.sh dev          # Retrieves from KV

GitHub Actions (❌ NOT Using Key Vault Yet)

Location: .github/workflows/deploy-*.yml

Current Behavior: - ❌ Does NOT use Key Vault - ❌ Queries deployment outputs directly (slow) - ❌ Uses PowerShell instead of Bash - ⏱️ 30-60 seconds per dependency query

Why: - Workflows use inline PowerShell deployment - Don't call Bash scripts with Key Vault logic - Service principal may need Key Vault permissions

Workaround: GitHub Actions still work, just slower due to deployment queries.

Future Improvement: Update workflows to call Bash scripts (inherit Key Vault integration).

Manual Key Vault Queries

View all deployment outputs:

# Set variables
keyVaultName="kv-ldfdev"

# List all secrets
az keyvault secret list --vault-name $keyVaultName \
  --query "[].name" -o table

# Get specific output
az keyvault secret show --vault-name $keyVaultName \
  --name "foundation-vnetId" --query value -o tsv

# Get all Foundation outputs
az keyvault secret list --vault-name $keyVaultName \
  --query "[?starts_with(name, 'foundation-')].{Name:name}" -o table

View outputs by layer:

# Foundation outputs
az keyvault secret list --vault-name $keyVaultName \
  --query "[?starts_with(name, 'foundation-')].[name]" -o tsv

# Substrate outputs
az keyvault secret list --vault-name $keyVaultName \
  --query "[?starts_with(name, 'substrate-')].[name]" -o tsv

# AI Models outputs
az keyvault secret list --vault-name $keyVaultName \
  --query "[?starts_with(name, 'ai-models-')].[name]" -o tsv

Troubleshooting

Key Vault secrets not found

Symptom: Script falls back to deployment queries

Causes: 1. First deployment (Key Vault not populated) 2. Key Vault access denied 3. Wrong Key Vault name

Solution:

# Verify Key Vault exists
az keyvault show --name kv-ldfdev

# Check your access
az keyvault secret list --vault-name kv-ldfdev

# Manually populate if needed (after Foundation deployment)
az keyvault secret set --vault-name kv-ldfdev \
  --name "foundation-vnetId" \
  --value "$(az deployment group show ...)"

Access denied to Key Vault

Symptom: Forbidden error when accessing secrets

Solution:

# Check access policies
az keyvault show --name kv-ldfdev --query properties.accessPolicies

# Add yourself (if needed)
az keyvault set-policy --name kv-ldfdev \
  --upn your-email@domain.com \
  --secret-permissions get list set

# For service principal (GitHub Actions)
az keyvault set-policy --name kv-ldfdev \
  --spn <service-principal-id> \
  --secret-permissions get list set

Outdated values in Key Vault

Symptom: Deployment uses old resource IDs

Solution:

# Delete old secrets
az keyvault secret delete --vault-name kv-ldfdev \
  --name "substrate-acrName"

# Redeploy layer to repopulate
./deploy-substrate.sh dev

Performance Comparison

Without Key Vault (Deployment Queries)

# Query 5 outputs from previous layer
vnetId=$(az deployment group show ...)        # 30-60 seconds
subnetId=$(az deployment group show ...)      # 30-60 seconds
acrName=$(az deployment group show ...)       # 30-60 seconds
aiProject=$(az deployment group show ...)     # 30-60 seconds
managedId=$(az deployment group show ...)     # 30-60 seconds

# Total: 150-300 seconds (2.5-5 minutes) just for queries

With Key Vault (Current)

# Query 5 outputs from Key Vault
vnetId=$(az keyvault secret show ...)         # 1 second
subnetId=$(az keyvault secret show ...)       # 1 second
acrName=$(az keyvault secret show ...)        # 1 second
aiProject=$(az keyvault secret show ...)      # 1 second
managedId=$(az keyvault secret show ...)      # 1 second

# Total: 5 seconds

Improvement: 30-60x faster! 🚀

Security Considerations

Access Control

Key Vault Access Policies: - Developers: get, list (read-only) - Deployment scripts: get, list, set (read-write) - Service principals: get, list (GitHub Actions: read-only currently)

Best Practices: - ✅ Use managed identities where possible - ✅ Limit secrets to deployment outputs only - ✅ Don't store application secrets here (use separate Key Vault) - ✅ Audit log enabled (track who accessed what)

Secrets vs. Deployment Outputs

What goes in Key Vault (ADR-048): - ✅ Resource IDs (VNet, subnet, ACR) - ✅ Resource names (AI project, managed identity) - ✅ Non-sensitive deployment outputs

What does NOT go here: - ❌ Application secrets (API keys, connection strings) - ❌ User credentials - ❌ Certificates (use dedicated Key Vault)

ADR-048: Key Vault for Deployment Outputs - Decision rationale
Direct Azure Deployment Layers - Layer-by-layer guide
Azure Deployment Architecture - System overview

See Also: - Bash Scripts README - Script documentation - GitHub Actions Deployment - CI/CD workflows

Key Vault for Cross-Layer Deployment Outputs

Overview

How It Works

1. Foundation Layer (Stores Outputs)

2. Substrate Layer (Retrieves + Stores)

3. AI Models Layer (Retrieves)

4. Apps Layer (Retrieves)

Key Naming Convention

Fallback Mechanism

Deployment Method Comparison

Bash Scripts (✅ Using Key Vault)

GitHub Actions (❌ NOT Using Key Vault Yet)

Manual Key Vault Queries

View all deployment outputs:

View outputs by layer:

Troubleshooting

Key Vault secrets not found

Access denied to Key Vault

Outdated values in Key Vault

Performance Comparison

Without Key Vault (Deployment Queries)

With Key Vault (Current)

Security Considerations

Access Control

Secrets vs. Deployment Outputs

Related Documentation