Skip to navigation
Account Payable to Fluxx HLD

Complete build and deploy web api documents ai to UAT and PROD

Background  

Currently, the build-deploy.yml workflow deploys to:

  • SBX (Sandbox)  
  • DEV (Development)  
  • QA (Quality Assurance)  

We need to extend this to support UAT (User Acceptance Testing) and PROD (Production) environments with appropriate approval gates, terraform infrastructure validation, and production-grade configurations.

Acceptance Criteria

1. Terraform Infrastructure Verification (DevOps)

  • AC 1.1: Verify terraform state exists for UAT environment resources:

  - AKS cluster: uat-eus2-adc-aks-001

  - Resource group: uat-eus2-adc-app-rg-001

  - SQL Database: uat-eus2-adc-sql-002.database.windows.net

  - Storage accounts: uateus2adcinst001, uateus2adcdocst001

  - User Assigned Managed Identity for workload identity

  - Azure OpenAI endpoint (if environment-specific)

  - Azure Form Recognizer endpoint (if environment-specific)

  • AC 1.2: Verify terraform state exists for PROD environment resources (DevOps)

  - AKS cluster: prod-eus2-adc-aks-001

  - Resource group: prod-eus2-adc-app-rg-001

  - SQL Database: prod-eus2-adc-sql-002.database.windows.net

  - Storage accounts: prodeus2adcinst001, prodeus2adcdocst001

  - User Assigned Managed Identity for workload identity

  - Azure OpenAI endpoint (production instance)

  - Azure Form Recognizer endpoint (production instance)

  • AC 1.3: Document required terraform modules/resources in a terraform/README.md file

  • AC 1.4: Ensure all infrastructure resources have appropriate tags (environment, application, cost-center, etc.)

2. Helm Chart Configuration Files (DevOps)

  • AC 2.1: Create helm/values-uat.yaml with UAT-specific configurations:

  - Environment set to ‘uat’

  - Appropriate replica count (2+ for high availability)

  - UAT-specific resource limits and requests

  - UAT SQL database connection string

  - UAT storage account endpoints

  - UAT Azure AD client ID

  - UAT workload identity client ID

  - Istio host: documents-ai-api.uat.clait.com (or appropriate domain)

  - Autoscaling configuration (min: 2, max: 15)

  • AC 2.2: Create helm/values-prod.yaml with PROD-specific configurations:

  - Environment set to ‘prod’

  - Higher replica count (3+ for high availability)

  - Production resource limits and requests (higher than UAT)

  - PROD SQL database connection string

  - PROD storage account endpoints

  - PROD Azure AD client ID

  - PROD workload identity client ID

  - Istio host: documents-ai-api.clait.com (or appropriate production domain)

  - Autoscaling configuration (min: 3, max: 20)

  - Production-grade monitoring and alerting enabled

  • AC 2.3: Remove hardcoded API keys and secrets from values files

  - Integrate with Azure Key Vault for secrets management

  - Use Key Vault references for:

    - AZURE_OPENAI_API_KEY

    - AZURE_FR_KEY

  - Document Key Vault setup requirements

  • AC 2.4: Enable production monitoring in helm values:

  - monitoring.serviceMonitor.enabled: true

  - Configure appropriate Prometheus scraping intervals

  - Set up Grafana dashboards (if applicable)

3. GitHub Actions Workflow Updates (DevOps)

  • AC 3.1: Add UAT deployment job to .github/workflows/build-deploy.yml:

  - Job name: package-deploy-uat

  - Runs after successful package-deploy-qa job

  - Requires manual approval via GitHub environment protection rules

  - Environment: uat

  - Uses correct AKS cluster and resource group naming

  - Deploys to doc-ai namespace

  • AC 3.2: Add PROD deployment job to .github/workflows/build-deploy.yml:

  - Job name: package-deploy-prod

  - Runs after successful package-deploy-uat job

  - Requires manual approval via GitHub environment protection rules

  - Environment: prod

  - Uses correct AKS cluster and resource group naming

  - Deploys to doc-ai namespace (or production-specific namespace)

  - Includes smoke tests before marking deployment as complete

  • AC 3.3: Configure deployment conditions:

  - UAT deployment only triggered from main branch

  - PROD deployment only triggered from main branch

  - Both require explicit approval from authorized personnel

  • AC 3.4: Add deployment verification steps:

  - Health check validation after deployment

  - Readiness probe verification

  - API endpoint smoke test

  - Rollback procedure documented in case of failure

4. GitHub Environment Configuration (DevOps)

  • AC 4.1: Create GitHub environment uat with:

  - Required reviewers (minimum 1 DevOps/Lead Engineer)

  - Environment secrets:

    - AZURE_CLIENT_ID

    - AZURE_TENANT_ID

    - AZURE_SUBSCRIPTION_ID

    - ACR_SUBSCRIPTION_ID

  - Environment variables:

    - ACR_NAME

    - AZURE_CLIENT_ID

    - AZURE_TENANT_ID

    - ACR_SUBSCRIPTION_ID

  - Protection rules: Require manual approval

  • AC 4.2: Create GitHub environment prod with:

  - Required reviewers (minimum 2 approvers: 1 DevOps + 1 Lead/Manager)

  - Environment secrets (same as UAT but PROD-specific values)

  - Environment variables (same as UAT but PROD-specific values)

  - Protection rules:

    - Require manual approval

    - Wait timer: 5 minutes (optional, for review period)

    - Restrict to main branch only

5. Service Principal & RBAC Configuration (DevOps)

  • AC 5.1: Ensure service principal has required permissions in UAT subscription:

  - AKS Cluster User role on AKS cluster

  - ACR Pull role on container registry

  - Contributor role on resource group (or specific resource-level permissions)

  • AC 5.2: Ensure service principal has required permissions in PROD subscription:

  - AKS Cluster User role on AKS cluster

  - ACR Pull role on container registry

  - Contributor role on resource group (or specific resource-level permissions)

  - Document separate PROD service principal if required by security policy

  • AC 5.3: Configure workload identity federation:

  - UAT managed identity federated credentials for GitHub Actions

  - PROD managed identity federated credentials for GitHub Actions

  - Validate identity token exchange works correctly

6. Documentation & Runbooks

  • AC 6.1: Create deployment runbook documentation:

  - Pre-deployment checklist

  - Approval process workflow

  - Rollback procedures

  - Monitoring and alerting setup

  - Incident response contacts

  • AC 6.2: Update main README.md with:

  - Deployment pipeline overview diagram

  - Environment promotion flow (DEV → QA → UAT → PROD)

  - Required approvers for each environment

  - Links to deployment runbooks

  • AC 6.3: Create DEPLOYMENT.md file documenting:

  - How to trigger deployments

  - How to approve deployments

  - How to monitor deployment status

  - How to verify deployment health

  - How to perform rollbacks

  - Emergency procedures

  • AC 6.4: Document terraform infrastructure requirements:

  - All required resources per environment

  - Naming conventions

  - Required terraform variables

  - Backend configuration

7. Testing & Validation

  • AC 7.1: Validate UAT deployment:

  - Deploy successfully to UAT environment

  - Verify pods are running and healthy

  - Test API endpoints respond correctly

  - Verify database connectivity

  - Verify Azure OpenAI integration works

  - Verify Form Recognizer integration works

  - Verify blob storage access works

  - Check logs for errors

  • AC 7.2: Validate PROD deployment:

  - Deploy successfully to PROD environment (with approval)

  - Verify pods are running and healthy

  - Test API endpoints respond correctly

  - Verify database connectivity

  - Verify Azure OpenAI integration works

  - Verify Form Recognizer integration works

  - Verify blob storage access works

  - Check logs for errors

  - Monitor for 24 hours post-deployment

  • AC 7.3: Validate rollback procedure:

  - Document and test rollback to previous version

  - Verify rollback can be completed within RTO (Recovery Time Objective)

  - Validate data consistency after rollback

8. Security & Compliance

  • AC 8.1: Security scanning enabled for production:

  - Enable Veracode scanning (currently commented out)

  - Container vulnerability scanning

  - Dependencies security audit

  - Secrets scanning validation

  • AC 8.2: Network policies configured:

  - Appropriate Istio authorization policies for UAT/PROD

  - Network policies for namespace isolation

  - Ingress/egress rules documented

  • AC 8.3: Compliance validation:

  - All production secrets stored in Key Vault (not in helm values)

  - Audit logging enabled

  - Compliance with organizational security policies

9. Monitoring & Observability

  • AC 9.1: Configure application monitoring for UAT/PROD:

  - Application Insights or equivalent APM

  - Custom metrics for API endpoints

  - Performance monitoring

  - Error rate tracking

  • AC 9.2: Configure infrastructure monitoring:

  - AKS cluster metrics

  - Pod resource utilization

  - Node health monitoring

  - Database connection pool monitoring

  • AC 9.3: Set up alerting:

  - High error rate alerts

  - Performance degradation alerts

  - Resource exhaustion alerts

  - Deployment failure notifications

10. Disaster Recovery

  • AC 10.1: Document backup strategy:

  - Database backup schedule

  - Blob storage backup/replication

  - Helm chart versioning strategy

  - Container image retention policy

  • AC 10.2: Define and document:

  - Recovery Time Objective (RTO)

  - Recovery Point Objective (RPO)

  - Disaster recovery procedures

  - Multi-region failover (if applicable)

Technical Notes

Current Pipeline Flow

Build Container → Package Helm Charts → Deploy to SBX/DEV (parallel) → Deploy to QA

Target Pipeline Flow

Build Container → Package Helm Charts →  
  Deploy to SBX/DEV (parallel) →  
    Deploy to QA →  
      Deploy to UAT (manual approval) →  
        Deploy to PROD (manual approval + stricter gates)

Environment Naming Convention

  • AKS Cluster: {env}-{region}-adc-aks-001

  • Resource Group: {env}-{region}-adc-app-rg-001

  • SQL Server: {env}-{region}-adc-sql-002.database.windows.net

  • Storage: {env}{region}adc{type}st001

Where:

  • env = sbx, dev, qa, uat, prod

  • region = eus2 (East US 2)

  • type = inst (instance), doc (documents)

Dependencies

  1. Infrastructure Team: Provision UAT and PROD Azure resources via Terraform

  2. Security Team: Approve service principal permissions and Key Vault setup

  3. Database Team: Set up UAT and PROD database instances with proper schemas

  4. Networking Team: Configure DNS entries for UAT/PROD domains

  5. Platform Team: Create GitHub environments and configure protection rules

Definition of Done

  • All acceptance criteria are met

  • Code changes peer-reviewed and approved

  • Documentation is complete and reviewed

  • UAT deployment successful with manual validation

  • PROD deployment successful with manual validation

  • Rollback procedure tested and documented

  • Monitoring and alerting configured and validated

  • Runbooks created and reviewed by operations team

  • Security review completed

  • Sign-off from Lead Engineer and Product Owner

Estimated Effort

  • Infrastructure Setup: 8-16 hours (depends on terraform state and existing resources)

  • Pipeline Development: 4-8 hours

  • Testing & Validation: 8-12 hours

  • Documentation: 4-6 hours

  • Security Review: 2-4 hours

Total Estimate: 26-46 hours (3-6 sprints depending on team capacity and dependencies)

Risks & Mitigations

| Risk | Impact | Probability | Mitigation |

|------|--------|-------------|------------|

| Terraform state drift | High | Medium | Audit existing infrastructure, reconcile state before deployment |

| Missing infrastructure resources | High | Medium | Complete infrastructure audit before starting development |

| Service principal permission issues | Medium | High | Test permissions in lower environments first |

| Secrets not in Key Vault | High | Low | Prioritize Key Vault integration before PROD deployment |

| Insufficient approval process | Medium | Medium | Define clear approval workflow with stakeholders upfront |

| Production data exposure | High | Low | Use production-specific secrets, never reuse lower environment credentials |

Success Metrics

  • Successful deployment to UAT on first attempt

  • Successful deployment to PROD with zero downtime

  • Rollback procedure tested and documented

  • All security scans pass

  • Zero production incidents in first 30 days post-deployment