> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://cla-demo.postmansolutions.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://cla-demo.postmansolutions.com/_mcp/server. # Complete build and deploy web api documents ai to UAT and PROD **Background** Currently, the build-deploy.yml workflow deploys to: * SBX (Sandbox) * DEV (Development) * QA (Quality Assurance) We need to extend this to support UAT (User Acceptance Testing) and PROD (Production) environments with appropriate approval gates, terraform infrastructure validation, and production-grade configurations. **Acceptance Criteria** ### 1. Terraform Infrastructure Verification (DevOps) * [ ] **AC 1.1**: Verify terraform state exists for UAT environment resources: \- AKS cluster: `uat-eus2-adc-aks-001` \- Resource group: `uat-eus2-adc-app-rg-001` \- SQL Database: `uat-eus2-adc-sql-002.database.windows.net` \- Storage accounts: `uateus2adcinst001`, `uateus2adcdocst001` \- User Assigned Managed Identity for workload identity \- Azure OpenAI endpoint (if environment-specific) \- Azure Form Recognizer endpoint (if environment-specific) * [ ] **AC 1.2**: Verify terraform state exists for PROD environment resources (DevOps) \- AKS cluster: `prod-eus2-adc-aks-001` \- Resource group: `prod-eus2-adc-app-rg-001` \- SQL Database: `prod-eus2-adc-sql-002.database.windows.net` \- Storage accounts: `prodeus2adcinst001`, `prodeus2adcdocst001` \- User Assigned Managed Identity for workload identity \- Azure OpenAI endpoint (production instance) \- Azure Form Recognizer endpoint (production instance) * [ ] **AC 1.3**: Document required terraform modules/resources in a `terraform/README.md` file * [ ] **AC 1.4**: Ensure all infrastructure resources have appropriate tags (environment, application, cost-center, etc.) ### 2. Helm Chart Configuration Files (DevOps) * [ ] **AC 2.1**: Create `helm/values-uat.yaml` with UAT-specific configurations: \- Environment set to 'uat' \- Appropriate replica count (2+ for high availability) \- UAT-specific resource limits and requests \- UAT SQL database connection string \- UAT storage account endpoints \- UAT Azure AD client ID \- UAT workload identity client ID \- Istio host: `documents-ai-api.uat.clait.com` (or appropriate domain) \- Autoscaling configuration (min: 2, max: 15) * [ ] **AC 2.2**: Create `helm/values-prod.yaml` with PROD-specific configurations: \- Environment set to 'prod' \- Higher replica count (3+ for high availability) \- Production resource limits and requests (higher than UAT) \- PROD SQL database connection string \- PROD storage account endpoints \- PROD Azure AD client ID \- PROD workload identity client ID \- Istio host: `documents-ai-api.clait.com` (or appropriate production domain) \- Autoscaling configuration (min: 3, max: 20) \- Production-grade monitoring and alerting enabled * [ ] **AC 2.3**: Remove hardcoded API keys and secrets from values files \- Integrate with Azure Key Vault for secrets management \- Use Key Vault references for: \- `AZURE_OPENAI_API_KEY` \- `AZURE_FR_KEY` \- Document Key Vault setup requirements * [ ] **AC 2.4**: Enable production monitoring in helm values: \- `monitoring.serviceMonitor.enabled: true` \- Configure appropriate Prometheus scraping intervals \- Set up Grafana dashboards (if applicable) ### 3. GitHub Actions Workflow Updates (DevOps) * [ ] **AC 3.1**: Add UAT deployment job to `.github/workflows/build-deploy.yml`: \- Job name: `package-deploy-uat` \- Runs after successful `package-deploy-qa` job \- Requires manual approval via GitHub environment protection rules \- Environment: `uat` \- Uses correct AKS cluster and resource group naming \- Deploys to `doc-ai` namespace * [ ] **AC 3.2**: Add PROD deployment job to `.github/workflows/build-deploy.yml`: \- Job name: `package-deploy-prod` \- Runs after successful `package-deploy-uat` job \- Requires manual approval via GitHub environment protection rules \- Environment: `prod` \- Uses correct AKS cluster and resource group naming \- Deploys to `doc-ai` namespace (or production-specific namespace) \- Includes smoke tests before marking deployment as complete * [ ] **AC 3.3**: Configure deployment conditions: \- UAT deployment only triggered from `main` branch \- PROD deployment only triggered from `main` branch \- Both require explicit approval from authorized personnel * [ ] **AC 3.4**: Add deployment verification steps: \- Health check validation after deployment \- Readiness probe verification \- API endpoint smoke test \- Rollback procedure documented in case of failure ### 4. GitHub Environment Configuration (DevOps) * [ ] **AC 4.1**: Create GitHub environment `uat` with: \- Required reviewers (minimum 1 DevOps/Lead Engineer) \- Environment secrets: \- `AZURE_CLIENT_ID` \- `AZURE_TENANT_ID` \- `AZURE_SUBSCRIPTION_ID` \- `ACR_SUBSCRIPTION_ID` \- Environment variables: \- `ACR_NAME` \- `AZURE_CLIENT_ID` \- `AZURE_TENANT_ID` \- `ACR_SUBSCRIPTION_ID` \- Protection rules: Require manual approval * [ ] **AC 4.2**: Create GitHub environment `prod` with: \- Required reviewers (minimum 2 approvers: 1 DevOps + 1 Lead/Manager) \- Environment secrets (same as UAT but PROD-specific values) \- Environment variables (same as UAT but PROD-specific values) \- Protection rules: \- Require manual approval \- Wait timer: 5 minutes (optional, for review period) \- Restrict to main branch only ### 5. Service Principal & RBAC Configuration (DevOps) * [ ] **AC 5.1**: Ensure service principal has required permissions in UAT subscription: \- AKS Cluster User role on AKS cluster \- ACR Pull role on container registry \- Contributor role on resource group (or specific resource-level permissions) * [ ] **AC 5.2**: Ensure service principal has required permissions in PROD subscription: \- AKS Cluster User role on AKS cluster \- ACR Pull role on container registry \- Contributor role on resource group (or specific resource-level permissions) \- Document separate PROD service principal if required by security policy * [ ] **AC 5.3**: Configure workload identity federation: \- UAT managed identity federated credentials for GitHub Actions \- PROD managed identity federated credentials for GitHub Actions \- Validate identity token exchange works correctly ### 6. Documentation & Runbooks * [ ] **AC 6.1**: Create deployment runbook documentation: \- Pre-deployment checklist \- Approval process workflow \- Rollback procedures \- Monitoring and alerting setup \- Incident response contacts * [ ] **AC 6.2**: Update main `README.md` with: \- Deployment pipeline overview diagram \- Environment promotion flow (DEV → QA → UAT → PROD) \- Required approvers for each environment \- Links to deployment runbooks * [ ] **AC 6.3**: Create `DEPLOYMENT.md` file documenting: \- How to trigger deployments \- How to approve deployments \- How to monitor deployment status \- How to verify deployment health \- How to perform rollbacks \- Emergency procedures * [ ] **AC 6.4**: Document terraform infrastructure requirements: \- All required resources per environment \- Naming conventions \- Required terraform variables \- Backend configuration ### 7. Testing & Validation * [ ] **AC 7.1**: Validate UAT deployment: \- Deploy successfully to UAT environment \- Verify pods are running and healthy \- Test API endpoints respond correctly \- Verify database connectivity \- Verify Azure OpenAI integration works \- Verify Form Recognizer integration works \- Verify blob storage access works \- Check logs for errors * [ ] **AC 7.2**: Validate PROD deployment: \- Deploy successfully to PROD environment (with approval) \- Verify pods are running and healthy \- Test API endpoints respond correctly \- Verify database connectivity \- Verify Azure OpenAI integration works \- Verify Form Recognizer integration works \- Verify blob storage access works \- Check logs for errors \- Monitor for 24 hours post-deployment * [ ] **AC 7.3**: Validate rollback procedure: \- Document and test rollback to previous version \- Verify rollback can be completed within RTO (Recovery Time Objective) \- Validate data consistency after rollback ### 8. Security & Compliance * [ ] **AC 8.1**: Security scanning enabled for production: \- Enable Veracode scanning (currently commented out) \- Container vulnerability scanning \- Dependencies security audit \- Secrets scanning validation * [ ] **AC 8.2**: Network policies configured: \- Appropriate Istio authorization policies for UAT/PROD \- Network policies for namespace isolation \- Ingress/egress rules documented * [ ] **AC 8.3**: Compliance validation: \- All production secrets stored in Key Vault (not in helm values) \- Audit logging enabled \- Compliance with organizational security policies ### 9. Monitoring & Observability * [ ] **AC 9.1**: Configure application monitoring for UAT/PROD: \- Application Insights or equivalent APM \- Custom metrics for API endpoints \- Performance monitoring \- Error rate tracking * [ ] **AC 9.2**: Configure infrastructure monitoring: \- AKS cluster metrics \- Pod resource utilization \- Node health monitoring \- Database connection pool monitoring * [ ] **AC 9.3**: Set up alerting: \- High error rate alerts \- Performance degradation alerts \- Resource exhaustion alerts \- Deployment failure notifications ### 10. Disaster Recovery * [ ] **AC 10.1**: Document backup strategy: \- Database backup schedule \- Blob storage backup/replication \- Helm chart versioning strategy \- Container image retention policy * [ ] **AC 10.2**: Define and document: \- Recovery Time Objective (RTO) \- Recovery Point Objective (RPO) \- Disaster recovery procedures \- Multi-region failover (if applicable) ## Technical Notes ### Current Pipeline Flow ``` Build Container → Package Helm Charts → Deploy to SBX/DEV (parallel) → Deploy to QA ``` ### Target Pipeline Flow ``` Build Container → Package Helm Charts →   Deploy to SBX/DEV (parallel) →     Deploy to QA →       Deploy to UAT (manual approval) →         Deploy to PROD (manual approval + stricter gates) ``` ### Environment Naming Convention * AKS Cluster: `{env}-{region}-adc-aks-001` * Resource Group: `{env}-{region}-adc-app-rg-001` * SQL Server: `{env}-{region}-adc-sql-002.database.windows.net` * Storage: `{env}{region}adc{type}st001` Where: * `env` = sbx, dev, qa, uat, prod * `region` = eus2 (East US 2) * `type` = inst (instance), doc (documents) ## Dependencies 1. **Infrastructure Team**: Provision UAT and PROD Azure resources via Terraform 2. **Security Team**: Approve service principal permissions and Key Vault setup 3. **Database Team**: Set up UAT and PROD database instances with proper schemas 4. **Networking Team**: Configure DNS entries for UAT/PROD domains 5. **Platform Team**: Create GitHub environments and configure protection rules ## Definition of Done * [ ] All acceptance criteria are met * [ ] Code changes peer-reviewed and approved * [ ] Documentation is complete and reviewed * [ ] UAT deployment successful with manual validation * [ ] PROD deployment successful with manual validation * [ ] Rollback procedure tested and documented * [ ] Monitoring and alerting configured and validated * [ ] Runbooks created and reviewed by operations team * [ ] Security review completed * [ ] Sign-off from Lead Engineer and Product Owner ## Estimated Effort * **Infrastructure Setup**: 8-16 hours (depends on terraform state and existing resources) * **Pipeline Development**: 4-8 hours * **Testing & Validation**: 8-12 hours * **Documentation**: 4-6 hours * **Security Review**: 2-4 hours **Total Estimate**: 26-46 hours (3-6 sprints depending on team capacity and dependencies) ## Risks & Mitigations \| Risk | Impact | Probability | Mitigation | \|------|--------|-------------|------------| \| Terraform state drift | High | Medium | Audit existing infrastructure, reconcile state before deployment | \| Missing infrastructure resources | High | Medium | Complete infrastructure audit before starting development | \| Service principal permission issues | Medium | High | Test permissions in lower environments first | \| Secrets not in Key Vault | High | Low | Prioritize Key Vault integration before PROD deployment | \| Insufficient approval process | Medium | Medium | Define clear approval workflow with stakeholders upfront | \| Production data exposure | High | Low | Use production-specific secrets, never reuse lower environment credentials | ## Success Metrics * Successful deployment to UAT on first attempt * Successful deployment to PROD with zero downtime * Rollback procedure tested and documented * All security scans pass * Zero production incidents in first 30 days post-deployment