Skip to navigation

GitHub Workflow Troubleshooting

This issue occurs when a Helm release becomes stuck in a pending state due to an interrupted or failed deployment. The fix requires identifying the pending revision and deleting the corresponding Helm secret

Why Helm Releases Get Stuck

Helm can create a new revision before a deployment fully completes. If something interrupts the process, the revision remains in a pending state and blocks all future upgrades. Common causes include:

  • The GitHub workflow was canceled during a deploy
  • Network interruption between the GitHub runner and AKS
  • Helm chart validation failed after the revision was created

When any of these conditions occur, the pending revision must be manually cleared before Helm can continue.


1. Confirm You Are Targeting the Correct AKS Cluster

Before checking Helm history, verify that kubectl and Helm are operating against the correct environment

Check the current Kubernetes context:

kubectl config current-context

Example output:

$ kubectl config current-context
sbx-eus2-adc-aks-001

Confirm the namespace exists in the cluster:

kubectl get ns <namespace>

Example fluxx output:

$ kubectl get ns fluxx
NAME STATUS AGE
fluxx Active 23d

If these commands return the expected cluster and namespace, you are connected to the correct AKS environment.


2. Verify Access Using PIM

To modify Helm release history, you must have permission to read and delete Kubernetes secrets in the target AKS cluster

Activate the following role in Azure AD PIM for your identified AKS resource:

Azure Kubernetes Service RBAC Cluster Admin

After activation, wait for permissions to propagate before running Helm or kubectl commands


3. Identify the Stuck Helm Revision

Run the following command to view the Helm release history:

helm history <release-name> -n <namespace>

Example fluxx-extractor output:

$ helm history fluxx-extractor -n fluxx
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
13 Thu Nov 20 14:07:26 2025 superseded fluxx-extractor-1.0.0 1.0.0 Upgrade complete
14 Thu Nov 20 15:06:43 2025 deployed fluxx-extractor-1.0.0 1.0.0 Upgrade complete
15 Thu Nov 20 16:02:50 2025 pending-upgrade fluxx-extractor-1.0.0 1.0.0 Preparing upgrade

Look for a revision with a status such as pending-upgrade, pending-install, or pending-rollback

15 Thu Nov 20 16:02:50 2025 pending-upgrade fluxx-extractor-1.0.0 1.0.0 Preparing upgrade

This indicates that revision 15 is blocking new Helm operations


4. Remove the Locked Revision

Each Helm revision is stored as a Kubernetes secret. To unblock the release, delete only the secret associated with the stuck revision

kubectl delete secret sh.helm.release.v1.<release-name>.v<revision> -n <namespace>

Example below:

  • namespace: fluxx
  • release-name: fluxx-extractor
  • revision: 15

Example output:

$ kubectl delete secret sh.helm.release.v1.fluxx-extractor.v15 -n fluxx
secret "sh.helm.release.v1.fluxx-extractor.v15" deleted from fluxx namespace

This does not affect the deployed workload. It only removes the corrupted history entry

Warning: Only delete the Helm secret for the specific stuck revision. Deleting other revisions can corrupt the release and require a full redeploy


5. Retry the Deployment

After clearing the pending revision, retry the GitHub workflow. Helm will generate a new revision and continue normally:

$ helm history fluxx-extractor -n fluxx
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
13 Thu Nov 20 14:07:26 2025 superseded fluxx-extractor-1.0.0 1.0.0 Upgrade complete
14 Thu Nov 20 15:06:43 2025 superseded fluxx-extractor-1.0.0 1.0.0 Upgrade complete
15 Thu Nov 20 17:22:08 2025 deployed fluxx-extractor-1.0.0 1.0.0 Upgrade complete

Success 🎉🎉