Deprecated HorizontalPodAutoscaler API in Helm Charts for Kubernetes 1.26+

01 February 2025 - 7 mins read time
Tags: DevOps Helm Kubernetes

Fixing Deprecated HorizontalPodAutoscaler API in Helm Charts for Kubernetes 1.26+ autoscaling/v2beta2 autoscaling/v2 ✓

TL;DR

Suddenly got no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2" or no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta1" when running helm upgrades? This is caused by deprecated API versions for HPA in Kubernetes 1.26+. Run the command below to automatically fix the API version in your Helm release secrets:

curl -sL https://gist.githubusercontent.com/nh4ttruong/172936eed756ca11e60efacf42117d2c/raw/31637ec5fed7a41ae202711304671df2de0bc1cd/fix-api-version.sh | bash -s

Note: Always review scripts before piping them directly to bash. The script above updates Helm release secrets to use autoscaling/v2 instead of deprecated API versions.

The Problem

If you’ve recently upgraded your Kubernetes cluster to version 1.26 or later, you might have encountered this frustrating error:

no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2"

This error occurs because Kubernetes 1.26 has officially removed support for the autoscaling/v2beta2 API version, which was previously marked as deprecated. The current recommended version is autoscaling/v2.

The Helm Chart Challenge

This issue is particularly problematic when dealing with Helm charts for several reasons:

  1. Embedded API Versions: Helm charts often have the API version hard-coded in their templates. When these charts were created with older API versions, they won’t automatically adapt to newer Kubernetes versions.

  2. Release History Storage: Helm stores the deployed resources’ state (including API versions) in secrets within your Kubernetes cluster. Even if you update your chart’s templates, the previously deployed release information still contains references to the deprecated API.

  3. Update Mechanism Issues: When you attempt to upgrade a release with helm upgrade, Helm compares the new manifest against the stored release data. If it encounters incompatible API versions, the upgrade fails.

  4. No Automatic Migration: Helm doesn’t automatically migrate resources from deprecated APIs to their replacements, requiring manual intervention.

The result is that even after updating your Helm chart’s templates to use autoscaling/v2, you might still face errors because the stored release data references the now-removed autoscaling/v2beta2 API.

Step-by-Step Solution

Here’s a comprehensive guide to fixing this issue within your Helm releases:

1. Identify the Affected Helm Release

First, locate the secret that stores your Helm release configuration:

kubectl get secret -l owner=helm,status=deployed --all-namespaces

From the output, identify the secret related to your specific Helm release. It will typically follow a naming pattern that includes your release name.

2. Extract and Decode the Release Configuration

Export the secret to a file:

kubectl get secret <helm_release_secret> -n <namespace> -o yaml > release.yaml

Then decode the stored Helm release data:

cat release.yaml | grep -oP '(?<=release: ).*' | base64 -d | base64 -d | gzip -d > release.data.decoded

This command extracts the release data, decodes it from base64 (twice, as Helm doubly encodes it), and decompresses it.

3. Modify the HorizontalPodAutoscaler API Version

Open the decoded file in your preferred text editor:

nano release.data.decoded  # or vim, or any other editor

Search for all occurrences of autoscaling/v2beta2 and replace them with autoscaling/v2.

Be sure to verify that the structure of any HorizontalPodAutoscaler resources is compatible with the v2 API. In particular, note that the metrics specification might need adjustments.

4. Re-encode and Update the Helm Secret

After making your changes, you need to re-encode the file:

cat release.data.decoded | gzip | base64 | base64 | tr -d '\n' > encoded_release.txt

Create a patch file named patch.json:

{
  "data": {
    "release": "<NEW_ENCODED_RELEASE_DATA>"
  }
}

Replace <NEW_ENCODED_RELEASE_DATA> with the content of your encoded_release.txt file.

Apply the patch to update the Helm release secret:

kubectl patch secret <helm_release_secret> -n <namespace> --patch-file=patch.json

5. Verify and Redeploy

Finally, run a Helm upgrade to ensure the fix is applied:

helm upgrade <release_name> <chart_name> -n <namespace>

To verify that the HorizontalPodAutoscaler is now using the correct API version:

kubectl get hpa -n <namespace> -o yaml | grep apiVersion

This should show apiVersion: autoscaling/v2 for your HorizontalPodAutoscaler resources.

Prevention Measures

To avoid similar issues in the future:

  1. Stay Informed About API Deprecations: Before upgrading Kubernetes, always check the release notes for deprecated APIs that will be removed.

  2. Update Your Helm Charts: Ensure your charts are regularly updated to support the latest Kubernetes API versions.

  3. Use Automated Tools: Consider implementing tools like pluto to detect deprecated API usage in your cluster before upgrading.

  4. Test in Non-Production: Always test Kubernetes upgrades in a staging environment first to catch these kinds of issues.

  5. Helm Chart Maintenance: For teams maintaining their own charts, establish a process to regularly review and update API versions.

Conclusion

API deprecations are a common challenge when upgrading Kubernetes, and Helm’s release storage mechanism can make these issues particularly tricky to resolve. By understanding how to diagnose and fix these issues directly within Helm’s stored release data, you can ensure smoother upgrades and minimize disruption to your services.

This specific fix for the HorizontalPodAutoscaler API version issue highlights the importance of keeping up with Kubernetes’ evolution while maintaining compatibility with your deployment tools like Helm.

Have you encountered other interesting challenges when upgrading Kubernetes or working with Helm charts? Let me know