Changing Sealed Secrets Passwords in Kubernetes

9 min readDec 17, 2022

Introduction

There are plenty of articles that focus on installing Bitnami’s Sealed Secrets application and creating your first Sealed Secret within Kubernetes. However, most article do not cover how to change passwords defined within a Sealed Secret.

Sealed Secrets is a product offered by Bitnami Labs. This allow Kubernetes secrets to be safely committed to Git repositories. Even public ones.

How to install and configure Sealed Secrets is outside the scope of this article. I’m going to assume you already have a Kubernetes cluster with Sealed Secrets installed, you’ve converted your standard secrets to Sealed Secrets and committed your Sealed Secrets YAML files to your Git repository which are deployed to your cluster via a GitOps process such as ArgoCD.

The Password Problem

Kubernetes does not store secrets encrypted, it only Base64 encodes secrets which is trivial to decode back into clear text. This is analogous to a lock on a glass door; it is only intended to keep the honest people out. Anyone else would use a brick and not worry about the lock.

Since secrets encoded with Base64 are so trivial to decode, they cannot be included with your application deployment to Git repositories. Even if you use private repositories it is unwise to store your Base64 encoded secrets within. Anyone who has access to the repository will have access to the contents of the secret.

This is where products such as Sealed Secrets come in. Standard Kubernetes secrets can be encrypted and “converted” into a Sealed Secret which is safe to store in a public repository (I personally still insist on using private repositories for secrets). The Sealed Secret is a Custom Resource Definition (CDR) installed with Sealed Secrets. These YAML files can be identified by object type kind: SealedSecret.

To be clear, this only protects your secrets stored at rest in the Git Repository. Anyone with access to use kubectl get secrets will still see the trivial to decode base64 encoded secret. Anyone with access to etcdctl get /registry/secrets/… will see secrets in clear text (by default).

Vendor Password Complexity

Due to this Password Problem, vendors have tried to address this in various ways such as using bcrypt to store a hash of the password. That hash is then stored within a Kubernetes secret using Base64 encoding.

This creates a new problem, in that you now have to figure out if the password is stored encoded with a hash or some other method. Then you have to figure out how to encode or hash a new password in a way that can be stored within a Sealed Secret that the application will understand.

The name of the key within the secret is not standardized. It can really be named anything the developer wants. Hopefully the key name used will have some hint that it is for storing a password.

Finally once that is done, the new password is pushed to Git. Your GitOps process runs and deploys the new secret. You still have to give the application a nudge in someway to reload the new secret. This often requires figuring out which Pod in the application is responsible for loading secrets and restarting that pod.

Starting to understand why nobody talks about how to change your passwords?? There is not a universal process that works on all applications. However, once you figure out how an application works, you can document it in your processes and should be able to automate the process fairly easily.

Sometimes you can ignore all this and just notify your deployment controller that secrets are allowed to differ from their Git repository contents. This allows you to use the “change password” feature within an application. The updated password stored internally will no longer match what is defined by Git. Normally something like ArgoCD would see this difference and rollback the change as Git is the “source of truth”. You can define what is allowed to be different using ArgoCD’s “Spec.IgnoreDifferences”.

I’ll demonstrate changing a Sealed Secret that holds my ArgoCD administrative password. This is the password used to log into the ArgoCD web administration or CLI. ArgoCD stores a bcrypt hash of the password and a timestamp of when the password was changed. Should something go wrong, you simply revert the commit to rollback the change and go back to the original Sealed Secret that has your old password.

Tools Needed

You probably do not have a tool to create the bcrypt hash. Even if you use a Linux desktop, this is not usually installed by default. The easiest method is to install a python version of it:

$ pip3 install bcrypt

To generate a bcrypt hash of a new password of value “newPassw0rd”:

$ python3 -c "import bcrypt; print(bcrypt.hashpw(b'newPassw0rd', bcrypt.gensalt()).decode())"

$2b$12$75PyWqCR7WL4HMwkA59dk.9/X9OyLmF0A5gQOHPEv87YCOwnk5sLu

You can execute the above command over and over, each time you will get a different bcrypt hash value. It does not matter which one you pick. The “salt” used within is included as part of the bcrypt hash.

You will also need the kubeseal CLI utility which is part of the Sealed Secret installation process.

What does the ArgoCD Sealed Secret Look like?

This is simply the file generated previously by “kubeseal” utility and stored within the git repository. It looks like the following:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: argocd-secret
  namespace: argocd
spec:
  encryptedData:
    admin.password: AgALADcrA0+eqnUGJRBVtJLpbA8LAWJt0KWaFz82K9H4K2qCQ/SeKBKqYDff/hMfmMgqmzdcbg4TXpOGpeJWbVHDmpUCzWY7T/WqPNuo9PCpk6huBiuPOMV9UF5ei92vZwjYG4IqCJpB13ds9QZPB9aQWfpJGtP9oTddMBBmfkhNZDLXURxQKXfBzUJtIUU86r0NLYRWC2wjcA1h0oq45K6Fa4WU33WeQpWsajRT97ekdsL4QqKw0X+nZ9PdGKoLwU+GZRHZFFxqLxzib28Y8IKWocYf/qEtvT+gAwyZKUMSPPTODTovvBcjoMOb3O244JZn/+HZmox4VfTlK/l3xT1aO+SXbmhZqgBI9ywdD1NduTgHNPIwmHMgqIPrZk0U67XrBgQ4XjNVlm2lOc741ipAf/y4HUIzJBE22lv4lAuU+MsZGSmrHH6fBQ1GiTfwg+toiTcKA/C/rNks5HbuC7WuGTm+DilMU3L0Z5MMwXYOzceS2TDaTGFZp3yhcBqm9rn2VrhXU0bHSJBcImUNpE1ZaaJdkMFELHm6hq15RZ0ghJTR7yzLTmWeEcFcKUbRFJiQuUB1E3PRivr1e964rKu9iQgGExYgwnQzo2w2/A/3C3Wrf4gW+8H7PvheC0w9FcMBdqMHCCaXn7JJgXj7xx6zCpD5yncwYJnikpPbUIZHwT16Y3WVGyb8QE80iON/ZrfNo9lMQx5vrq0Z5J6wmebOmT7wjMqHwJW5XH+H9mQZn6cI/7lDqxvBYMKZRYmoBub5v60whNNM01A789s=
    admin.passwordMtime: AgDAriIqkv8SlJj0v1RNWfEVOLopM0mIRvgtZVxRq8ahA9EedJMBQQDQDVE5qeIDmgVKO1FDkZFhDc10iilSLwaHSa+/qx1Rd005TCNPRqjqMY9CXVE59auvfPOd7eCRDfH10RL6HNxdCvMFAg5GQrrvbzH3mTyK7+HEy3OXCO8Ht/YyD/aAv32y8FMm68Ugh/zzGazEv8uftM+KNv/I8lu0c830ukRxrLpT818PVEGbIr0nJo3h93nWv3BVsWmw+QJA3AToUrkXohui2gG6Sw0Yua3oTWDLnM0xCFh555/9NBVZoQiOBRMWzwSK8L8WNlXdn/cX1PTcURU2ED6/Cs12L1X5AJE5tCstyi0/cTlvRTUwwrzM8z2sBPE8g2eHgBxBmxhMkEMdzCZ1h9wABKRyzR3ew0JfVBYvs9UjBWHSVf86qzOg36mkN39T1IHxRg2ffq6GfgBt8Px9Q09vxUMEPTbE/8acMYQVBUsZzoQNBC7YirQdmjNgC7houXj/BdAaD0hZw3ftCitA6IRa+s0LAuZfs8ZorbF1/jgAGnXZYsrC4nU+pJr2ibwLu2Cgr1d/kPJ+x8ZMQfstJv47/mhhb5IeqqYIHV3J7qmDVjrNzrGl6uFJ8MGu5rVudMd2xIEnhoDZMUKCzCVbnIVoeG0lPzehxahZPslXj1utcjssz4B01PnyAZH1JxWwPXtdbIEkul4/oGZUigAjSMOuy4gHWf1LuC6m
    server.secretkey: AgBUK9IW7G8vvZD3RQDeAWkcn+TaMlTkOeCSgqaPa9ZDta75BI8GhF3dLprXv2adRAPFEhgKyL4D533lJ5AQQ+VKAqJQvYxPo/5wAmzCyLToVCcg6mgLMBgkkkltJAh+/FwTfl7qZbBJoy8WgvowiI1GyHcVWnWo67pjsATwCDqcBvvKMpizKpBxuI+NlPitaDQNBLum/rXTOmdBtS520a9cm9+BRkHcAwiLL0KHlkvV5YUUnovO24GnX0A3rkLCe4QlGoXQyy8Kc317w4zWbCZ2POsWEQfyP55Ey+y3fFN9en9EbsUxzrZis+fvYwAsRjfHqTYWoNxD24Rzk+VPOYt/jmuWdeTzJEF5+CBISZqzysjbjVVOfprBhOONBPLvNHO6TWpR6myKdPglQE8VsbfZJCejslob0KKn+exhFjBFTHgTa+agJLFqIZegR9IBy6F3x1sffY/ZL+w7vUp+x5/GV3qyoecldyQy68Df5twzL/OQ6JRItQbIWJIHBrQXRq+Xc3tBdwZRY39XGutVEO9IqX1+ttST4ZqgxRhsD26VqLqh7oUfWaDD0a1H33g9/IfDnbkk06kMmVjjDRk6tVBKjrt7OVM7TAehAF5BtORF5u8YCg0qafu0oPHuPU/6IHU3OAHKyL0nMQVid1+8Vevw53flNoZ/b1tV34PeP12x9uoyR/Z175SaG/ZRT/RHB4s=
  template:
    metadata:
      creationTimestamp: null
      name: argocd-secret
      namespace: argocd
    type: Opaque

We will be interested in changing the values of the “admin.password” and “admin.passwordMtime” of this Sealed Secret.

The Process

In the conclusion of this article I will provide the one shot full set of commands needed for the actual day to day work. But before we get there, I’ll break down the process step by step so you’ll have some understanding on how this works and can apply this to other applications.

Store the Password

The hashed passwords are rather long. We will store it in a Bash variable named NEWPASS to keep the syntax of our command line work more tidy. For simplicity the value of the new password is “newPassw0rd”. You will obviously use a much more complex password.

$ NEWPASS=$(python3 -c "import bcrypt; print(bcrypt.hashpw(b'newPassw0rd', bcrypt.gensalt()).decode())")

Create a Kubernetes Secret Template

We will use “kubectl create secret” command to generate a generic secret and inject it with our new values. The output of this command will be JSON which “kubeseal” can be read as input later on. We will pass three flags to this command:

— dry-run=client : which means kubectl will render this locally only. This is not to be submitted to the server for processing.
— from-literal=admin.password= : kubectl shall populate the value for “admin.password” based on the contents of our NEWPASS Bash variable.
— from-literal=admin.passwordMtime="$(date +%FT%T%Z)” : will populate the value based on the current timestamp using the Linux date command.

$ kubectl create secret generic tempxxx --dry-run=client \
  --from-literal=admin.password="${NEWPASS}" \
  --from-literal=admin.passwordMtime="$(date +%FT%T%Z)" -o json

{
    "kind": "Secret",
    "apiVersion": "v1",
    "metadata": {
        "name": "tempxxx",
        "creationTimestamp": null
    },
    "data": {
        "admin.password": "JDJiJDEyJDc1UHlXcUNSN1dMNEhNd2tBNTlkay45L1g5T3lMbUYwQTVnUU9IUEV2ODdZQ093bms1c0x1",
        "admin.passwordMtime": "MjAyMi0xMi0xNlQxMjo1MjoxNkVTVA=="
    }
}

The values of our two keys are simply Base64 encoded at this point. To see the value of the “admin.passwordMtime”:

$ echo -n "MjAyMi0xMi0xNlQxNDoyMzo0M0VTVA==" | base64 -d

2022-12-16T14:23:43EST

Convert Secret to Sealed Secret

The JSON above can be read directly by kubeseal. We will simply pipe this output as input to kubeseal. We will need to pass kubeseal three flags:

— controller-namespace=sealed-secrets : the namespace where Sealed Secrets is installed.
— controller-name=sealed-secrets : the name of the Sealed Secrets Controller.
— format yaml : how we want the output to be formatted.

$ kubectl create secret generic tempxxx --dry-run=client \
  --from-literal=admin.password="${NEWPASS}" \
  --from-literal=admin.passwordMtime="$(date +%FT%T%Z)" -o json | kubeseal \
  --controller-namespace=sealed-secrets --controller-name=sealed-secrets \
  --format yaml


apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: tempxxx
  namespace: default
spec:
  encryptedData:
    admin.password: AgCvgPkdLaRRxX/UTHD1/zE3ozahe0soKrzlilB2kbWqhgI78yDytB+MPXLjDy9A6iB8WetYWcyK5qhUEH03xmO6dWp00oeIOIlgM+X3fJy1bAQEp8K0uHpti9qpWj43HlR2OREqYTBV6z54lkxjNMGQ/4WttEslAMOVVs2v5eQJVhlMbfEFT8PoN6/f4SrA9+VzFF9ZiB2XSaNy5JSPBZxReryBETG0hl0I4iQs8D3s043M7J1NGLI1gtXipeGthW77ZsRYt3d9ynwgNMpqTqK6FlODE6v22+gGXFmUIh8BSlrtIaV3hT2vnMYlViglHjilO9BPFNacxJ3fvsvH+ZCQ0Q8GdmZMD9Fa8ZHC5xcl/3feHv5o7sUingBkZ00/pg6kDlBVMOktU+d9e6M+PAgY2t61KFB0FoyVGzsKjDXjQgVYEXaTLuczX2quVjjCzfxet0kraIImD5lpbaWYcxZwkvudRuphbVG4xLBwUJO01VnVOL2SqaC06GZQIqfXHpXUqKg5/gXV2a7O1xgLnzmB+/ERongYSdvqTulvEHVhOjVv+7/tLlev0o+PMFogW43Agnyy8tOFvS0KTbcvzC9REpk9q28cmAjZ2SxwoYNlfNyt8s12F9vHA4WjK5PAuI/51ZAn17uZ/8wXzauDIBtQvdk2E/X3lZjd4VKKWH8Fs3R7SOOcgTD4p3w6Ft6lEQxI9l6btaOa9EiWVT38zpUsKH3qL4iy3uWLSxFWLoQZIKNzJ0hcIo95UQpLkTl0eo4uRLlB6GtWwz7i58Q=
    admin.passwordMtime: AgAqrNjEWuXm7hrV8mJcUZ30J1K7u0201JWDYE638dgrksS0dHNH+6JqybmkvvyWKuN2Vt2Z3kl9mFX928nEIcOcG++cqRZUPg/QIHdRhnk4L9p0WSn40m2gKyxZ6fZgDOzryHm0D7L4qacF5T7Gl6DucRxHftKnXlanqjoWhqMIluthjdBilinKbpVqrJCGQnVb389CYN07vXrG1jEIp0GcfOdgLzKieQl2lU+MiqUH2NEKbEw2DhO4v0mMobrM912wXYRcMZH6dCzrsZyBq7lrN+uJpC85qiAXv5mmcAtt86ppv8zCgSenHrs+EKjYxcXHV53/xPNbQh1f4yKSwb8VgB5jgnnj4lMF9qyRVFc2karnwIJltOpSOMfleMxIitDERmgl+Uu7epT6OC5ni3Z1Ges9zDNNRirmqrOqQug7XUOM7agC3Xb5h84R6TIE7NdVdKkLD1Jotrm9Txt+A2nIOpjsGbxGo3MkrZRXRlwMrTghbNf4cV5dlIJZ+bQWq9jNhJBuloCPE6WVYhfP92Wt5yz515WORNHkKYnTueA+fci8uPP/9MatkBQZ6PWTPajwhNpa0G74uID5ajimRurZcNOMXuVnmvPEebSkdN7sK8MviTVAA6iifJBtfRpMyU7E1ZWOGry0iLlunfVIBPsyI5xtrPSkmbQ195jzX+QQO7cOFtBKkuiIWKWW1rBoZPokvA9prcKMyLsppNFV9tNx3vylGx50
  template:
    metadata:
      creationTimestamp: null
      name: tempxxx
      namespace: default

The YAML is now of “kind: SealedSecret” and the two values are encrypted. These values can only be decrypted within the cluster that contains the private encryption key. Even the person who encrypted this using the kubeseal utility (with the cluster’s public key), cannot decrypt this without access to the private key.

At this point you could just cut & paste the two lines of “admin.password” and “admin.passwordMtime” replacing your old values and save that to Git. Who wants to do that manual work?

Merge Updated Values into Existing Secret

The workflow to update the Sealed Secret in Git would have a copy of the previous secret. So lets merge our updated vales into the existing secret. We can do this by passing one more flag to kubeseal:

— merge-into : specifies the path and name of the existing Sealed Secret YAML file to merge the changes into.

$ kubectl create secret generic tempxxx --dry-run=client \
  --from-literal=admin.password="${NEWPASS}" \
  --from-literal=admin.passwordMtime="$(date +%FT%T%Z)" -o json | kubeseal \
  --controller-namespace=sealed-secrets --controller-name=sealed-secrets \
  --format yaml --merge-into argocd_admin_secret-sealed.yaml

The updated “argocd_admin_secret-sealed.yaml” file can now be committed into Git for deployment via ArgoCD.

Adding New Keys Value Pairs to a Sealed Secret:
During a merge if the new key name matches an existing key name then it replaces the old key value with the updated key value. If the new key name does not exist then merge will add the key name and its value into the new Sealed Secret. For example, if your application does not track a timestamp of when the password was last changed you can still store (add) the “admin.passwordMtime” key name and value pair to the secret. You can name the key anything you wish as it is only for your reference.

Confirming Updated Secrets

You can easily tell if the updated password was applied by checking the contents of the “admin.password” to see if it matches the contents of the NEWPASS variable we stored it in:

$ echo ${NEWPASS}

$2b$12$75PyWqCR7WL4HMwkA59dk.9/X9OyLmF0A5gQOHPEv87YCOwnk5sLu

$ kubectl get secrets -n argocd argocd-secret \ 
  -o jsonpath='{.data.admin\.password}' | base64 -d

$2b$12$75PyWqCR7WL4HMwkA59dk.9/X9OyLmF0A5gQOHPEv87YCOwnk5sLu

If the two values do not match, then something went wrong. It’s possible ArgoCD has not gotten around to sync the change yet or perhaps you provided kubeseal with the wrong public key and the cluster does not have the respective private key to decrypt it (fetch the public key and try again).

Restarting ArgoCD

The password change will take effect based on when your application reloads secrets. That typically means the next time the Pod is restarted. We can give ArgoCD a nudge by reducing its ApplicationSet Controller replica to zero:

$ kubectl scale -n argocd --replicas=0 \ 
  replicaset argocd-applicationset-controller-b44754f9d

replicaset.apps/argocd-applicationset-controller-b44754f9d scaled

ArgoCD will see this change and roll it back to the proper number of replica as defined within its Git based “source of truth”.

At this point you can logout of the ArgoCD and login again using the new password.

Conclusion

Sometimes passwords are just for the initial installation, and never used again. Perhaps once the application is up and running the user accounts are stored in a database or LDAP.

However, sometimes you are stuck with a password stored in a Kubernetes Secret and that password has been encrypted and committed to a Git repository. This does not make it immune to regular password rotations. You should still be changing passwords and making sure your encryption keys have defined rotations (changes) etc.

The main steps to changing the password are pretty simple. This assumes you have already fetched the original Sealed Secret YAML file from Git and you know the key name to update:

$ NEWPASS=$(python3 -c "import bcrypt; print(bcrypt.hashpw(b'newPassw0rd', bcrypt.gensalt()).decode())")

$ kubectl create secret generic tempxxx --dry-run=client \
  --from-literal=admin.password="${NEWPASS}" \
  --from-literal=admin.passwordMtime="$(date +%FT%T%Z)" -o json | kubeseal \
  --controller-namespace=sealed-secrets --controller-name=sealed-secrets \
  --format yaml --merge-into argocd_admin_secret-sealed.yaml

Use a better password than “newPassw0rd”. 😝
Commit the updated the Sealed Secret YAML file to Git (named argocd_admin_secret-sealed.yaml in this example).
Restart the application to reload the new secret.
Test the new password (or rollback to previous YAML file version if it does not work).

Lastly, if your application does not require something like bcrypt hash, then simply:

$NEWPASS="newPassw0rd"

I’m trying to get past Medium’s minimum follower requirement. If you found this article interesting or helpful then a clap 👏🏻 and follow 🤳🏼 would be appreciated.