Skip to main content

External Secrets Operator

Now let's explore integrating with AWS Secrets Manager using the External Secrets operator. This has already been installed in our EKS cluster:

~$kubectl -n external-secrets get pods
NAME                                                READY   STATUS    RESTARTS   AGE
external-secrets-6d95d66dc8-5trlv                   1/1     Running   0          7m
external-secrets-cert-controller-774dff987b-krnp7   1/1     Running   0          7m
external-secrets-webhook-6565844f8f-jxst8           1/1     Running   0          7m
~$kubectl -n external-secrets get sa
NAME                  SECRETS   AGE
default               0         7m
external-secrets-sa   0         7m


The operator uses a ServiceAccount named external-secrets-sa which is tied to an IAM role via IRSA, providing access to AWS Secrets Manager for retrieving secrets:


~$kubectl -n external-secrets describe sa external-secrets-sa | grep Annotations
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/eks-workshop-external-secrets-sa-irsa


We need to create a ClusterSecretStore resource - this is a cluster-wide SecretStore that can be referenced by ExternalSecrets from any namespace:


~/environment/eks-workshop/modules/security/secrets-manager/cluster-secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: "cluster-secret-store"
spec:
  provider:
    aws:
      service: SecretsManager
      region: $AWS_REGION
      auth:
        jwt:
          serviceAccountRef:
            name: "external-secrets-sa"
            namespace: "external-secrets"



~$cat ~/environment/eks-workshop/modules/security/secrets-manager/cluster-secret-store.yaml \
  | envsubst | kubectl apply -f -


Let's examine the specifications of this newly created resource:


~$kubectl get clustersecretstores.external-secrets.io
NAME                   AGE   STATUS   CAPABILITIES   READY
cluster-secret-store   81s   Valid    ReadWrite      True
~$kubectl get clustersecretstores.external-secrets.io cluster-secret-store  -o yaml | yq '.spec'
provider:
  aws:
    auth:
      jwt:
        serviceAccountRef:
          name: external-secrets-sa
          namespace: external-secrets
    region: us-west-2
    service: SecretsManager
 


The ClusterSecretStore uses a JSON Web Token (JWT) referenced to our ServiceAccount to authenticate with AWS Secrets Manager.


Next, we'll create an ExternalSecret that defines what data should be fetched from AWS Secrets Manager and how it should be transformed into a Kubernetes Secret. We'll then update our catalog Deployment to use these credentials:


~/environment/eks-workshop/modules/security/secrets-manager/external-secrets/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../../../base-application/catalog
  - external-secret.yaml
patches:
  - path: deployment.yaml


~$kubectl kustomize ~/environment/eks-workshop/modules/security/secrets-manager/external-secrets/ \
  | envsubst | kubectl apply -f-
~$kubectl rollout status -n catalog deployment/catalog --timeout=120s


Let's examine our new ExternalSecret resource:


~$kubectl -n catalog get externalsecrets.external-secrets.io
NAME                      STORE                  REFRESH INTERVAL   STATUS         READY
catalog-external-secret   cluster-secret-store   1h                 SecretSynced   True


The SecretSynced status indicates successful synchronization from AWS Secrets Manager. Let's look at the resource specifications:


~$kubectl -n catalog get externalsecrets.external-secrets.io catalog-external-secret -o yaml | yq '.spec'
dataFrom:
  - extract:
      conversionStrategy: Default
      decodingStrategy: None
      key: eks-workshop/catalog-secret
refreshInterval: 1h
secretStoreRef:
  kind: ClusterSecretStore
  name: cluster-secret-store
target:
  creationPolicy: Owner
  deletionPolicy: Retain


The configuration references our AWS Secrets Manager secret via the key parameter and the ClusterSecretStore we created earlier. The refreshInterval of 1 hour determines how often the secret values are synchronized.


When we create an ExternalSecret, it automatically creates a corresponding Kubernetes secret:


~$kubectl -n catalog get secrets
NAME                      TYPE     DATA   AGE
catalog-db                Opaque   2      21h
catalog-external-secret   Opaque   2      1m
catalog-secret            Opaque   2      5h40m


This secret is owned by the External Secrets Operator:


~$kubectl -n catalog get secret catalog-external-secret -o yaml | yq '.metadata.ownerReferences'
- apiVersion: external-secrets.io/v1beta1
  blockOwnerDeletion: true
  controller: true
  kind: ExternalSecret
  name: catalog-external-secret
  uid: b8710001-366c-44c2-8e8d-462d85b1b8d7


We can verify our catalog pod is using the new secret values:


~$kubectl -n catalog get pods
NAME                       READY   STATUS    RESTARTS   AGE
catalog-777c4d5dc8-lmf6v   1/1     Running   0          1m
catalog-mysql-0            1/1     Running   0          24h
~$kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.containers[] | .env'
- name: DB_USER
  valueFrom:
    secretKeyRef:
      key: username
      name: catalog-external-secret
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: catalog-external-secret


Conclusion​


There is no single "best" choice between AWS Secrets and Configuration Provider (ASCP) and External Secrets Operator (ESO) for managing AWS Secrets Manager secrets.


Each tool has distinct advantages. ASCP can mount secrets directly from AWS Secrets Manager as volumes, avoiding exposure as environment variables, though this requires volume management. ESO simplifies Kubernetes Secrets lifecycle management and offers cluster-wide SecretStore capability, but doesn't support volume mounting. Your specific use case should drive the decision, and using both tools can provide maximum flexibility and security in secrets management.