What is K8s External Secrets and how it will make your life easier?
Before saying about External Secrets we will say about k8s secrets and how it will work. In k8s secrets we will create key value pairs of the secrets and set this as either pod env variables or mount them as volumes to pods. For more details about k8s secrets you can check my blog http://jinojoseph.blogspot.com/2020/08/k8s-secrets-explained.html
So in this case if developers wants to change the ENV variables , then we have to edit the k8s manifest yaml file, then we have to apply the new files to the deployment.
This is a tiresome process and also chances of applying to the wrong context is high if you have multiple k8s clusters for dev / stage and Prod deployments.
So in-order to make this easy , we can add all the secrets that is needed in the deployment, in the AWS Secret Manager and with the help of External secrets we can fetch and create those secrets in the k8s cluster.
So what is K8s external Secret?
It is an API Extension from Godaddy to integrate [(AWS / GCP) secrets into K8s]. Kubernetes External Secrets extends the Kubernetes Secrets API by adding a Custom Resource Definition (CRD) called “External Secret” which is internally mapped to the native Kubernetes secret. It uses a controller to fetch secrets from the external secret manager service, thereby linking the native K8s secrets with the external secrets.
Pre-requisites
###########
- Kubectl installed
- Helm installed
- Running EKS cluster.
- AWS IAM Role for service account.
Integrating Kubernetes External Secrets
###############################
This steps contains the following sub steps:
A. Getting the OpenID connect issuer URL from the AWS account where the EKS cluster resides.
B. Creating an OIDC Identity Provider in the AWS Account where you will create the secrets using SM.
C. Creating an IAM Role in the AWS Account, same as B.
D. Installing the kubernetes external secret API Controller using helm chart.
E. Adding the secrets in the AWS Secrets Manager.
F. Creating corresponding “ExternalSecrets” object in k8s cluster.
With IAM roles for service accounts on Amazon EKS clusters, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account.
A) Getting the OpenID connect issuer URL
To use IAM roles for service accounts in your cluster, you must create an OIDC identity provider
If your cluster supports IAM roles for service accounts, it will have an OpenID Connect issuer URL associated with it. You can view this URL in the Amazon EKS console, or you can use the following AWS CLI command to retrieve it.
# aws eks describe-cluster --name <cluster_name> --query "cluster.identity.oidc.issuer" --output text
B) Creating an OIDC Identity Provider.
1. Open the IAM console at https://console.aws.amazon.com/iam/.
2. In the navigation panel, choose Identity Providers, and then choose Create Provider.
3. For Provider Type, choose Choose a provider type, and then choose OpenID Connect.
4. For Provider URL, paste the OIDC issuer URL for your cluster.
5. For Audience, type sts.amazonaws.com and choose Next Step.
6. Verify that the provider information is correct, and then choose Create to create your identity provider.
C) Creating an IAM Role.
- Open the IAM console at https://console.aws.amazon.com/iam/.
- In the navigation panel, choose Roles, Create Role.
- In the Select type of trusted entity section, choose Web identity.
- In the Choose a web identity provider section:
- For Identity provider, choose the URL for your cluster.
- For Audience, choose sts.amazonaws.com.
5. Choose Next: Permissions. You can attach the SecretsManagerReadWrite Policy
6. In the Attach Policy section, select the policy to use for your service account. Choose Next: Tags.
7. On the Add tags (optional) screen, you can add tags for the account. Choose Next: Review.
8. For Role Name, enter a name for your role and then choose Create Role.
9. After the role is created, choose the role in the console to open it for editing.
10. Choose the Trust relationships tab, and then choose Edit trust relationship.
- Edit the OIDC provider suffix and change it from :aud to :sub.
- Replace sts.amazonaws.com with your service account ID.
- If necessary, change <region-code> to the Region code returned in the output from step
- The resulting line should look like this.
"oidc.eks.<region-code>.amazonaws.com/id/<EXAMPLED539D4633E53DE1B716D3041E>:sub": "system:serviceaccount:<SERVICE_ACCOUNT_NAMESPACE>:<SERVICE_ACCOUNT_NAME>"
Here replace the SERVICE_ACCOUNT_NAMESPACE with the name of the namespace that you have created for this purpose.
Also replace the SERVICE_ACCOUNT_NAME with any name , because we will be creating this Service Account with this name in the next steps using helm chart.
11. Choose Update Trust Policy to finish.
D) Installing the kubernetes external secret API Controller using helm chart.
For this we have to give the below helm command in the EKS cluster via the kubectl server.
jino@kubectl-001:~$ helm install my-release external-secrets/kubernetes-external-secrets --skip-crds --namespace my-kube-ext-secret --set securityContext.fsGroup=65534 --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"='arn:aws:iam::<ACCOUNT-ID>:role/my-external-secret-ssm-role' --set serviceAccount.create=true --set serviceAccount.name=my-test-external-secret-sa --set env.AWS_REGION=ap-northeast-1
This command will give an output like below:
####################################
NAME: my-test-release
LAST DEPLOYED: Sun Sep 27 12:55:08 2020
NAMESPACE: my-kube-ext-secret
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The kubernetes external secrets has been installed. Check its status by running:
kubectl --namespace my-kube-ext-secret get pods -l "app.kubernetes.io/name=kubernetes-external-secrets,app.kubernetes.io/instance=my-release"
Visit https://github.com/godaddy/kubernetes-external-secrets for instructions on how to use kubernetes external secrets
####################################
jino@kubectl-001:~$ kubectl --namespace my-kube-ext-secret get pods -l "app.kubernetes.io/name=kubernetes-external-secrets,app.kubernetes.io/instance=my-release"
NAME READY STATUS RESTARTS AGE
my-release-kubernetes-external-secrets-7d65dcb5db-wtr7h 1/1 Running 0 85s
jino@kubectl-001:~$
jino@kubectl-001:~$ kubectl get serviceaccount -n my-kube-ext-secret
NAME SECRETS AGE
default 1 5d3h
my-test-external-secret-sa 1 5d3h
jino@kubectl-001:~$
jino@kubectl-001:~$ kubectl describe -n my-kube-ext-secret sa my-test-external-secret-sa
Name: my-test-external-secret-sa
Namespace: my-kube-ext-secret
Labels: app.kubernetes.io/instance=my-release
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=kubernetes-external-secrets
helm.sh/chart=kubernetes-external-secrets-5.2.0
Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/my-external-secret-ssm-role
Image pull secrets: <none>
Mountable secrets: my-test-external-secret-sa-token-nrmdq
Tokens: my-test-external-secret-sa-token-nrmdq
Events: <none>
jino@kubectl-001:~$
You can check the godaddy git hub for each parameter meanings.
E) Adding the secrets in the AWS Secrets Manager.
This you can either do this by going to the AWS Console and add it manually or using the below awscli command:
# aws secretsmanager create-secret --name MyTestDatabaseSecret \
--description "My test database secret created with the CLI" \
--secret-string file://mycreds.json
Where the mycreds.json file will be like below:
[
{
"Key": “username”,
"Value": “mytestuser”
},
{
"Key": “password”,
"Value": “StrongPword”
}
]
You can also give the name of the secrets like
/MYCLIENT/DEV/CONTROLLER_APP_DEPLOYMENT > For the Dev Env secrets of your clients controller-app deployment.
/MYCLIENT/STAGE/CONTROLLER_APP_DEPLOYMENT > For the Stage Env secrets of your clients controller-app deployment.
/MYCLIENT/PROD/CONTROLLER_APP_DEPLOYMENT > For the Prod Env secrets of your clients controller-app deployment.
F. Creating corresponding “ExternalSecrets” object in k8s cluster.
Now you want to create an ExternaSecret object in k8s cluster under the Namespace of your deployment where you want to create its secrets.
vi myclient-external-secret.yaml
apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
name: myclientfrontend-app-externalsecrets
namespace: myclient-namespace
spec:
backendType: secretsManager
dataFrom:
- /MYCLIENT/DEV/CONTROLLER_APP_DEPLOYMENT
Wq!
jino@kubectl-001:~$ kubectl apply -f myclient-external-secret.yaml
externalsecret.kubernetes-client.io/myclientfrontend-app-externalsecrets created
jino@kubectl-001:~$ kubectl get externalsecret -n myclient-namespace
NAME LAST SYNC STATUS AGE
myclientfrontend-app-externalsecrets 5s SUCCESS 2d5h
jino@kubectl-001:~$ kubectl get secrets -n myclient-namespace
NAME TYPE DATA AGE
default-token-5462b kubernetes.io/service-account-token 3 220d
myclientfrontend-app-externalsecrets Opaque 29 2d5h
Now each time you add a new key value entry to the Secret “/MYCLIENT/DEV/CONTROLLER_APP_DEPLOYMENT” in AWS console, the externalSecret will automatically fetch that secret value from the AWS via the role and Service Account and apply the same to the Kubernetes secrets.
So now you can give the AWS console access to the developers and they will manage the env variable editing and addition by themselves, which is a great timesaver for us.
Cheers :-)
References:
#########
https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html
https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html
https://github.com/godaddy/kubernetes-external-secrets
Comments