Installing cert-manager with LetsEncrypt on k3s and RKEv2

This article gives a good explanation of installing cert-manager+LetsEncrypt(LE). However it used HTTP verification and I prefer DNS verification because :

I want a private Kubernetes cluster (so no access for LE to contact our http port).
less chance of LE being blocked by a firewall (AWS WAF etc...).
can use a single wildcard domain certificate (so a single DNS record required).

So this article is meant to supplement and not duplicate the original article

I am using a single node k3s (was created from my terraform module).

Traefik (which comes pre-bundled with k3s) actually has Let's Encrypt support built-in, so why use cert-manager? At the time of this writing, Traefik's Let's Encrypt support retrieves certificates and stores them in files. Cert-manager retrieves certificates and stores them in Kubernetes secrets, which is preferable.

In the beginning there was IAM

You will need IAM on your kubernetes EC2 instances (nodes) to allow cert-manager to update your Route53 DNS. Instructions here

Next you need to create an IAM User, give it programmatic access only, and use the AccessKey and SecretKey in the next section.

Create the below IAM policy and attach it to the IAM User :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "route53:GetChange",
      "Resource": "arn:aws:route53:::change/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/*"
    },
    {
      "Effect": "Allow",
      "Action": "route53:ListHostedZonesByName",
      "Resource": "*"
    }
  ]
}

Installing cert-manager in k3s

Using their helm chart with these values :

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.8.0 \
  --set installCRDs=true \
  --set podDnsConfig.nameservers[0]="8.8.8.8" \
  --set podDnsConfig.nameservers[1]="1.1.1.1"

I added the podDnsConfig.nameservers so cert-manager is using a public DNS server (not the private Route53 DNS zone as defined in my kubernetes nodes EC2 DHCP OptionSet). Cert-manager uses DNS to check that the DNS record it creates is present and propogating to prevent LE using too many DNS checks and erroring.

Cert-manager's ACME (LetsEncrypt) plugin will create a TXT record in our Route53 DNS Public Hosted Zone. This will prove that we own the domain and its OK for LE to issue us a certificate.

To check it installed ok you can use the cmctl tool.

(To install this cmctl tool see docs)

$ cmctl check api

The cert-manager API is ready

Now you need to create an Issuer to test the webhook works okay.

Create a Secret for our AWS access secret key :

apiVersion: v1
Kind: Secret
metadata:
    name: cert-manager-aws-secret-key
    namespace: cert-manager
type: Opaque
stringData:
  secret-access-key: ....put your aws secret key here ...

Create a ClusterIssuer for LE :

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    # server: https://acme-staging-v02.api.letsencrypt.org/directory  ... use staging for initial tests
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: [email protected]
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the DNS challenge provider
    solvers:
    - selector:
        dnsZones:
          - 'backbone.parkrunpointsleague.org'
          - 'parkrunpointsleague.org'
      dns01:
        # Valid values are None and Follow
        cnameStrategy: Follow
        route53:
          region: eu-west-1
          accessKeyID: ....put your aws access key here ...
          hostedZoneID: ... put your route53 public hosted zone id here ...
          secretAccessKeySecretRef: 
            name: cert-manager-aws-secret-key
            key: secret-access-key

The above uses the production LE server. You may want to configure the ClusterIssuer with the staging LE server, for initial tests. Its server url is https://acme-staging-v02.api.letsencrypt.org/directory. However it will give an invalid certificate with issue ERR_CERT_AUTHORITY_INVALID.

To check it worked with you can :

Look at the certificate for Events line "The certificate has been successfully issued" with :
```
kubectl describe certificate -n cert-manager-test
```
To check the certificate progress look at logs on the kubernetes deployment for cert-manager

Or can check with :

kubectl -n app get challenges.acme.cert-manager.io
# and then
kubectl -n app describe challenges.acme.cert-manager.io <NAME>

For the new certificate to be used on our ingress (traefik) we need to add annotation cert-manager.io/cluster-issuer: letsencrypt-prod and the tls as below :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
  annotations:
    kubernetes.io/ingress.class: traefik
    cert-manager.io/cluster-issuer: letsencrypt-prod
  labels:
    app: demo
spec:
  rules:
    - host: kube.backbone.parkrunpointsleague.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: demo
                port:
                  number: 8080
  tls:
    - hosts:
        - kube.backbone.parkrunpointsleague.org
      secretName: kube-backbone-prpl-tls

Load up your browser and go to our ingresses host+path of https://kube.backbone.parkrunpointsleague.org. This should have an SSL/TLS certificate, issued by LetsEncrypt.

Alternative : Using with Nginx Ingress.

Should only need to change the Ingresses annotation to :

kubernetes.io/ingress.class: nginx

Alternative : Using with kube2iam

Using kube2iam means we can avoid the need to create an IAM User to allow cert-manager to have permission to manipulate our Route53 DNS.

Remove from the ClusterIssuer yaml the following :

  accessKeyID: ....put your aws access key here ...
  secretAccessKeySecretRef: 
    name: cert-manager-aws-secret-key
    key: secret-access-key

Remove the Secret for the AWS secret-access-key, since no longer needed.

Instead kube2iam will grant cert-manager permission assuming an IAM role to it.

More details in my Gist on it here.

Alternative : Using cert-manager in RKEv2 (not k3s).

It is very similar with only slight modification if also using kube2iam (see above).

Summary

Hopefully this has helped you to install and configure and troubleshoot cert-manager in k3s or RKEv2.

davidcallen/cert-manager-k3s-and-lets-encrypt.md