IBM Cloud Private Certificate Cleanup . Useful when encountering failure with token renewal via "bx pr cluster-config mycluster"

cert-exhaustion-cleanup.sh

#!/bin/bash
# Let's install jq to help with json parsing

if [ "${OS}" == "rhel" ]; then
  sudo yum install epel-release -y
  sudo yum install jq -y
else
  sudo apt-get -qq install jq -y
fi

CLUSTER_HOST="mycluster.icp"

# Get the list of certificates
CERT_TOKEN=$(bx pr tokens | grep "Access token:" | cut -d' ' -f3- | sed -e 's/^[[:space:]]*//')

# NOTE:  Double quotes are important for token variable expansion
CACHED_CERT_IDS=$(curl -X GET -sk --header "Accept: application/json" --header "Content-Type: application/json" --header "Authorization: ${CERT_TOKEN}" "https://${CLUSTER_HOST}:8443/idmgmt/identity/api/v1/certificates" | jq .[] | jq .id)

NUM_CACHED_CERT_IDS=$(curl -X GET -sk --header "Accept: application/json" --header "Content-Type: application/json" --header "Authorization: ${CERT_TOKEN}" "https://${CLUSTER_HOST}:8443/idmgmt/identity/api/v1/certificates" | jq .[] | jq .id | wc -l)

echo -e "  Before cleanup, we HAD ${NUM_CACHED_CERT_IDS} certificates"

for cert in ${CACHED_CERT_IDS} ; do
  # Cleansing the value of prefix and suffix double quotes
  cert="${cert%\"}"
  cert="${cert#\"}"
  echo -e "  Deleting CERT ID: ${cert} ...\n"
  curl -X DELETE -sk --header "Accept: application/json" --header "Content-Type: application/json" --header "Authorization: ${CERT_TOKEN}" "https://${CLUSTER_HOST}:8443/idmgmt/identity/api/v1/certificates/${cert}"
  echo -e ""
done

NUM_CURRENT_CACHED_CERT_IDS=$(curl -X GET -sk --header "Accept: application/json" --header "Content-Type: application/json" --header "Authorization: ${CERT_TOKEN}" "https://${CLUSTER_HOST}:8443/idmgmt/identity/api/v1/certificates" | jq .[] | jq .id | wc -l)

echo "  After cleanup, we CURRENTLY have ${NUM_CURRENT_CACHED_CERT_IDS} certificates"

We upgraded a server from 2.1.0.2 to 2.1.0.3 and this script did not work because the auth endpoint was returning 0 certs but it still was erroring that there were more than 25. It may be important to run this before upgrading.

I found another temporary workaround:

Do the following on each master:

1. Create a Dockerfile containing:
   2.1.0.3 is image tag 1.0.0 and 2.1.0.2 is image tag 2.1.0.2

FROM ibmcom/icp-identity-manager:1.0.0

RUN echo '{ "certificate_limit": 2500 }' > /opt/ibm/identity-mgmt/config/identity_prop.json && cat /opt/ibm/identity-mgmt/config/identity_prop.json

2. Build a new image:

docker build -t ibmcom/icp-identity-manager:1.0.0-cert-fix .

After the image is created on the masters, edit the daemonset and change the Image tag value:

kubectl -n kube-system edit daemonset auth-idp

The pods will restart and run the new image with an increased cert limit.