Bitnami Image Migration Action Plan for Energy Web Foundation - Critical ECR Credentials Fix Applied

bitnami-migration-action-plan.md

Bitnami Image Migration Action Plan for Energy Web Foundation

🚨 CRITICAL ISSUE IDENTIFIED

ArgoCD ECR credentials updater is failing due to bitnami/kubectl:1.23 image not found, causing authentication issues for pulling ECR images.

📊 Current Status Analysis

Dev Cluster (k8s-misc)

FAILING COMPONENTS:

• ❌ argocd-ecr-credentials-updater - CronJob using bitnami/kubectl:1.23 (CRITICAL)
  • Error: failed to pull and unpack image "docker.io/bitnami/kubectl:1.23": not found
  • Impact: ECR authentication broken, causing cascade failures

WORKING COMPONENTS:

• ✅ pod-monitor - Already updated to bitnamisecure/kubectl:latest

OTHER BITNAMI IMAGES IN USE:

• docker.io/bitnami/redis:6.2.6-debian-10-r120 (did-auth-proxy, iam-did-auth-proxy)
• docker.io/bitnami/redis:7.2.4-debian-12-r13 (gp4btc-paypal)
• docker.io/bitnami/postgresql:14.4.0-debian-11-r9 (gp4btc, tgc)
• docker.io/bitnami/postgresql:17.0.0-debian-12-r11 (gpdive)
• docker.io/bitnami/postgresql:16.1.0-debian-11-r26 (ika)
• docker.io/bitnami/nginx:1.29.1-debian-12-r0 (ewx staking-dapp)
• docker.io/bitnami/nginx:1.21.6-debian-10-r93 (iam-did-auth-proxy)
• bitnami/sealed-secrets-controller:v0.18.0 (kube-system)

Staging k8s-misc Cluster

FAILING COMPONENTS:

• ❌ argocd-ecr-credentials-updater - Same issue as dev

OTHER BITNAMI IMAGES:

• docker.io/bitnami/postgresql:14.4.0-debian-11-r9 (gp4btc)
• docker.io/bitnami/redis:6.2.6-debian-10-r120 (iam-did-auth-proxy)
• docker.io/bitnami/nginx:1.21.6-debian-10-r93 (iam-did-auth-proxy)
• bitnami/sealed-secrets-controller:v0.18.0 (kube-system)

Staging v2 Cluster

• ✅ docker.io/bitnami/sealed-secrets-controller:v0.24.0 (newer version, likely working)

🎯 IMMEDIATE ACTIONS REQUIRED

1. CRITICAL - Fix ArgoCD ECR Credentials Updater (PRIORITY 1)

Repository: energywebfoundation/argocd-ecr-credentials-updater File: templates/cronJob.yaml

Current:

image: bitnami/kubectl:1.23

Fix:

image: bitnamisecure/kubectl:1.23

Deployment Commands:

# Update both clusters immediately
kubectl --context=dev-cluster patch cronjob argocd-ecr-credentials-updater -n argocd --type='merge' -p='{"spec":{"jobTemplate":{"spec":{"template":{"spec":{"containers":[{"name":"kubectl","image":"bitnamisecure/kubectl:1.23"}]}}}}}}'

kubectl --context=staging-k8s-misc patch cronjob argocd-ecr-credentials-updater -n argocd --type='merge' -p='{"spec":{"jobTemplate":{"spec":{"template":{"spec":{"containers":[{"name":"kubectl","image":"bitnamisecure/kubectl:1.23"}]}}}}}}'

2. Update Repository Code (PRIORITY 2)

Repository Updates Needed:

A. energywebfoundation/argocd-ecr-credentials-updater

• File: templates/cronJob.yaml line ~23
• Change: bitnami/kubectl:1.23 → bitnamisecure/kubectl:1.23

B. energywebfoundation/erpc-helm

• File: values.yaml lines 23-26
• Current:

  kubectl:
    image:
      repository: bitnami/kubectl
      pullPolicy: IfNotPresent
      tag: "1.21"

• Fix:

  kubectl:
    image:
      repository: bitnamisecure/kubectl
      pullPolicy: IfNotPresent
      tag: "1.21"

C. energywebfoundation/operating-envelope-dle-node

• File: helm/sandbox/substrate-node-0/values.yaml lines 55-58
• Current:

  kubectl:
    image:
      repository: bitnami/kubectl
      tag: latest

• Fix:

  kubectl:
    image:
      repository: bitnamisecure/kubectl
      tag: latest

3. Monitor Other Bitnami Images (PRIORITY 3)

These images are currently working but should be monitored for future migration:

• PostgreSQL images (multiple versions in use)
• Redis images (multiple versions in use)
• Nginx images (multiple versions in use)
• Sealed Secrets Controller (consider upgrading to newer version)

🔍 Root Cause Analysis

1. Bitnami Registry Migration: Bitnami announced migration from bitnami/ to bitnamisecure/ namespace
2. Pod Monitor Fix: Already applied - pod-monitor cronjob updated to bitnamisecure/kubectl:latest
3. Cascade Effect: ECR credentials updater failure prevents pulling other ECR images, causing scheduling issues

📝 Verification Steps

After applying fixes:

# Check ArgoCD ECR updater status
kubectl --context=dev-cluster get pods -n argocd -l job-name -o wide

# Verify ECR credentials are being updated
kubectl --context=dev-cluster get secret aws-ecr-creds -n argocd -o yaml

# Check for image pull errors
kubectl --context=dev-cluster get events --all-namespaces | grep -i "failed to pull\|ImagePull"

🚀 Implementation Timeline

1. Immediate (< 30 minutes): Apply kubectl patches to both clusters
2. Same day: Create PRs for repository updates
3. This week: Monitor other bitnami images for deprecation notices
4. Ongoing: Set up monitoring for registry migration announcements

📊 Node Group Analysis

Current Dev Cluster Nodes: 22 nodes (all in default group)

• No issues found with node availability
• The mentioned EWX_marketplace_EKS-2024041809412703080000001c worker group is not visible
• Current scheduling working fine on default node group

🔧 Quick Fix Commands

# IMMEDIATE FIX - Run these now:

# Dev cluster
kubectl --context=dev-cluster patch cronjob argocd-ecr-credentials-updater -n argocd --type='merge' -p='{"spec":{"jobTemplate":{"spec":{"template":{"spec":{"containers":[{"name":"kubectl","image":"bitnamisecure/kubectl:1.23"}]}}}}}}'

# Staging cluster  
kubectl --context=staging-k8s-misc patch cronjob argocd-ecr-credentials-updater -n argocd --type='merge' -p='{"spec":{"jobTemplate":{"spec":{"template":{"spec":{"containers":[{"name":"kubectl","image":"bitnamisecure/kubectl:1.23"}]}}}}}}'

# Verify the fix
kubectl --context=dev-cluster get cronjob argocd-ecr-credentials-updater -n argocd -o yaml | grep image:
kubectl --context=staging-k8s-misc get cronjob argocd-ecr-credentials-updater -n argocd -o yaml | grep image:

---

Status: ⚠️ CRITICAL - Execute immediate fixes first, then proceed with repository updates Last Updated: $(date) Clusters Analyzed: dev-cluster, staging-k8s-misc, staging-v2-cluster