Skip to content

Instantly share code, notes, and snippets.

@heartshare
Forked from MarWeUMR/rke2-ha.md
Created January 28, 2025 18:25

Revisions

  1. @MarWeUMR MarWeUMR revised this gist Sep 7, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion rke2-ha.md
    Original file line number Diff line number Diff line change
    @@ -281,5 +281,5 @@ echo "export KUBECONFIG=$HOME/.kube/rke2.yaml" >> $HOME/.bashrc
    Here are some possible things that might resolve issues because of old/faulty entries:
    - `iptables -F`
    - `rm -rf /var/lib/rancher/rke2/server/db`
    - `rm /run/k3s/containerd/*`
    - `rm -rf /run/k3s/containerd/*`
    - `rm /etc/rancher/rke2/rke2.yaml`
  2. @MarWeUMR MarWeUMR revised this gist Aug 31, 2023. 1 changed file with 3 additions and 2 deletions.
    5 changes: 3 additions & 2 deletions rke2-ha.md
    Original file line number Diff line number Diff line change
    @@ -278,7 +278,8 @@ echo "export KUBECONFIG=$HOME/.kube/rke2.yaml" >> $HOME/.bashrc
    # Troubleshooting
    Here are some possible things that might resolve issues:
    Here are some possible things that might resolve issues because of old/faulty entries:
    - `iptables -F`
    - `rm -rf /var/lib/rancher/rke2/server/db`
    - `rm /run/k3s/containerd/*`
    - `rm /run/k3s/containerd/*`
    - `rm /etc/rancher/rke2/rke2.yaml`
  3. @MarWeUMR MarWeUMR revised this gist Aug 31, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion rke2-ha.md
    Original file line number Diff line number Diff line change
    @@ -280,5 +280,5 @@ echo "export KUBECONFIG=$HOME/.kube/rke2.yaml" >> $HOME/.bashrc
    Here are some possible things that might resolve issues:
    - `iptables -F`
    - `rm -rf /var/lib/rancher/rke/server/db`
    - `rm -rf /var/lib/rancher/rke2/server/db`
    - `rm /run/k3s/containerd/*`
  4. @MarWeUMR MarWeUMR revised this gist Aug 31, 2023. 1 changed file with 8 additions and 1 deletion.
    9 changes: 8 additions & 1 deletion rke2-ha.md
    Original file line number Diff line number Diff line change
    @@ -274,4 +274,11 @@ echo "export PATH=$PATH:/var/lib/rancher/rke2/bin" >> $HOME/.bashrc
    mkdir ~/.kube
    cat /etc/rancher/rke2/rke2.yaml > ~/.kube/rke2.yaml
    echo "export KUBECONFIG=$HOME/.kube/rke2.yaml" >> $HOME/.bashrc
    ```
    ```
    # Troubleshooting
    Here are some possible things that might resolve issues:
    - `iptables -F`
    - `rm -rf /var/lib/rancher/rke/server/db`
    - `rm /run/k3s/containerd/*`
  5. @MarWeUMR MarWeUMR revised this gist Aug 31, 2023. 1 changed file with 3 additions and 4 deletions.
    7 changes: 3 additions & 4 deletions rke2-ha.md
    Original file line number Diff line number Diff line change
    @@ -100,7 +100,7 @@ upstream backend {
    least_conn;
    server CP_NODE_1:9345 max_fails=3 fail_timeout=5s;
    server CP_NODE_2:9345 max_fails=3 fail_timeout=5s;
    #server CP_NODE_3:9345 max_fails=3 fail_timeout=5s;
    server CP_NODE_3:9345 max_fails=3 fail_timeout=5s;
    }

    # This server accepts all traffic to port 9345 and passes it to the upstream.
    @@ -113,7 +113,7 @@ upstream ef_cx_api {
    least_conn;
    server CP_NODE_1:6443 max_fails=3 fail_timeout=5s;
    server CP_NODE_2:6443 max_fails=3 fail_timeout=5s;
    #server CP_NODE_3:6443 max_fails=3 fail_timeout=5s;
    server CP_NODE_3:6443 max_fails=3 fail_timeout=5s;
    }
    server {
    listen 6443;
    @@ -264,9 +264,8 @@ curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -
    And start the service:
    ```bash
    systemctl start rke2-server
    systemctl enable rke2-server
    systemctl start rke2-server
    ```
    Now add the kubeconfig and binaries path:
  6. @MarWeUMR MarWeUMR revised this gist Aug 31, 2023. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions rke2-ha.md
    Original file line number Diff line number Diff line change
    @@ -7,6 +7,10 @@ Helpful links:
    We need all machines up and running.
    At first we will setup the loadbalancer.

    ##### Note:
    Keep in mind to use `rke2-killall.sh` if you need to restart the service.
    You also may need to delete `/var/lib/rancher/rke/server/db` before setting up the cluster again.

    ## Loadbalancer Nginx Setup

    ### Configure /etc/hosts
  7. @MarWeUMR MarWeUMR revised this gist Aug 30, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion rke2-ha.md
    Original file line number Diff line number Diff line change
    @@ -179,7 +179,7 @@ Now we install rke2:
    curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -
    ```
    And start the service:
    And start the service (ONE BY ONE!):
    ```bash
    systemctl start rke2-server
  8. @MarWeUMR MarWeUMR revised this gist Aug 30, 2023. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion rke2-ha.md
    Original file line number Diff line number Diff line change
    @@ -268,5 +268,7 @@ Now add the kubeconfig and binaries path:
    ```bash
    echo "export PATH=$PATH:/var/lib/rancher/rke2/bin" >> $HOME/.bashrc
    echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml" >> $HOME/.bashrc
    mkdir ~/.kube
    cat /etc/rancher/rke2/rke2.yaml > ~/.kube/rke2.yaml
    echo "export KUBECONFIG=$HOME/.kube/rke2.yaml" >> $HOME/.bashrc
    ```
  9. @MarWeUMR MarWeUMR revised this gist Aug 30, 2023. 1 changed file with 147 additions and 1 deletion.
    148 changes: 147 additions & 1 deletion rke2-ha.md
    Original file line number Diff line number Diff line change
    @@ -123,4 +123,150 @@ And (re)start the server `systemctl restart nginx`.
    ## Prepare the k8s Cluster Servers

    Now, before we start our cluster, all machines must be prepared.
    As a first step, lets create the necessary config files and directories.
    As a first step, lets create the necessary config files and directories.

    ```bash
    mkdir -p /etc/rancher/rke2/
    mkdir -p /var/lib/rancher/rke2/server/manifests/
    ```

    and now we create the rke config (remember to change ips and fqdn):

    ```bash
    cat<<EOF|tee /etc/rancher/rke2/config.yaml
    tls-san:
    # Loadbalancer
    - devops67.ef.com
    - 10.192.168.67
    # Cluster
    - devops61.ef.com
    - 10.192.168.61
    - devops62.ef.com
    - 10.192.168.62
    - devops63.ef.com
    - 10.192.168.63
    EOF
    ```
    Next create the nginx ingress config:
    ```bash
    cat<<EOF| tee /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml
    ---
    apiVersion: helm.cattle.io/v1
    kind: HelmChartConfig
    metadata:
    name: rke2-ingress-nginx
    namespace: kube-system
    spec:
    valuesContent: |-
    controller:
    metrics:
    service:
    annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "10254"
    config:
    use-forwarded-headers: "true"
    allowSnippetAnnotations: "true"
    EOF
    ```
    Now we install rke2:
    ```bash
    curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -
    ```
    And start the service:
    ```bash
    systemctl start rke2-server
    systemctl enable rke2-server
    ```
    Now add the kubeconfig and binaries path:
    ```bash
    echo "export PATH=$PATH:/var/lib/rancher/rke2/bin" >> $HOME/.bashrc
    echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml" >> $HOME/.bashrc
    ```
    And as the last step get the token for the other Cluster Nodes to join:
    ```bash
    cat /var/lib/rancher/rke2/server/node-token
    ```
    ### Configuring the other Cluster Nodes
    We start again by creating the directories:
    ```bash
    mkdir -p /etc/rancher/rke2/
    mkdir -p /var/lib/rancher/rke2/server/manifests/
    ```
    and creaing the rke config file:
    ```bash
    cat<<EOF|tee /etc/rancher/rke2/config.yaml
    server: https://10.192.168.67:9345 # Loadbalancer ip
    token: [token from /var/lib/rancher/rke2/server/node-token on server node 1]
    tls-san:
    - devops67.ef.com
    - 10.192.168.67
    - devops61.ef.com
    - 10.192.168.61
    - devops62.ef.com
    - 10.192.168.62
    - devops63.ef.com
    - 10.192.168.63
    EOF
    ```
    And the nginx config again.
    ```bash
    cat<<EOF| tee /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml
    ---
    apiVersion: helm.cattle.io/v1
    kind: HelmChartConfig
    metadata:
    name: rke2-ingress-nginx
    namespace: kube-system
    spec:
    valuesContent: |-
    controller:
    metrics:
    service:
    annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "10254"
    config:
    use-forwarded-headers: "true"
    allowSnippetAnnotations: "true"
    EOF
    ```
    Now we install rke2:
    ```bash
    curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -
    ```
    And start the service:
    ```bash
    systemctl start rke2-server
    systemctl enable rke2-server
    ```
    Now add the kubeconfig and binaries path:
    ```bash
    echo "export PATH=$PATH:/var/lib/rancher/rke2/bin" >> $HOME/.bashrc
    echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml" >> $HOME/.bashrc
    ```
  10. @MarWeUMR MarWeUMR revised this gist Aug 30, 2023. 1 changed file with 8 additions and 0 deletions.
    8 changes: 8 additions & 0 deletions rke2-ha.md
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,7 @@
    Helpful links:
    - https://docs.expertflow.com/cx/rke2-deployment-in-high-availability-with-nginx-haproxy-195855073.html
    - https://docs.rke2.io/install/ha

    # Preparations

    We need all machines up and running.
    @@ -116,3 +120,7 @@ server {

    And (re)start the server `systemctl restart nginx`.

    ## Prepare the k8s Cluster Servers

    Now, before we start our cluster, all machines must be prepared.
    As a first step, lets create the necessary config files and directories.
  11. @MarWeUMR MarWeUMR revised this gist Aug 30, 2023. 1 changed file with 97 additions and 1 deletion.
    98 changes: 97 additions & 1 deletion rke2-ha.md
    Original file line number Diff line number Diff line change
    @@ -5,6 +5,7 @@ At first we will setup the loadbalancer.

    ## Loadbalancer Nginx Setup

    ### Configure /etc/hosts
    ```bash
    $ cat /etc/hosts

    @@ -19,4 +20,99 @@ $ cat /etc/hosts
    <k8s server 1 ip> <k8s server 1 fqdn> CP_NODE_1 # <- We need the alias names in the nginx config
    <k8s server 2 ip> <k8s server 1 fqdn> CP_NODE_2
    <k8s server 3 ip> <k8s server 1 fqdn> CP_NODE_3
    ```
    ```
    ### Configure /etc/nginx/nginx.conf

    Now lets setup the nginx config:

    ```bash
    user nginx;
    worker_processes 4;
    worker_rlimit_nofile 40000;
    error_log /var/log/nginx/error.log;
    pid /run/nginx.pid;


    include /etc/nginx/modules-enabled/*.conf;
    # Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
    include /usr/share/nginx/modules/*.conf;

    events {
    worker_connections 8192;
    }


    http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    types_hash_max_size 2048;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
    }

    stream {
    upstream backend {
    least_conn;
    server CP_NODE_1:9345 max_fails=3 fail_timeout=5s;
    server CP_NODE_2:9345 max_fails=3 fail_timeout=5s;
    #server CP_NODE_3:9345 max_fails=3 fail_timeout=5s;
    }

    # This server accepts all traffic to port 9345 and passes it to the upstream.
    # Notice that the upstream name and the proxy_pass need to match.
    server {
    listen 9345;
    proxy_pass backend;
    }
    upstream ef_cx_api {
    least_conn;
    server CP_NODE_1:6443 max_fails=3 fail_timeout=5s;
    server CP_NODE_2:6443 max_fails=3 fail_timeout=5s;
    #server CP_NODE_3:6443 max_fails=3 fail_timeout=5s;
    }
    server {
    listen 6443;
    proxy_pass ef_cx_api;
    }
    }
    ```

    And (re)start the server `systemctl restart nginx`.

  12. @MarWeUMR MarWeUMR created this gist Aug 30, 2023.
    22 changes: 22 additions & 0 deletions rke2-ha.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,22 @@
    # Preparations

    We need all machines up and running.
    At first we will setup the loadbalancer.

    ## Loadbalancer Nginx Setup

    ```bash
    $ cat /etc/hosts

    ####################################################################
    # IP FQDN ALIASES
    #-------------- --------------------------- ------------------------

    # Loadbalancer
    <lb ip> <lb fqdn> <lb alias>

    # K8s Server
    <k8s server 1 ip> <k8s server 1 fqdn> CP_NODE_1 # <- We need the alias names in the nginx config
    <k8s server 2 ip> <k8s server 1 fqdn> CP_NODE_2
    <k8s server 3 ip> <k8s server 1 fqdn> CP_NODE_3
    ```