AWS Spot Instance Termination Monitor

monitor-termination.service

[Unit]
Description=Monitor AWS Spot Termination Event
After=network.target

[Service]
ExecStart=/opt/monitor_termination.sh
Restart=always
User=root

[Install]
WantedBy=multi-user.target

monitor_termination.sh

#!/bin/bash

LOG_FILE="/var/log/spot_termination_monitor.log"
AWS_REGION="ap-southeast-3"

# Get the instance ID from metadata
INSTANCE_ID=$(/usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id)

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE"
}

# Function to check termination event
check_termination_event() {
    STATUS_CODE=$(/usr/bin/curl -s -o /dev/null -w "%{http_code}" http://169.254.169.254/latest/meta-data/spot/termination-time)
    echo "$STATUS_CODE"
}

# Function to set the instance as unhealthy
set_instance_unhealthy() {
    log "Setting instance $INSTANCE_ID to Unhealthy in region $AWS_REGION..."
    /usr/bin/aws autoscaling set-instance-health --instance-id "$INSTANCE_ID" --health-status Unhealthy --region "$AWS_REGION"
}

log "Monitoring for Spot Instance termination events..."

while true; do
    STATUS_CODE=$(check_termination_event)
    
    if [ "$STATUS_CODE" -eq 200 ]; then
        log "Termination event detected! Marking instance as Unhealthy..."
        set_instance_unhealthy
        exit 0
    else
        log "No termination event detected. Checking again in 30 seconds..."
    fi

    sleep 30
done

AWS Spot Instance Termination Monitor

📌 Description

This script monitors AWS Spot Instance termination events and automatically sets the instance’s health status to Unhealthy in an Auto Scaling Group. This ensures the instance is removed from the load balancer before AWS reclaims it.

🔹 Features:

✅ Checks for termination events every 30 seconds
✅ Logs all actions to /var/log/spot_termination_monitor.log
✅ Sets the instance Unhealthy in ap-southeast-3 region
✅ Runs as a systemd service for automatic monitoring

🚀 How to Use

1️⃣ Download & Set Up the Script

Save the script to /opt/monitor_termination.sh:

sudo curl -o /opt/monitor_termination.sh https://gist.githubusercontent.com/cipulan/409ee59b19ed7746567e417922ede8d5/raw/18b6903f2b6ee1390dc0ab15d55097dc6fd74ef6/monitor_termination.sh
sudo chmod +x /opt/monitor_termination.sh

2️⃣ (Optional) Run the Script Manually

To test the script manually:

sudo /opt/monitor_termination.sh

3️⃣ Run as a systemd Service (Recommended)

1. Create a systemd service file:

sudo nano /etc/systemd/system/monitor-termination.service

2. Add the following content:

[Unit]
Description=Monitor AWS Spot Termination Event
After=network.target

[Service]
ExecStart=/opt/monitor_termination.sh
Restart=always
User=root

[Install]
WantedBy=multi-user.target

3. Enable & start the service:

sudo systemctl daemon-reload
sudo systemctl enable monitor-termination
sudo systemctl start monitor-termination

4️⃣ Check Logs & Status
• View Service Status:

sudo systemctl status monitor-termination

• Check Logs:

tail -f /var/log/spot_termination_monitor.log

5️⃣ (Optional) Stop or Restart
• To stop the service:

sudo systemctl stop monitor-termination

• To restart after modifying the script:

sudo systemctl restart monitor-termination

🛠️ Troubleshooting
• If AWS CLI is not installed, install it:

sudo apt update && sudo apt install awscli -y

• Ensure AWS credentials are configured for the instance:

aws configure

• Check if the instance has permissions to update Auto Scaling health:
Attach the following policy to the IAM role:

{
  "Effect": "Allow",
  "Action": "autoscaling:SetInstanceHealth",
  "Resource": "*"
}

This script helps ensure smooth scaling and prevents unexpected 502 errors due to AWS spot termination. 🚀 Let me know if you need any modifications!