Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save jfinstrom/60011630ed586a79f5b9c78313e0d708 to your computer and use it in GitHub Desktop.
Save jfinstrom/60011630ed586a79f5b9c78313e0d708 to your computer and use it in GitHub Desktop.
FreePBX Endpoint Zero-Day — incident note, detection & remediation checklist
#!/usr/bin/env bash
# collect_forensics_freepbx.sh
# Guarded forensic artifact collection for FreePBX CVE-2025-57819 incidents.
# PURPOSE: copy timestamps and preserve ownership/permissions of key artifacts into a single destination for offline review.
# SAFETY: script WILL NOT run unless run as root, a destination directory is provided, and --confirm is passed.
# USAGE: ./collect_forensics_freepbx.sh --dest /secure/location --confirm
# DO NOT RUN this script unless you intend to collect artifacts on the target host.
set -euo pipefail
usage() {
cat <<EOF
Usage: $0 --dest /path/to/output --confirm [--dry-run]
--dest PATH Destination directory where artifacts will be collected (must be empty or non-existent).
--confirm Required; prevents accidental execution.
--dry-run Show what will be collected without copying files.
EOF
exit 1
}
DEST=""
CONFIRM=0
DRYRUN=0
while [[ $# -gt 0 ]]; do
case "$1" in
--dest) DEST="$2"; shift 2;;
--confirm) CONFIRM=1; shift;;
--dry-run) DRYRUN=1; shift;;
-h|--help) usage;;
*) echo "Unknown arg: $1"; usage;;
esac
done
if [[ $CONFIRM -ne 1 || -z "$DEST" ]]; then
echo "ERROR: --dest and --confirm are required."
usage
fi
if [[ $EUID -ne 0 ]]; then
echo "ERROR: run as root (sudo). Exiting."
exit 2
fi
TIMESTAMP="$(date -u +%Y%m%dT%H%M%SZ)"
OUTDIR="${DEST%/}/freepbx_forensics_${TIMESTAMP}"
LOG="${OUTDIR}.collect.log"
echo "Collection target: $OUTDIR"
echo "Logfile: $LOG"
if [[ $DRYRUN -eq 1 ]]; then
echo "[DRY-RUN] Would create: $OUTDIR"
else
if [[ -e "$OUTDIR" ]]; then
echo "ERROR: destination $OUTDIR already exists. Aborting to avoid overwrite."
exit 3
fi
mkdir -p "$OUTDIR"
fi
# Helper to record actions
log() { printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" | tee -a "$LOG"; }
# List of files/dirs to collect (non-destructive)
declare -a COLLECT_PATHS=(
"/var/www/html"
"/var/www/html/.clean.sh"
"/var/www/html/cache"
"/dev/shm"
"/var/log/asterisk"
"/var/log/httpd"
"/var/log/apache2"
"/var/log/messages"
"/var/log/secure"
"/var/log/auth.log"
"/etc/passwd"
"/etc/shadow"
"/etc/rsyslog.conf"
"/etc/crontab"
"/var/spool/cron"
"/etc/ssh/sshd_config"
)
# Commands to capture output of (safe-to-run read-only)
declare -a COMMANDS=(
"fwconsole ma list"
"fwconsole --version"
"ps auxf"
"ss -tulpn"
"lsof -nP | egrep '/var/www|php|/dev/shm' || true"
"crontab -l 2>/dev/null || true"
"for u in \$(cut -f1 -d: /etc/passwd); do echo '--- crontab for:' \$u; crontab -u \$u -l 2>/dev/null; done"
"rpm -qa 2>/dev/null || dpkg --list 2>/dev/null"
"find / -xdev -type f -mtime -7 -ls 2>/dev/null | head -n 500"
)
log "BEGIN collection (dry-run=$DRYRUN)"
# Copy files/directories safely
for p in "${COLLECT_PATHS[@]}"; do
if [[ -e "$p" ]]; then
rel="$(echo "$p" | sed 's#/##' | tr / _)"
target_dir="$OUTDIR/collected$(dirname "$p")"
if [[ $DRYRUN -eq 1 ]]; then
log "[DRY-RUN] Would copy: $p -> $target_dir/"
else
mkdir -p "$target_dir"
# use rsync to preserve permissions, owners, xattrs; treat files and dirs
if command -v rsync >/dev/null 2>&1; then
rsync -aHAX --numeric-ids --progress "$p" "$target_dir/" >>"$LOG" 2>&1 || true
else
# fallback to cp -a (best-effort)
cp -a "$p" "$target_dir/" >>"$LOG" 2>&1 || true
fi
log "Collected: $p"
fi
else
log "Missing: $p (skipping)"
fi
done
# Capture outputs of read-only commands
mkdir -p "$OUTDIR/command-outputs"
for cmd in "${COMMANDS[@]}"; do
safe_name="$(echo "$cmd" | tr ' /|$' '_____')"
outfile="$OUTDIR/command-outputs/${safe_name}.${TIMESTAMP}.txt"
if [[ $DRYRUN -eq 1 ]]; then
log "[DRY-RUN] Would run: $cmd -> $outfile"
else
log "Running: $cmd"
# shellcheck disable=SC2086
bash -lc "$cmd" > "$outfile" 2>&1 || true
fi
done
# Record basic system metadata
if [[ $DRYRUN -eq 1 ]]; then
log "[DRY-RUN] Would record system metadata (uname, df, mount, ip addr)"
else
uname -a > "$OUTDIR/uname.txt" 2>&1 || true
df -h > "$OUTDIR/df-h.txt" 2>&1 || true
mount > "$OUTDIR/mounts.txt" 2>&1 || true
ip addr show > "$OUTDIR/ip-addr.txt" 2>&1 || true
echo "Timestamp: $TIMESTAMP" > "$OUTDIR/collected_timestamp.txt"
fi
# Generate checksums for collected files
if [[ $DRYRUN -eq 1 ]]; then
log "[DRY-RUN] Would compute checksums for collected files"
else
log "Computing SHA256 checksums for collected files..."
(cd "$OUTDIR" && find . -type f -print0 | xargs -0 sha256sum > "$OUTDIR/checksums.sha256") || true
fi
# Create archive (optional)
ARCHIVE="${OUTDIR}.tar.gz"
if [[ $DRYRUN -eq 1 ]]; then
log "[DRY-RUN] Would create archive: $ARCHIVE"
else
log "Creating compressed archive: $ARCHIVE"
tar -C "$(dirname "$OUTDIR")" -czf "$ARCHIVE" "$(basename "$OUTDIR")" >>"$LOG" 2>&1 || true
log "Archive created at: $ARCHIVE"
fi
log "Collection complete. Artifacts: $OUTDIR"
log "Logfile: $LOG"
log "Do NOT transmit archive to untrusted networks. Transfer offline or over trusted, audited channels."
exit 0

FreePBX Emergency Checklist — Minimal immediate actions (copy/paste for chat/Slack) CVE-2025-57819

  1. Isolate
  • Immediately block public access to Admin UI (80/443). Example (iptables):
    • iptables -I INPUT -p tcp --dport 443 -s x.x.x.x -j ACCEPT # allow your admin IP
    • iptables -I INPUT -p tcp --dport 443 -j DROP # drop all other 443
    • iptables -I INPUT -p tcp --dport 80 -j DROP # drop HTTP
  • If possible: place the PBX behind VPN or on a management VLAN.
  1. Confirm Endpoint module presence
  • fwconsole ma list | egrep -i "endpoint"
  1. If Endpoint installed AND you have valid Sangoma support: update immediately
  • fwconsole ma downloadinstall endpoint --edge
  • OR (vendor-fixed tags)
    • fwconsole ma downloadinstall endpoint --tag 17.0.2.31
    • fwconsole ma downloadinstall endpoint --tag 16.0.88.19
  1. If you do NOT use Endpoint: uninstall & remove it
  • fwconsole ma uninstall endpoint
  • fwconsole ma remove endpoint
  1. Quick Indicators-of-Compromise (IoC) checks (run now)
  • Find attacker files:
    • find /var/www/html -maxdepth 2 -type f ( -name ".clean.sh" -o -name "monitor.php" -o -name "backend.php" ) -ls
  • Check shared memory/temp:
    • ls -al /dev/shm
    • find /tmp /var/tmp /dev/shm -type f -mtime -7 -ls
  • Search logs for Symfony error:
    • grep -R "Symfony\Component\Console\Application" /var/log 2>/dev/null || true
  • Check fwconsole failure and module list:
    • fwconsole --version
    • fwconsole ma list | egrep -i "endpoint"
  • Quick process/network snapshot:
    • ps auxf | egrep -i "php|httpd|asterisk|fwconsole"
    • ss -tulpn | egrep -i "http|asterisk|sip|5060|5061"
  1. Preserve simple artifacts before any removals
  • If /var/www/html/.clean.sh exists, copy it immediately (do NOT execute it):
    • mkdir -p /root/forensics
    • cp --preserve=mode,ownership,timestamps /var/www/html/.clean.sh /root/forensics/.clean.sh.$(date -u +%Y%m%dT%H%M%SZ)
    • stat /var/www/html/.clean.sh > /root/forensics/.clean.sh.$(date -u +%Y%m%dT%H%M%SZ).stat
  • Save recent logs and a few command outputs:
    • cp -a /var/www/html /root/forensics/www-html.$$ # quick copy
    • tail -n 500 /var/log/asterisk/full > /root/forensics/asterisk-full.$(date -u +%s).log
    • tail -n 500 /var/log/httpd/error_log > /root/forensics/http-error.$(date -u +%s).log
    • fwconsole ma list > /root/forensics/fwconsole-ma-list.$(date -u +%s).txt
    • fwconsole --version > /root/forensics/fwconsole-version.$(date -u +%s).txt
  1. If you find IoCs or any suspicious items: assume compromise
  • Isolate host (network-level) and snapshot/dump disks (cloud: create image/snapshot).
  • Rotate all credentials (web admin, root, SIP/trunk credentials, API keys).
  • Rebuild from known-good backup or fresh OS image. Do not trust in-place cleanup.
  1. Monitoring and provider checks
  • Check provider CDRs/trunk usage for unusual outgoing calls.
  • Notify/tell SIP provider to rotate trunk credentials if compromise suspected.
  1. For immediate follow-up actions (after containment)
  • Collect full forensic set (see script provided).
  • If compromised: rebuild, restore only from verified pre-compromise backups, rotate secrets, revoke/reissue TLS certs if needed.

Minimal notes:

  • The vendor-supplied EDGE fix prevents new exploitation but does NOT clean already-compromised systems.
  • If you have a complicated environment, schedule a controlled rebuild and forensic analysis.

FreePBX Endpoint Zero-Day — incident note, detection & remediation checklist

Status: Active incident. Public reports began ~2025-08-21. Vendor advisory posted 2025-08-26. Limited public disclosure and no CVE at time of writing.

UPDATE: CVE-2025-57819

This document consolidates vendor guidance and community reports into an operator-friendly incident response guide. It is intentionally pragmatic: short prioritized actions first, then detection, containment, remediation, forensic guidance and hardening.

Summary (plain language)

  • Around 2025-08-21 multiple FreePBX systems began showing errors and later confirmed compromises.
  • Vendor (Sangoma/FreePBX Security Team) published an advisory on 2025-08-26 urging administrators to restrict public access to the Administrator Control Panel and offering EDGE module fixes.
  • The vulnerability is associated with the commercial Endpoint Manager (Endpoint) and appears to be an unauthenticated privilege escalation/RCE that can be exploited when the Administrator UI is exposed to hostile networks.
  • The EDGE fix prevents new exploitation but does not clean already-compromised systems.

Key vendor guidance (as posted by Sangoma in community discussions)

  • Users are strongly advised to limit Administrator access to only known/trusted hosts (use the Firewall module, VPN, or IP restrictions).
  • EDGE module fixes were provided for testing; they are intended to be included in a formal security release after QA.
  • Commands provided by Sangoma (apply if you have a valid support/contract and Endpoint installed):
    • For FreePBX v16 (PBXAct v16): fwconsole ma downloadinstall endpoint --tag 16.0.88.19
    • For FreePBX v17 (PBXAct v17): fwconsole ma downloadinstall endpoint --tag 17.0.2.31
    • For FreePBX v16 or v17 as a testing option: fwconsole ma downloadinstall endpoint --edge

Important community-confirmed facts

  • A PHP fatal error was a common early indicator: PHP Fatal error: Uncaught Error: Class "Symfony\Component\Console\Application" not found in /var/www/html/admin/libraries/FWApplication.class.php:11 ...
  • Community members found attacker artifacts on multiple systems; repeatedly-observed files include:
    • /var/www/html/.clean.sh
    • /var/www/html/cache/monitor.php
    • /var/www/html/cache/backend.php
    • /dev/shm/.conf_
  • Content of .clean.sh (excerpt observed in multiple systems):
    • A bash script enumerating common log files and sed-ing out traces (removes lines referencing modular.php, monitor.php, backend.php, .cache, etc.), then rm $0
    • This script aims to remove log evidence and then delete itself.
  • Some community members reported large-scale impact in affected environments (example report: thousands of SIP extensions and many trunks impacted in one group’s infrastructure).
  • The EDGE fix is for future protection; Sangoma and community repeatedly noted that the fix will not "undo" changes on systems already compromised.

What we know / what we do not know

  • Known:
    • Affects commercial Endpoint Manager:
      • Version 16: prior to 16.0.88.19
      • Version 17: prior to 17.0.2.31
    • Exploit correlates with exposed FreePBX admin UI and the Endpoint module being installed.
    • Attackers leave cleanup scripts (.clean.sh) to remove traces and leave persistence artifacts.
  • Unknown (publicly):
    • Full technical exploit details and exact attack vector (no public, full disclosure yet).
    • Whether certain unsupported / older versions beyond v16/v17 are affected.
    • Exact persistence mechanisms used in all incidents (varies by attacker).

Prioritized quick actions (do these now)

  1. Isolate suspected hosts from untrusted networks (block public access to the web UI immediately).
  2. If Endpoint is installed and you have a support contract: update Endpoint now using vendor commands below.
  3. If you do NOT use Endpoint: uninstall and remove the module.
  4. If you suspect compromise: collect forensic artifacts and then rebuild from a known-good backup or fresh install — do not rely on an in-place clean.
  5. Rotate all credentials (admin, root, SIP accounts, trunk credentials, API keys). Assume compromise if any IoCs are found.
  6. Monitor both local and provider call records for suspicious outbound activity.

Immediate commands and checks

  • Check if Endpoint is installed:
    • fwconsole ma list | egrep -i "endpoint"
  • Update Endpoint (if supported and advised):
    • fwconsole ma downloadinstall endpoint --edge
    • fwconsole ma downloadinstall endpoint --tag 17.0.2.31
    • fwconsole ma downloadinstall endpoint --tag 16.0.88.19
  • Uninstall/remove Endpoint (if you do not use it):
    • fwconsole ma uninstall endpoint
    • fwconsole ma remove endpoint
  • Basic detection commands (run as root or sudo):
    • find /var/www/html -type f -name ".clean.sh" -o -name "monitor.php" -o -name "backend.php"
    • ls -al /dev/shm
    • find /tmp /var/tmp /dev/shm -type f -mtime -7 -ls
    • grep -R --line-number --binary-files=without-match -e "monitor.php" -e "backend.php" -e ".clean.sh" / 2>/dev/null
    • tail -n 200 /var/log/httpd/error_log # or /var/log/apache2/error.log
    • tail -n 200 /var/log/asterisk/full
    • grep -i "Symfony\Component\Console\Application" -R /var/log 2>/dev/null
    • ps auxf | egrep -i "php|httpd|asterisk|fwconsole"
    • ss -tulpn | egrep -i "http|asterisk|sip|5060|5061"
    • crontab -l; ls -al /etc/cron.* /var/spool/cron
    • for u in $(cut -f1 -d: /etc/passwd); do echo "crontab for $u"; crontab -u $u -l 2>/dev/null; done

Detecting .clean.sh and its behavior

  • Community found .clean.sh in /var/www/html and its purpose is to sanitize logs then delete itself. Example behavior:
    • Iterates a list of common log paths, sed -i --follow-symlinks to delete lines matching strings like "monitor.php", "backend.php", ".cache", and removes itself.
  • If you find .clean.sh:
    • Do not execute it.
    • Preserve it as an artifact (copy it off the host to your forensics server).
    • Collect modification times and file owner/permissions: stat /var/www/html/.clean.sh

Containment guidance

  • Block public access to FreePBX Admin (ports 80/443). Require VPN or allow only trusted IPs.
  • If you cannot isolate the host immediately:
    • Use firewall rules to restrict connections to the Admin UI only to management addresses.
    • Example iptables (replace x.x.x.x with admin IP):
      • iptables -I INPUT -p tcp -s x.x.x.x --dport 443 -j ACCEPT
      • iptables -I INPUT -p tcp --dport 443 -j DROP
  • If the system is compromised: take snapshots/images before performing remediation actions so you can investigate.

Remediation (patch/update and rebuild)

  • If Endpoint is installed and you have vendor support:
    • Apply the vendor fix (EDGE or tagged release) immediately (see commands above).
  • If Endpoint is not installed:
    • The community consensus is that a system without the Endpoint module installed and without the admin page exposed is unlikely to be affected.
    • Use fwconsole ma list | grep -i endpoint to confirm installation status.
  • If you have evidence of compromise:
    • Best practice: rebuild from known-good media or restore a verified pre-compromise backup. In-place cleaning is risky — attackers may have installed backdoors.
    • After rebuild, rotate all passwords and trunk credentials; notify your provider if trunk credentials may have been exposed.

Forensic collection checklist (preserve before cleaning)

  • Produce a disk image or snapshot (cloud image or dd) before changing the system if you intend to investigate or report the incident.
  • Collect these artifacts:
    • /var/www/html (especially /var/www/html/admin and /var/www/html/cache)
    • /var/www/html/.clean.sh (copy the file and record its timestamps)
    • /dev/shm contents
    • /etc/passwd and /etc/shadow (handle securely)
    • /var/log/httpd* or /var/log/apache2*; /var/log/asterisk/*; /var/log/messages; /var/log/auth.log; /var/log/secure
    • fwconsole outputs: fwconsole ma list, fwconsole --version
    • ps aux, ss -tulpn, lsof -nP
    • crontab entries for all users and /etc/cron.*, /var/spool/cron
    • Package list (rpm -qa or dpkg --list)
    • Any suspicious user accounts, SSH keys, and SUID/SGID files
  • Look for persistence:
    • New systemd units, /etc/init.d scripts, /etc/ld.so.preload, unexpected authorized_keys entries
  • Run rootkit checks (only after snapshotting or from trusted media):
    • rkhunter --check
    • chkrootkit

Observed attacker behaviors (community reports)

  • Clean-up script to scrub logs (.clean.sh).
  • Script removes references to modular.php, monitor.php, backend.php, and similar traces.
  • Attackers may attempt to re-add cleanup scripts, or touch the same files repeatedly (mtime vs ctime differences can indicate re-add).
  • Some reported incidents showed systems with visible admin failures (fwconsole failing because of missing Symfony class), other systems continued functioning while being compromised.

Hardening recommendations (preventive and ongoing)

  • Never expose the FreePBX Admin UI to the public internet. Use VPN or strict IP allow-lists.
  • If remote admin is required, use two-factor authentication and IP restrictions.
  • Remove/uninstall modules you do not use (commercial modules included).
  • Use Fail2Ban or similar to protect web UI and SSH.
  • Keep FreePBX modules and underlying OS patched; maintain a patch cadence.
  • Monitor logs centrally (SIEM, Wazuh/OSSEC) and alert on suspicious changes.
  • Periodically validate and test backups by performing restorations in an isolated environment.

Notes on backups and restores

  • Community members reported that performing a "full backup restore" to recover did not necessarily mean the system was clean — attackers persisted in places outside the restored data in some cases.
  • A true full restore must include the OS and system-level binaries, or you must rebuild the system and then re-apply configuration and data from known-good sources.

Operator TL;DR

  • If you run Endpoint and have a valid support contract: update Endpoint immediately to the vendor-specified fixed versions (EDGE or tagged).
  • If you do not use Endpoint: uninstall and remove it.
  • If you see the PHP fatal Symfony error or find .clean.sh or the listed IoCs: assume compromise, isolate the host, preserve artifacts, rotate credentials, and rebuild from a known-good backup or image.
  • Lock down your admin UI immediately — require VPN or restrict to trusted IPs.

Appendix — key commands (summary)

  • Check endpoint:
    • fwconsole ma list | egrep -i "endpoint"
  • Update endpoint (v17 example):
    • fwconsole ma downloadinstall endpoint --edge
    • fwconsole ma downloadinstall endpoint --tag 17.0.2.31
  • Update endpoint (v16 examole):
    • fwconsole ma downloadinstall endpoint --tag 16.0.88.19
  • Uninstall endpoint:
    • fwconsole ma uninstall endpoint
    • fwconsole ma remove endpoint
  • Find suspicious files:
    • find /var/www/html -type f -name ".clean.sh" -o -name "monitor.php" -o -name "backend.php"
  • Snapshot (cloud): create a block device or instance snapshot before doing remediation.

Closing note

  • This file is an operational consolidation of Sangoma's advisory and the corroborating community reports. The EDGE module fix is intended to stop further exploitation but it does not remove backdoors or evidence on already-compromised servers. Treat any hosts showing IoCs as likely compromised and follow a rebuild/restore approach after capture of forensic artifacts.

Legal Mumbo Jumbo:

DISCLAIMER: This is an unofficial, community-generated incident guide. It is NOT endorsed by or authorized by Sangoma or FreePBX. Trademarks and product names are used only for reference and do not imply any official affiliation or endorsement.

LICENSE:

  • This document and any non-code text files are licensed under Creative Commons Attribution (CC BY 4.0).
  • The forensic collection script included with these materials is provided under the Affero GPL v3 (AGPLv3).

NOTICE ABOUT CONTENT CREATION:

  • This document was written and edited with assistance from generative technologies to improve readability, gather facts from supplied community discussions, and clarify recommended steps. Human review was applied to validate and organize the information. Use your own judgement and, if necessary, consult vendor support or professional incident responders for critical decisions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment