GCE Metadata server emulator with the GCP OpsAgent

ops_agent.md

The following describes how to run the GCE Metadata Server Emulator with the GCP OpsAgent

Note that the opsAgent is only supported on specific GCP VMs as described here ("The Ops Agent is not supported on Amazon Elastic Compute Cloud (Amazon EC2) instances or on-premises machines.")

However, if you really want to, you can coax it to run with this emulator and the following steps details such a configuration where the ops agent run in a local docker container alongside the emulator.

(Needless to say, this is unsupported and can result in unpredictable results (eg, the logs appear as if from a gce_vm). Forr on-prem, you should use bindplane)).

### First allow the service account the metadata server runs as access to write logs and metrics
export PROJECT_ID=`gcloud config get-value core/project`

gcloud projects  add-iam-policy-binding \
   $PROJECT_ID --member=serviceAccount:metadata-sa@$PROJECT_ID.iam.gserviceaccount.com \
   --role=roles/monitoring.metricWriter

gcloud projects  add-iam-policy-binding \
   $PROJECT_ID --member=serviceAccount:metadata-sa@$PROJECT_ID.iam.gserviceaccount.com \
   --role=roles/logging.logWriter

now run the container. Remember to import the service account JSON file

docker run -ti -v ./certs:/certs/ \
     -e GCE_METADATA_ROOT=127.0.0.1:80 \
     --add-host metadata.google.internal:127.0.0.1  debian:latest /bin/bash

Within the container install the metadata server

apt-get update
apt-get install curl vim systemctl wget net-tools rsyslog -y
systemctl start rsyslog

wget https://github.com/salrashid123/gce_metadata_server/releases/download/v4.1.1/gce_metadata_server_4.1.1_linux_amd64
wget https://raw.githubusercontent.com/salrashid123/gce_metadata_server/refs/heads/master/config.json

## make sure your config.json reflects the correct service account name and correct projectId, numericProjectId where you want to emit the logs/metrics to:

## then start
chmod u+x gce_metadata_server_4.1.1_linux_amd64
./gce_metadata_server_4.1.1_linux_amd64 --configFile=config.json -metricsEnabled  \
     -alsologtostderr -v 50  -port :80 --serviceAccountFile=/certs/metadata-sa.json

Now configure and start the ops agent.

Create a new shell on the container:

$ docker ps
CONTAINER ID   IMAGE           COMMAND       CREATED         STATUS         PORTS     NAMES
79e05042a2f4   debian:latest   "/bin/bash"   4 minutes ago   Up 4 minutes             epic_keller

$ docker exec -ti 79e05042a2f4 /bin/bash
root@79e05042a2f4:/# 

### add entry to /etc/hosts (though this step isn't necessary since we started the container with --add-host switch)
vi /etc/hosts
127.0.0.1 metadata.google.internal

## verify connectivity
curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token

### install the ops agent
curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
bash add-google-cloud-ops-agent-repo.sh --also-install

Important: Edit the three ops agent config files and add Environment="GCE_METADATA_HOST=localhost:80" override under the [Service] section:

vi /usr/lib/systemd/system/google-cloud-ops-agent.service
vi /usr/lib/systemd/system/google-cloud-ops-agent-opentelemetry-collector.service
vi /usr/lib/systemd/system/google-cloud-ops-agent-fluent-bit.service

Edit /etc/google-cloud-ops-agent/config.yaml and configure as shown below.

What this configuration does is emits syslog data and metrics from the host.

In addition, it collects prometheus metrics from the gce emulator and emits them to GCP

$ cat  /etc/google-cloud-ops-agent/config.yaml

logging:
  receivers:
    syslog:
      type: files
      include_paths:
      - /var/log/messages
      - /var/log/syslog
  service:
    pipelines:
      default_pipeline:
        receivers: [syslog]
metrics:
  receivers:
    hostmetrics:
      type: hostmetrics
      collection_interval: 60s

    prometheus_receiver:
      type: prometheus
      config:
        scrape_configs:
          - job_name: 'gce-metadata-emulator'
            scrape_interval: 60s
            enable_http2: false
            metrics_path: /metrics
            static_configs:
              - targets: ['localhost:9000']
            metric_relabel_configs:
              - source_labels: [ __name__ ]
                regex: '^(metadata_endpoint_latency_seconds_bucket|go_|promhttp_|process_)'
                action: drop

  processors:
    metrics_filter:
      type: exclude_metrics
      metrics_pattern: []
  service:
    pipelines:
      default_pipeline:
        receivers: [hostmetrics]
        processors: [metrics_filter]
      emulator_pipeline:
        receivers: [prometheus_receiver]

Now finally restart the components and verify theyr'e running:

systemctl stop google-cloud-ops-agent
systemctl stop google-cloud-ops-agent-fluent-bit.service
systemctl stop google-cloud-ops-agent-opentelemetry-collector.service

systemctl start google-cloud-ops-agent
systemctl start google-cloud-ops-agent-fluent-bit.service
systemctl start google-cloud-ops-agent-opentelemetry-collector.service

systemctl status "google-cloud-ops-agent*"

    google-cloud-ops-agent-fluent-bit.service - Google Cloud Ops Agent - Logging Agent
        Loaded: loaded (/usr/lib/systemd/system/google-cloud-ops-agent-fluent-bit.service, static)
        Active: active (running)

    google-cloud-ops-agent-opentelemetry-collector.service - Google Cloud Ops Agent - Metrics Agent
        Loaded: loaded (/usr/lib/systemd/system/google-cloud-ops-agent-opentelemetry-collector.service, static)
        Active: active (running)

    google-cloud-ops-agent.service - Google Cloud Ops Agent
        Loaded: loaded (/usr/lib/systemd/system/google-cloud-ops-agent.service, enabled)
        Active: active (running)

# to view logs
cat /var/log/journal/google-cloud-ops-agent-fluent-bit.service.log
cat /var/log/journal/google-cloud-ops-agent-opentelemetry-collector.service.log 
cat /var/log/journal/google-cloud-ops-agent.service.log 

Now write some log lines:

logger -p local0.notice -t ${0##*/}[$$] Hello world

which should how up like this:

{
  "insertId": "16gghi0f48cchh",
  "jsonPayload": {
    "message": "2025-09-20T16:00:12.793753+00:00 720a0530f98b bash[1]: Hello world"
  },
  "resource": {
    "type": "gce_instance",
    "labels": {
      "zone": "us-central1-a",
      "project_id": "core-eso",
      "instance_id": "5775171277418378000"
    }
  },
  "timestamp": "2025-09-20T16:00:12.794284395Z",
  "labels": {
    "compute.googleapis.com/resource_name": "720a0530f98b"
  },
  "logName": "projects/core-eso/logs/syslog",
  "receiveTimestamp": "2025-09-20T16:00:13.863606351Z"
}

echo $PROJECT_ID

### to list the emulator metircs:
curl -s -H "Authorization: Bearer `gcloud auth print-access-token`" \
    "https://monitoring.googleapis.com/v3/projects/$PROJECT_ID/metricDescriptors?filter=metric.type=starts_with(\"prometheus.googleapis.com/\")"

curl -s -H "Authorization: Bearer `gcloud auth print-access-token`" \
    "https://monitoring.googleapis.com/v3/projects/$PROJECT_ID/metricDescriptors/prometheus.googleapis.com/service_account_name/gauge"

## adjust the start/end window
export startTime="2025-09-20T14:25:58.653Z"
export endTime="2025-09-20T14:28:58.653Z"

## then print emulator metrics
curl -s -H "Authorization: Bearer `gcloud auth print-access-token`" \
    "https://monitoring.googleapis.com/v3/projects/$PROJECT_ID/timeSeries?filter=metric.type=starts_with(\"prometheus.googleapis.com/service_account_name\")&interval.startTime=$startTime&interval.endTime=$endTime"

### and standard cpu metrics
curl -s -H "Authorization: Bearer `gcloud auth print-access-token`" \
    "https://monitoring.googleapis.com/v3/projects/$PROJECT_ID/timeSeries?filter=metric.type=starts_with(\"agent.googleapis.com/cpu/load_1m\")&interval.startTime=$startTime&interval.endTime=$endTime"

{
  "timeSeries": [
    {
      "metric": {
        "labels": {
          "instance_name": "instance-1",
          "machine_type": "projects/708288290784/machineTypes/e2-standard-4",
          "otel_scope_name": "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver",
          "email": "[email protected]",
          "otel_scope_version": ""
        },
        "type": "prometheus.googleapis.com/service_account_name/gauge"
      },
      "resource": {
        "type": "prometheus_target",
        "labels": {
          "namespace": "5775171277418378000/instance-1",
          "project_id": "core-eso",
          "location": "us-central1-a",
          "job": "gce-metadata-emulator",
          "cluster": "__gce__",
          "instance": "localhost:9000"
        }
      },
      "metricKind": "GAUGE",
      "valueType": "DOUBLE",
      "points": [
        {
          "interval": {
            "startTime": "2025-09-20T14:28:58.653Z",
            "endTime": "2025-09-20T14:28:58.653Z"
          },
          "value": {
            "doubleValue": 1
          }
        }
      ]
    }
  ],
  "unit": "{not_a_unit}"
}

{
  "timeSeries": [
    {
      "metric": {
        "type": "agent.googleapis.com/cpu/load_1m"
      },
      "resource": {
        "type": "gce_instance",
        "labels": {
          "project_id": "core-eso",
          "instance_id": "5775171277418378000",
          "zone": "us-central1-a"
        }
      },
      "metricKind": "GAUGE",
      "valueType": "DOUBLE",
      "points": [
        {
          "interval": {
            "startTime": "2025-09-20T14:28:51.643057Z",
            "endTime": "2025-09-20T14:28:51.643057Z"
          },
          "value": {
            "doubleValue": 0.42
          }
        }
      ]
    }
  ],
  "unit": "1"
}

You should see the logs and metrics show up in the cloud console. Prometheus Target and VM Instance