A critical failure of a production workload can carry a significant impact on a company, and it matters how soon the administrator or developers get notified. Even though Google Cloud offers powerful monitoring and alerting capabilities, sometimes projects require more than a standard email notification. Custom alert formatting, application-specific filtering, integration with external platforms, and automated responses all require additional processing logic.
Silent Watcher is a serverless, event-driven production alerting pipeline built using Google Cloud services. It monitors error logs generated by a production Compute Engine VM, filters them using Cloud Logging, routes relevant events through Pub/Sub, and triggers an ephemeral Cloud Run Function to process and deliver the alert to an external communication platform such as Discord or Slack.
1. The Problem With Traditional Production Alerting
Production applications can fail in many different ways, such as throwing an unhandled exception, server uptime failure, a process running out of memory, database connection failure, or a configuration issue. Sometimes, the first challenge is not always fixing the problem; it is getting to know that the problem exists in a timely manner.
Google Cloud already provides powerful monitoring and alerting capabilities, including notification channels for common operational use cases. However, there are situations where a standard notification is not enough. Such as filtering logs using application-specific logs, transforming raw log entries into custom alerts, routing different types of alerts to different destinations, etc.
Furthermore, Google Cloud’s built-in Cloud Monitoring alert policies can route emails or Pub/Sub triggers natively, but the default notifications are notoriously painful to digest on a phone screen at 2:00 AM. They often arrive as wall-of-text JSON dumps or generic "Policy Triggered" emails that lack critical context, such as the exact VM Instance ID, human-readable stack traces, or localized timestamps, forcing engineers to log into the console just to understand what broke.
This led to the design of Silent Watcher.
2. Architecture & Zero-Trust Security Design
The goal behind Silent Watcher is to create a monitoring system that is separate and isolated from the production system. A monitoring system should never increase the production environment's attack surface, expose internal architecture or add weight to the production workload.
Without inventing the wheel again, this architecture uses Google Cloud's log ingestion through OpsAgent and a series of Google Cloud-managed services to filter and direct error events from production VMs to an external notification channel such as Slack or Discord.
This architecture can be divided into two layers.
- Detection and Processing Layer (Log Router Sinks and Pub/Sub Filtering)
- Notification Dispatcher Layer (Cloud Run Function and Secret Manager)
3. Detection and Processing Layer (Log Router Sinks and Pub/Sub Filtering)
Google Cloud VMs generate logs from different sources, including the application, web server, and operating system. CloudOps Agent runs in the VM as a background service on the host operating system. CloudOps Agent collects the relevant logs and streams them to Cloud Logging in real-time.
Cloud Logging evaluates these incoming log records against a Log Router filter, and only matching event logs are routed to the Pub/Sub topic. This architecture uses Cloud Logging inbuilt filters to filter through the logs before reaching the serverless processing layer. Then the Pub/Sub topic acts as a buffer between the logging system and the Cloud Run Function.
resource.type="gce_instance" labels.instance_name="YOUR-VM-NAME" -- You can use either instance_name or instance_id to identify the VM -- resource.labels.instance_id="YOUR_VM_ID_HERE" severity>=ERROR
This creates a level of decoupling between the log producer and consumer. VM does not communicate directly with the notification platform, logging system does not know how the logs are processed and delivered to the user.
4. Notification Dispatcher Layer (Cloud Run Function and Secret Manager)
When a matching log record is published into the Pub/Sub topic, the Cloud Run function is triggered automatically. The Cloud Run Function is not running continuously. It gets up when a log record enters the Pub/Sub topic, processes the log record by extracting relevant information, dispatches the notification via a webhook URL and terminates once completed successfully.
Cloud Run Function needs the Discord/Slack Webhook URL to dispatch the notification successfully to the channel. Since this webhook URL contains secret tokens for authentication, the webhook URL is stored in the Google Cloud Secret Manager and accessed from the Cloud Run Function.
{
"name": "silent-watcher",
"version": "1.0.0",
"description": "Zero-cost real-time alerting engine for GCP Cloud Logging",
"main": "index.js",
"scripts": {
"start": "functions-framework --target=watchErrorEvents"
},
"dependencies": {
"@google-cloud/functions-framework": "^3.3.0",
"@google-cloud/secret-manager": "^5.6.0"
}
}const functions = require('@google-cloud/functions-framework');
const { SecretManagerServiceClient } = require('@google-cloud/secret-manager');
const secretClient = new SecretManagerServiceClient();
let cachedWebhookUrl = null;
/**
* Retrieve the webhook URL from Secret Manager.
*
* The value is cached so that warm Cloud Run instances do not need
* to contact Secret Manager for every log event.
*/
async function getWebhookUrl() {
if (cachedWebhookUrl) {
return cachedWebhookUrl;
}
// GOOGLE_CLOUD_PROJECT and GCP_PROJECT_ID are defined as environment variables
const projectId =
process.env.GOOGLE_CLOUD_PROJECT ||
process.env.GCP_PROJECT_ID;
if (!projectId) {
throw new Error('Google Cloud project ID is not available.');
}
const secretName =
`projects/${projectId}/secrets/monitoring-webhook-url/versions/latest`;
const [version] = await secretClient.accessSecretVersion({
name: secretName
});
cachedWebhookUrl = version.payload.data
.toString('utf8')
.trim();
if (!cachedWebhookUrl) {
throw new Error('Webhook URL secret is empty.');
}
return cachedWebhookUrl;
}
/**
* Triggered by Pub/Sub
*/
functions.cloudEvent(
'watchErrorEvents',
async (cloudEvent) => {
try {
/* Extract Pub/Sub message data */
const base64Data =
cloudEvent.data?.message?.data;
if (!base64Data) {
console.warn(
'Received CloudEvent without Pub/Sub message data.'
);
return;
}
/* Decode the original Cloud Logging entry */
const rawLogMessage =
Buffer
.from(base64Data, 'base64')
.toString('utf8');
const logEntry =
JSON.parse(rawLogMessage);
/* Extract useful diagnostic information */
const severity =
logEntry.severity || 'ERROR';
const timestamp =
logEntry.timestamp ||
new Date().toISOString();
const instanceId =
logEntry.resource?.labels?.instance_id ||
logEntry.resource?.labels?.instance_name ||
'Unknown Instance';
const message =
logEntry.textPayload ||
logEntry.jsonPayload?.message ||
JSON.stringify(logEntry.jsonPayload) ||
'Undefined system anomaly detected.';
/* Retrieve webhook URL */
const webhookUrl = await getWebhookUrl();
/* Build Discord/Slack alert payload */
const discordPayload = {
username: 'Silent Watcher',
avatar_url: 'https://projectnullbyte.com/images/icons/logo.png',
embeds: [
{
title: `GCP System Alert: ${severity}`,
color:
severity === 'CRITICAL' ||
severity === 'EMERGENCY'
? 15158332
: 15105570,
fields: [
{
name: 'VM Instance',
value: `\`${instanceId}\``,
inline: true
},
{
name: 'Severity',
value: `\`${severity}\``,
inline: true
},
{
name: 'Timestamp',
value: `\`${timestamp}\``,
inline: false
},
{
name: 'Diagnostic Log Exception',
value:
`\`\`\`text\n` +
`${message.substring(0, 1000)}` +
`\n\`\`\``,
inline: false
}
],
footer: {
text:
'Project Null Byte | Silent Watcher Engine'
}
}
]
};
/* Send alert to Discord */
const response =
await fetch(
webhookUrl,
{
method: 'POST',
headers: {
'Content-Type':
'application/json'
},
body:
JSON.stringify(discordPayload)
}
);
/* Treat webhook failures as function failures */
if (!response.ok) {
const responseBody = await response.text();
throw new Error(
`Webhook delivery failed. ` +
`HTTP ${response.status}: ` +
`${responseBody}`
);
}
console.log(
`Successfully dispatched ${severity} alert ` +
`for timestamp ${timestamp}`
);
} catch (error) {
console.error(
'Fatal execution failure in Silent Watcher function:',
error
);
/* Re-throw the error so Pub/Sub/Eventarc can retry the event if the execution failed. */
throw error;
}
}
);Therefore, the production VM is separated from the notification platform. Production VM does not need to know about the existence of the webhook URL or the monitoring platform.
Each component has its assigned responsibility, and components communicate with each other using controlled, relevant Google Cloud services and IAM permissions.
Note:- Cloud Run Function should be set up to be strictly internal only. (No unauthenticated public access because threat actors might use this to spam the notification pipeline if the open endpoint is discovered)
Furthermore, to follow the principle of least privilege, Cloud Run Function should be assigned to a separate service account that is only used for the notification pipeline assigned with necessary IAM permissions, such as "Secret Manager Secret Accessor" and "Cloud Run Invoker".
5. Absolute Observability for $0/Month
The most interesting part of this alerting system architecture is not that it uses serverless services, but that the entire pipeline costs $0 monthly as the baseline. It leverages Google Cloud's free tier allowances to keep the cost to $0.
No dedicated monitoring services, no continuously running services, no paid third-party services. The architecture simply waits for an eligible event to fire a notification.
The important point is that the system is event-driven rather than resource-driven.
+-----------------------------+-----------------------------+----------------------+------------------+ | GCP Component | Monthly Free Allowance | Estimated Usage | Estimated Cost | +-----------------------------+-----------------------------+----------------------+------------------+ | Google Cloud Ops Agent | No separate agent charge; | 1 production VM | $0.00* | | | runs on the Compute Engine | | | | | host | | | +-----------------------------+-----------------------------+----------------------+------------------+ | Cloud Logging | First 50 GiB of log | ~2 GiB of retained | $0.00 | | | storage per project/month | logs per month | | +-----------------------------+-----------------------------+----------------------+------------------+ | Pub/Sub | First 10 GiB of message | ~0.01 GiB/month | $0.00 | | | delivery throughput/month | | | +-----------------------------+-----------------------------+----------------------+------------------+ | Cloud Run Functions | 2 million requests/month | ~5,000 invocations | $0.00 | | | | | | +-----------------------------+-----------------------------+----------------------+------------------+ | Cloud Run Compute | 180,000 vCPU-seconds and | ~250 vCPU-seconds | $0.00 | | | 360,000 GiB-seconds/month | ~125 GiB-seconds | | +-----------------------------+-----------------------------+----------------------+------------------+ | Secret Manager | 6 active secret versions | 1 active version | $0.00 | | | and 10,000 access | ~50-100 access calls | | | | operations/month | per month | | +-----------------------------+-----------------------------+----------------------+------------------+ | TOTAL | | | $0.00 | +-----------------------------+-----------------------------+----------------------+------------------+
Final Thoughts: Hardened, Cost-Effective Observability by Design
The architecture utilises the Google Cloud Free Tier and serverless services to create an observability and notification pipeline that is completely customizable and completely decoupled from the production environments.
This architecture is not intended to replace Google Cloud's native monitoring and alerting capabilities. The purpose of this architecture is to demonstrate how production logs can become programmable events that can be customized to match our requirements.
Sending a Discord/Slack alert is one of the possible outcomes. This architecture can be used to log support tickets automatically, send security notifications, trigger a separate pipeline process, etc.
