Update: the latest revision of Observium now supports slack notifications (as well as a few other services)! You can add a slack channel in a contact with the format channel::#channel-name||username::username||url::webhookapi-url-for-slack and associate it as usual. Happy alerting! For anyone who has not updated to the latest rev or can’t, I’m keeping the below for posterity.
The first task on my list for my chatops project is to centralize alerting and slowly reduce the ridiculous flow of email to my inbox.
As background, my email philosophy is loosely organized as a to-do list. If a message is in my inbox, it is something I need to follow up on, reply to, fix, or otherwise is actionable. Newsletters, discussion lists, and other notifications go directly into folders with an elaborate set of rules. In this way, I can open my email client and know exactly what I have on my plate for the day.
Most of the notifications I get via email are not for mission critical things – they are “pre-failure” notifications, really. These kinds of alerts include:
- printer toner will run out in a few days
- disk space on a file server is below the acceptable threshold
- PDU voltage is sagging
- internal dev/QA environments are down
While important, none of those are items I need to prioritize to the top. Notifications of down network ports, down servers, or other major infrastructure problems will continue to flow to my inbox. For the rest of these pre-failure notifications, chat is the perfect medium. It notifies me without clogging up my day, and will continue to notify me at the interval I set until the issue is resolved. It’s the difference between a soft nagging reminder and a massive flashing neon “DANGER” sign.
Enough background, let’s get to the good stuff!
Observium is my software of choice for monitoring network-connected devices. It has a broad range of support, an alerting platform, and is written in a language I’m decent with (PHP). I decided to quick-and-dirty hack in the ability for Observium to push notifications to a Slack channel using Slack’s incoming webhook API. The code is split into three parts – a line in config.php to specify the webhook URL, an include call in the alerting processor, and the actual webhook call itself. Note – you must be a subscriber to the Observium Professional edition as of this writing to get the alerting functionality. Here’s how to set up the alerting.
Get a Slack Webhook URL
Log in to your slack instance and click the name in the top left and select “Configure Integrations” from the drop down menu. Select “Incoming Webhooks” from the list, and choose the channel you want to integrate with from the list. If you want, select a good label, nickname, and upload an icon. Copy down the webhook URL for the next step.
Modifying config.php
This is the easiest. Add the following line before the end of config.php:
$config['alerts']['slack']['webhook_url'] = "YOUR-SLACK-WEBHOOK-URL";
Modify the alerting processor
Right now, this bit of logic is in the file observium/includes/alerts.inc.php in the alert_notify() function. We don’t use the contacts table, so I’m ignoring it for now. This isn’t really the proper way – we should really add Slack as a contact method. The lead dev of the project has said he’s re-writing this area soon, so I’m okay with a quick and dirty hack.
Locate the alert_notify function in alerts.inc.php. Just before the switch statement, insert the following line:
include('includes/alerting/slack.inc.php');
It is important to note the placement is correct – if put elsewhere, Slack will not follow the same ignore / disable notify settings as email and you’ll end up getting spammed.
You will also need to modify the calls to the alert_notify function to also pass the $message_tags array to slack.inc.php, as well as add it to the arguments in the function definition for alert_notify. Again, this is pretty sloppy but if it will be rewritten in a few weeks I’ll wait until that is done to make it smooth and pass as little as possible back and forth.
Add the alerting include to call the webhook
Place the following file in includes/alerting and name it slack.inc.php.
Trigger an alert if you want to – it will hopefully show up in your Slack channel, and look a little something like this:
