Update: the latest revision of Observium now supports slack notifications (as well as a few other services)! You can add a slack channel in a contact with the format channel::#channel-name||username::username||url::webhookapi-url-for-slack and associate it as usual. Happy alerting! For anyone who has not updated to the latest rev or can’t, I’m keeping the below for posterity.
The first task on my list for my chatops project is to centralize alerting and slowly reduce the ridiculous flow of email to my inbox.
As background, my email philosophy is loosely organized as a to-do list. If a message is in my inbox, it is something I need to follow up on, reply to, fix, or otherwise is actionable. Newsletters, discussion lists, and other notifications go directly into folders with an elaborate set of rules. In this way, I can open my email client and know exactly what I have on my plate for the day.
Most of the notifications I get via email are not for mission critical things – they are “pre-failure” notifications, really. These kinds of alerts include:
- printer toner will run out in a few days
- disk space on a file server is below the acceptable threshold
- PDU voltage is sagging
- internal dev/QA environments are down
While important, none of those are items I need to prioritize to the top. Notifications of down network ports, down servers, or other major infrastructure problems will continue to flow to my inbox. For the rest of these pre-failure notifications, chat is the perfect medium. It notifies me without clogging up my day, and will continue to notify me at the interval I set until the issue is resolved. It’s the difference between a soft nagging reminder and a massive flashing neon “DANGER” sign.
Enough background, let’s get to the good stuff!
Observium is my software of choice for monitoring network-connected devices. It has a broad range of support, an alerting platform, and is written in a language I’m decent with (PHP). I decided to quick-and-dirty hack in the ability for Observium to push notifications to a Slack channel using Slack’s incoming webhook API. The code is split into three parts – a line in config.php to specify the webhook URL, an include call in the alerting processor, and the actual webhook call itself. Note – you must be a subscriber to the Observium Professional edition as of this writing to get the alerting functionality. Here’s how to set up the alerting.
Get a Slack Webhook URL
Log in to your slack instance and click the name in the top left and select “Configure Integrations” from the drop down menu. Select “Incoming Webhooks” from the list, and choose the channel you want to integrate with from the list. If you want, select a good label, nickname, and upload an icon. Copy down the webhook URL for the next step.
Modifying config.php
This is the easiest. Add the following line before the end of config.php:
$config['alerts']['slack']['webhook_url'] = "YOUR-SLACK-WEBHOOK-URL";
Modify the alerting processor
Right now, this bit of logic is in the file observium/includes/alerts.inc.php in the alert_notify() function. We don’t use the contacts table, so I’m ignoring it for now. This isn’t really the proper way – we should really add Slack as a contact method. The lead dev of the project has said he’s re-writing this area soon, so I’m okay with a quick and dirty hack.
Locate the alert_notify function in alerts.inc.php. Just before the switch statement, insert the following line:
include('includes/alerting/slack.inc.php');
It is important to note the placement is correct – if put elsewhere, Slack will not follow the same ignore / disable notify settings as email and you’ll end up getting spammed.
You will also need to modify the calls to the alert_notify function to also pass the $message_tags array to slack.inc.php, as well as add it to the arguments in the function definition for alert_notify. Again, this is pretty sloppy but if it will be rewritten in a few weeks I’ll wait until that is done to make it smooth and pass as little as possible back and forth.
Add the alerting include to call the webhook
Place the following file in includes/alerting and name it slack.inc.php.
Trigger an alert if you want to – it will hopefully show up in your Slack channel, and look a little something like this:
Having an issue where it’s not pulling the data into slack:
observ-alertsBOT [4:46 PM]
– []
Go to Observium page
– []
happen to know why this would be?
Double check you are passing $message_tags to the alerting include as described?
That, and – if you have kept your observium professional subscription up – there is actually slack support now baked in by default. There is some light documentation on their wiki. I’ll update the post later to reflect this.
Thank you, I’ve found their integration in contacts. Could you provide the link to the wiki, I’m having trouble finding it.
There apparently is no wiki yet – the format was given to me on IRC, sorry! You can take a look at includes/alerting/slack.inc.php for the code that actually parses the contact. The format should be:
username::username-of-bot||channel::channel-to-post-in||url::slack-web-hook-url in the destination field of the contacts. Make sure to associate the contact with the alert checkers (click on the destination after saving).
Awesome thank you!
Hello! I could not run or with the site’s script nor the current slack.inc.php the observium.
follows some informations
$config[‘alerts’][‘slack’][‘username’] = “Metro”;
$config[‘alerts’][‘slack’][‘channel’] = “#cgr”;
$config[‘alerts’][‘slack’][‘url’] = “https://hooks.slack.com/services/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”;
$config[‘alerts’][‘slack’][‘webhook_url’] = “https://hooks.slack.com/services/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”;
$config[‘alerts’][‘slack’][‘default’][‘contact’] = “channel::#cgr||username::Metro||url::https://hooks.slack.com/services/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“;
has any idea what can be?
You don’t need to add anything to your config file to use slack anymore – simply add a contact via the web interface with the same syntax as you used in the config file and assign it to the appropriate alerts.
I don’t see the slack option in the contacts option, why? How do I enable it to appear? or where do I modify so that I can send notification to slack? I did the detailed in this manual but it doesn’t work for me. helpme
When you add a new transport in the Contacts page you should see Slack as one of the options. If not, I’d ask on the Observium mailing list – it is also possible that Slack is one of the Observium Pro features if you’re running community edition? (We run Pro)