Jenkins + smartctl + Pushover: Drive Monitoring w/Push Notifications and Jenkins Job History

Could you use smartd to send emails via Postfix/sendmail on your Linux box?  Sure.  Could you use Jenkins to trigger individual smartctl queries and then tell Pushover to send a push notification to your devices saying that you have a drive failure?  Also, yes.  Why do the latter?  Because you're bored, of course!

On my NAS at home, I run a ZFS volume with cache, log, and data drives.  The data portion of the ZFS volume contains multiple mirrors (don't @ me, that's just how it's grown).  Being served from this NAS are, just like every other homelab enthusiast, multiple services including but not limited to: file shares (SMB/CIFS, NFS, AFP, S3), Jenkins, Plex, Tautulli, Sonarr, Radarr, SABnzbd, etc, etc.  I wish to be notified ASAP if/when there is ever a drive failure.  smartd, by default, can be configured to send emails using your system's mail relay service of choice (such as Postfix) but what is the smartd service crashes and you aren't notified when trouble is brewing?  Or, what if you want a history of your drives' health?

Introducing the highly unnecessary, yet quick and easy, Jenkins + smartctl + Pushover stack!  Whoooo!  Here are our goals:

• Scan all disks in our NAS and make sure they are healthy
• Keep a record of the history of our disks' health
• Notify us immediately if a problem is discovered
• Run this on a schedule

Again, this all could be accomplished with smartd but why not have some fun?

First, we'll go ahead and create a new Freestyle Project in Jenkins.  For me, I have nested this job under: Storage Jobs > SMART Scans > Scan sd(x).

Second, we'll set the following options:

• Discard old builds (setting Max # of builds to keep to 5)
• Build periodically (setting Schedule to @daily)

Last, but not least, we'll add an Execute Shell step under the Build stage.  This shell script will look something similar to this:

#!/bin/bash

RESULTS="$(smartctl -H /dev/sda)"

if [[ $RESULTS != *"PASSED"* ]]; then
curl -s --form-string "token=TOKEN" --form-string "user=USER" --form-string "message=SMART Alarm: sda on NAS" https://api.pushover.net/1/messages.json
exit 1
fi

As you can see, this script will execute daily and notify the user via Pushover if sda is not healthy.  The Jenkins build job will also be marked as a failure if this happens.  Please be sure to replace the TOKEN and USER variables (you can find these values on your Pushover dashboard).

And that's it!  Simple, easy, and mostly unnecessary!  Of course, you can accomplish this same setup with a simple cronjob and maybe a Slack webhook or a WhatsApp webhook or a Philips Hue command...the possibilities are endless!