Auto-Remediation with StackStorm & Splunk

Oct 21, 2016
by Siddharth Krishna

Splunk is a great tool for collecting and analyzing log data. StackStorm is a great tool for automated event-driven remediation. So what happens when we stick them together? Here’s how to use Splunk to collect syslog data and trigger event-based network remediation workflows using StackStorm!

Syslog as an event source

Syslog for events and errors from the network devices can be tapped to trigger troubleshooting and auto-remediation actions on those devices. For example, if a ‘link down’ event occurs, the syslog message can be used to auto-trigger an action that would log into the device and try to bring the interface back up. In parallel, an IT ticket can also be auto-created with the relevant interface details. Notification of the event and the automated workflow can optionally be sent to a Slack channel (ChatOps).

One option is to have StackStorm itself act as the syslog server and run a sensor that polls the log file to match specific log strings. This method, although workable, has its performance limitations, and it doesn’t give us a nice tool for searching historical logs. Instead we can use something like Splunk, which is a log aggregation and analysis system. Splunk includes alert functionality – we can filter syslog messages, extract relevant fields, and trigger actions such as making a webhook request to StackStorm.

Here’s how to configure an auto-remediation workflow using Brocade VDX switches, Splunk and StackStorm:

NB this guide assumes that you have a working StackStorm/BWC system, and a Splunk server.

VDX Configuration

Configure all VDX switches to send syslog messages to the Splunk server:

logging syslog-server <splunk-ip-address> use-vrf mgmt-vrf

Setting up Splunk

We’re using the default Splunk Search & Reporting App here. You could also use the Brocade Splunk App.

If your server does not currently accept syslog input, add it by going to:

Settings -> Data -> Data Inputs -> UDP -> New

Input UDP port 514 (default syslog port) and Source Type as ‘syslog’

Refer here for more on configuring data inputs on Splunk.

Splunk should now start displaying live syslog events from the switches in its search result.

Search -> Data Summary -> Search Type: syslog

Use the output of show logging raslog on a switch to identify an event or error of interest from its log string. In this case, we want to dynamically detect and act upon a link flap for which we need to tap syslog

[NSM-1003], 48201, SW/0 | Active | DCE, INFO, LEAF2,  Interface TenGigabitEthernet 102/0/48 is link down.

On splunk, create a search criteria to filter out the event. Here, we use the following:

sourcetype=syslog NSM-1003 process=raslogd | fields + host, interface, syslog_text

Stricter search criteria are recommended for greater filter accuracy. Once your search pattern is set up correctly, save it as a Splunk Alert with alert type as “Real-time” in order to capture the event live.

Save As -> Alert | Alert-type: Real-time, Trigger alert when: Per-Result

Splunk provides multiple options for Trigger Actions – Run a script, Send an Email, Webhook. You can run a script that makes a cURL call to StackStorm webhook URL or, for simplicity, use Splunk’s webhook option. StackStorm’s custom webhook URL is:

https://{STACK_STORM_HOSTNAME}/api/v1/webhooks/splunk_link_flap?st2-api-key=XXXXXXXXXXXXXXXXXXXXXXXX

For successful action execution, relevant fields or parameters need to be correctly extracted from the event log message and passed via the webhook call to StackStorm. For example, to be able to auto-remediate a ‘link down’ event i.e. do a “shut; not shut” on the switch, the switch’s IP address and an interface name are must. Splunk’s field extraction capability is useful in auto-generating regexp to pull out the field values from the log string. These “field:value” pairs are then passed in the webhook JSON payload. More on field extractions can be found here.

StackStorm/BWC Configuration

A custom webhook rule mapping the webhook trigger to the action workflow is created within StackStorm/BWC.

The custom webhook rule for ‘link flap’ defines the following:

• Webhook trigger URL: “splunk_link_flap”, complete webhook URL to be configured on Splunk is “https://bwc/api/v1/webhooks/splunk_link_flap”
• Action Reference: Name of the workflow to be executed when the webhook is triggered
• Action Parameters (originally extracted by Splunk and passed via webhook):
• Host: IP Address of the switch/device
• Interface: Interface that went down

Rule Definition:

splunk_link_flap.yaml

Rule Details:

When the ‘link down’ syslog is detected by Splunk, and the webhook to StackStorm called, the trigger instance payload contains the values for the various Splunk fields and parameters. These include the host IP address and the interface name (as per configured field extractions) along with other standard ones such as search link, search ID, raw log message etc. See this example below:

These field values can now be directly accessed from the StackStorm webhook rule using {{trigger.body.result.xxx}} e.g. {{trigger.body.result.host}}.

Upon successful enforcement, the rule executes the auto-remediation workflow with execution ID as shown below:

In this example, the workflow for link flap remediation does the following:

1. Notify “link down” event on Slack with the Host IP address and Interface name
2. Pull configuration details for the given interface from the switch and post it to the Slack channel
3. Try to bring the interface back up by executing “shut; no shut” on the switch
4. Pull interface details show interface detail from the switch and post the output to Slack
5. Create a Zendesk IT ticket for the event occurrence and attach relevant logs

Slack Channel

All code examples used here are available in the StackStorm st2_demos repo on GitHub.

This is just one example of a syslog-driven auto-remediation workflow. Workflows can be custom-built with actions and integrations for your environment. Use Jira instead of Zendesk? No problem! Quickly modify the workflow to use Jira instead! Use Hipchat? No problem, modify your st2chatops2 setup to use that! Easy.

What are you doing with event-driven remediation with StackStorm? Jump into our community and let us know!