Simplified Network Performance Tests with PerfSonar and SLX

May 3, 2018
by Lindsay Hill

PerfSonar is a super handy toolkit for measuring network performance between any two points. Combine this with the Guest VM built into the Extreme SLX series of switches, and you can easily run performance tests between any two points on your network, measuring performance, latency, jitter, MTU, path taken, etc. Combine that with StackStorm, and you can easily run those tests from Slack. No need to even login to a switch.

Demo: Network Performance Tests via Slack

Check out the video here. From Slack, we can trigger different tests between any two switches – performance, one-way latency measurement, or trace the path, showing the path MTU. The results are then shown in Slack:

Read on for more about how to set this up.

Technologies in Use

We’ve put together a few technologies here:

1. Extreme Insight Architecture: the Extreme SLX series of switches run a separate ‘Guest VM’ in addition to the default SLX-OS VM. This Guest VM is running Ubuntu 14.04. There is an onboard hardware path that connects the packet processor ASIC(s) to this VM. This can act as another server attached to the data plane.
2. PerfSonar: This is an Open Source toolkit of network measurement tools.
3. Slack: IRC for hipsters, of course.

Plus of course StackStorm, which is the operational ‘glue’, stitching together these elements, co-ordinating inputs, actions and outputs across our environment.

Setup: System Configuration

Here’s the basics for how to set this up:

1. StackStorm & ChatOps: Install & configure StackStorm. Create a Slack team if you don’t already have one, and add a Hubot configuration, and get a HUBOT_TOKEN. Edit the /opt/stackstorm/chatops/st2chatops.env file on your StackStorm server, and add that HUBOT_TOKEN. Set HUBOT_ADAPTER=slack, and restart the st2chatops service. Invite the bot to your Slack channel, and it should start responding to !help.
2. SLX Setup: Enable the Guest VM on your SLX switches. Copy stanley’s public SSH key to authorized_keys on the Guest VM. Add entries to DNS or your hosts file so that you can ssh from your StackStorm server to the switches and their guest VM. Use the format switch_name-tpvm for the guest VM DNS entries.Set an IP address on eth1 on each Guest VM.

   Set a corresponding IP on the SLX side, and configure the Insight port. Set up routing such that each Guest VM can ping every other Guest VM’s eth1 interface, across the data plane, not via the management network.
   Install the perfSonar toolkit using these commands:

   cd /etc/apt/sources.list.d/
   wget http://downloads.perfsonar.net/debian/perfsonar-wheezy-release.list
   wget -qO - http://downloads.perfsonar.net/debian/perfsonar-debian-official.gpg.key | apt-key add -
   apt-get install perfsonar-toolkit
   

Setup: Packs, Workflows & Aliases

Install the st2_demos pack with st2 pack install https://github.com/StackStorm/st2_demos.

This contains the alias file /opt/stackstorm/packs/st2_demos/aliases/bwctl.yaml, and the workflow metadata and definitions, /opt/stackstorm/packs/st2_demos/actions/bwctl.yaml and /opt/stackstorm/packs/st2_demos/actions/workflows/bwctl.yaml.

You can of course edit these files, and move them to other packs.

Try it Out!

You should now be ready to test it out. From Slack, run !help performance.

If this is responding, try running a test. Start with a simple iperf test, e.g.: bwctl iperf slx1 slx2.

This will run an iperf test using the default timeout (60s) and bandwidth (1Gb). After a little over a minute, you should see a response in Slack, showing the measured performance between SLX1 and SLX2. This should be 1Gbps.

You can try running it with unlimited bandwidth, i.e. bandwidth=0. You should see maximum performance of ~8.3Gbps. This is around the maximum you can achieve with a Linux VM, using a 10GbE connection.

Try running some other tests:

• owamp – e.g. bwctl owamp slx2 slx1This will measure the one-way latency and hops between slx2 and slx1 (recall that ping measures round-trip time). This can be useful in situations where asymmetric routing may be happening.
• tracepath – e.g. bwctl tracepath slx1 slx2 – this will show you the path between slx1 and slx2. It will also show you the path MTU – this is very handy if you’re investigating possible MTU issues.

Try it out, let us know how it goes!

You’ll also note that you don’t need to use the SLX Guest VM for this. If you have access to existing servers, you can run those tests between those systems too. You just need to install the perfSonar toolkit.