Rapid Integration Development With StackStorm

April 3, 2015

by James Fryman

Day in and day out, the team at StackStorm is building tools that take away pain from the daily lives of Operations and Developers in IT Departments everywhere. We do this by focusing our efforts on absorbing all the Glue Code, the small snippets of code that tie together tools in your organization, and relieving the developer and operator of much of the traditional management overhead associated with automations. We also need to make sure that the tools we provide around StackStorm are equally enjoyable and frictionless to use. To that end, I would like to share with you how to rapidly build integrations with our integrated development environment, st2workroom.

What is StackStorm?

If you’re just getting started with StackStorm, or are curious what it is, we have a great primer on the product… Take a moment, head over there, and give it a quick read or watch. We’ll be right here when you get back.

In a nutshell, StackStorm provides Event Driven Automation. StackStorm integrates with your various tools, and manages orchestration based on events that occur in your environment.

st2workroom

The st2workroom is a self-contained project that has some custom additions that allow us to quickly and easily prototype infrastructures of all types and sizes. This project can be downloaded from our GitHub repository at https://github.com/StackStorm/st2contrib. In this project, there are a few primary components:

  • Vagrant – This project needs little introduction. Vagrant allows us to ship a consistent Dev Platform, regardless of the Host Operating System.
  • Stacks – This addition to Vagrant allows us to quickly and easily define many nodes in a Vagrant environment.
  • Guard – This ruby project keeps watch on changes to the filesystem and manages restarts as necessary.

Using this project, we can quickly and easily set up a fresh StackStorm server, associate our code with that node, and rapidly prototype/develop. Let’s build our first pack.

Pre-Work

In this example, I am going to work on the trello integration with StackStorm. I have a few things already set up on my host computer. These steps really should only have to be done one time. Let’s go over them:

  • Download and Install Vagrant: This is a manual step currently. Head to https://www.vagrantup.com/downloads.html and download the latest version for your OS/Architecture.
  • Download and Install VirtualBox/VMWare Fusion (Optional): If you prefer to do all development on your local machine, you’ll want to get one of these Virtual Machine executors. We’ll go over how to develop with StackStorm if your local machine isn’t suited to run Virtual Machines.
  • st2contrib and st2incubator projects are downloaded to my computer: These projects are where we keep our main packs. We encourage community members to submit Pull Requests against this repository. To learn more about Pull Requests, head over to https://help.github.com/articles/fork-a-repo/.
  • st2workbench project is downloaded to my computer: This is the actual place work will be done.
Setup

Now, in this example, let’s work on developing the trello pack. What we’ll want here is to set up st2workroom to allow us to develop on our local machine with all of our environment settings, but still run the code in our clean-room development environment. This consists of two steps: the first, to tell st2workroom where the pack files are located, and the second, to optionally configure any configuration for the integration itself.

Setting up the Pack File Location

In this example, I’ll be contributing to the st2contrib repository. Let’s take advantage of our Stacks and tell the development environment where it can expect to find our trello pack files, which on my computer is located at /Users/jfryman/stackstorm/st2contrib/packs/trello.

  • Navigate to the st2workbench directory: cd ~/stackstorm/st2workbench
  • Edit the file stacks/st2.yaml and add the file-mount to the st2express key. We want to target the pack directory to /opt/stackstorm/packs/trello.

<

pre>

# stacks/st2.yaml

Defaults can be defined and reused with YAML anchors

defaults: &defaults
domain: stackstorm.net
memory: 1024
cpus: 1
box: puppetlabs/ubuntu-14.04-64-puppet
st2express: <<: *defaults hostname: st2express sync_type: nfs puppet: facts: role: st2express mounts: - “/opt/stackstorm/packs/trello:/Volumes/Repositories/stackstorm/st2contrib/packs/trello”

  • Save the file and exit.

Note: By default in the st2workroom, the st2 stack is loaded up (stacks/st2.yaml). If you want to create your own infrastructure definitions, simply create a new Stack, and change the environment variable stack to your newly named stack. For example, if I wanted to create a new stack called frymanio, I would create a new file stacks/frymanio.yaml, and update my environment variable. See the file STACKS.md in the root of the st2workbench repository for more information.

Setting Pack Configuration

To bootstrap StackStorm, st2workbench leverages our Puppet module. (https://github.com/StackStorm/puppet-st2). Because of this, we can leverage Hiera to set configuration variables about our packs that will be downloaded. The benefit here is that once you are done developing against this repository, the hiera data is easily portable to a production environment, reinforcing the Infrastructure as Code concept.

In the trello pack, we have determined before hand that we need two configuration variables: api_key and token. Let’s set those now.

  • Navigate to the st2workbench directory: cd ~/stackstorm/st2workbench
  • Edit the hieradata/workbench.yaml file, and add the pack configuration to the st2::packs key.
# hieradata/workbench.yaml

st2::packs:
  trello:
    ensure: present
    config:
      api_key: ""
      token: ""

Cloud Configuration

This is an optional step. However, if you find yourself wanting to develop on StackStorm, but your local computing environment is not up to the challenge? No fear, you can quickly and easily get a quick development environment setup.

Digital Ocean

To enable Digital Ocean support, you’ll need to do the following:

1) Install the Digital Ocean Vagrant plugin: vagrant plugin install vagrant-digitalocean

2) Add the following environment variables to the file .env at the root of the st2workroom directory.

# .env

DO_SSH_KEY_PATH=XXX # Path to the SSH Key to be used for installation. (e.g.: ~/.ssh/id_rsa)

DO_TOKEN=YYY        # Digital Ocean API Token

sync_type=rsync     # Copies all bootstrap items to the Digital Ocean node, and provisions.

3) Make sure the attribute sync_type: nfs is removed from the file stacks/st2.yaml.

You can get your Digital Ocean API Token from https://cloud.digitalocean.com/settings/applications and generate a new token.

Proceed to Start it up!, paying specific attention to the --provisioner parameter.

AWS
To enable AWS support, you’ll need to do the following:

1) Install the AWS Vagrant Plugin: vagrant plugin install vagrant-aws
2) Download the dummy AWS box: vagrant box add dummy <a href="https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box" target="_blank">https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box</a>
3) Setup a New Keypair in AWS. Download this keypair and store it locally.
4) Add the following environmental variables to the file .env at the root of the st2workroom directory.

# .env

AWS_ACCESS_KEY_ID=XXX     # AWS Access Key

AWS_SECRET_ACCESS_KEY=YYY # AWS Secret Access Key

AWS_SSH_KEY_PATH=ZZZ      # Path to the SSH Key Generated in Step #3 (e.g.: ~/.ssh/aws_ssh)

AWS_KEYPAIR_NAME=NNN      # Use the keypair generated in Step #3

sync_type=rsync           # Copies all bootstrap items to the Digital Ocean node, and provisions.

5) Make sure the attribute sync_type: nfsis removed from the file stacks/st2.yaml.

Head to the AWS Web Console and look for the “Security Credentials” page. You may need to get in touch with your administrator for key.

Proceed to Start it up!, paying specific attention to the --provisioner parameter.

Note: By default, this provisions an Ubuntu 14.04 node in us-west-2. If you would like to change this, set the AWS_REGION and “AWS_AMIvariables in.envas detailed above.
<h6>Start it Up!</h6>
At this point, we're all set to get started! Go ahead and spin up
st2expressvia the workroom, and let's get started. You can do this with thevagrant upcommand in the root of thest2workroom` directory.

$ cd ~/stackstorm/st2workroom

$ vagrant up st2express (--provisioner )

Note: If you are leveraging the DO/AWS Cloud Provisoner, take note of the provisioner parameter. You will need to supply this. Use the --provisioner digital_ocean for DO, and --provisioner aws.

If this is your first time booting, it will take a few minutes depending on the speed of your computer and internet connection. On average, first boot takes approximately ~5 minutes. During this process, StackStorm is being downloaded to the latest version, setup on your computer in the development environment, and validated. It is rare that you’ll need to re-create the st2workbench after the first boot.

Start Guard

Guard is our Filesystem monitor. Its job is to monitor the /opt/stackstorm directory (and children, which in this case includes our new trello pack) for filesystem changes and automatically reloads. Anytime that you save a file in your pack directory, Guard will automatically restart the appropriate components of StackStorm to have your changes live loaded.

To do this, log into your new StackStorm box, and click on guard.

$ cd ~/stackstorm/st2workroom

$ vagrant ssh st2express

$ cd /opt/puppet

$ bundle exec guard -i -w /opt/stackstorm/packs

This will now automatically restart the appropriate component of StackStorm as you save files to the filesystem. Focus on development!

Start Developing!

The hard work is over! We now have a full fledged development environment to work with. If you happen to be on OSX, or have the avahi tool installed on your Linux box, you can head over to https://st2express.local:9101 and gain access to the WebUI. Likewise, you now have access to the StackStorm st2express environment, and can kick off commands from there.

Head over to our documentation website at http://docs.stackstorm.com, and read up on creating Actions and Sensors. From here, you can begin creating all the integrations that you desire.

In coming articles, we’ll have tutorials and walk throughs on developing your first action, developing ActionChain workflows, developing Mistral workflows, sensor development and more! In the meantime, be sure to take a look at our existing community packs online at https://github.com/StackStorm/st2contrib and https://github.com/StackStorm/st2 incubator for example packs.

Until next time!