August 10, 2016
By Tomaz Muraus
A good month after releasing StackStorm v1.5.0 we are happy to announce the release of StackStorm v1.6.0.
This release includes various new features and improvements described below.
We have listened to feedback from our users and updated action concurrency policies to allow action execution to be canceled instead of just delayed when a concurrency threshold is reached.
This feature comes handy in situations where multiple users request the same action, but the action is already running because of the first request.
This was achieved by adding new “action” attribute to the action.concurrency and action.concurrency.attr policy definition. Valid values for this attribute include “delay” and “cancel”. This new feature is fully backward-compatible and if this attribute is not specified, it defaults to “delay” which is the same as before.
In StackStorm v1.5.0 we have introduced improved configuration which allows user to more easily version control the config files, utilize secrets and values from the datastore in the config and more.
In this release we have built on top of the existing foundation and introduced strict validation for new-style configs. This functionality is built on top of pack configuration schema which allows packs authors to define valid attributes and types for all the config options.
When running st2ctl reload –register-configs / st2-register-content reload –register-configs script, StackStorm will now validate new-style configs located in /opt/stackstorm/configs/ directory against the config schema located inside the pack directory (if a schema doesn’t exist, no validation is performed).
Validation failing on aws pack config since attribute “region” is an integer and not a string.
This feature allows users to catch common errors and typos faster and allows for a better user-experience and faster development cycle. Previously, typos and other common errors which are now caught during register / load phase were only caught during run-time (when action runs). This meant slower development and testing cycle since the user needed to run the action to see if all the configuration values are correct.
In addition to that, this feature also makes pack author’s life a bit easier since it offloads some of the work from them to StackStorm. Previously, pack authors would need to put code which performs config validation (does config contain value X, is it of a right type, etc.) inside the action.
When StackStorm development started, 2.4 was the latest stable and supported release of MongoDB.
Since then, MongoDB has undergone a lot of development and released multiple new major releases (2.6, 3.0 and 3.2) which include many improvements. The underlying libraries StackStorm uses to talk to MongoDB (pymongo and mongoengine) were updated to support MongoDB 3.x, but some of the changes were backward incompatible so updating StackStorm to support it wasn’t trivial.
In this release we upgraded to the latest versions of underlying libraries and updated our code base so it now supports both – MongoDB 2.x and MongoDB 3.x.
We believe this is an important change for our users since the latest version of MongoDB (3.2) includes a new storage engine, WiredTiger, which includes many performance and reliability improvements (document level concurrency and more granular locking, snapshots and checkpoints, compression, encryption, etc.).
In addition to that, we have also updated our installer script and documentation so new installations will install and use MongoDB 3.2 by default.
As far as existing users go – for the time being our codebase supports MongoDB 2.x and 3.x so we won’t force existing users to upgrade. Having said that, because of the various improvements in MongoDB 3.2 we would still encourage existing users who have the time, resources and can afford short down-time to upgrade.
It’s worth noting that because of the storage engine and on-disk format changes, the upgrade is not totally straight-forward. It requirements incremental upgrade (first to 3.0, then to 3.2) and data migration.
If you decide to upgrade, make sure you create a consistent backup of your data (that’s something you should be doing periodically anyway) so you can restore it in case things go wrong. In addition to that, also make sure you periodically test the recovery process and restore from backups since backups which aren’t tested are the same as no backups.
Python runner actions can now return a status (succeeded, failed) along with with the result by returning a tuple from the run() method.
Previously the only way for the Python runner action to be considered as failed was by throwing an exception or exiting with a non-zero exit code. Problem with this approach was that it wasn’t directly possible to return data (aka result) with the failed actions. This was especially problematic inside workflows where you usually also want to pass some data along even if the action fails.
| import sys | |
| from st2actions.runners.pythonrunner import Action | |
| class MyEchoAction(Action): | |
| def run(self, message): | |
| print(message) | |
| if message == 'working' | |
| # If value "working" is passed for parameter "message", action status will | |
| # succeed (first item in the tuple indicates success - success flag is True) | |
| return (True, message) | |
| # If any other value is passed in for parameter "message", action will fail (second | |
| # item in the tuple indicates failure - success flag is False) | |
| return (False, message) |
Users utilized various approaches to work-around this limitation, but due to native support this is now not needed anymore.
This feature is fully backward-compatible unless you have an existing Python runner action which returns a tuple with two items. If that’s the case, you need to update your action to either return a list or return a tuple with status as described in the Upgrade Notes.
One of the long requested features from our users was ability to access the values in the datastore inside the Mistral workflows. This was already possible in action-chain workflows using Jinja template notations (e.g. {{system.my_key}}), but not in Mistral workflows.
Since StackStorm v1.6.0 you can now access datastore variables inside the Mistral workflows using st2kv() function as shown below.
| version: '2.0' | |
| examples.mistral-yaql-st2kv-system-scope: | |
| vars: | |
| foobar: unspecified | |
| tasks: | |
| task1: | |
| action: std.noop | |
| publish: | |
| foobar: <% st2kv('system.foobar') %> | |
| on-complete: | |
| - fail: <% $.foobar != foobar %> |
| version: '2.0' | |
| examples.mistral-yaql-st2kv-user-scope: | |
| vars: | |
| polo: unspecified | |
| tasks: | |
| task1: | |
| action: std.noop | |
| publish: | |
| polo: <% st2kv('marco') %> | |
| on-complete: | |
| - fail: <% $.polo != polo %> |
For more information, please refer to the documentation – Mistral + YAQL.
In some edge cases when running a lot of concurrent action-chain workflows coupled with action.concurrency delay policy, StackStorm action runner processes could dead lock.
The issue was hiding in the action-chain implementation and a size of an internal green-thread pool used by the action runner process.
To alleviate this issue, we have changed implementation a bit to utilize two green-thread pools (one for regular actions and one for workflows). In addition to that, we also exposed two new config options which allow StackStorm administrators to tune those pool sizes to their specific workloads – Tuning action runner dispatcher pool size.
This issue was originally reported by Brian Martin who we would like to thank. Brian opened a bug report which can serve as a great example to other uses. The bug report included all the necessary information (environment, log files) and code to reproduce it. In addition to that, Brian even suggested some possible solutions / workarounds for the problem.
st2-register-content and st2ctl reload script will now exit with non-zero exit code and fail by default if a failure such as invalid resource metadata is detected.
Previously the script didn’t fail on failure by default and user needed to pass in “–register-fail-on-failure” flag if they wanted this to happen. That behavior was problematic and often caused users to miss out on important errors.
With a more sane default behavior we hope to provide a better user-experience and make it easier and faster to spot common errors such as typos and invalid resource metadata.
If for some reason you want to achieve old behavior you can do so by using “–register-no-fail-on-failure” flag although there should rarely be a need for that since broken / invalid resource metadata usually indicates a problem which should be resolved immediately.
In addition to the changes mentioned above, StackStorm v1.6.0 also includes various ChatOps related improvements and more. For a full list of changes, please see the changelog.