Mar 16, 2021
by Tomaz Muraus and @blag
Today we are announcing StackStorm v3.4.1, a bug fix release which fixes a security issue which has been uncovered recently.
The issue affects anyone who is running StackStorm under Python 3 and doesn’t have a system locale / encoding which is used for StackStorm service processes (st2api, st2actionrunner, etc.) set to UTF-8. Under such conditions, if StackStorm receives a payload with unicode characters which also results in the payload being logged, StackStorm process would go into an infinite-loop trying to decode that payload. This would cause the affected service to have a high CPU utilization and also service log file to grow either until the process is killed or all the available disk space is exhausted (basically result in a DoS).
If you are running StackStorm under Python 3, you are strongly encouraged to upgrade to this release.
For more details, please see the sections below.
Any StackStorm installation which is using Python 3 and doesn’t have the system locale / encoding used for StackStorm services set to utf-8 is affected.
Until StackStorm v3.4.0 which was released recently, StackStorm was still using Python 2.7 on most distros except RHEL / CentOS 8 and Ubuntu Bionic 18.04. StackStorm already used Python 3 on those operating systems.
It’s worth noting that nowadays that the system and processes locales are almost always set to utf-8, and when locales are either not being set, or set to C.ascii, is is usually a system misconfiguration.
The actual bug and security vulnerability is located in the piece of code which is responsible for routing any data which is written to standard error (those are mostly just exception stack traces) to the logger method and logging this data under ERROR log level.
Under normal scenario when encountering unicode data and system / process locale (encoding) not being set to utf-8, trying to decode this utf-8 data to an ascii string would result in an exception because the character we are trying to decode is out of range for the current locale (which is usually ascii in such scenarios). That exception would then simply just be logged to the log file like any other.
Due to the bug in the code, under such scenario, the code would incorrectly try to decode this data in an infinite-loop instead of simply bailing out on first attempt trying to decode this data.
The bug was initially uncovered by one of the end to end tests, but at that time it was assumed it was just a flaky test and not an actual bug since we could not reproduce it on all operating systems, or during manual user testing for the 3.4.0 release.
A couple of days later, I was working on a different change when I encountered the same issue. After digging in a lot, I was able to track down and pinpoint the root cause.
One of the main reasons why the issue wasn’t detected faster was because of the nature of it – as mentioned above, these days it’s quite uncommon to have a system which is not configured with utf-8 locale.
The fix includes modifying our custom logger method described above to decode the underlying byte string and removing any characters which are not in the ascii range before passing that data to the underlying log formatter method.
Now if this issue occurs, the process will simply just log an error / exception when trying to log unicode data when process encoding is not set to utf-8 (as it should have done out of the box).
Having said that, even though it will now work correctly under non-utf-8 encodings, you are still strongly recommended to set your system locale (and encoding for StackStorm services processes) to utf-8 to ensure a good experience when working with unicode data.
To do so, ensure that the encoding part of LC_MESSAGES or LANG in /etc/locale.conf is set to UTF-8. To set your locale for American English with UTF-8encoding, use this:
LANG=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
Or, for example, a German locale with American English log messages, you can use:
LANG=de_DE.UTF-8 LC_MESSAGES=en_US.UTF-8
For more information on locales, check the documention on freedesktop.org.
On that note, we have also added a small change to the code which will print a warning on service startup in case the encoding for that service is not set to utf-8 as shown below.
WARNING [-] Detected a non utf-8 locale / encoding (fs encoding: ascii, default encoding: utf-8, locale: unable to retrieve locale: unknown locale: invalid). Using a non utf-8 locale while working with unicode data will result in exceptions and undefined behavior. You are strongly encouraged to configure all the StackStorm services to use utf-8 encoding (e.g. LANG=en_US.UTF-8).
If you fall under affected deployments (you are running StackStorm under Python 3 and using locale / encoding other than utf-8), you are strongly encouraged to upgrade to v3.4.1.
If you are a researcher or user that discovers a security issue please reach out to moc.mrotskcatsnull@ytiruces (/security/).