StackStorm: Event-driven automation

(github.com)

103 points | by gjvc 8 days ago

10 comments

  • debarshri 8 days ago
    Stackstorm has been around since 2014 [1].

    Rundeck is another closest competitors that worth mentioning. Things have evolved a lot in the ops space since then.

    With advent of container orchestration of the operational automation pieces have also moved to the orchestration layer. There have been quite some new tools in the space I have been tracking for instance Fylament [2] which got acquired by netapp, blinkops etc. Post rendezvous with this domain, some bigger problems most of the major framework tools like this have is that, it is very hard to get started. There are lot of pre-requisites given the nature of the ops automation, this leads to a problem where it is very hard to active a customer. Secondly, these frameworks are always competing with it is bash/python script deployed with some container orchestration or some cronjobs. I have also seen people write CRDs and reconciliation loops in k8s doing lot of the tasks these automation frameworks would do.

    [1] https://en.wikipedia.org/wiki/StackStorm

    [2] https://www.netapp.com/newsroom/press-releases/news-rel-2022...

    • chillfox 7 days ago
      Every time I need to automate something and I look at these solutions it becomes clear that I can put together a Ruby script to handle it much faster than getting one of these automation solutions up and running.
      • debarshri 7 days ago
        I agree that writing a ruby script is tempting. Where I disagree with you is that, when you have an organisation of 100s of devs, these scripts become tech debt. You need to have standard operating procedures delivered via one platform, that's when rundeck et al plays an important role.
        • chillfox 7 days ago
          Never have been able to get management buy in for automation.

          I even had one manager tell me that there was value in having someone do it manually instead of automatic, I suspect part of their pay or career progression was based on the size of their team.

          • debarshri 7 days ago
            That is precisely what I mentioned in my first post. Activating the customer and getting them to value is the biggest challenge these frameworks have. It is hard to buy into a framework from get go.
            • chillfox 6 days ago
              So part of the problem from a sysadmin/developer perspective here is that these automation solutions are too difficult/time consuming to setup on the side and prove their value to management. They basically can’t start as a one or two person on the side initiative and then grow the way other technologies like introducing version control and CI can.
              • gjvc 6 days ago
                They basically can’t start as a one or two person on the side initiative and then grow the way other technologies like introducing version control and CI can.

                They absolutely can. Find a spare host or create a VM. Write a shell script to run docker, or much better, use "docker compose" to create the setup. Have it run on the quiet for a while and get familiar with it, a little more every day, before using it in production. It's crucial that you are confident about using it, so that when you come to sell it to anyone else, your confidence transfers to them. (Sales, in large part, is about the transfer of feeling.) Once you have got the deployment and backup/restore, and tool-specific critical procedures clear in your mind, you're ready.

                It's not a five-minute job; it's about demonstrating your strategic value to management by getting out of the toil loop. But, to be clear, if you feel you don't have the backing of anyone about such things, you certainly have my sympathy. The best way I have found out of the mire is to work on a workflow (create/destroy, backup/restore, LDAP integration) a little a day, until it's there, and then suddenly, your work put in over the days and weeks will be an overnight success.

                Good luck!

                • chillfox 6 days ago
                  The competition is cron and a folder of scrips which provides a productive result from day one.

                  I am not in an ops role at the moment, but in the past nothing has been able to beat cron. And the main reason is the time required to get automation tools up and running.

  • zyphr 8 days ago
    Red Hat is working towards a similar feature for anisble automation platform.

    https://www.ansible.com/blog/getting-started-with-event-driv...

  • linsomniac 8 days ago
    I played around with StackStorm maybe 5 years ago, but in the end I decided to use RunDeck instead. https://www.rundeck.com/. Used that for a few years but ended up abandoning it, but am thinking about resurrecting it, it was pretty good.
    • gzapp 7 days ago
      Ex-Rundeck engineer here. Would be very interested to here why you initially dropped it and why you are thinking about picking it back up!
      • linsomniac 7 days ago
        It was just that we weren't using Rundeck very effectively. I set it up so that others could run a few Ansible jobs, and so that a slack bot could fire off one of those jobs. The Slack API changed a few years later and our old bot completely broke and I couldn't get it working again without rewriting. As far as other people running them, nobody else seemed to use it. So when it came time to upgrade the EOL OS that was hosting Rundeck, I just shut it down.

        But, now we have some Jenkins jobs that would benefit from being isolated from Ansible runs, so Rundeck might be a good way to run those.

  • lgas 8 days ago
    This is interesting. It's like IFTTT for ops. The list of adopters is intriguing too. I poked around their blog for a few minutes and it seemed to be targeted more at people who have already adopted StackStorm. It would be great to be able to read some success stories from the people using it.
  • kinow 8 days ago
    I tested StackStorm sometime ago to confirm you can have cycles in the workflow graph: https://kinoshita.eti.br/2021/10/01/cyclic-workflows-with-cy...

    I work with climate and weather workflows, where cycles in workflows are common (ecFlow and Cylc are popular tools that support it).

    Haven't checked in a while but I expect StackStorm's Orchesta engine to have even more interesting features now.

  • marcrosoft 8 days ago
    I wouldn’t trust any tool that claims to have anything to do with security that has a curl pipe to bash install advertised.
    • eugenc 8 days ago
      One of the maintainers here and I second that.

      Other options available:

        - deb/rpm packages
        - Vagrant
        - Docker-compose
        - Ansible/Puppet
        - Kubernetes/Helm
      
      https://docs.stackstorm.com/install/index.html
      • ilyt 8 days ago
        ...probably better idea to adveritse docker container, not everyone wants to have half-rotten corpse of python app spreading shit around your system forever (there is never an uninstall...) just to test something.

        ...or at least one that fucking works

            zzz@hydra:~$ curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe
            *** Detected Distro is Debian ***
            *** Detected flavor bookworm ***
            Unsupported ubuntu codename bookworm. Please use 16.04 (xenial) or Ubuntu 18.04 (bionic) or Ubuntu 20.04 (focal) as base system!
        
        I've tried to drop the check for distro only to be asked for fucking sudo which is like... WTF ?

            zzz@hydra:/tmp$  ./install.sh --user=st2admin --password=Ch@ngeMe
            *** Detected Distro is Debian ***
            *** Detected flavor bookworm ***
            Downloading deployment script from: https://raw.githubusercontent.com/StackStorm/st2-packages/v3.8/scripts/st2bootstrap-deb.sh...
            Running deployment script for st2 ...
            OS specific script cmd: bash st2bootstrap-deb.sh  --stable   --user=st2admin --password=****
            [sudo] password for zzz: 
        
        curl|sh that downloads more curl|sh is also interesting choice.
        • gjvc 7 days ago
          `curl | sh` "installs" are not a good look on any package, and should be dropped -- you are absolutely right -- there is never an "uninstall" process.
        • undefined00z 8 days ago
          It's supporting a few OS platforms and require a clean server for installation: https://docs.stackstorm.com/install/system_requirements.html

          Expect to get MongoDB, RabbitMQ and Redis installed as dependencies.

    • letsg0 8 days ago
      I wouldn't recommend it as a security tool

      Works well for DevOps and SRE operations and like K8s, needs a whole team to run it

    • ilyt 8 days ago
      I've tried on clean account

      It is curl|sh that also requires sudo

  • candiddevmike 8 days ago
    If you're looking for a similar, commercially supported product, checkout RackN Digital Rebar: https://www.rackn.com. Similar concept, vastly easier deployment/HA/GitOps/multi-tenancy story.
    • armab 8 days ago
      StackStorm was an open-core project with a paid enterprise offering on top of it, belonging to a single vendor. In 2019 the project was donated to the Linux Foundation as a neutral umbrella, also open-sourcing the enterprise features like RBAC, LDAP, UI Workflow Designer.

      Today StackStorm has an open-source ecosysem with several partner companies (5) providing consulting, commercial support, custom solutions or training for the clients that may need it: https://stackstorm.com/partners/

      These partners also contribute back to the core and help supporting the project together with the other maintainers.

  • v3ss0n 7 days ago
    How it compares to:

    - Prefect.io

    - Apache Airflow

    - Ansible Tower

    • eugenc 7 days ago
      Prefect and Apache Airflow defines logic in a pure Python code, while StackStorm uses a YAML-structured Workflow called Orquesta [1] supporting different logic patterns (parallel, merge, concurrency, join, retries, pause, ask, nested subworkflows, etc) [2] with Jinja and YAQL [3] support. YAQL is especially interesting one where Jinja is not enough.

      There's also a bi-directional ChatOps framework you can tie with the workflows to make the experience more interactive.

      So you can listen on events (could be a chat command) from 3rd party services, do some rule filtering & matching (all in YAML), and trigger workflows with the logic that runs actions in other tool(s) and service(s).

      StackStorm has Exchange [4], - a collection of 100+ plugin integrations with different tools and services contributed and supported by community to make that building blocks juggling easier.

      [1] https://docs.stackstorm.com/orquesta/index.html

      [2] https://github.com/StackStorm/st2/tree/master/contrib/exampl...

      [3] https://yaql.readthedocs.io/en/latest/

      [4] https://exchange.stackstorm.org/

  • nyellin 8 days ago
    Nice to see this on here.

    I founded a startup that does something similar for Kubernetes:

    http://github.com/robusta-dev/robusta

    We've seen a bit of StackStorm too with our users!

  • throwawaaarrgh 8 days ago
    I love the design, but it just seems mostly abandoned. Maybe it needs better marketing?
    • eugenc 8 days ago
      One of the maintainers during EU late night here.

      * The release management team is preparing the v3.8.0 release right now, - coming out soon next week.

      * There are 15 Maintainers (https://github.com/StackStorm/st2/blob/master/OWNERS.md) in the Technical Steering Committee (TSC), a very diverse group, but I'd say only 1/3 or half are really active.

      * There are monthly TSC meetings and we invite curious to join us: https://github.com/StackStorm/community/issues/33 (next meeting is on 6th Dec).

      * Past Meetings: https://github.com/StackStorm/community/issues?q=is%3Aissue+...

      * The project is under the neutral Linux Foundation umbrella with no single commercial owner, so probably why less marketing. We have blog/twitter/linkedin with the mostly engineering updates for engineers.

      But I agree with the sentiment overall, - the development velocity could be higher and we definitely welcome new contributors and interested folks. The tool is written in Python if someone is willing to join, play with it and start contributing.

    • tempest_ 8 days ago
      In what sense?

      The repo linked here appears to have contributions merged in that last month and it has a bunch of active pull requests.