In my view, a great alternative would be the use of workflow tools, which have a graphical pipeline where teams can follow each step of the process along with its respective input and output, in a visual interface.
I see this type of design as an evolution of how software is developed, but I don't see a wide adoption by companies/startups when they are starting to develop their applications. What I see is the same old recipe being repeated infinitely, new project = microservices + cloud.
Note: to give examples of such tools, I will mention two that I found on the awesome-workflow-engines list on GitHub. - Apache Airflow - temporal(dot)io
The problem I saw was the same problem that (almost) every framework everywhere has. It makes expected use cases easy, small variations difficult, and unexpected use cases impossible. The escape hatch was generally via custom code that used some horrible DSL evolved from a config file.
Workflows are not zero-cost, they have their own tradeoffs compared to microservices. State management / bootstrapping logic becomes non-trivial, execution order though easier to visualize is also slightly not deterministic, workflows are not as well suited for request-response style replies due to the latency involved in total execution etc (but I think they are great alternatives to async / background workfllows) - and shared underlying infrastructure means increased chances of SPOFs.
The state must have improved much since then. Also, adoption of anything new to require remodelling your application into a different paradigm must be worth the value delivered. For example, modular monoliths became popular because they reduced operational complexity by reducing # of pieces involved. At the time, that value prop vs effort involved was unclear to our teams IMO
I use Step Functions over Lambda for one big reason: no dependencies or runtimes to update. Secondary reasons include the lack of “cold start” and the unique capabilities. However, they have huge downsides and one of them is the complexity of building anything that isn’t choreographing calls to AWS services. It can get crazy ugly, and would quickly if it was the only tool I had.
Used in combination with imperative code, declarative workflow is a gift. On its own, a nightmare.
Don't be confused that you _can_ call Lambada from StepFunctions, but that's a different beast (and a bit of an anti-patter IMHO).
Both have advantages and disadvantages, but in the case you’re talking about the compromise is that you end up being constrained by someone else’s design goals, concepts and principles.
When you use multiple SaaS systems together the constraints are cumulative.
Hence, you are forced to constrain the constraints and the analogy “jack of all trades, master of none”, springs to mind.
One of the many SaaS offerings I regularly have to work with is for customers leveraging Microsoft D365 CRM. It classically demonstrates this point. There are two types of people who “develop” using it. The first are integrators that have a more technical background writing code, and locoders who sit in enterprises as a quasi business analyst / click-and-drag developer. The solutions they develop work, but often badly through horrific spaghetti designs, because they are forced to workaround the constraints of the locode system.
As long as those trade-offs are understood by CTO’s making these decisions then it’s fine, however many don’t grok it at all.
One thing many of these systems fail at is observability. Connecting your own code up to DataDog or Prometheus is easy, not often the case for a SaaS or third party system. Although in the case of Temporal, that is somewhat available in your own activities and workflows.
There is the term "citizen integrator/developer". The business guys who like computers. 1: most of the time real-world problems are to complex for them to solve as it needs more software engineering skills. 2: Low/No Code is a nice argument to sell such products. In the end techies do develop with it and have a hard time working around the supplied "framework" 3: In some cases it does save time, on the other hand you lose the ability to solve problems as you are going proprietary with this platforms and that makes you dependent.
What could work is if you configure the "business view" workflow in no code and do the implementation in a casual programming language, but this isn't a requirement very often, and there are not so many platforms as you can't lock-in customers that way.
Do you guys know any open WMS/BPM, integration platforms from your own experience that you would recommend? Thanks.
The whole point of micro-service architectures is that everything talks over very standardized interfaces with very clear boundaries so you can insert a wide variety of instrumentation. That instrumentation can even be added after the fact and progressively enhanced when chasing problems instead of prematurely optimizing.
I point it out because I feel this is a common enough misunderstanding, microservices isn't a technical solution it's just an exercise in organization.
To this point, so are workflows. Workflows better emulate business logic and business data flows, depending on how much of that you have to deal with, worflows might be a good way to organize that core requirement.
You would then leverage a WMS to create sequential tasks to maybe: trigger an app deployment, send notifications to developers and stakeholders, update a database, send out SMS messages to customers, etc... although many ci/cd services offer these capabilities as well, WMSs are more comprehensive in what they can do.
But if you don't need all of those extra bells and whistles, you might actually add more complexity going to a WMS.
For help with a complex codebase, i like to adhere to a lot of principles in the 12 factor app https://12factor.net/
...Although thats easier said than done to "clean up" a codebase