This is very interesting. I've wanted something like this for a while. When debugging SSR in Next.js, I have a script to spin up mitmproxy and configure Next.js to send requests through it. I also have a script that can dump the logs of all containers while sending some request, but it's such a hack I hardly ever use it. I've toyed with the idea of scaling this up to include more services in the stack with some tcpdump or eBPF magic, but it gets complicated really fast.
At work we have about two dozen images which spawn about 60+ containers (including replicas) in prod, and 30 in dev. The services run the gamut in terms of role and language, including everything from stock haproxy images to esoteric Python or JS images. In development we use docker-compose and in prod we use Nomad.
I'm not familiar with OpenTelemetry. Does this require adding some instrumentation to every downstream service, or do you/OpenTelemetry sidestep that somehow (eg with eBPF on the shared network interfaces)?
How easy would it be to add this into our e2e tests within our JS stack? For example, say we want to make assertions that Next.js SSR results in an HTTP request through haproxy (standard), then web-openresty (customized), then auth-service (Python). Can we write the tests in JS, only worrying about the Next.js service, without touching the other services, or do we need to add instrumentation to them as well? Ideally, we could add the instrumentation once in the development stack for docker-compose, and once in "prod" (temporary staging environments where we run e2e tests) for Nomad. (And ideally "adding it once" wouldn't mean adding two dozen sidecar services, but rather once per network interface, eg task groups in Nomad and Docker networks in docker-compose.)
Also, I see you have a bunch of "coming soon" services - is this just about writing the assertion adapters for their protocols, or do they require some low level adaptation to even work at all?
Really interesting because we considered eBPF when making this but started with OpenTelemetry (for various reasons, but mostly its increasing adoption). We still might incorporate eBPF if it adds value.
OpenTelemetry does need to instrument each service in your env (at least anything you want to make assertions on, you can also start with a subset). That said, they have an incredible community and have made installing it in Kubernetes very simple. The "Open Telemetry Operator" has auto-instrumentation for Node, .Net, Java, Python.
So its very easy to add (and gives you lots of value in terms of observability in general).
We chose JS and Jest for our testing framework (again because of popularity) but the trace data produced by Otel is framework/language agnostic, so it could really have been anything. And as you said, adding functionality is just a matter of writing more assertion adapters for different protocols. Its pretty straight forward, and we welcome any contributions.
Yes, it's exactly the same setup.
I think OpenTelemetry gives a whole world of possibilities you can build on top which are much more than just observability.
We're creating the backend equivalent of what cypress is doing. So in cypress you're generating FE tests where you click buttons and assert on FE components. Our framework can do that for checking BE flows. Let's say a user registered on your website, you can then create a test that make sure that there's a new entry for her in all your DBs, that you send her a welcome e-mail, and that you reported a BI event with all the information (like referrer etc.).
At work we have about two dozen images which spawn about 60+ containers (including replicas) in prod, and 30 in dev. The services run the gamut in terms of role and language, including everything from stock haproxy images to esoteric Python or JS images. In development we use docker-compose and in prod we use Nomad.
I'm not familiar with OpenTelemetry. Does this require adding some instrumentation to every downstream service, or do you/OpenTelemetry sidestep that somehow (eg with eBPF on the shared network interfaces)?
How easy would it be to add this into our e2e tests within our JS stack? For example, say we want to make assertions that Next.js SSR results in an HTTP request through haproxy (standard), then web-openresty (customized), then auth-service (Python). Can we write the tests in JS, only worrying about the Next.js service, without touching the other services, or do we need to add instrumentation to them as well? Ideally, we could add the instrumentation once in the development stack for docker-compose, and once in "prod" (temporary staging environments where we run e2e tests) for Nomad. (And ideally "adding it once" wouldn't mean adding two dozen sidecar services, but rather once per network interface, eg task groups in Nomad and Docker networks in docker-compose.)
Also, I see you have a bunch of "coming soon" services - is this just about writing the assertion adapters for their protocols, or do they require some low level adaptation to even work at all?
OpenTelemetry does need to instrument each service in your env (at least anything you want to make assertions on, you can also start with a subset). That said, they have an incredible community and have made installing it in Kubernetes very simple. The "Open Telemetry Operator" has auto-instrumentation for Node, .Net, Java, Python.
So its very easy to add (and gives you lots of value in terms of observability in general).
We chose JS and Jest for our testing framework (again because of popularity) but the trace data produced by Otel is framework/language agnostic, so it could really have been anything. And as you said, adding functionality is just a matter of writing more assertion adapters for different protocols. Its pretty straight forward, and we welcome any contributions.
Are you able to use the same prod monitoring setup as integration test time? Or does it require any differences in setup?