A Distributed System On Fire: Diagnosing Failures with otel4s
Learn to diagnose realistic production failures in a distributed Typelevel streaming platform. Through a curated set of incidents, you will debug failures using telemetry and iteratively improve observability through manual instrumentation with otel4s, leaving with a practical mental model for telemetry-driven debugging in Scala.
This interactive lab puts you in front of a running distributed streaming platform built on the Typelevel stack (cats-effect, fs2, http4s, smithy4s, skunk, otel4s) and asks you to diagnose what just broke.
The system is realistic: several microservices communicating over HTTP, an fs2 traffic generator, Postgres, and a full observability stack (metrics, traces and logs) already wired up with otel4s and visualized with Grafana dashboards.
Everything is pre-built, pre-deployed, and runs locally with a single command. If you would rather not run anything locally, the lab will be fully available as a GitHub Codespace.
We will work through a curated series of incidents covering two flavours of production pain:
- System-wide degradation: A big chunk of the platform slows down or stops working for everyone.
- Tail-latency mysteries: most users are fine, but a small fraction see slow or failing requests.
For each incident, you will get the symptoms, access to the running system and its observability backend, and time to investigate. We will guide you through a repeatable diagnosis pipeline — metrics → traces → logs — that scales from trivial bugs to genuinely tricky ones. At least one scenario is designed so that default instrumentation is not enough: you will add your own spans and attributes with otel4s to narrow down the problem.
After each round we will walk through the diagnosis together and discuss what telemetry made visible that logs alone would not have.
By the end you will have a practical mental model for telemetry-driven debugging in Scala, hands-on familiarity with otel4s in a realistic distributed stack, and a take-home repository you can run locally to keep practising.
Setup requirements: Docker, sbt, java 17+ or use github codespace.
Nice to have (but not required):
- Comfortable working with Scala and Cats Effect.
- Familiarity with the basic ideas of distributed systems.
This session will not be livestreamed or recorded.
Antonio is a Scala engineer, EPFL graduate, and functional programming enthusiast. A Typelevel contributor and Scala Ambassador, he's participated in Google Summer of Code as both student and mentor.
I'm a Scala engineer and open source maintainer focused on functional systems, observability, and distributed systems. I'm the maintainer of otel4s, a Typelevel OpenTelemetry instrumentation library for Scala, and I contribute across the Typelevel ecosystem. My work centers on making production systems easier to understand, diagnose, and operate through practical observability.
