Reliable Architectures through Observability

()

Architecture Reliability Observability

Architecture reliability observability involves handling change and being resilient to changes in the environment and customer behavior.
Telemetry and observability are necessary to understand systems in the cloud.
Metrics provide pre-aggregated answers but are limited for debugging.
Logs require clever queries and correlation between systems and events.
Tracing provides more information about the system, including hierarchy, ordering, sequencing, and timing.
Tracing is crucial for understanding cross-service interactions and requires propagating trace information.

OpenTelemetry (OTel) is an open-source vendor-independent standard for telemetry data formats and protocols.
OTel includes standards for traces, metrics, and logs, with libraries available in various programming languages.
The OpenTelemetry Collector is a processing proxy that receives data from various sources, processes it, and exports it to different destinations.
Auto-instrumentation and manual instrumentation are two approaches for collecting telemetry data.
Auto-instrumentation works well for low-level components like operating systems and containers.
Manual instrumentation is valuable at the application level to capture specific details.
OpenTelemetry provides a way to trace user behavior and monitor applications, potentially replacing traditional RUM (Real User Monitoring) tools.

Tracing helps understand cause-and-effect relationships in asynchronous architectures.
Span links and secondary traces connect traces across different processes or services.
Baggage allows data propagation between services but should be filtered to prevent unintended propagation.
Planning for observability and integrating tracing from the beginning of a project is valuable for debugging and understanding application behavior.
OpenTelemetry is a cost-effective solution for tracing, and it's recommended to record too much data initially and then trim it down as needed.

The OTel Collector receives telemetry data from sources and forwards it to exporters.
Processors transform and filter telemetry data before it is exported.
Exporters send telemetry data to various destinations, such as Prometheus, third-party vendors, or custom systems.

Sampling reduces the amount of telemetry data collected and processed.
Service Level Objectives (SLOs) define the expected performance of a system and track how well it meets those expectations.

Observability can be added incrementally to legacy systems by starting with a collector and then adding instrumentation as needed.
OpenTelemetry is an open-source project supported by a large community of contributors.

To avoid tracking issues with large batches, set limits on batch size and duration.
Decorate all traces with a batch ID to easily query and identify related traces within a batch.
Tracing is relatively mature and works well in most languages.
Matrix is still stabilizing, with some languages having better support than others.
Logs are still in development, especially in areas like sampling.
Despite these challenges, investing in OpenTelemetry now is worthwhile, starting with tracing and gradually adopting other features as they mature.