{"id":97534,"date":"2025-06-09T08:00:00","date_gmt":"2025-06-09T15:00:00","guid":{"rendered":""},"modified":"2025-06-06T10:29:13","modified_gmt":"2025-06-06T17:29:13","slug":"microsoft-and-f5-join-forces-on-opentelemetry-with-apache-arrow-in-rust","status":"publish","type":"post","link":"https:\/\/opensource.microsoft.com\/blog\/2025\/06\/09\/microsoft-and-f5-join-forces-on-opentelemetry-with-apache-arrow-in-rust\/","title":{"rendered":"Microsoft and F5 join forces on OpenTelemetry with Apache Arrow in Rust"},"content":{"rendered":"\n

Microsoft and F5 are collaborating on Phase 2 of the OpenTelemetry<\/a> with Apache Arrow<\/a> project. This project is developing a high-performance telemetry pipeline integrated with the Apache Arrow ecosystem, with a range of applications for small- and large-scale uses. We are creating new telemetry pipeline libraries in the Rust language, and we aim to deliver this functionality through integration with the OpenTelemetry Collector. <\/p>\n\n\n\n

In Phase 1 of the project, we developed OpenTelemetry-Arrow Protocol (OTAP) libraries meant for optimizing network usage between Collectors based on Apache Arrow IPC streams. In this phase of the project, we are developing OTAP libraries for optimizing the cost of telemetry inside Collectors, in terms of CPU usage, using Apache Arrow data frames as the unit of transport between producers and consumers. <\/p>\n\n\n\n

We are bringing OpenTelemetry use-cases into the Apache Arrow ecosystem, with a promise of full compatibility. OpenTelemetry with Apache Arrow gives OpenTelemetry users a way to exchange OpenTelemetry data with a wide range of data-oriented applications. We think it will be good for OpenTelemetry to have access to the broader Apache Arrow and Rust ecosystems. <\/p>\n\n\n\n

A look back: Phase 1 success <\/h2>\n\n\n\n

Phase 1 of OpenTelemetry with Apache Arrow project began with a reference implementation of OTAP streams for the OpenTelemetry Collector. Now included in the OpenTelemetry Collector-Contrib distribution, these components act as drop-in replacements for the OTLP receiver and exporter. Anywhere OTLP requests are being used for bulk telemetry transport between Collectors, a \u201ccompression bridge\u201d using OTAP streams instead could be established to achieve substantially better compression. <\/p>\n\n\n\n

For users, this means sending 30 to 70% less data over the network, simply from using OTAP instead of OTLP. The network savings result from translating OTLP data out of its hierarchical representation, based on Protocol Buffers, into a tabular representation based on Apache Arrow, the results achieved first through sorting and de-duplication, then through Arrow IPC encoding optimizations. <\/p>\n\n\n\n

Phase 1 was just the beginning. It proved the power of combining OpenTelemetry and Apache Arrow and set the stage for Phase 2. <\/p>\n\n\n\n

Phase 2: Enter Rust and Arrow <\/h2>\n\n\n\n

In Phase 2, Microsoft and F5 are building a fully native, embeddable OpenTelemetry pipeline powered by Rust and Apache Arrow. The goal? Dramatic improvements in CPU utilization, memory efficiency, throughput, and latency\u2014all of which contribute to a more efficient data processing ecosystem. <\/p>\n\n\n\n

The decision to go “all in” with Rust has stirred excitement\u2014and a bit of controversy\u2014among the technical community. After all, OpenTelemetry\u2019s Collector project is traditionally built using Golang. So why Rust? <\/p>\n\n\n\n

Rust and Apache Arrow are like best friends in the data processing world. Both technologies complement each other profoundly, making Rust the ideal choice for Phase 2 development. Here are just a few reasons:  <\/p>\n\n\n\n