The State of OpenTelemetry Profiling: A Deep Dive into Signal Integration and Technical Evolution

Introduction

OpenTelemetry has emerged as a cornerstone of observability in cloud-native ecosystems, providing standardized tools for tracing, metrics, and logging. Recently, profiling has been introduced as a new signal type within the OpenTelemetry framework, aiming to enhance performance analysis and debugging capabilities. This article explores the current state of OpenTelemetry Profiling, its technical architecture, challenges, and future directions, emphasizing its role within the CNCF ecosystem.

Defining Profiling and Its Core Concepts

Profiling involves visualizing resource consumption during program execution, including CPU, memory, and latency metrics, using tools like flamegraphs, perfetto, and Chrome Developer Tools. Its primary applications include:

• Anomaly Diagnosis: Identifying root causes of unexpected CPU spikes, such as OpenSSL regressions.
• Performance Optimization: Pinpointing high-cost functions for resource allocation adjustments.
• Tail Latency Analysis: Resolving mutex contention issues in distributed systems.
• Thread Interaction Monitoring: Detecting bottlenecks in multi-threaded environments.

OpenTelemetry’s Progress in Profiling

OpenTelemetry’s journey toward profiling began in 2020 with a GitHub issue proposing profiling as a new signal type. Initial delays occurred due to community focus on metrics and locks. Key milestones include:

• 2022: Formation of the SIG (Special Interest Group) with 100+ participants, leading to the development of data models (OTB-212) and format standards (EP-239).
• 2024: Elastic’s donation of an eBPF profiler marked a critical step, resolving standardization and toolchain integration challenges.
• 2024–2025: Integration of the eBPF profiler into the Collector, enabling resource detection and attribute tagging via OTL (OpenTelemetry Protocol).

Technical Architecture and Implementation

Collector Configuration

• Cluster-wide Collector: Aggregates logs, metrics, traces, and profiles, converting data via OTL to external storage (e.g., Elasticsearch).
• Node-level Collector: Deploys eBPF profilers per node to avoid resource overuse, ensuring architectural flexibility.

Data Processing Pipeline

1. Profiling Data Generation: Captures stack traces and timestamps.
2. Context Enrichment: Adds resource detection (e.g., host ID) and attributes (K attributes) for contextual analysis.
3. Query Support: Utilizes OTL query syntax for data transformation and analysis.

Technical Challenges and Future Directions

Data Model Optimization

• Stack Trace Representation: Current frame-list format may be replaced with more efficient structures.
• Timestamp Handling: Balancing aggregation views (e.g., flamegraphs) with time-series data.

Protocol Standardization

• VLP Protocol: Finalizing the OpenTelemetry Profiling Protocol (VLP) for cross-toolchain compatibility.
• Integration: Harmonizing eBPF with other profilers (e.g., JFR, PPR) for unified data formats.

Production Readiness

• Current Status: Still in development, requiring improvements in data conversion, performance, and stability.
• Future Goals: Establishing a complete OpenTelemetry Profiling ecosystem for real-time monitoring and visualization.

Collector Configuration and Operation

• Clusterwide Collector: Configures OTL receivers and Elasticsearch exporters for data visualization.
• eBPF Profiler: Enables eBPF modules for performance data collection.
• Resource Detection: Adds host attributes (e.g., host ID, name) to profiling data.
• Data Export: Integrates with TLP Collector for comprehensive analysis.

Data Visualization and Analysis

• Flame Graphs: Display CPU usage and stack traces for performance insights.
• Time-series Views: Supports real-time and historical analysis of CPU and non-CPU states.
• Elasticsearch Integration: Visualizes data through the Elastic search interface.

Current Challenges and Open Issues

1. Stack Trace Representation: Frame lists require slicing; evaluating optimized formats.
2. Timestamp Support: Enhancing protocol handling for real-time and historical analysis.
3. Resource Tagging: Improving tagging and segmentation for focused analysis on specific components.
4. Symbol Information Upload: Standardizing symbol upload protocols for function name and line number decoding, even when symbols are not available locally.

Future Work and Roadmap

• Official Release: Developing lightweight Collector distributions focused on eBPF and OTL data export.
• Trace-Profile Correlation: Linking trace and profile data for seamless navigation between datasets.
• Protocol Finalization: Completing the VLP protocol and expanding toolchains for enhanced data processing.

Current Status and Recommendations

OpenTelemetry Profiling remains under development and is not yet production-ready. Users are advised to test in controlled environments and provide feedback. The community remains active, encouraging contributions through Slack channels, SIG meetings, and observability showcases. Collaboration across SIGs will be critical to addressing protocol representation challenges and advancing the ecosystem.