Securing OpenTelemetry Telemetry in Transit with TLS

Introduction

OpenTelemetry has become a cornerstone for observability in modern distributed systems, enabling the collection and analysis of telemetry data such as traces, metrics, and logs. However, the transmission of this data without encryption poses significant risks, including exposure of sensitive information and potential exploitation by attackers. This article explores the critical need for encrypting telemetry data in transit using TLS, aligns with regulatory requirements, and provides a comprehensive guide to implementing secure configurations within the OpenTelemetry ecosystem.

Problem Overview

Information Exposure Risks

Unencrypted telemetry data transmitted via OpenTelemetry contains sensitive details such as hostnames, operating systems, Java versions, database types, connection strings, and SQL queries. These details can be easily extracted using tools like Wireshark, enabling attackers to exploit vulnerabilities or target specific systems.

Regulatory Compliance

Multiple regulations mandate encryption during data transmission:

• FedRAMP SC8 (United States)
• GDPR, UK GDPR, ENISA NIS2 (European Union/UK)
• HIPAA (Healthcare sector)

TLS Configuration for OpenTelemetry

TLS Features

TLS provides bidirectional authentication and encrypted communication channels, ensuring data confidentiality and integrity. Servers must validate authenticity, and data remains visible only to authorized endpoints.

OpenTelemetry Collector Setup

Configure TLS in collector.yaml to enforce secure communication:

• Minimum TLS Version: Recommend TLS 1.3 for enhanced security.
• Server Certificates: Use .crt for certificates and .key for private keys.
• FIPS Compliance: Utilize OpenSSL versions with FIPS extensions and enable -provider fips for algorithm restrictions.

Certificate Generation and Configuration

Self-Signed Certificate Creation

Generate self-signed certificates using OpenSSL:

openssl req -new -x509 -nodes -days 365 -out collector.crt -keyout collector.key

• Certificate Fields: Define organizational details (e.g., O, L, C, CN) to ensure DNS name alignment.

Trust Chain Validation

• Public Certificates: Rely on trusted Certificate Authorities (e.g., Let's Encrypt) for chain validation.
• Self-Signed Certificates: Manually add root certificates to truststores to avoid unable to find valid certification path errors in Java environments.

Testing and Trust Chain Challenges

TLS Connection Verification

Test TLS 1.3 connectivity to OpenTelemetry Collector's 4318 port using openssl s_client. Self-signed certificates may trigger trust chain failures in Java clients.

Trust Chain Management

Ensure certificates are validated through a trusted chain. In isolated networks, verify certificate authenticity to prevent spoofing attacks.

Configuration Challenges and Best Practices

Java Client Configuration

• Transition endpoints from http to https.
• Address certificate trust issues by explicitly managing truststores.

Network Isolation Considerations

• Validate certificate sources to prevent spoofing.
• Regularly update certificates and ensure secure storage of private keys.

Technical Implementation Details

Port and TLS Configuration

• Default Port: OpenTelemetry uses port 4318 for plaintext. Configure a dedicated TLS port (e.g., 465) for clarity.
• TLS Overhead: Acknowledge increased latency and resource usage in production environments.

mTLS Recommendations

• Implement mutual TLS (mTLS) for enhanced authentication. Clients must embed certificates in requests, as outlined in OpenTelemetry Collector documentation.

Security Considerations

Default Security Mechanisms

• Advocate for default TLS activation in OpenTelemetry to mitigate plaintext risks.

Trust Chain Challenges

• Self-signed certificates lack full trust chains, requiring manual CA management.

Development vs. Production Environments

• Development: Use self-signed certificates for rapid validation.
• Production: Adhere to strict certificate management protocols, including CA infrastructure and automated renewal processes.

Conclusion

Encrypting telemetry data in transit is essential to comply with regulatory standards and protect sensitive information. Implementing TLS with proper certificate management, trust chain validation, and mTLS configurations ensures secure OpenTelemetry operations. Addressing challenges such as Java truststore management and network isolation is critical for robust security. By prioritizing encryption and adhering to best practices, organizations can safeguard their observability infrastructure against potential threats.