Deciphering Cassandra’s Prophecy: Preventing the Fall of Your Application

Cassandra, an open-source NoSQL database managed by the Apache Foundation, has emerged as a critical tool for financial analytics due to its scalability, fault tolerance, and ability to handle large-scale data workloads. However, its complexity demands meticulous configuration and management. This article explores four critical challenges in Cassandra deployment, their root causes, and actionable solutions to ensure application stability and performance.

Problem 1: TTL and Compaction Strategy Conflict Leading to Disk Space Explosion

Symptoms

• High disk usage
• Abnormal increase in SSTable count
• Droppable tombstone ratio exceeding 100%

Root Causes

• TTL (Time To Live) causes tombstones for expired data
• Time Window Compaction Strategy (TWCS) fails to merge SSTables when TTL ranges are inconsistent (e.g., 2 minutes vs. 6 months)
• Incompatible TTL ranges prevent safe deletion of fully expired SSTables

Solutions

• Adjust Compaction Strategy:
  • Enable unchecked_tombstone_compaction to force deletion of non-overlapping tombstones
  • Use allow_unsafe_aggressive_ss_table_expiration (ensure data is append-only)
• Data Model Optimization:
  • Partition data with different TTLs into separate tables
  • Switch to Levelled Compaction Strategy (LCS) for improved efficiency
• Cassandra 5.0 introduces Unified Compaction Strategy, promising enhanced resolution

Problem 2: Cross-Datacenter Write Timeout and JVM Thread Exhaustion

Symptoms

• Timeout in specific datacenter nodes
• Out of Memory (Native Thread) errors

Root Causes

• Default JVM configuration fails to terminate nodes when Native Threads are exhausted
• Gossip protocol continues pinging failed nodes, causing misjudgment of node status
• Mutate messages block at failed nodes, leading to coordinator timeouts

Solutions

• Terminate Failed Nodes Immediately:
  • Use JVM Agent (e.g., JvmQuake) to handle Out of Memory states automatically
• Tune JVM Configuration:
  • Set task_max parameter correctly to avoid systemd thread limits
  • Avoid misconfigurations related to enpr (system resource limits)

Problem 3: Authorization Errors and Cache Invalidation Leading to Query Timeout

Symptoms

• Occasional authorization errors
• Query timeouts under high load

Root Causes

• Local node caches (Authorization Policies, Roles, Credentials) expire, requiring queries to System Tables
• Overloaded nodes may experience contention between authorization queries and other queries
• Expired authorization errors are classified as UnauthorizedException, preventing retries

Solutions

• Application Layer Retry Mechanism:
  • Configure retries for specific error types (e.g., AuthFailure)
• Optimize Cache Configuration:
  • Increase roles_validity, permissions_validity, and credentials_validity
  • Enable asynchronous cache refresh (async_refresh) to reduce sync query pressure
• Upgrade to Cassandra 4.1:
  • Supports synchronous cache refresh, reducing system load

Problem 4: Data Migration and Volume Anomalies

Symptoms

• Post-migration data volume is only 50% of the original cluster

Root Causes

• Original cluster used Size-Tiered Compaction (STCS), leading to fragmented SSTables from multiple inserts
• Post-migration single writes caused compaction, reducing data redundancy
• Original cluster had higher compression due to data fragmentation

Solutions

• Switch to Levelled Compaction Strategy (LCSS):
  • Reduces redundant row storage, improving compression
• Optimize Insert Logic:
  • Minimize multiple inserts; batch updates into a single operation
• Trigger Compaction Manually:
  • Use nodetool compact to reduce data fragmentation post-migration

Technical Summary

• Compaction Strategy Selection: Choose based on data characteristics (TWCS, LCS, Unified)
• TTL Management: Avoid large TTL ranges to prevent compaction failures
• JVM Tuning: Address Native Thread exhaustion and thread limits
• Authorization Optimization: Enhance cache validity and error handling
• Data Migration: Ensure compaction strategy alignment and data consistency

Key Lessons

• Observability: Enable query tracing to detect multi-insert issues early
• Configuration: Avoid default settings; tailor compaction strategies to use cases
• Proactive Monitoring: Track tombstone ratios, compaction progress, and disk usage anomalies

By addressing these challenges, organizations can harness Cassandra’s full potential for financial analytics, ensuring reliability, scalability, and cost efficiency in their data infrastructure.