Cassandra Performance Tuning

Apache Cassandra is designed for high performance at scale, but achieving optimal performance requires careful configuration,

monitoring, and optimization. In this lesson, we'll explore comprehensive techniques for tuning Cassandra

to achieve maximum throughput, minimal latency, and efficient resource utilization.

Performance tuning is a continuous, iterative process. The techniques we'll cover should be applied

systematically with careful monitoring to measure the impact of each change.

By the end of this lesson, you'll understand:

Hardware and infrastructure considerations for Cassandra

JVM and system-level tuning for optimal performance

Data model optimization techniques

Compaction strategies and their performance impacts

Caching mechanisms and configuration

Query optimization patterns

Performance monitoring and benchmarking

Capacity planning and scaling strategies

Cassandra's performance is significantly impacted by the hardware it runs on. Understanding Cassandra's resource

utilization patterns helps in making informed hardware decisions.

Cassandra benefits from multiple cores rather than higher clock speeds, as many operations can run in parallel:

Use processors with at least 8-16 cores per node

Newer CPUs with improved instruction sets can significantly boost performance

For read-heavy workloads, ensure sufficient CPU resources for handling read requests

For write-heavy workloads, CPU is less critical compared to memory and disk I/O

Memory is crucial for Cassandra performance, particularly for caching and preventing disk I/O:

Minimum 16GB RAM per node, with 32-64GB recommended for production

Allocate 8GB minimum for JVM heap, but not more than 32GB to avoid long GC pauses

Off-heap memory is used for Cassandra's internal caching

Memory should be allocated based on your workload (read-heavy vs. write-heavy)

1# Memory allocation in cassandra.yaml
2jvm_opts:
3  -Xms: 16G        # Initial heap size
4  -Xmx: 16G        # Maximum heap size
5  # Other JVM options...
6
7file_cache_size_in_mb: 512  # Off-heap memory for file caching

Storage is often the limiting factor for Cassandra performance:

SSDs provide significant performance advantages over HDDs

NVMe drives offer even better performance for high-throughput workloads

Separate disks for commitlog, data directory, and system files

RAID-0 can improve throughput for data directories

Avoid RAID-5 due to write penalties

Consider local storage over network storage for better performance

1# Storage configuration in cassandra.yaml
2data_file_directories:
3  - /var/lib/cassandra/data
4
5commitlog_directory: /var/lib/cassandra/commitlog
6
7saved_caches_directory: /var/lib/cassandra/saved_caches
8
9hints_directory: /var/lib/cassandra/hints

Network performance impacts both client operations and inter-node communication:

Use 10Gbps or faster network interfaces

Configure jumbo frames (MTU 9000) when possible

Use dedicated networks for client traffic and inter-node communication

Minimize latency between nodes, especially in multi-datacenter deployments

Configure network buffer sizes appropriately

Avoid sharing network infrastructure with other high-traffic services. Network congestion can cause Cassandra nodes

to temporarily appear down to other nodes, triggering unnecessary repair operations.

As a Java application, Cassandra's performance is significantly affected by JVM configuration.

Garbage collection pauses can cause latency spikes and affect throughput:

Use G1GC for Cassandra 3.0+ (default in newer versions)

Set heap size to an appropriate value based on your node's RAM

Monitor GC activity and tune parameters if necessary

Keep heap sizes under 32GB to avoid long GC pauses

1# Example JVM options in jvm.options or cassandra-env.sh
2-XX:+UseG1GC
3-XX:G1RSetUpdatingPauseTimePercent=5
4-XX:MaxGCPauseMillis=500
5-XX:InitiatingHeapOccupancyPercent=70
6-XX:+ParallelRefProcEnabled
7-XX:+PerfDisableSharedMem
8-XX:+AlwaysPreTouch
9-XX:+HeapDumpOnOutOfMemoryError

Operating system settings can significantly impact Cassandra performance:

Increase file descriptor limits (minimum 32768, recommended 100000+)

Optimize disk I/O scheduler (deadline or noop for SSDs)

Disable swap to prevent performance degradation

Configure transparent huge pages according to your workload

Set appropriate vm.max_map_count (at least 1048575)

Optimize TCP settings for high-throughput networks

Always test system-level changes thoroughly before applying them to production environments.

The impact of these changes can vary depending on your specific hardware and workload.

1# Example sysctl settings
2fs.file-max = 1000000
3vm.max_map_count = 1048575
4vm.swappiness = 1
5net.core.rmem_max = 16777216
6net.core.wmem_max = 16777216
7net.ipv4.tcp_rmem = 4096 87380 16777216
8net.ipv4.tcp_wmem = 4096 65536 16777216
9net.core.netdev_max_backlog = 2500

Even with optimal hardware and system configuration, a poor data model can severely limit Cassandra's performance.

Controlling partition size is critical for Cassandra performance:

Target partition sizes between 10-100MB

Avoid partitions larger than 200MB

Use composite partition keys to distribute data more evenly

Implement time-based bucketing for time-series data

1-- Poor design: Unbounded partition growth
2CREATE TABLE user_events (
3  user_id UUID,
4  event_time TIMESTAMP,
5  event_type TEXT,
6  event_data TEXT,
7  PRIMARY KEY (user_id, event_time)
8);
9
10-- Better design: Time bucketing to limit partition size
11CREATE TABLE user_events_by_month (
12  user_id UUID,
13  year_month TEXT,  -- Format: 'YYYY-MM'
14  event_time TIMESTAMP,
15  event_type TEXT,
16  event_data TEXT,
17  PRIMARY KEY ((user_id, year_month), event_time)
18);

Clustering columns affect how data is stored on disk and can impact read performance:

Choose clustering columns based on your most frequent access patterns

Use clustering order that aligns with your query patterns

Limit the number of clustering columns to minimize overhead

Order clustering columns from most selective to least selective

1-- Example with optimized clustering column ordering
2CREATE TABLE user_sessions (
3  user_id UUID,
4  session_date DATE,
5  session_id UUID,
6  device TEXT,
7  ip_address TEXT,
8  duration INT,
9  PRIMARY KEY ((user_id), session_date, session_id)
10) WITH CLUSTERING ORDER BY (session_date DESC, session_id DESC);
11
12-- Query pattern: Get most recent sessions for a user
13SELECT * FROM user_sessions
14WHERE user_id = 123e4567-e89b-12d3-a456-426614174000
15LIMIT 10;

Use these features cautiously as they can impact performance:

Avoid secondary indexes on high-cardinality columns

Use materialized views for read-heavy workloads when appropriate

Consider additional tables instead of secondary indexes for critical queries

Be aware that materialized views add write overhead

1-- Secondary index (use cautiously)
2CREATE INDEX ON users(email);  -- Only if email has low cardinality
3
4-- Better approach: Separate lookup table
5CREATE TABLE users_by_email (
6  email TEXT PRIMARY KEY,
7  user_id UUID
8);
9
10-- Materialized view (adds write overhead)
11CREATE MATERIALIZED VIEW products_by_category AS
12  SELECT * FROM products
13  WHERE category IS NOT NULL AND product_id IS NOT NULL
14  PRIMARY KEY (category, product_id);

Compaction is the process of merging SSTables and purging tombstones. The choice of compaction

strategy significantly impacts read and write performance.

The default strategy, optimized for write-heavy workloads:

Triggers compaction when similar-sized SSTables reach a threshold

Good for insert-heavy workloads with few updates or deletes

May lead to read performance degradation over time

Tunable via min_threshold and max_threshold parameters

1CREATE TABLE metrics (
2  metric_name TEXT,
3  timestamp TIMESTAMP,
4  value DOUBLE,
5  PRIMARY KEY (metric_name, timestamp)
6) WITH compaction = {
7  'class': 'SizeTieredCompactionStrategy',
8  'min_threshold': 4,
9  'max_threshold': 32,
10  'min_sstable_size': 50
11};

Optimized for read-heavy workloads:

Organizes SSTables into levels with increasing sizes

Guarantees ~90% of reads touch at most one SSTable

Higher write amplification than STCS

Better for tables with frequent updates and deletes

Tunable via sstable_size_in_mb parameter

1CREATE TABLE users (
2  user_id UUID PRIMARY KEY,
3  username TEXT,
4  email TEXT,
5  last_login TIMESTAMP
6) WITH compaction = {
7  'class': 'LeveledCompactionStrategy',
8  'sstable_size_in_mb': 160
9};

Optimized for time-series data:

Buckets SSTables into time windows

Uses STCS within each time window

Excellent for time-series data with a TTL

Reduces compaction overhead for data that is written in time order

Tunable via compaction_window_size and compaction_window_unit parameters

Choose your compaction strategy based on your access patterns and data lifecycle.

Monitor compaction performance and adjust as needed based on your observations.

1CREATE TABLE temperature_by_device (
2  device_id TEXT,
3  date DATE,
4  timestamp TIMESTAMP,
5  temperature FLOAT,
6  PRIMARY KEY ((device_id, date), timestamp)
7) WITH compaction = {
8  'class': 'TimeWindowCompactionStrategy',
9  'compaction_window_size': 1,
10  'compaction_window_unit': 'DAYS'
11};

Cassandra offers several caching mechanisms to improve read performance.

Caches entire rows in memory:

Best for tables with frequent reads of the same rows

Memory-intensive as it stores complete rows

Most effective for tables with small rows that don't change often

Invalidated when rows are updated

1CREATE TABLE product_details (
2  product_id UUID PRIMARY KEY,
3  name TEXT,
4  description TEXT,
5  price DECIMAL
6) WITH caching = {
7  'keys': 'ALL',           -- Cache all partition keys
8  'rows_per_partition': 'ALL' -- Cache all rows
9};

Caches the partition key to SSTable offset mapping:

Enabled by default

Reduces disk seeks for read operations

Less memory-intensive than row cache

Beneficial for almost all workloads

1CREATE TABLE frequent_reads (
2  id UUID PRIMARY KEY,
3  data BLOB
4) WITH caching = {
5  'keys': 'ALL',            -- Cache all partition keys
6  'rows_per_partition': 'NONE' -- Don't cache row data
7};

Caches frequently accessed counters:

Specific to counter columns

Reduces lock contention for counter updates

Important for high-throughput counter workloads

System-wide cache settings in cassandra.yaml:

Row cache is memory-intensive and should be used selectively. Monitor memory usage carefully when enabling

row cache, especially on tables with large rows.

1# Cache size settings
2key_cache_size_in_mb: 100
3row_cache_size_in_mb: 0  # Disabled by default
4counter_cache_size_in_mb: 50
5
6# Cache save periods
7key_cache_save_period: 14400
8row_cache_save_period: 0
9counter_cache_save_period: 7200

Optimizing how you interact with Cassandra can significantly improve application performance.

Adjusting consistency levels can dramatically impact performance:

Lower consistency levels (ONE, LOCAL_ONE) provide better performance

Higher levels (QUORUM, ALL) provide stronger consistency at the cost of performance

Use different consistency levels for different operations based on requirements

Consider using LOCAL_QUORUM instead of QUORUM for multi-datacenter deployments

1-- Lower consistency for non-critical reads
2CONSISTENCY ONE;
3SELECT * FROM product_catalog WHERE category = 'Electronics';
4
5-- Higher consistency for critical operations
6CONSISTENCY LOCAL_QUORUM;
7UPDATE user_accounts SET balance = balance - 100 WHERE user_id = uuid();

Always use prepared statements with client drivers:

Reduces parsing overhead

Enables more efficient request routing

Prevents CQL injection

Improves performance for repeated queries

1// Java driver example of prepared statements
2PreparedStatement prepared = session.prepare(
3  "SELECT * FROM users WHERE user_id = ?"
4);
5
6// Reuse for multiple executions
7BoundStatement bound = prepared.bind(userId);
8ResultSet results = session.execute(bound);

Use batches appropriately:

Unlogged batches for operations on the same partition

Avoid cross-partition logged batches when possible

Keep batch sizes small (generally under 100 statements)

Use batches for atomic operations, not for bulk loading

1-- Unlogged batch for operations on the same partition (recommended)
2BEGIN UNLOGGED BATCH
3  INSERT INTO user_events (user_id, event_id, event_type)
4  VALUES (123e4567-e89b-12d3-a456-426614174000, uuid(), 'login');
5
6  UPDATE user_stats SET login_count = login_count + 1
7  WHERE user_id = 123e4567-e89b-12d3-a456-426614174000;
8APPLY BATCH;
9
10-- Logged batch across partitions (use sparingly)
11BEGIN BATCH
12  INSERT INTO users (user_id, username)
13  VALUES (123e4567-e89b-12d3-a456-426614174000, 'alice');
14
15  INSERT INTO user_roles (role_id, user_id, role)
16  VALUES (uuid(), 123e4567-e89b-12d3-a456-426614174000, 'admin');
17APPLY BATCH;

Implement proper pagination for large result sets:

Always use LIMIT to constrain result size

Use token-based pagination for large datasets

Consider using driver-level paging features

1-- Basic pagination with LIMIT
2SELECT * FROM products WHERE category = 'Electronics' LIMIT 25;
3
4-- Token-based pagination
5SELECT * FROM products
6WHERE category = 'Electronics'
7AND token(product_id) > token(last_seen_id)
8LIMIT 25;

Regular monitoring and benchmarking are essential for maintaining optimal Cassandra performance.

Read/write latency (p95, p99, max)

Throughput (operations per second)

Garbage collection frequency and duration

Compaction statistics

Cache hit rates

Thread pool utilization

Disk space usage and growth rate

OS-level metrics (CPU, memory, network, disk)

nodetool (cfstats, tpstats, tablestats, etc.)

JMX monitoring tools (jconsole, jmc)

Prometheus with Cassandra exporter

Grafana for visualization

Cassandra-specific tools like DataStax OpsCenter

1# Common nodetool commands for monitoring
2nodetool status        # Cluster status
3nodetool tablestats    # Table statistics
4nodetool cfhistograms  # Table performance statistics
5nodetool tpstats       # Thread pool statistics
6nodetool compactionstats  # Compaction information
7nodetool info          # Node information

Regular benchmarking helps identify performance regressions:

Use cassandra-stress for general benchmarking

Develop custom benchmarks that model your actual workload

Benchmark after significant configuration changes

Test both peak throughput and sustained load scenarios

Monitor system metrics during benchmarks

1# Basic cassandra-stress example
2cassandra-stress write n=1000000 -rate threads=50
3
4# More complex mixed workload
5cassandra-stress mixed ratio(write=1,read=3) n=1000000 -rate threads=50
6  -pop dist=gaussian(1..1000000) -schema keyspace=keyspace1 -mode native cql3

Understanding when and how to scale your Cassandra cluster is crucial for maintaining performance.

Key indicators that you need to scale your cluster:

Disk space utilization approaching 50-70%

Sustained high CPU utilization (>70-80%)

Increasing read/write latencies

Growing compaction backlog

Consistently high memory pressure

Horizontal scaling (adding nodes)

Vertical scaling (upgrading existing nodes)

Adding a new datacenter

Workload splitting (separating workloads into different clusters)

Double the cluster size rather than adding single nodes

Use token allocation planning for balanced data distribution

Run nodetool cleanup after adding nodes

Monitor and adjust streaming settings for efficient bootstrap

Scale before hitting critical capacity limits

1# Commands for node addition
2nodetool status  # Check current cluster status
3# Add new node with proper token range allocation
4# After bootstrap completes:
5nodetool cleanup  # Clean up data on existing nodes
6nodetool repair   # Ensure data consistency

Learn about Common Performance Issues and Solutions.

Issue: Excessive tombstones slow down read operations and increase storage requirements.

Solutions:

Use TTL on data when appropriate

Adjust gc_grace_seconds (default: 10 days)

Run regular repairs to clean up tombstones

Monitor tombstone thresholds in logs

Consider TimeWindowCompactionStrategy for time-series data

Issue: Garbage collection pauses cause latency spikes.

Solutions:

Tune JVM heap size (8-32GB recommended)

Use G1GC garbage collector

Monitor and adjust GC parameters based on observations

Consider JEMalloc for off-heap memory management

Issue: Uneven data distribution leads to certain nodes handling more traffic.

Solutions:

Review partition key design to ensure even distribution

Use vnodes to improve token distribution

Avoid monotonically increasing/decreasing partition keys (like timestamps)

Monitor token range ownership with nodetool status

Issue: Performing reads before writes increases latency and load.

Solutions:

Use lightweight transactions (IF EXISTS) for conditional operations

Design the data model to avoid read-before-write patterns

Use application-level caching for frequently accessed data

Learn about Advanced Performance Techniques.

Vnodes improve token distribution and operational simplicity:

Default: 256 tokens per node (num_tokens in cassandra.yaml)

Higher values improve distribution but increase memory usage and startup time

Consider reducing for very large clusters or memory-constrained nodes

Typically adjust between 64-256 based on cluster size

Proper compression can improve performance by reducing disk I/O:

LZ4 offers good balance between compression ratio and CPU usage

Adjust chunk_length_in_kb based on your access patterns

Larger chunk sizes improve compression ratio but may increase read amplification

Monitor compression savings vs. CPU impact

1CREATE TABLE compressed_table (
2  id UUID PRIMARY KEY,
3  data TEXT
4) WITH compression = {
5  'sstable_compression': 'LZ4Compressor',
6  'chunk_length_in_kb': 64
7};

Consider specialized indexes for specific query patterns:

SAI (Storage-Attached Indexing) for more efficient indexing

Custom index implementations for specific use cases

SASI (SSTable Attached Secondary Index) for text searches

Optimize your client application for better Cassandra performance:

Use connection pooling with appropriate pool sizes

Implement request throttling to prevent overloading nodes

Configure appropriate timeouts and retry policies

Use token-aware load balancing policies

Implement client-side caching for frequently accessed data

Use this checklist as a reference when optimizing Cassandra performance:

CPU: Sufficient cores for parallelism

Memory: Properly sized JVM heap and system memory

Storage: Fast SSDs with appropriate configuration

Network: High-bandwidth, low-latency connections

OS: Optimized Linux settings for high-throughput databases

JVM: Properly tuned heap size and GC settings

Compaction: Strategy appropriate for workload

Caching: Configured based on access patterns

Concurrency: Thread pools sized for expected load

Timeout settings: Appropriate for network and workload

Commitlog: Optimized for write performance

Partition size: Controlled to avoid hotspots

Clustering columns: Ordered for efficient queries

Denormalization: Appropriate for query patterns

Secondary indexes: Used sparingly and appropriately

TTL: Applied where appropriate

Consistency levels: Tuned for performance vs. consistency needs

Prepared statements: Used for all queries

Connection pooling: Properly configured

Batching: Used appropriately

Pagination: Implemented for large result sets

Error handling: Proper retry policies

Repairs: Scheduled regularly

Monitoring: Comprehensive metrics collection

Benchmarking: Regular performance testing

Cleanup: After adding/removing nodes

Upgrades: Planned and tested

Cassandra performance tuning requires a holistic approach that addresses hardware, configuration,

data modeling, and application interaction patterns. Key points to remember:

Hardware choices significantly impact Cassandra performance, particularly storage and memory

JVM tuning is critical for stable performance

Data model design is often the most important factor in Cassandra performance

Compaction strategies should be chosen based on workload characteristics

Caching configuration can dramatically improve read performance

Query optimization techniques like prepared statements and appropriate consistency levels are essential

Regular monitoring and benchmarking are necessary for maintaining performance

Scaling should be planned proactively based on capacity metrics

Performance tuning is an ongoing process. As your data volume, access patterns, and requirements evolve,

you'll need to continuously monitor and adjust your Cassandra configuration and practices.

Remember that trade-offs are inherent in any distributed system. Understand your specific requirements

for consistency, availability, and partition tolerance, and optimize accordingly.