Cassandra Security & Operations

As Cassandra deployments move from development to production, security and operational concerns

become critically important. In this lesson, we'll explore comprehensive security measures and

operational best practices for running Cassandra clusters in production environments.

Securing a distributed database system like Cassandra requires a layered approach that includes

authentication, authorization, encryption, and network security. Additionally, maintaining operational

excellence requires robust monitoring, backup strategies, and well-defined procedures for common scenarios.

By the end of this lesson, you'll understand:

Authentication mechanisms in Cassandra

Role-based access control

Encryption options for data at rest and in transit

Network security measures

Auditing and compliance

Backup and recovery strategies

Monitoring and alerting best practices

Common operational procedures

Disaster recovery planning

Upgrade and maintenance strategies

Authentication in Cassandra verifies the identity of clients connecting to the cluster.

By default, Cassandra uses the AllowAllAuthenticator, which doesn't require authentication.

In production, you should use a more secure authenticator.

The most common authentication mechanism is PasswordAuthenticator, which validates usernames and passwords

stored in Cassandra's system tables.

After configuring PasswordAuthenticator, you must create a superuser account, as the default 'cassandra'

user will be disabled:

1# In cassandra.yaml
2authenticator: PasswordAuthenticator

1-- Create a superuser (run as cassandra user before restarting)
2CREATE ROLE admin WITH PASSWORD = 'strong_password' AND SUPERUSER = true AND LOGIN = true;
3
4-- After restart, connect with the new admin account:
5-- cqlsh -u admin -p strong_password
6
7-- Change cassandra user password (recommended)
8ALTER ROLE cassandra WITH PASSWORD = 'new_strong_password';

Cassandra supports custom authenticators through its pluggable architecture. Common custom authenticators include:

LDAP Authenticator

: Authenticate against an LDAP directory

Kerberos Authenticator

: Use Kerberos for authentication

Multi-factor Authentication

: Implement MFA with custom authenticators

To implement a custom authenticator, create a class that implements the

IAuthenticator

interface

and configure it in cassandra.yaml.

Use strong, unique passwords for all accounts

Implement password rotation policies

Avoid sharing credentials among users

Consider using a password manager for credential storage

Remove or disable unused accounts

Audit authentication attempts regularly

Changes to the authenticator require a restart of all Cassandra nodes. Plan this change carefully,

and ensure you have created administrative users before restarting with PasswordAuthenticator enabled.

Cassandra implements role-based access control through its authorization system. By default, Cassandra

uses AllowAllAuthorizer, which grants all permissions to all users. In production, you should use

CassandraAuthorizer.

1# In cassandra.yaml
2authorizer: CassandraAuthorizer

In Cassandra, roles can represent users or groups. Roles can be granted to other roles,

creating a hierarchical structure.

1-- Create a role for application access
2CREATE ROLE app_user WITH PASSWORD = 'app_password' AND LOGIN = true;
3
4-- Create a role for read-only operations
5CREATE ROLE read_only WITH LOGIN = false;
6
7-- Grant read_only role to app_user
8GRANT read_only TO app_user;
9
10-- List all roles
11LIST ROLES;
12
13-- List roles of a specific role
14LIST ROLES OF app_user;

Cassandra permissions follow a hierarchical structure that includes:

ALL PERMISSIONS

: Grants all permissions

: Permission to create resources

: Permission to modify resources

: Permission to remove resources

: Permission to read data

: Permission to insert, update, or delete data

: Permission to grant/revoke permissions

: Permission to view schema definitions

: Permission to execute functions

Learn about Granting and Revoking Permissions.

1-- Grant read access to a table
2GRANT SELECT ON TABLE keyspace_name.table_name TO read_only;
3
4-- Grant write access to a table
5GRANT MODIFY ON TABLE keyspace_name.table_name TO app_user;
6
7-- Grant all permissions on a keyspace
8GRANT ALL PERMISSIONS ON KEYSPACE keyspace_name TO admin;
9
10-- Revoke permission
11REVOKE SELECT ON TABLE keyspace_name.table_name FROM read_only;
12
13-- List all permissions
14LIST ALL PERMISSIONS;
15
16-- List permissions of a role
17LIST ALL PERMISSIONS OF app_user;

Follow the principle of least privilege

Use role hierarchies to manage permissions efficiently

Create separate roles for different application components

Regularly audit role permissions

Implement a process for role lifecycle management

Document role purposes and permission grants

When running a multi-datacenter Cassandra cluster, role and permission changes are replicated across all

datacenters. However, authentication credentials in

system_auth

keyspace should use NetworkTopologyStrategy

with appropriate replication factors for all datacenters.

Cassandra supports encryption for both data in transit (node-to-node and client-to-node communications)

and data at rest (on disk).

Client-to-node encryption secures communication between clients (like cqlsh or application drivers)

and Cassandra nodes.

1# In cassandra.yaml
2client_encryption_options:
3  enabled: true
4  optional: false  # Require encryption for all clients
5  keystore: /etc/cassandra/conf/keystores/cassandra.keystore
6  keystore_password: keystore_password
7  require_client_auth: false  # Set to true for mutual TLS

Node-to-node encryption secures inter-node communication within the Cassandra cluster.

1# In cassandra.yaml
2server_encryption_options:
3  internode_encryption: all  # Options: all, none, dc, rack
4  keystore: /etc/cassandra/conf/keystores/cassandra.keystore
5  keystore_password: keystore_password
6  truststore: /etc/cassandra/conf/keystores/cassandra.truststore
7  truststore_password: truststore_password

Cassandra provides transparent data encryption (TDE) to protect data stored on disk.

This can be implemented at different levels:

1. File System Encryption

Use encrypted file systems like LUKS (Linux Unified Key Setup) to encrypt Cassandra data directories:

1. Commit Log Encryption

Encrypt commit logs using the CommitLogEncryptionContext:

1. Full Data Directory Encryption

For comprehensive data-at-rest encryption, use a combination of approaches:

Encrypt the commit log as shown above

Encrypt SSTables using third-party tools or encryption providers

Ensure all Cassandra data directories are on encrypted volumes

1# Create encrypted volume
2cryptsetup luksFormat /dev/sdb1
3
4# Open encrypted volume
5cryptsetup luksOpen /dev/sdb1 cassandra-data
6
7# Create file system
8mkfs.xfs /dev/mapper/cassandra-data
9
10# Mount volume
11mount /dev/mapper/cassandra-data /var/lib/cassandra/data

1# In cassandra.yaml
2commitlog_encryption_options:
3  enabled: true
4  algorithm: AES/CBC/PKCS5Padding
5  secret_key_strength: 128
6  key_name: commitlog_encryption_key
7  key_provider:
8    - class_name: org.apache.cassandra.security.JKSKeyProvider
9      parameters:
10        - keystore: /etc/cassandra/conf/keystores/commitlog.keystore
11        - keystore_password: commitlog_keystore_password
12        - key_password: commitlog_key_password

Proper key management is crucial for maintaining security:

Store encryption keys securely, separate from the encrypted data

Implement key rotation procedures

Use a hardware security module (HSM) for key storage when possible

Back up encryption keys securely

Document key recovery procedures

Encryption adds processing overhead that can impact performance. Benchmark your workload with encryption

enabled to understand the impact. Modern CPUs with AES-NI instruction sets minimize this overhead.

Securing Cassandra's network communication is essential for protecting data and preventing unauthorized access.

Restrict network access to Cassandra ports, allowing only legitimate clients and nodes to connect:

: Inter-node communication (7001 for SSL)

: CQL native transport (client connections)

: JMX monitoring

: Thrift clients (legacy, typically disabled)

1# Example iptables rules for Cassandra
2# Allow inter-node communication only from other Cassandra nodes
3iptables -A INPUT -p tcp -s 192.168.1.0/24 --dport 7000 -j ACCEPT
4iptables -A INPUT -p tcp --dport 7000 -j DROP
5
6# Allow CQL connections only from application servers
7iptables -A INPUT -p tcp -s 10.0.1.0/24 --dport 9042 -j ACCEPT
8iptables -A INPUT -p tcp --dport 9042 -j DROP
9
10# Restrict JMX access to monitoring servers only
11iptables -A INPUT -p tcp -s 10.0.2.5 --dport 7199 -j ACCEPT
12iptables -A INPUT -p tcp --dport 7199 -j DROP

Implement network segmentation to isolate Cassandra traffic:

Use separate networks for client traffic and inter-node communication

Place Cassandra nodes in a private subnet not directly accessible from the internet

Use a bastion host for administrative access

Implement VLANs or network security groups to segment traffic

JMX (Java Management Extensions) provides a powerful interface for monitoring and managing Cassandra,

but it can be a security risk if not properly secured:

For the JMX password and access files:

1# In cassandra-env.sh, configure JMX authentication
2JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
3JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
4JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.access.file=/etc/cassandra/jmxremote.access"
5
6# Bind JMX to localhost or specific interface (not 0.0.0.0)
7LOCAL_JMX=yes  # Or specify JMX_PORT and RMI_PORT for remote access

1# jmxremote.password
2monitorRole password1
3controlRole password2
4
5# jmxremote.access
6monitorRole readonly
7controlRole readwrite

Implement multiple layers of network security:

Use application-level firewalls or proxies for Cassandra access

Implement intrusion detection systems to monitor for suspicious activity

Use VPN for remote administrative access

Consider network traffic encryption beyond Cassandra's built-in TLS

Implement host-based security measures on Cassandra nodes

Always validate your network security configuration by attempting to connect to Cassandra

from unauthorized networks and verifying that the connection is rejected.

Auditing database activity is essential for security monitoring, compliance requirements,

and troubleshooting.

Cassandra provides an audit logging facility to track database operations:

Configure the SLF4J logger to direct audit logs to a separate file:

1# In cassandra.yaml
2audit_logging_options:
3  enabled: true
4  logger: SLF4JAuditWriter  # Default audit logger
5  # Exclude keyspaces from audit logging
6  excluded_keyspaces: system,system_traces,system_auth,system_distributed
7  # Exclude categories of operations
8  excluded_categories: QUERY,DML,DDL,DCL,OTHER
9  # Include specific users' activity
10  included_users: app_user,admin_user
11  # Exclude specific users from audit
12  excluded_users: monitoring_user

1<!-- In logback.xml -->
2<appender name="AUDIT-FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
3  <file>/var/log/cassandra/audit.log</file>
4  <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
5    <fileNamePattern>/var/log/cassandra/audit-%d{yyyy-MM-dd}.log</fileNamePattern>
6    <maxHistory>30</maxHistory>
7  </rollingPolicy>
8  <encoder>
9    <pattern>%date{ISO8601} %msg%n</pattern>
10  </encoder>
11</appender>
12
13<logger name="org.apache.cassandra.audit" level="INFO" additivity="false">
14  <appender-ref ref="AUDIT-FILE"/>
15</logger>

For more advanced audit requirements, you can implement a custom audit logger:

Configure the custom logger in cassandra.yaml:

1public class CustomAuditLogger implements IAuditLogger {
2    @Override
3    public void setup() {
4        // Initialize logger
5    }
6
7    @Override
8    public void audit(AuditLogEntry entry) {
9        // Log the entry, potentially to external system
10        System.out.printf("User %s performed %s on %s%n",
11                         entry.getUser(),
12                         entry.getOperation(),
13                         entry.getScope());
14    }
15
16    @Override
17    public void stop() {
18        // Cleanup
19    }
20}

1audit_logging_options:
2  enabled: true
3  logger: com.example.CustomAuditLogger

Different industries have specific compliance requirements for database systems:

: For systems handling payment card data

: For healthcare data

: For personal data of EU residents

: For financial reporting systems

: For California consumer privacy

Common compliance requirements for Cassandra include:

Authentication and access control

Encryption for sensitive data

Audit trails for database activity

Secure key management

Data retention and deletion policies

Regular security assessments

Collect audit logs in a central location

Implement log rotation and archiving

Protect audit logs from unauthorized access or modification

Set up alerting for suspicious activity

Establish log retention policies based on compliance requirements

Regularly review audit logs for security anomalies

Audit logging can generate significant amounts of data and impact performance. Carefully

configure what activities to audit and monitor the performance impact in your environment.

A robust backup and recovery strategy is essential for protecting data against loss and

ensuring business continuity.

Cassandra offers several approaches to data backup:

1. Snapshots

Snapshots create point-in-time backups by hard-linking SSTables:

Snapshots are stored in data directories alongside live data. You should copy them to external storage:

1. Incremental Backups

Incremental backups store copies of SSTables when they're flushed from memory:

Incremental backup files are stored in a backups directory alongside the data. You should regularly

copy these files to external storage.

1. CommitLog Backups

For point-in-time recovery, back up the commit logs:

1# Create a snapshot of all keyspaces
2nodetool snapshot
3
4# Create a snapshot of a specific keyspace
5nodetool snapshot keyspace_name
6
7# Create a snapshot with a custom name
8nodetool snapshot --tag daily_backup keyspace_name
9
10# List snapshots
11nodetool listsnapshots

1# Example script to copy snapshots
2for node in cassandra-node1 cassandra-node2 cassandra-node3; do
3  ssh $node "find /var/lib/cassandra/data -name snapshots -type d | tar czf /backup/snapshot-$(date +%Y%m%d).tar.gz -T -"
4  scp $node:/backup/snapshot-*.tar.gz /backup/repository/
5done

1# In cassandra.yaml
2incremental_backups: true

1# In cassandra.yaml
2commitlog_archiving:
3  enabled: true
4  archive_command: "cp %path /backup/commitlogs/%name"
5  restore_command: "cp /backup/commitlogs/%name %path"
6  restore_directories:
7  - /backup/commitlogs

Recovery options depend on the backup method and failure scenario:

1. Restoring from Snapshots
2. Point-in-Time Recovery

For more complex point-in-time recovery using commit logs:

1# Stop Cassandra
2systemctl stop cassandra
3
4# Clear data directory for the keyspace/table to restore
5rm -rf /var/lib/cassandra/data/keyspace_name/table_name-*/*
6
7# Copy snapshot files
8cp -r /var/lib/cassandra/data/keyspace_name/table_name-*/snapshots/snapshot_name/* /var/lib/cassandra/data/keyspace_name/table_name-*/
9
10# Start Cassandra
11systemctl start cassandra
12
13# Run repair to ensure consistency
14nodetool repair keyspace_name table_name

1# Stop Cassandra
2systemctl stop cassandra
3
4# Restore snapshot as above
5
6# Configure commitlog restore in cassandra.yaml
7# commitlog_archiving:
8#   restore_directories:
9#   - /backup/commitlogs/2023-07-15
10
11# Start Cassandra with point-in-time recovery
12systemctl start cassandra

Regularly verify backups to ensure they can be successfully restored:

Perform test restores to a separate cluster

Validate data integrity after restore

Document and time recovery procedures

Test different recovery scenarios

Use a combination of snapshot and incremental backups

Store backups on separate storage systems

Implement off-site backup storage

Automate backup procedures

Monitor backup completion and sizes

Implement backup retention policies

Encrypt backup data if it contains sensitive information

In a multi-datacenter Cassandra cluster, you can often simplify backup strategy by treating

the second datacenter as a backup. However, this doesn't protect against logical data corruption

or accidental deletion, so traditional backups are still necessary.

Comprehensive monitoring is essential for maintaining Cassandra's health and performance.

1. System-level Metrics

CPU utilization

Memory usage

Disk usage and I/O

Network throughput and latency

Open file descriptors

System load average

1. JVM Metrics

Heap usage

Garbage collection frequency and duration

Thread counts

Old generation collection time

1. Cassandra Metrics

Read/write latency

: p50, p95, p99 latencies

Compaction

: Pending tasks, bytes compacted

Cache hit rates

: Key cache, row cache

Thread pools

: Blocked tasks, active tasks

: Load, estimated partitions

: Hints in progress, failed deliveries

Client connections

: Connected native clients

1. Built-in Tools

Cassandra provides several command-line tools for monitoring:

1. JMX Monitoring

Cassandra exposes detailed metrics via JMX, which can be accessed using tools like JConsole,

JMXTerm, or through monitoring systems that support JMX.

1. Prometheus and Grafana

A popular monitoring setup for Cassandra involves:

Prometheus for metrics collection

JMX Exporter to expose Cassandra metrics to Prometheus

Grafana for visualization and dashboards

1# Check node and cluster status
2nodetool status
3
4# View table statistics
5nodetool tablestats keyspace_name
6
7# Check thread pool statistics
8nodetool tpstats
9
10# View compaction statistics
11nodetool compactionstats
12
13# View client connections
14nodetool clientstats
15
16# Check storage info
17nodetool info

1# prometheus.yml
2scrape_configs:
3  - job_name: 'cassandra'
4    static_configs:
5      - targets: ['cassandra-node1:9103', 'cassandra-node2:9103', 'cassandra-node3:9103']

Implement alerts for critical conditions that require attention:

1. System Alerts

High CPU usage (> 80% for extended periods)

Low disk space (< 30% free)

High load average (exceeding CPU count)

Memory pressure

1. JVM Alerts

High heap usage (> 75%)

Long GC pauses (> 500ms)

Frequent full GC events

1. Cassandra Alerts

Node down or unreachable

High read/write latencies (exceeding SLAs)

Growing compaction backlog

Blocked thread pools

Low cache hit rates

Read/write timeouts

Repair failures

1# Example Prometheus alert rule
2groups:
3- name: cassandra_alerts
4  rules:
5  - alert: CassandraNodeDown
6    expr: up{job="cassandra"} == 0
7    for: 5m
8    labels:
9      severity: critical
10    annotations:
11      summary: "Cassandra node down"
12      description: "Node {{ $labels.instance }} has been down for more than 5 minutes."
13
14  - alert: CassandraHighLatency
15    expr: quantile_over_time(0.99, cassandra_client_request_latency_seconds{operation="read"}[5m]) > 0.1
16    for: 10m
17    labels:
18      severity: warning
19    annotations:
20      summary: "High read latency"
21      description: "99th percentile read latency on {{ $labels.instance }} is above 100ms for 10 minutes."

Monitor all nodes in the cluster

Collect historical metrics for trend analysis

Set up dashboards for different aspects (system, JVM, Cassandra)

Implement multi-level alerting (warning, critical)

Monitor both cluster-wide and node-specific metrics

Include application-level metrics for end-to-end visibility

Regularly review and tune alert thresholds

Implement alert notification channels (email, Slack, PagerDuty)

When monitoring Cassandra, look for trends rather than isolated metrics. For example, a single

GC pause might not be concerning, but increasing GC frequency over time could indicate a memory leak.

Well-defined operational procedures help ensure smooth and consistent management of Cassandra clusters.

Expanding a cluster requires careful planning and execution:

1# 1. Install Cassandra on the new node and configure it:
2# - Set the same cluster name
3# - Configure the seed nodes
4# - Set the same snitch type
5# - Set appropriate auto_bootstrap: true
6
7# 2. Start Cassandra on the new node
8systemctl start cassandra
9
10# 3. Monitor streaming progress
11nodetool netstats
12
13# 4. Verify the node has joined the cluster
14nodetool status
15
16# 5. Run cleanup on all existing nodes after streaming completes
17nodetool cleanup

To safely remove a node from a cluster:

1# 1. Decommission the node to stream its data to other nodes
2nodetool decommission
3
4# 2. Monitor the decommission progress
5nodetool netstats
6
7# 3. After decommission completes, stop Cassandra
8systemctl stop cassandra
9
10# 4. Remove the node from the seeds list on other nodes
11# 5. Run cleanup on remaining nodes
12nodetool cleanup

When a node fails permanently:

1# 1. Install Cassandra on the replacement node with the same IP address if possible
2
3# 2. Configure cassandra.yaml:
4# - Set replace_address or replace_address_first_boot to the IP of the dead node
5# - Use replace_address_first_boot for a one-time replacement setting
6
7# 3. Start Cassandra on the replacement node
8systemctl start cassandra
9
10# 4. Monitor bootstrap progress
11nodetool netstats
12
13# 5. Verify the node has joined the cluster
14nodetool status

To restart each node in a cluster with minimal impact:

1# For each node in the cluster, one at a time:
2
3# 1. Disable client traffic (optional)
4nodetool disablegossip
5nodetool disablethrift  # If using Thrift
6nodetool disablebinary  # Disables native transport (CQL)
7
8# 2. Drain the node to flush data to disk
9nodetool drain
10
11# 3. Stop Cassandra
12systemctl stop cassandra
13
14# 4. Start Cassandra
15systemctl start cassandra
16
17# 5. Wait for the node to be fully up
18nodetool status
19
20# 6. Enable client traffic (if disabled earlier)
21nodetool enablegossip
22nodetool enablethrift  # If using Thrift
23nodetool enablebinary  # Enables native transport (CQL)
24
25# 7. Move to the next node and repeat

Repairs are essential for maintaining data consistency:

1# Full repair of all keyspaces
2nodetool repair
3
4# Repair a specific keyspace
5nodetool repair keyspace_name
6
7# Repair a specific table
8nodetool repair keyspace_name table_name
9
10# Sequential repair (less intensive)
11nodetool repair -seq
12
13# Incremental repair (only repair data since last repair)
14nodetool repair -inc
15
16# Primary range repair (only repair ranges this node is primary for)
17nodetool repair -pr
18
19# Subrange repair (repair specific token ranges)
20nodetool repair -st <start_token> -et <end_token>
21
22# Repair only local datacenter
23nodetool repair -dc local_datacenter_name

Schema changes should be managed carefully to maintain consistency:

1-- 1. Before making schema changes, check schema agreement
2-- nodetool describecluster
3
4-- 2. Make schema changes during low-traffic periods
5
6-- 3. Use lightweight transactions for safety
7CREATE TABLE IF NOT EXISTS keyspace_name.table_name (
8  id UUID PRIMARY KEY,
9  name TEXT,
10  value INT
11);
12
13-- 4. Verify schema propagation
14-- nodetool describecluster

Regular cleanup tasks help maintain performance:

1# Run cleanup to remove data that no longer belongs to a node
2nodetool cleanup
3
4# Remove snapshot data after backup verification
5nodetool clearsnapshot
6
7# Remove old commit logs
8nodetool cleanup

When a node fails temporarily:

Always test operational procedures in a staging environment before applying them to production.

Document each procedure with specific commands and expected outcomes.

1# 1. Check node status
2nodetool status
3
4# 2. Determine the cause of failure
5# - Check logs (/var/log/cassandra/system.log)
6# - Check system resources (disk, memory, etc.)
7# - Check network connectivity
8
9# 3. Address the underlying issue
10
11# 4. Start Cassandra
12systemctl start cassandra
13
14# 5. Monitor the node as it rejoins the cluster
15nodetool status
16nodetool netstats  # Check streaming progress
17
18# 6. Run repair after the node is fully up
19nodetool repair -pr

A comprehensive disaster recovery (DR) plan is essential for maintaining business continuity

in the event of major failures.

Plan for various disaster scenarios:

Single node failure

: Loss of one Cassandra node

Rack failure

: Loss of multiple nodes in the same rack

Datacenter failure

: Loss of an entire datacenter

Network partition

: Network failure isolating parts of the cluster

Corrupted data

: Data corruption due to bugs or human error

Complete cluster loss

: Catastrophic failure requiring full rebuild

A multi-datacenter deployment provides built-in disaster recovery capabilities:

1# cassandra.yaml configuration for multi-DC
2endpoint_snitch: GossipingPropertyFileSnitch
3
4# cassandra-rackdc.properties (for each DC)
5# DC1 nodes:
6# dc=dc1
7# rack=rack1
8
9# DC2 nodes:
10# dc=dc2
11# rack=rack1

1-- Create keyspace with multi-DC replication
2CREATE KEYSPACE application_keyspace WITH replication = {
3  'class': 'NetworkTopologyStrategy',
4  'dc1': 3,
5  'dc2': 3
6};

Define recovery objectives based on business requirements:

: Maximum acceptable time to restore service after a failure

: Maximum acceptable data loss measured in time

Examples of different tiers of DR strategy:

: Active-Active multi-DC (RTO: minutes, RPO: seconds)

: Active-Passive multi-DC (RTO: hours, RPO: minutes)

: Backup-based recovery (RTO: days, RPO: hours)

Create comprehensive DR documentation that includes:

Contact information for the DR team

Decision tree for determining response

Step-by-step procedures for each scenario

Infrastructure details and diagrams

Recovery validation procedures

Estimated recovery times

Regularly test disaster recovery procedures:

Table-top exercises: Discuss responses to simulated scenarios

Functional testing: Test individual components of the DR plan

Full simulation: Perform complete DR procedures in a controlled environment

Chaos engineering: Deliberately introduce failures to test resilience

Disaster recovery is not just a technical challenge but also an organizational one.

Ensure that all stakeholders understand the DR plan, and that there are clear roles

and responsibilities during a disaster scenario.

1# 1. Verify primary datacenter is down
2nodetool status
3
4# 2. Switch application traffic to secondary datacenter
5# (Update load balancers, DNS, etc.)
6
7# 3. Configure clients to use LOCAL_QUORUM consistency in the secondary DC
8
9# 4. Monitor cluster health and performance
10
11# 5. Begin recovery procedures for the primary datacenter
12
13# 6. After primary DC recovery:
14# - Verify replication is working
15# - Run repairs to synchronize data
16# - Gradually shift traffic back
17# - Restore original consistency levels

Regular upgrades and maintenance are necessary to keep Cassandra secure and performant.

Review release notes for new features and breaking changes

Test upgrades in a staging environment first

Create a detailed upgrade plan with rollback procedures

Schedule upgrades during low-traffic periods

Communicate the upgrade schedule to all stakeholders

Cassandra supports rolling upgrades, allowing nodes to be upgraded one at a time:

1# For each node in the cluster, one at a time:
2
3# 1. Disable client traffic
4nodetool disablegossip
5nodetool disablethrift
6nodetool disablebinary
7
8# 2. Drain the node
9nodetool drain
10
11# 3. Stop Cassandra
12systemctl stop cassandra
13
14# 4. Backup configuration files
15cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
16cp /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
17
18# 5. Update Cassandra packages
19apt-get update
20apt-get install cassandra=<new_version>  # For Debian/Ubuntu
21# or
22yum update cassandra-<new_version>  # For RHEL/CentOS
23
24# 6. Apply any needed configuration changes
25
26# 7. Start Cassandra with the new version
27systemctl start cassandra
28
29# 8. Wait for the node to fully initialize
30nodetool status
31
32# 9. Enable client traffic
33nodetool enablegossip
34nodetool enablethrift
35nodetool enablebinary
36
37# 10. Upgrade the next node only after this one is fully up

Learn about Post-Upgrade Tasks.

1# After all nodes are upgraded:
2
3# 1. Verify cluster health
4nodetool status
5
6# 2. Run nodetool upgradesstables to update SSTables to the new format
7nodetool upgradesstables
8
9# 3. Run a full repair
10nodetool repair -seq
11
12# 4. Test the application functionality
13
14# 5. Monitor for any issues related to the upgrade
15
16# 6. Update monitoring and alerting for new metrics or changed thresholds

Prepare for the possibility of needing to roll back an upgrade:

1# If issues are encountered after upgrading a node:
2
3# 1. Stop Cassandra
4systemctl stop cassandra
5
6# 2. Restore configuration files
7cp /etc/cassandra/cassandra.yaml.bak /etc/cassandra/cassandra.yaml
8cp /etc/cassandra/cassandra-env.sh.bak /etc/cassandra/cassandra-env.sh
9
10# 3. Downgrade Cassandra packages
11apt-get install cassandra=<previous_version>  # For Debian/Ubuntu
12# or
13yum downgrade cassandra-<previous_version>  # For RHEL/CentOS
14
15# 4. Start Cassandra
16systemctl start cassandra
17
18# 5. Verify the node rejoins the cluster
19nodetool status
20
21# 6. Enable client traffic
22nodetool enablegossip
23nodetool enablethrift
24nodetool enablebinary
25
26# 7. Revise upgrade plan based on issues encountered

Establish a schedule for routine maintenance tasks:

1. Daily Tasks

Check cluster status:

nodetool status

Monitor disk space usage

Review error logs

Verify backup completion

1. Weekly Tasks

Run incremental repairs

Cleanup old snapshots

Review performance metrics

Check for outdated SSTables:

nodetool tablestats

1. Monthly Tasks

Run full repairs

Review user accounts and permissions

Test backup restores

Validate capacity planning

Apply security patches

1. Quarterly Tasks

Review and update monitoring thresholds

Test disaster recovery procedures

Evaluate performance tuning options

Check for Cassandra version updates

Audit schema design

Establish regular maintenance windows for tasks that may impact service:

Schedule during periods of lowest traffic

Communicate maintenance windows to all stakeholders

Define the scope of each maintenance window

Prepare rollback procedures for all maintenance activities

Document all changes made during maintenance

Never upgrade or apply major changes to all nodes at once. Always use a rolling approach

that maintains cluster availability, and verify each step before proceeding to the next node.

Operating Cassandra in production requires attention to security and operational excellence.

In this lesson, we covered:

Authentication mechanisms to verify client identities

Role-based access control for limiting data access

Encryption options for protecting data in transit and at rest

Network security measures to prevent unauthorized access

Auditing to track database activities

Backup and recovery strategies to protect against data loss

Monitoring and alerting best practices

Common operational procedures for managing the cluster

Disaster recovery planning for major failures

Upgrade and maintenance strategies

By implementing these security measures and operational practices, you can ensure that your

Cassandra clusters remain secure, reliable, and performant in production environments.

Remember that security and operations are ongoing processes, not one-time tasks. Regularly

review and update your security configurations and operational procedures to address new

threats and changing requirements.