Learn how to evaluate and select the right NoSQL database for your specific use case by understanding different types, analyzing requirements, and following a structured decision framework.
Learn how to evaluate and select the right NoSQL database for your specific use case by understanding different types, analyzing requirements, and following a structured decision framework.
With the proliferation of NoSQL database options, selecting the right database for your specific
use case has become increasingly complex. Different NoSQL databases are optimized for different
workloads, data models, and operational requirements. Making the right choice can significantly
impact your application's performance, scalability, and development efficiency.
There is no one-size-fits-all NoSQL database. The best choice depends on your specific data
characteristics, access patterns, consistency requirements, and operational constraints.
In this lesson, we'll explore a structured approach to NoSQL database selection, covering:
Review of different NoSQL database types and their strengths
Framework for gathering and analyzing requirements
Decision criteria for database selection
Evaluation methodology and proof-of-concept approaches
Common use case patterns and suitable database matches
Polyglot persistence strategies
Migration considerations and approaches
Let's start by reviewing the major types of NoSQL databases and their respective strengths
and weaknesses. Understanding these fundamental differences is crucial for making an informed
selection.
Document databases store data in flexible, semi-structured documents (typically JSON or BSON).
Examples include MongoDB, Couchbase, and Amazon DocumentDB.
Strengths:
Flexible schema that can evolve over time
Intuitive data model matching object-oriented programming
Support for complex nested structures
Good for content management, catalogs, user profiles
Generally good query capabilities with indexing support
Horizontal scaling through sharding
Weaknesses:
Atomicity limited to single document operations (in most implementations)
Join operations can be inefficient
Not ideal for highly interconnected data
May require denormalization, leading to data duplication
Document size limitations (e.g., 16MB in MongoDB)
Key-value stores are the simplest form of NoSQL databases, storing data as pairs of keys and
values. Examples include Redis, Amazon DynamoDB, and Riak.
Strengths:
Simple data model with very fast read/write operations
Highly scalable with excellent performance
Ideal for caching, session storage, user preferences
Low latency for high-throughput applications
Often support in-memory storage for maximum speed
Weaknesses:
Limited query capabilities (primarily key-based access)
No built-in support for relationships between data items
No schema enforcement or data validation
Not suitable for complex data structures (without serialization)
Often limited secondary indexing capabilities
Wide-column stores organize data in tables with rows and columns, but with flexible schema per row.
Examples include Apache Cassandra, ScyllaDB, and HBase.
Strengths:
Highly scalable and designed for massive datasets
Excellent write throughput
Linear scalability with high availability
Good for time-series data, IoT, and event logging
Optimized for specific query patterns defined by partition key
Tunable consistency levels (in Cassandra)
Weaknesses:
Query patterns must be known in advance for effective modeling
Limited support for ad-hoc queries
Not ideal for highly interconnected data
Data modeling complexity and learning curve
Typically eventual consistency by default
Joins and aggregations are challenging
Graph databases specialize in managing highly connected data with nodes, edges, and properties.
Examples include Neo4j, Amazon Neptune, and JanusGraph.
Strengths:
Native support for relationship-rich data
Efficient traversal of connected data
Intuitive modeling of networks and hierarchies
Excellent for social networks, recommendations, fraud detection
Powerful query languages for graph traversal
Ability to discover patterns and hidden relationships
Weaknesses:
Higher learning curve for graph data modeling and query languages
Typically more complex to scale horizontally than other NoSQL types
Often more resource-intensive
Not ideal for simple key-value or document data
Less standardization across implementations
While primarily for search, many search engines now function as document-oriented databases with
rich indexing and query capabilities. Examples include Elasticsearch and Solr.
Strengths:
Full-text search capabilities
Advanced text analysis (stemming, synonyms, etc.)
Fast retrieval of documents based on complex criteria
Support for faceting and aggregations
Geospatial search capabilities
Near real-time indexing and search
Weaknesses:
Eventually consistent by design
Not ideal as a primary transactional database
Higher storage overhead for inverted indices
More complex to maintain for high availability
Can be resource-intensive, particularly for memory
Time-series databases are optimized for data points indexed by time. Examples include InfluxDB,
TimescaleDB, and Amazon Timestream.
Strengths:
Optimized for time-based data ingestion and retrieval
Efficient storage with time-based compression
Built-in time-based aggregation functions
Retention policies for managing old data
Ideal for metrics, monitoring, and IoT data
Often include specialized visualization tools
Weaknesses:
Limited flexibility for non-time-series data
Not suited for complex relationships between entities
Some implementations have limited query capabilities
May have scaling limitations compared to distributed NoSQL systems
Often specialized for specific time-series use cases
Modern database systems increasingly blend features from different categories. For example,
MongoDB has graph-like capabilities with $graphLookup, and Couchbase provides both document
and key-value access patterns. Consider the overall feature set rather than just the category.
The foundation of a good database selection process is a thorough understanding of your requirements.
Let's explore the key areas to consider:
Structure complexity
: How complex is your data? Does it have a rigid structure or does it vary?
Relationships
: How connected is your data? Are there many relationships between entities?
Schema flexibility
: Will your schema evolve frequently? Do different items have different attributes?
: What is the size of individual data items? Are they small or large documents?
Hierarchical nature
: Does your data have natural nesting or parent-child relationships?
Read vs write ratio
: Is your application read-heavy, write-heavy, or balanced?
Query patterns
: What are your most common queries? Are they simple lookups or complex queries?
Read consistency
: How fresh must the data be when read? Can you tolerate eventual consistency?
Access frequency
: How often is data accessed? Are there hot spots in your data?
Sorting requirements
: Do you need to retrieve data in a specific order?
Aggregation needs
: Do you need to perform calculations across multiple records?
Throughput
: How many operations per second does the system need to handle?
: What are the response time requirements for different operations?
Concurrent users
: How many simultaneous users will access the database?
Scaling projections
: How will data and traffic grow over time?
Resource constraints
: Are there limitations on hardware or infrastructure?
Availability requirements
: What is the required uptime? Can you tolerate any downtime?
Consistency needs
: What level of data consistency is required? ACID or BASE?
Backup and recovery
: What are the requirements for disaster recovery and data protection?
Geographic distribution
: Do you need multi-region or multi-datacenter deployment?
Monitoring and management
: What operational visibility and control is needed?
Security requirements
: What authentication, authorization, and encryption needs exist?
Team expertise
: What databases is your team already familiar with?
Ecosystem and tools
: What integration requirements exist with other systems?
Programming language support
: Are there good client libraries for your languages?
Development speed
: How quickly do you need to build and iterate?
Community and support
: What level of community activity and vendor support exists?
Cost constraints
: What is your budget for licensing, infrastructure, and operations?
Licensing preferences
: Do you prefer open-source or commercial solutions?
Vendor lock-in concerns
: How important is database portability?
Long-term viability
: Will the database be supported and developed for years to come?
Cloud vs on-premises
: Where will the database be deployed?
Don't rush the requirements analysis phase. Many database selection mistakes stem from
misunderstanding the true nature of data, access patterns, or operational needs. Involve
stakeholders from development, operations, and business to get a comprehensive view.
1// Example of requirement gathering structure
2const databaseRequirements = {
3 dataModel: {
4 structure: "Semi-structured JSON documents with varying fields",
5 relationships: "Moderate - some references between entities",
6 schemaFlexibility: "High - schema evolves frequently",
7 dataSizes: "Small to medium documents (1KB-100KB)"
8 },
9 accessPatterns: {
10 readWriteRatio: "80% reads, 20% writes",
11 queryPatterns: [
12 "Lookup by ID (very frequent)",
13 "Filter by 2-3 properties (frequent)",
14 "Text search on content (occasional)",
15 "Aggregations for analytics (infrequent, batch)"
16 ],
17 consistencyNeeds: "Eventual consistency acceptable for most operations"
18 },
19 performance: {
20 peakThroughput: "5,000 reads/second, 1,000 writes/second",
21 maxLatency: "Read: 50ms, Write: 100ms",
22 concurrentUsers: "Up to 10,000",
23 growthProjection: "Data doubling yearly, traffic increasing 50% yearly"
24 },
25 operational: {
26 availability: "99.95% uptime required",
27 backupStrategy: "Point-in-time recovery, max 1 hour data loss acceptable",
28 geographicDistribution: "Multi-region deployment across US and Europe"
29 }
30};Based on your requirements analysis, you can use a structured framework to narrow down the database options.
Here's a step-by-step approach:
Begin by listing non-negotiable requirements that will immediately eliminate certain databases.
These could include:
Specific consistency guarantees (e.g., strong consistency)
Required query capabilities (e.g., full-text search)
Critical operational features (e.g., multi-region replication)
Essential scaling capabilities (e.g., horizontal scaling)
Security requirements (e.g., field-level encryption)
Based on your data model analysis, determine which database types are most suitable:
If your data is...
Consider...
Semi-structured with varying attributes
Document databases (MongoDB, Couchbase)
Simple objects with primary key access
Key-value stores (Redis, DynamoDB)
Time-series data with timestamps
Time-series databases (InfluxDB, TimeScaleDB)
Highly interconnected with relationships
Graph databases (Neo4j, Neptune)
Large datasets with predictable queries
Wide-column stores (Cassandra, HBase)
Text-heavy with search requirements
Search engines (Elasticsearch, Solr)
Evaluate how well different databases support your access patterns:
If you need...
Consider...
High write throughput
Cassandra, ScyllaDB, DynamoDB
Low-latency reads
Redis, Memcached (caching); MongoDB with indexing
Complex querying and indexing
MongoDB, Couchbase
Path/traversal queries
Neo4j, JanusGraph
Full-text search
Elasticsearch, Solr, MongoDB with Atlas Search
Time-based aggregations
InfluxDB, TimescaleDB, Cassandra (with limitations)
Consider how well different databases meet your operational requirements:
If you require...
Consider...
High availability (99.99%+)
Cassandra, MongoDB (replica sets), AWS DynamoDB
Strong consistency
MongoDB, CockroachDB, Redis
Multi-region deployment
Cassandra, CockroachDB, MongoDB Atlas, DynamoDB Global Tables
Easy scaling
DynamoDB, Cassandra, MongoDB sharding
Simple operations
Managed services: MongoDB Atlas, DynamoDB, Azure Cosmos DB
Finally, evaluate the practical aspects of each remaining option:
Team expertise
: Does your team have experience with this database?
Ecosystem maturity
: Are there good tools, libraries, and resources available?
Cost implications
: What are the licensing, infrastructure, and operational costs?
Community and support
: Is there active community support or professional support available?
Future roadmap
: Does the database's development direction align with your needs?
Create a scoring matrix to objectively evaluate databases against your criteria. Weight
criteria according to their importance to your application. This approach can help reduce
bias and make the decision more transparent to stakeholders.
After narrowing down your options, it's essential to test the most promising candidates
with a proof of concept (POC). Here's how to structure this evaluation phase:
Identify critical scenarios
: Select 3-5 key data access patterns or operations that are most important to your application.
Create realistic data
: Generate a representative dataset with realistic volume and characteristics.
Define clear metrics
: Establish measurable criteria for performance, ease of development, etc.
Design tests
: Create tests that simulate your real-world workload and access patterns.
Plan for operational testing
: Include tests for scaling, failover, and recovery scenarios.
For each database, explore how you would model your data:
Create a sample schema or data model for your core entities
Evaluate how well the model supports your access patterns
Consider implications for data consistency, duplication, and integrity
Assess flexibility for future changes and extensions
1// Document model example (MongoDB)
2const productSchema = {
3 _id: ObjectId(),
4 name: "Ergonomic Chair",
5 category: "Furniture",
6 price: 299.99,
7 attributes: {
8 color: "Black",
9 material: "Mesh",
10 adjustable: true
11 },
12 inventory: {
13 inStock: 34,
14 warehouses: [
15 { location: "East", quantity: 20 },
16 { location: "West", quantity: 14 }
17 ]
18 },
19 tags: ["office", "ergonomic", "chair"],
20 createdAt: ISODate("2023-07-15")
21}
22
23// Key-value model example (Redis)
24SET product:1234 "{\"name\":\"Ergonomic Chair\",\"price\":299.99,\"inStock\":34}"
25HSET product:1234:attributes color "Black" material "Mesh" adjustable "true"
26SADD product:1234:tags "office" "ergonomic" "chair"
27
28// Wide-column model example (Cassandra)
29CREATE TABLE products (
30 product_id uuid,
31 name text,
32 category text,
33 price decimal,
34 color text,
35 material text,
36 adjustable boolean,
37 in_stock int,
38 created_at timestamp,
39 PRIMARY KEY (product_id)
40);
41
42CREATE TABLE products_by_category (
43 category text,
44 product_id uuid,
45 name text,
46 price decimal,
47 PRIMARY KEY (category, product_id)
48);
49
50// Graph model example (Neo4j)
51CREATE (p:Product {id: "1234", name: "Ergonomic Chair", price: 299.99})
52CREATE (c:Category {name: "Furniture"})
53CREATE (t1:Tag {name: "office"})
54CREATE (t2:Tag {name: "ergonomic"})
55CREATE (t3:Tag {name: "chair"})
56CREATE (w1:Warehouse {location: "East", quantity: 20})
57CREATE (w2:Warehouse {location: "West", quantity: 14})
58CREATE (p)-[:BELONGS_TO]->(c)
59CREATE (p)-[:HAS_TAG]->(t1)
60CREATE (p)-[:HAS_TAG]->(t2)
61CREATE (p)-[:HAS_TAG]->(t3)
62CREATE (p)-[:STOCKED_AT]->(w1)
63CREATE (p)-[:STOCKED_AT]->(w2)Conduct performance testing to verify the database meets your requirements:
Test read performance for your key query patterns
Test write performance for your main write operations
Measure latency at different levels of concurrency
Test scaling behavior as data volume or traffic increases
Evaluate performance under realistic load conditions
Assess the operational characteristics of each database:
Test backup and restore procedures
Simulate node failures to observe failover behavior
Test horizontal scaling by adding nodes
Evaluate monitoring and management capabilities
Test schema changes and data migration procedures
Evaluate the developer experience with each database:
Assess the quality of client libraries for your programming languages
Measure development time for implementing key features
Evaluate query language expressiveness and readability
Test debugging and troubleshooting capabilities
Gather feedback from developers on ease of use
Create a comprehensive evaluation matrix to compare databases:
Database A
Database B
Database C
Data model fit
Query performance
Scalability
Operational ease
Developer experience
Weighted Score
Don't skip the POC phase, even if you have a strong preference based on your initial analysis.
Real-world behavior can differ significantly from theoretical expectations, and a POC can reveal
unexpected strengths or limitations.
Certain use cases tend to align well with specific database types. Here are some common patterns:
Content management systems typically involve structured content with metadata, versioning, and search capabilities.
Primary option
: Document databases (MongoDB, Couchbase)
: Flexible schema for varied content types, good query capabilities, indexing
Considerations
: Add search engine (Elasticsearch) for advanced text search
Example pattern
: Store content as documents with nested metadata, use text indexes for search
Applications that store user profiles, settings, and preferences need flexible storage with fast retrieval by user ID.
Primary options
: Document databases, Key-value stores
: Simple lookup by user ID, flexible schema for varying preferences
Considerations
: Cache frequently accessed profiles in Redis for performance
Example pattern
: Store complete profile documents by user ID, with secondary indexes for searches
Product catalogs typically have products with various attributes, categories, and search requirements.
Primary options
: Document databases, Search engines
: Flexible schema for different product types, rich indexing, faceted search
Considerations
: Combine MongoDB with Elasticsearch for advanced search features
Example pattern
: Store products as documents with nested attributes, use text and geospatial indexing
Web applications need to store session data with quick access and automatic expiration.
Primary option
: Key-value stores (Redis, Memcached)
: Ultra-fast access by session ID, built-in TTL for expiration
Considerations
: Consider persistence options based on session importance
Example pattern
: Store serialized session data with session ID as key and appropriate TTL
Social networks involve complex relationships between users, content, and activities.
Primary options
: Graph databases, combination of document and graph
: Native support for relationship queries, efficient path traversal
Considerations
: Use document database for content, graph for relationships
Example pattern
: Store user and content nodes with relationship edges for follows, likes, etc.
IoT applications generate time-series data from many devices that need efficient storage and analysis.
Primary options
: Time-series databases, Wide-column stores
: Optimized for time-ordered data, efficient storage, time-based queries
Considerations
: Consider data retention policies and downsampling for historical data
Example pattern
: Store measurements with device ID and timestamp, with automatic retention policies
Applications that need to perform analytics on streaming or rapidly changing data.
Primary options
: Wide-column stores, Time-series databases
: High write throughput, time-based aggregations, scalability
Considerations
: Pre-aggregate data when possible to optimize query performance
Example pattern
: Use wide-column store with time bucketing in row keys, counter columns for metrics
Systems that generate personalized recommendations based on relationships between users and items.
Primary options
: Graph databases, Combination approaches
: Relationship traversal, pattern matching, similarity detection
Considerations
: Combine with document or relational storage for item details
Example pattern
: Graph model with user and item nodes, weighted relationship edges for preferences
E-commerce platforms have complex data needs spanning products, inventory, orders, and customers.
Primary options
: Multiple databases (polyglot persistence)
: Optimize each data domain with appropriate storage
Considerations
: Increased complexity of managing multiple databases
Example pattern
: Document store for products, relational for orders, key-value for sessions, search engine for product search
Don't feel restricted to a single database type. Many applications benefit from a polyglot
persistence approach, using different databases optimized for different parts of the application.
Polyglot persistence refers to using multiple database technologies within a single application,
each chosen to best match specific data storage and retrieval needs.
Optimize each data domain with the most suitable storage technology
Better performance for specific access patterns
Greater flexibility to evolve different parts of the system
Ability to adopt new database technologies incrementally
Avoid compromising on key requirements for any data domain
Increased operational complexity with multiple technologies to manage
Data consistency challenges across different databases
Higher learning curve for the development team
More complex backup, recovery, and monitoring processes
Potential for increased infrastructure costs
Divide your application into bounded contexts and use different databases for each domain:
Separate read and write operations, potentially using different databases:
Use fast in-memory databases in front of persistent storage:
1// Example architecture with domain-driven database separation
2
3// User domain - Document database (MongoDB)
4const userService = {
5 getUserProfile: async (userId) => {
6 // Query MongoDB for user profile
7 return await userCollection.findOne({ _id: userId });
8 }
9};
10
11// Product catalog domain - Search engine (Elasticsearch)
12const productService = {
13 searchProducts: async (query, filters) => {
14 // Query Elasticsearch for products
15 return await elasticClient.search({
16 index: 'products',
17 body: { query: { match: query }, ...filters }
18 });
19 }
20};
21
22// Order processing domain - Relational database (PostgreSQL)
23const orderService = {
24 createOrder: async (order) => {
25 // Use transaction in PostgreSQL for order creation
26 return await db.transaction(async (trx) => {
27 // Create order and line items atomically
28 });
29 }
30};
31
32// Session management - Key-value store (Redis)
33const sessionService = {
34 getSession: async (sessionId) => {
35 // Get session from Redis
36 return await redisClient.get(\`session:\${sessionId}\`);
37 }
38};1// Example of CQRS with different databases for reads and writes
2
3// Write operations go to primary database (MongoDB)
4const productCommandService = {
5 createProduct: async (product) => {
6 // Write to MongoDB
7 return await productCollection.insertOne(product);
8 },
9 updateProduct: async (productId, updates) => {
10 // Write to MongoDB
11 return await productCollection.updateOne(
12 { _id: productId },
13 { $set: updates }
14 );
15 }
16};
17
18// Read operations come from read-optimized database (Elasticsearch)
19const productQueryService = {
20 findProducts: async (criteria) => {
21 // Read from Elasticsearch
22 return await elasticClient.search({
23 index: 'products',
24 body: { query: criteria }
25 });
26 },
27 getProductDetails: async (productId) => {
28 // Read from Elasticsearch
29 return await elasticClient.get({
30 index: 'products',
31 id: productId
32 });
33 }
34};
35
36// Synchronization process ensures data is replicated
37const syncService = {
38 syncToReadDatabase: async (changes) => {
39 // Update Elasticsearch based on MongoDB changes
40 // Often implemented via change streams or event sourcing
41 }
42};1// Example of layered caching with Redis in front of MongoDB
2
3const userService = {
4 getUserProfile: async (userId) => {
5 const cacheKey = \`user:\${userId}\`;
6
7 // Try cache first
8 const cachedUser = await redisClient.get(cacheKey);
9 if (cachedUser) {
10 return JSON.parse(cachedUser);
11 }
12
13 // Cache miss, get from primary database
14 const user = await userCollection.findOne({ _id: userId });
15
16 // Update cache for future requests
17 if (user) {
18 await redisClient.set(cacheKey, JSON.stringify(user), 'EX', 3600); // 1 hour TTL
19 }
20
21 return user;
22 },
23
24 updateUserProfile: async (userId, updates) => {
25 // Update primary database
26 const result = await userCollection.updateOne(
27 { _id: userId },
28 { $set: updates }
29 );
30
31 // Invalidate cache
32 await redisClient.del(\`user:\${userId}\`);
33
34 return result;
35 }
36};When using multiple databases, you need strategies to keep data synchronized:
Event-based synchronization
: Publish events when data changes and have consumers update other databases
Change data capture (CDC)
: Monitor database logs for changes and propagate to other systems
Batch synchronization
: Run periodic jobs to sync data between databases
Dual writes
: Update multiple databases within the application code (beware of partial failures)
API-based synchronization
: Use service APIs to ensure all changes go through a consistent interface
Be cautious when adopting polyglot persistence. Start with the minimum number of databases
needed to meet your requirements, and only add complexity when the benefits clearly outweigh
the operational costs.
1// Example of event-based synchronization
2
3// When data changes in the primary database, emit an event
4async function updateProduct(productId, updates) {
5 // Update primary database
6 await productCollection.updateOne(
7 { _id: productId },
8 { $set: updates }
9 );
10
11 // Publish event for other systems to consume
12 await eventBus.publish('product.updated', {
13 productId,
14 updates,
15 timestamp: new Date()
16 });
17}
18
19// Consumers listen for events and update their databases
20async function handleProductUpdated(event) {
21 // Update search index
22 await elasticClient.update({
23 index: 'products',
24 id: event.productId,
25 body: { doc: event.updates }
26 });
27
28 // Update cache
29 const cacheKey = \`product:\${event.productId}\`;
30 await redisClient.del(cacheKey);
31
32 // Update analytics database
33 await analyticsDb.execute(
34 'UPDATE product_metrics SET last_updated = ? WHERE product_id = ?',
35 [event.timestamp, event.productId]
36 );
37}When migrating from an existing database to a new NoSQL solution, consider these approaches
and best practices:
Replace the entire system at once:
: Complete data migration, switch all traffic to new system
: Faster completion, no need for long-term coexistence
: Higher risk, more complex rollback, potential extended downtime
: Smaller systems or systems where phased approach isn't practical
Move functionality in stages:
: Migrate one domain or feature at a time
: Lower risk, easier to troubleshoot, smaller impact per phase
: Longer timeline, needs dual-write during transition
: Larger systems that can be logically divided into domains
Gradually replace parts of the application:
: Build new functionality on new database, redirect traffic incrementally
: Very low risk, allows incremental value delivery
: Longest timeline, requires façade/proxy layer
: Critical systems that cannot afford downtime or risk
Split read and write traffic between old and new systems:
: Direct writes to old system, replicate to new, gradually move reads
: Allows validation of new system under real read traffic
: Complexity of replication, potential consistency issues
: Systems with significantly higher read than write traffic
Export data from the old system and import to the new one during a maintenance window:
Modify the application to write to both old and new databases, then gradually shift reads:
Use database logs or triggers to capture changes and apply them to the new system:
Then use a separate process to consume the change log and update the NoSQL database:
1# Example of offline migration process
2
3# 1. Export data from source database
4mysqldump --opt my_database > my_database.sql
5
6# 2. Process and transform the data
7python transform_data.py my_database.sql > nosql_format.json
8
9# 3. Import into target NoSQL database
10mongoimport --db new_database --collection users --file users.json
11mongoimport --db new_database --collection products --file products.json1// Example of dual-write approach
2
3async function createUser(userData) {
4 // Write to original database
5 const oldDbResult = await oldDb.query(
6 'INSERT INTO users (name, email, created_at) VALUES (?, ?, ?)',
7 [userData.name, userData.email, new Date()]
8 );
9
10 // Also write to new NoSQL database
11 const newDbResult = await userCollection.insertOne({
12 name: userData.name,
13 email: userData.email,
14 created_at: new Date(),
15 oldSystemId: oldDbResult.insertId // Keep reference to old system
16 });
17
18 return {
19 oldId: oldDbResult.insertId,
20 newId: newDbResult.insertedId
21 };
22}
23
24async function getUser(userId, useNewSystem = false) {
25 if (useNewSystem) {
26 // Try to get from new system
27 const user = await userCollection.findOne({ oldSystemId: userId });
28 if (user) return user;
29 }
30
31 // Fall back to old system
32 const [rows] = await oldDb.query('SELECT * FROM users WHERE id = ?', [userId]);
33 return rows[0];
34}1-- Example using database triggers for CDC
2
3-- Create a change log table
4CREATE TABLE change_log (
5 id INT AUTO_INCREMENT PRIMARY KEY,
6 table_name VARCHAR(255),
7 operation VARCHAR(10),
8 record_id INT,
9 data JSON,
10 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
11);
12
13-- Create a trigger for insert operations
14DELIMITER $$
15CREATE TRIGGER users_after_insert
16AFTER INSERT ON users
17FOR EACH ROW
18BEGIN
19 INSERT INTO change_log (table_name, operation, record_id, data)
20 VALUES ('users', 'INSERT', NEW.id, JSON_OBJECT(
21 'name', NEW.name,
22 'email', NEW.email,
23 'created_at', NEW.created_at
24 ));
25END$$
26DELIMITER ;
27
28-- Similar triggers for UPDATE and DELETE1// Process to consume change log and update NoSQL
2async function processChangeLogs() {
3 // Get unprocessed changes
4 const [rows] = await oldDb.query(
5 'SELECT * FROM change_log WHERE processed = 0 ORDER BY id ASC LIMIT 100'
6 );
7
8 for (const change of rows) {
9 try {
10 switch (change.table_name) {
11 case 'users':
12 await handleUserChange(change);
13 break;
14 case 'products':
15 await handleProductChange(change);
16 break;
17 // Other tables...
18 }
19
20 // Mark as processed
21 await oldDb.query('UPDATE change_log SET processed = 1 WHERE id = ?', [change.id]);
22 } catch (error) {
23 console.error(\`Error processing change \${change.id}:\`, error);
24 // Implement retry logic or alert
25 }
26 }
27}
28
29async function handleUserChange(change) {
30 const { operation, record_id, data } = change;
31
32 switch (operation) {
33 case 'INSERT':
34 case 'UPDATE':
35 await userCollection.updateOne(
36 { oldSystemId: record_id },
37 { $set: { ...data, oldSystemId: record_id, last_updated: new Date() } },
38 { upsert: true }
39 );
40 break;
41 case 'DELETE':
42 await userCollection.deleteOne({ oldSystemId: record_id });
43 break;
44 }
45}Thorough testing is critical for successful migrations:
Data integrity validation
: Verify that all data is correctly migrated
Performance testing
: Ensure the new system meets performance requirements
Functional testing
: Confirm all application features work with the new database
Operational testing
: Test backup/restore, scaling, and failure scenarios
Shadow testing
: Run operations on both systems and compare results
Always have a rollback plan in case issues arise:
Document detailed rollback procedures for each migration phase
Practice rollback procedures before the actual migration
Establish clear criteria for when to trigger a rollback
Consider keeping the old system running in parallel for a period
Set up monitoring to quickly detect issues that might require rollback
Data migrations are complex and risky. Always perform thorough testing in staging environments
that closely mirror production, and plan for more time than you think you'll need. Having
a solid rollback plan is essential.
Selecting the right NoSQL database is a crucial decision that significantly impacts your
application's performance, scalability, and development efficiency. In this lesson, we've
covered:
Different types of NoSQL databases and their respective strengths and weaknesses
A structured approach to gathering and analyzing requirements
A framework for evaluating and selecting databases
Proof of concept approaches and evaluation methodologies
Common use case patterns and their database matches
Polyglot persistence strategies for using multiple databases
Migration considerations and approaches
Remember these key principles when selecting a NoSQL database:
No one-size-fits-all
: Different databases excel at different things
Start with data access patterns
: How you'll access the data is often more important than how you'll store it
Consider the entire lifecycle
: Development, deployment, operations, and maintenance
Validate with real-world testing
: Conduct thorough POCs with realistic workloads
Be pragmatic
: Balance technical ideals with practical considerations like team expertise and support
With a thoughtful selection process, you can choose a NoSQL database that not only meets your
current requirements but also supports your application's growth and evolution for years to come.
Database technology is constantly evolving. Stay informed about new features and capabilities
in the databases you're considering, and be prepared to reassess your choices as your
application and requirements evolve.
Your progress is saved in this browser, on this device only.