Database sharding distributes data across multiple database instances, enabling horizontal scaling beyond single-machine limits.
What is Sharding?
Sharding partitions data across multiple databases based on a shard key. Each shard holds a subset of data.
Sharding Strategies
Hash-Based Sharding
typescript
function getShard(key: string, numShards: number): number {
const hash = crypto.createHash('md5').update(key).digest('hex');
const hashInt = parseInt(hash.substring(0, 8), 16);
return hashInt % numShards;
}
// Route queries to appropriate shard
const shardId = getShard(userId, 4);
const shard = shards[shardId];
const user = await db.query('SELECT * FROM users WHERE id = ?', [userId], shard);Range-Based Sharding
typescript
// Shard by user ID ranges
const shards = [
{ range: [0, 1000000], db: 'shard1' },
{ range: [1000001, 2000000], db: 'shard2' },
{ range: [2000001, 3000000], db: 'shard3' }
];
function getShardForUser(userId: number) {
return shards.find(s =>
userId >= s.range[0] && userId <= s.range[1]
);
}Directory-Based Sharding
typescript
// Lookup table in separate directory database
class ShardDirectory {
async getShardForKey(key: string): Promise<string> {
const result = await this.directoryDb.query(
'SELECT shard_id FROM shard_mapping WHERE key = ?',
[key]
);
return result.shard_id;
}
async assignShard(key: string, shardId: string) {
await this.directoryDb.query(
'INSERT INTO shard_mapping (key, shard_id) VALUES (?, ?)',
[key, shardId]
);
}
}Implementation Considerations
Cross-Shard Queries
typescript
// Query multiple shards in parallel
async function queryAllShards(query: string) {
const promises = shards.map(shard =>
shard.query(query)
);
const results = await Promise.all(promises);
return results.flat();
}Resharding
typescript
// Move data between shards
async function reshard(oldShard: Database, newShard: Database, keyRange: Range) {
// Copy data
const data = await oldShard.query(
'SELECT * FROM data WHERE id BETWEEN ? AND ?',
[keyRange.start, keyRange.end]
);
await newShard.batchInsert(data);
// Update directory
await directory.updateShardMapping(keyRange, newShard.id);
// Verify and delete from old shard
await oldShard.delete('id BETWEEN ? AND ?', [keyRange.start, keyRange.end]);
}Challenges
- Complex Queries: Cross-shard queries are difficult
- Resharding: Moving data between shards is complex
- Consistency: Maintaining ACID across shards
- Operations: Managing multiple database instances
Best Practices
- 1Choose shard keys carefully
- 2Plan for resharding from the start
- 3Implement proper monitoring
- 4Use connection pooling per shard
- 5Consider managed sharding solutions
Tools
- Vitess: MySQL sharding solution
- Citus: PostgreSQL extension
- MongoDB Sharding: Built-in sharding
- Amazon DynamoDB: Managed sharding
Conclusion
Sharding enables horizontal database scaling but adds complexity. Implement carefully with proper planning and monitoring.