The concept of database sharding has gained popularity over the past several years due to the enormous growth in transaction volume and size of business-application databases. This is particularly true for many successful online service vendors, software-as-a-service companies and social networking websites.
Database sharding can be simply defined as a "shared-nothing" partitioning scheme for large databases across a number of servers, enabling new levels of database performance and scalability. If you think of broken glass, you can get the concept of sharding-breaking your database down into smaller chunks called "shards" and spreading them across a number of distributed servers.
The term "sharding" was coined by Google and popularized through its publication of the "Big Table" architecture. However, the concept of "shared-nothing" database partitioning has been around for more than a decade and there have been many implementations during this period from in-house solutions to commercial database management tools.
What Drives the Need for Database Sharding?
Database sharding is a highly scalable approach for improving the throughput and overall performance of high-transaction, large, database-centric business applications. Since the inception of the relational database, application engineers and architects have required ever-increasing performance and capacity due to the simple observation that business databases generally grow in size over time. Adding to this trend is the extreme expansion of business data due to the evolution of the Internet economy, the Information Age and the prevalence of high-volume electronic commerce.
The business reasons for database sharding are also driven by cost. With today's stressful economy, organizations need to reduce infrastructure costs, while improving performance and scalability to remain competitive. At first glance, it may seem that "sharding" a single database across a number of independent servers could actually increase costs, but given a correct implementation, the financial benefits are significant.
As any experienced database administrator or application developer knows all too well, it is axiomatic that as the size and transaction volume of the database tier incurs linear growth, response times tend to grow logarithmically.
The reasons for the performance and scalability challenges are inherent to the fundamental design of the database management systems themselves. Databases rely heavily on the primary three components of any computer: CPU, memory and disk.
Each of these elements on a single server can only scale to a given point-after that, you need to take additional measures to improve performance. While it is common knowledge that disk I/O is the primary bottleneck, as database management systems have improved they also continue to take greater advantage of CPU and memory. In fact, we have observed that obtaining the correct balance of these three resources is the key to maximizing performance and scalability. In other words, you cannot add an unlimited number of CPUs (or processing cores) and see a commensurate increase in performance without also improving the memory capacity and performance of the disk-drive subsystem. It is also common to see a diminishing return as resources are added to a single database server. These factors are especially true in mixed-use business transaction systems and ones that perform a high volume of read and write transactions, as well as supporting generalized business reporting tasks.
Therefore, as business applications gain sophistication and continue to grow in demand, architects, developers and database administrators have been presented with a constant challenge of maintaining database performance for mission-critical systems. This landscape drives the need for database sharding.
Database Sharding-The "Shared-Nothing" Approach
Database sharding provides a method for scalability across independent servers, each with its own CPU, memory and disk. The technique allows the proper balancing of database size with system resources, resulting in dramatic performance improvements and scalability for a given application. Although the concept of a "shared-nothing" database implementation has been under discussion for more than 15 years, it appears that the business application market is just now identifying a generalized need for database sharding due to the exponential increase in data volumes over the past several years.
The basic concept of database sharding is very straightforward: take a large database and break it into a number of smaller databases across multiple servers. Each "shard" (running on its own server) contains a portion of the original monolithic database, and is "sharded" based on application-specific rules. For example, many organizations "shard" by customer, with each shard containing a specific group of customer-related information. Database queries and write operations are then performed in one of two modes:
- Single-Shard: Each database operation (read or write) is performed against a single "shard," such as for a single customer transaction.
- Multi-Shard: This typically applies to analytic queries, performed in parallel across one or more "shards," enabling impressive performance results.
How fast is database sharding? It depends on your application and its specific requirements. However, in general, you can plan on near-linear scalability as you add "shards" and servers for typical single-shard transactions. For multi-shard queries, especially analytic operations, the results can be many times faster when compared to a single monolithic database due to the parallel processing capability and the correct balancing of database size to system resources.
The obvious advantage of the shared-nothing database sharding approach is improved scalability, growing in a near-linear fashion as more servers are added to the network. However, there are several other advantages of smaller databases, which should not be overlooked when considering a sharding solution:
- Smaller Databases are Easier to Manage. Production databases must be fully managed for regular backups, database optimization and other common tasks. With a single large database these routine tasks can be very difficult to accomplish, if only in terms of the time window required for completion. Routine table and index optimizations can stretch from hours to days, in some cases making regular maintenance infeasible. By using the sharding approach, each individual "shard" can be maintained independently, providing a far more manageable scenario, performing such maintenance tasks in parallel.
- Smaller Databases are Faster. The scalability of sharding is apparent and achieved through the distribution of processing across multiple shards and servers in the network. What is less apparent is the fact that each individual shard database will outperform a single large database due to its smaller size. By hosting each shard database on its own server, the ratio between memory and data on disk is properly balanced, thereby reducing disk I/O and maximizing system resources. This results in less contention, greater join performance, faster index searches and fewer database locks. Therefore, not only can a sharded system scale to new levels of capacity, individual transaction performance is benefited as well.
- Database Sharding can Reduce Costs. Most database sharding implementations take advantage of low-cost open source databases and commodity databases. The technique can also take full advantage of reasonably priced "workgroup" versions of many commercial databases. Sharding works well with commodity multi-core server hardware, systems that are far less expensive when compared to high-end, multi-CPU servers and expensive storage area networks (SANs). The overall reduction in cost due to savings in license fees, software maintenance and hardware investment is substantial-in some cases 70% when compared to traditional solutions.
There is no doubt that database sharding is a viable solution for many organizations, supported by the number of large online vendors and SaaS organizations that have implemented the technology, including Amazon, eBay and, of course, Google.
This article has provided a brief overview of database sharding, what it is and why it's of benefit to today's business organizations. Database sharding has been proven in many organizations, large and small (but growing ...), and may very well be applicable to your specific application problems. When implemented properly, database sharding can deliver on the objective of cost-effective, near-linear scalability for high-volume business transaction applications.