Different Types of Replication
To get right to the point: there are three fundamental forms of data replication. For example, disk blocks can be replicated (block-based replication, such as in Redundant Arrays of Inexpensive Disks— RAID), records can be replicated (in databases), or entire files can be replicated (such as AWS S3 buckets, Google Cloud Storage, or Microsoft Azure Blob storage). All three have their strengths and weaknesses and, therefore, specific areas of application.
In “Cloud Speak,” we refer to these as“Patterns and Anti-Patterns.” Here, the “Patterns” are the correctly applied replication types, and the “Anti-Patterns” are incorrectly applied replication types in a specific scenario.
Block-based storage
Block-based storage is commonly found in the world of Storage Area Networks (SANs). This ensures that data is stored across multiple disks, so that the failure of a single disk does not result in data loss. It is a highly efficient form of data replication because only changed blocks are replicated, as opposed to replicating the entire file. This approach is not limited to SANs; there is also open-source software available for block-based replication (including over a network), such as DRBD.
Block-based replication is typically used to achieve redundancy (of disks, for example). When replication is not between local disks but to a different disk at a different location, the goal is not only disk redundancy but also data availability. The (physical) location of the data is then eliminated as a single point of failure (SPoF).
But even a solution like this has its limitations. If data is deleted on one side, it is also deleted on the other. So it doesn’t protect against that. It is not disaster recovery (DR). An off-site backup, for example, is.
Record-based replication
Record-based replication is used with databases. Databases, for example, involve read and write operations on records. In replication, only write operations are replicated, since read operations do not change anything. The write operations are then performed again on the other side so that both databases are once again in sync. Often, the replicated database is a “read-only” database (also called a “slave”). The replicated (slave) database can, for example, be used effectively for backups. This has no performance impact on the “primary” or “master” database.
Replication is therefore also used here to increase availability (and does not involve disaster recovery). However, the backup created from the “Slave” database is a DR measure. The replicated write operations can be stored in a redo log file (Oracle technology). These redo log files are then often included in the backup as well. By combining the database backup with the redo log files, a“point-in-time restore”can be performed.
Databases offer countless ways to replicate data. For example, MySQL and MariaDB provide extensive capabilities for this purpose. In the example above, the database replica was not used for production. It is used for backup purposes and has no end users. However, both databases (“Master” and “Slave”) can be used for read operations, as long as changes are written only to the “Master.”
In general, a database has 10 times as many read operations as write operations. A “Master” with 9 “Slaves” can therefore be a way to increase the overall performance of the entire system. But even though performance is higher, the “Master” is still a Single Point of Failure (SPoF). One of the replicas can become the master, but that takes time and therefore results in downtime for write operations.
Master-master setup
ACID
A term used in databases is ACID (Atomicity, Consistency, Isolation, and Durability). These are characteristics of a database that ensure the accuracy and completeness of data. The “Atomicity” property states that database operations are indivisible and are therefore either executed and saved correctly or not at all. It prevents operations from being partially executed and ensures that data is neither missing nor corrupted.
Another option is to set up a “Master-Master” configuration. In this setup, both “Master” databases can receive write operations and replicate them to the other “Master.” It is essential to prevent records with the same record number from being written. Therefore, one “Master” is designated to handle even-numbered inserts, and the other “Master” handles odd-numbered inserts. Both databases can be used with local performance—even when they are located far apart. This system can be expanded as needed, for example, with a 3-Master setup (see the figure). This increases both data availability and performance. However, a DROP TABLE command is replicated to all masters, so this is not a DR (Disaster Recovery) solution.
One final note about the multi-“Master” setup. When the databases are located far apart, replication latency is higher. Both databases can be used locally with excellent performance. This is a good setup for active-active (also known as multi-region) designs. In this scenario, replication may “lag behind” (consider the RPO; see the section on RPO and RTO).
File-based replication
In cloud environments, the (simple) replication of files is a basic feature. This existed even before the cloud (Ceph, for example ), but the cloud has certainly made it popular. These days, everyone knows what an AWS S3 bucket is. With file-based replication, files are stored on multiple storage systems (and often in multiple locations).
As soon as a file changes, a completely new version of the file (or object) is created; this is also known as“Copy-on-Write.”As a result, when a file (or object) changes, all copies must be updated immediately. This takes time, which is why it is often referred to as“Eventually Consistent”storage.
The file (the object) can be read from any location (once it has become consistent). This results in high performance. Version control is also (relatively) simple; after a “Copy-on-Write,” the old version is not deleted.
A drawback of the “Copy-on-Write” principle is that if even a single bit of a file is changed, the entire file must be rewritten. Storing a database file on an object store (such as S3) is therefore not efficient. Every record change results in a new copy of the entire database file.
Consequently, its performance is poor; it is what is known as an“anti-pattern.” It can be said that object storage (file-based replication) therefore improves both performance (in the right situations, the “patterns”) and availability. It can also provide disaster recovery (DR), provided that version control is enabled.
One final advantage of this type of storage technology is that it offers virtually unlimited scalability in terms of storage capacity. It works precisely because it is so simple.
But can an object storage file system be mounted?
Block-based file systems can be mounted “on” or “to” a computer. They are thus treated as local storage, even though they are not. This is not common with object storage. It has an Application Programming Interface (API) that can be used. Read, write, create, and delete operations can be performed via this API. However, if you really want to, it is possible to mount an object store. The performance of such a solution is low, mainly due to the “Copy-on-Write” principle. The object store behaves fundamentally differently from local storage, and as a result, it has few use cases (patterns).
RTO and RPO
When designing an environment, the required RTO and RPO are determined. The Recovery Point Objective (RPO) is the maximum amount of data loss an environment is allowed to experience. The Recovery Time Objective (RTO) is essentially the maximum downtime, but it can also be viewed as the environment’s availability. These values are often determined during a Business Impact Analysis (BIA).
RTO is achieved through all the measures taken to ensure or restore availability. RPO is primarily determined by the latency of data replication and backup/restore (DR). But if a network failure occurs before data has been fully replicated, how much data is lost?
How much data loss is actually acceptable? RPO is a complex topic, because how do you know which data has been replicated during an outage and which hasn’t? How do you determine which of the two sides contains the correct data if they are no longer in sync?
This is also referred to as the data “GAP.” This brings us to the topic of synchronous and asynchronous data replication. More on this in the “Timing” section.
Which data is now correct after we experienced a replication failure?
For a database, the “Time Stamp” can be used. The last record that was correctly written is the last valid data; everything after that is incorrect. This assumes that both sides have the correct time. In block-based replication, a technique called “fencing” is used. “Fencing” is used to determine which (storage) side contains the correct data.
What about backups?
You might—and rightly so—wonder whether backup/restore is a data replication technique. That’s certainly worth discussing. In most cases, however, backup/restore isn’t considered a replication technique. Backup/restore is more about retention and recovery.
It creates a point-in-time copy of the data (including errors). Data replication is more of a technique used to ensure“business continuity.”What’s confusing is that object storage can also be used to store backups. It offers high availability and supports version control. This debate is difficult to resolve. The key question, therefore, is: what needs to be achieved?
What about off-site backups?
Object storage offers good options for storing backups. Not only does it support version control, but it also allows you to use other regions. This brings us close to the concept of off-site backups.
Granularity
The three replication technologies all have their advantages and disadvantages, their “Patterns & Anti-Patterns.” One major difference between these techniques, however, is their granularity. The smallest replication unit is block-based replication, which involves, for example, 4-kilobyte blocks. Records are usually much larger, and files are often many times larger still.
Topology
The article previously discussed “one-way” replication and “two-way” replication. This is also referred to as the replication topology.
Deduplication
When file storage is analyzed, it becomes apparent that the same files are often stored multiple times. That’s a shame, because replication already keeps multiple copies. Copies of copies are unnecessary and costly. Some storage systems remove these duplicates without the end user even noticing. This is called deduplication. Deduplication saves storage capacity and, therefore, costs.
Compression
In addition to deduplication, data and files can also be compressed. This involves reducing the size of files through compression, thereby saving storage capacity. You’ll achieve the greatest benefit by combining deduplication and compression. However, both processes require computing power (overhead) and therefore introduce latency (albeit theoretically).
Encryption
One topic that comes up regularly is encryption. Most cloud providers and hyperscalers offer this“out of the box.”It can be applied to local (block) storage, databases, and object storage. It’s a good idea to use encryption—it’s easy and costs virtually nothing.
However, it does not protect you from the cloud provider or hyperscaler itself. They still have access to the encryption keys. If you want to prevent that, a company or organization will have to manage its own encryption keys. That’s doable, but it doesn’t integrate very well with the cloud environment.
For example, using a Hardware Security Module (HSM). This provides protection against the cloud provider or hyperscaler, as well as against information requests from governments and intelligence agencies. It is therefore an important factor in the discussion surrounding data sovereignty. This encryption of stored data is also referred to as“Data at Rest”encryption.
In addition to “Data at Rest” encryption, there is also“Data in Transit”encryption, which refers to network traffic. Both are forms of encryption, and both protect against different risks.
Caching
One topic that needs to be mentioned here is caching. Caching—in general—is the approach of storing frequently requested data on fast disk (SSD), in memory, or in a geographically close location. This allows the data to be delivered very quickly. Operating systems such as Linux, OS X, and Windows also use this concept. Databases also utilize caching, which enables frequently requested records to be retrieved quickly. This is particularly useful for session information that is frequently accessed. Caching is a topic in data replication because, on the one hand, it can speed up replication, and on the other hand, it can disrupt replication. We won’t go into further detail on this topic here, but you’ll cover topics such as “cache hits,” “cache misses,” and “flushing the cache.”
Timing
Another interesting aspect of data replication needs to be discussed in more detail:latency. This is important in replication. It was mentioned earlier in the context of RTO and RPO. The physical distance between the storage systems we are replicating determines how long it takes to replicate that data.
Data is secure only after it has been fully written to both sides. This becomes an issue when the distance is great. With synchronous replication, a confirmation is sent only after both sides have written the data. Because of the distance, an end user must then wait longer.
Over very long distances, this can become a problem. In such cases, it’s better to opt for asynchronous replication. With this approach, the end user can continue working once the data has been saved locally. The data is then replicated. This is good news for performance, but less so for data security.
After all, it’s possible that the data hasn’t been fully replicated yet and a disruption occurs. In that case, the storage is no longer the same. Which side is in the right when bidirectional replication was used? Of course, there are solutions to this, but they fall outside the scope of this article.
Data Sovereignty
And why was all of this so important for data sovereignty? In light of data sovereignty, companies and organizations might consider storing data in multiple locations—for example, to ensure the (geo)location of the data (data residency). Various techniques also offer the possibility of data recovery (DR).
This is certainly an issue that touches on data sovereignty. The techniques described are therefore important for achieving such objectives. And because no single technique solves all problems and situations vary, it is useful to be familiar with all of them and to “master” them. You may even need to use a combination of techniques.
Although it is not a replication technology, open-source software (OSS) should still be mentioned here. Because data sovereignty is about gaining “control” over data, the software used has a major impact on that. The best way to maintain (or regain) control is to maintain control over the software being used.
OSS is therefore the best choice one can make. An interesting detail, by the way, is that all cloud providers and hyperscalers use OSS. Sometimes they even go so far as to rename OSS and market it as their own product. That is, incidentally, legal.
Even if the renamed version is used, you can always revert to the OSS version, which ensures that the data remains accessible.
Another advantage of using OSS is that it is the best way to ensure the use of open standards. OSS has no choice but to use open standards, and that guarantees access to the data.
In summary, it can be said that understanding “Patterns and Anti-Patterns” and mastering data replication technologies are most likely essential to achieving data sovereignty. Data sovereignty involves mitigating geopolitical risks in data storage. These techniques will help you achieve that.
To learn how SUE secures cloud environments, visit the Cloud Security page. The Cloud Security Guide explains how to set up and maintain a secure cloud environment.
