Should minimum file size equal or exceed delta block transfer size?
Overview of delta-level replication
Delta-level replication is a byte-level replication technology that enables block or byte-level replication for a File Collaboration job. This feature allows PeerGFS to transmit only the changed bytes or blocks of a file, rather than transferring the entire file. This significantly reduces network bandwidth usage, which is especially beneficial when transferring files over a slow WAN or VPN, or across a high-volume LAN.
When editing File Collaboration, File Synchronization, or File Replication jobs, you can specify delta replication options, including Checksum Transfer Size and Delta Block Transfer Size. This article explains the relationship between these two options and the minimum file size required for delta encoding. Delta encoding stores or transmits data as differences (deltas) between sequential data, rather than complete files. More generally, this is known as data differencing.
Checksum Transfer Size and Delta Block Transfer Size
The Checksum Transfer Size and Delta Block Transfer Size fields are not related to the chunks of a file that PeerGFS uses to calculate checksums. By default, PeerGFS generates 64-byte MD4 checksums for each block, which can range from 8 KB to 1 MB, depending on the nature of the file. This checksum calculation is auto-adjusted based on the file, but it cannot be configured via the PeerGFS user interface.
How these fields work
Changing the Checksum Transfer Size affects the number of checksums that are sent from the target to the source in one transfer. Think of this as a buffer for checksum values. PeerGFS calculates a checksum for each block of a target file and stores the checksum in a 256 KB buffer. If the buffer becomes full, PeerGFS sends it to the source and places the remaining checksum values into another buffer. If all checksums for a file do not fill a single buffer, PeerGFS sends the partially filled buffer. The checksums for most files will fit within the default 256 KB.
Changing the Delta Block Transfer Size affects the number of bytes of file differences PeerGFS sends from the source to the target in one transfer. Similar to Checksum Transfer Size, this works with a buffer. For each checksum comparison, the source Agent adds either an in-sync flag or the actual 8 KB of the source file that needs to be sent to the target. Once the buffer is full, PeerGFS sends it to the target and begins filling a new buffer.
Note: Both settings affect delta compression logic for a single file only. Partially empty buffers are not shared across files; they are sent once the delta compression logic for a single file is complete.Recommendations for adjusting these values
We recommend increasing these values only when you are working with large files greater than 100 MB. Increasing these values can optimize delta compression performance, but it will also increase memory usage for all Peer Agents. If you decide to adjust these values, it is best practice to increase both values simultaneously and by the same increment.
We currently do not recommend lowering these values.