Accelerating File Transfer: Quicklinks
- compress data
- minimized data sets (other than compression)
- transmit over multiple streams
- use an alternate to FTP
CompressionUsing Zip, Rar, Gzip, or another compression algorithm, data sets are minimized. Compression algorithms find patterns and recreate the data using a shorter form. Just as it is quicker to write "five million zeroes in a row" than to actually type five million zeroes in a row, the number of bits needed to store the data is reduced. In order for the data to be used again, it must also be decoded (uncompressed) on the destination side.
Multiple StreamsBy sending data across more than one stream at a time, there are fewer moments of “downtime”. While there is housekeeping occurring on one stream (in-between files being sent for example), the other stream(s) can still be actively sending data.
Alternate ProtocolsFTP transfer occurs over TCP, a protocol with ordered delivery of packets. Since the order cannot be interrupted, latency and packet loss can cause it to slow to a crawl. Protocols such as UDP can blast out packets with disregard for order. However, UDP in and of itself does not reliably transfer data, which is why file transfer technologies based around UDP must also include their own retransmission and verification functionality.
Sending less dataCompression is one way of sending a smaller file than the original, but it is not the only way. For example, FileCatalyst offers an option to transfer file deltas, which are partial files which can be used to “patch” a file that has been sent previously. Sending a 4MB delta instead of an entire multi-GB file can save an absolutely staggering amount of time and bandwidth. Keep your eyes on this space as we talk a bit more about these and other ways to achieve fast file transfer. Cheers, Greg
#1 on 2010-Aug-17 Tue 08:58+-14400
#2 on 2010-Aug-17 Tue 08:28+-14400
#3 on 2010-Aug-17 Tue 08:12+-14400
#4 on 2009-Feb-23 Mon 02:00+-14400