Accelerating File Transfer

by , February 23, 2009

Accelerating File Transfer: Quicklinks

Overview | Part 1: Compression | Part 2: Minimizing Data Sets | Part 3: Multiple Streams | Part 4: Alternatives (like FileCatalyst!) to FTP/TCP

When people first find out about FileCatalyst, one of the first questions asked is always, "How does your product speed up file transfer?" often followed by a best guess, "Is it some sort of compression?" Although the deepest possible answer is going to be somewhat complicated, there are some very basic principles at play in the world of file transfer acceleration, with compression being just one. Almost all products claiming to provide fast file transfer will do one or more of the following:
  1. compress data
  2. minimized data sets (other than compression)
  3. transmit over multiple streams
  4. use an alternate to FTP
The first two are considered fairly standard for the industry. The third is the domain of specialized applications such as FileCatalyst, and the fourth is territory wide open to the creativity of developers. Over the next several days, we will explore each of these; we will elaborate, discuss realistic expectations to have, and how they factor into the FileCatalyst family. In the meantime, here's a brief look at each:

Compression

Using Zip, Rar, Gzip, or another compression algorithm, data sets are minimized. Compression algorithms find patterns and recreate the data using a shorter form. Just as it is quicker to write "five million zeroes in a row" than to actually type five million zeroes in a row, the number of bits needed to store the data is reduced. In order for the data to be used again, it must also be decoded (uncompressed) on the destination side.

Multiple Streams

By sending data across more than one stream at a time, there are fewer moments of “downtime”. While there is housekeeping occurring on one stream (in-between files being sent for example), the other stream(s) can still be actively sending data.

Alternate Protocols

FTP transfer occurs over TCP, a protocol with ordered delivery of packets. Since the order cannot be interrupted, latency and packet loss can cause it to slow to a crawl. Protocols such as UDP can blast out packets with disregard for order. However, UDP in and of itself does not reliably transfer data, which is why file transfer technologies based around UDP must also include their own retransmission and verification functionality.

Sending less data

Compression is one way of sending a smaller file than the original, but it is not the only way. For example, FileCatalyst offers an option to transfer file deltas, which are partial files which can be used to “patch” a file that has been sent previously. Sending a 4MB delta instead of an entire multi-GB file can save an absolutely staggering amount of time and bandwidth. Keep your eyes on this space as we talk a bit more about these and other ways to achieve fast file transfer. Cheers, Greg
Revision List

#1 on 2010-Aug-17 Tue  08:58+-14400

#2 on 2010-Aug-17 Tue  08:28+-14400

#3 on 2010-Aug-17 Tue  08:12+-14400

#4 on 2009-Feb-23 Mon  02:00+-14400

Tags: , , ,

No Comments Yet

Be the first to respond!

Sorry, comments for this entry are closed at this time.