Table of Contents

  1. Overview
    1. Requirements
    2. Installation Procedure
  2. Configurable Parameters
    1. Connection-related Settings
    2. FileCatalyst Transport Tuning
    3. Transfer Content Parameters
    4. Transfer Features
    5. GUI Behaviour and Presentation
    6. PostURL and Browser Redirection
    7. Debug Parameters
  3. Performance Tuning
    1. Know Your System
    2. Know Your Bandwidth
    3. Know Your Network Infrastructure
    4. Know Your Data
    5. Know the Client Endpoint
  4. Firewall Considerations
    1. Configuring for the Control Connection
    2. Configuring for the Data Connections
  5. Regular Expressions
    1. Example Usage
  6. Support
    1. Support Options

 


1 Overview

1.a Requirements

FileCatalyst will run on any Windows, Linux, OS X, or Solaris/Sparc computer that can run Java 1.4 or higher. Please visit http://java.com to get the latest version of Java for your platform.

To run a FileCatalyst applet, the user will need a web browser (e.g. Internet Explorer, Firefox, Safari) which has been enabled with Java Plugin version 1.4 or later. Once you have installed the applet, you may navigate to the install path to see the sample page. For example, if the files are at /wwwroot/filecatalyst then you would access the applet through http://www.server.com/filecatalyst , substituting your own domain or IP address for "www.server.com". The browser will find the "index.html" page and load it, which will in turn load the applet.

1.b Installation Procedure

  1. Unzip/decompress FileCatalyst Applet package (e.g. “fc_applet.zip”).
  2. Save the contents to an appropriate directory on your website (for example, “/uftp”).
  3. Add the following lines to the webpage that will contain the applet:
    <script language="JavaScript" src="deployJava.js"></script>
    <script language="JavaScript" src="fcupload.js"></script>
    The included document “fcupload.html” offers an example of usage. Change the src=“fcupload.js” to include the relative path, unless the webpage and the applet are located in the same directory. for example, it might become: src="../uftp/fcupload.js"
  4. Modify the JavaScript file (“fcupload.js”) to suit your needs, according to the parameters outlined in the next section of this guide.

2 Configurable Parameters

The following section describes the JavaScript parameters defined in the fcupload.html page. They include both manditory parameters (server, port, user, password) and optional features.

Note: For a JavaScript-only headless applet, please refer to the additional examples provided in the live.html or ajax_sample.html pages included in the applet bundle.

2.a Connection-related Settings

Provides basic connection configurations required for the applet to connect to a remote server.

Connection-related Settings
server

The host name or IP address of the FileCatalyst server.

Default value: ""
Valid values: valid hostname or IP address

In FileCatalyst Webmail and Workflow, this parameter is set automatically

port

The port on which to connect to the FileCatalyst server software

Default value: 21
Valid values: 1-65535

In FileCatalyst Webmail and Workflow, this parameter is set automatically

user

Specify username when authenticating to the server

Default value: "anonymous"

In FileCatalyst Webmail and Workflow, this parameter is set automatically

pass

will send the specified password when authenticating to the server

In FileCatalyst Webmail and Workflow, this parameter is set automatically

encrypt

If set to "true" you must use encrypted values for the server/user/pass parameters. FileCatalyst server will decrypt them and perform the login. The encryption is based on a key value that is specified by the "ek" parameter below. To generate encrypted parameters, use encrypt.html, included with the applet. This provides at least some measure of protection from prying eyes.

Although including the encryption tool may seem somewhat insecure, keep in mind this is only a way to hide credentials from users who are merely poking around in your HTML source. Determined hackers will easily be able to sniff the network to retrieve user and password info regardless of encryption done in the HTML. Use SSL on the FileCatalyst server to ensure full security.

Default value: false
Valid values: false, true

In FileCatalyst Webmail and Workflow, this parameter is set automatically

ek

This parameter works in conjunction with the "encrypt" parameter. This is the key that is used to encode/decode the string value you pass to the user, pass, and server parameters. Here is an example configuration. In this example, the server/user/pass strings are 192.168.1.100/anonymous/fc@FileCatalyst.com respectively. This illustrates how the parameters will look to anyone who inspects the connection values in your HTML:

var server = "8I7BLHN;8I8@5";
var port = "21";
var pass = "|vy&[)&yp*p&jx#((;j,t";
var user = "h~t$6!=$z";
var autoconnect = "true";
var autoreconnect = "true";
var passive = "true";
var encrypt = "true";
var ek = "encryptkey";

Default value: ""
Valid values: string value

In FileCatalyst Webmail and Workflow, this parameter is set automatically

clientConnectKey

obtain a key from your FileCatalyst representative in order to allow applet to communicate with non-FileCatalyst servers

Default value: ""
Valid values: Obtain key from FileCatalyst

In FileCatalyst Webmail and Workflow, this parameter is set automatically

enableSSL

Require applet to connect with SSL mode

Default value: false
Valid values: true, false

In FileCatalyst Webmail and Workflow, this parameter is set automatically

maxRetries

This value specifies how many times FileCatalyst will try, if a transfer error occurs, to reconnect and resume the transfer

Default value: 3
Valid values: integer value

waitRetry

This value (in milliseconds) specifies how long FileCatalyst will wait, if a transfer error occurs, before it tries to reconnect and resume the transfer

Default value: 3000 (3 seconds)
Valid values: integer value

2.b FileCatalyst Transport Tuning

These parameters allow you to configure FileCatalyst to perform properly over your specific network conditions.

FileCatalyst Transport Tuning
bandwidth

Maximum bandwidth (kbps) applet will attempt to transfer at. If congestionControl is false, this value represents the speed at which the applet will transfer data at.

Default value: 100000 (100 Mbps)
Valid values: integer value

In FileCatalyst Webmail and Workflow, this parameter is set automatically

mode

Specifies protocol used to transfer files.

UDP -- Utilizes the FileCatalyst protocol for fast network transfers. Immune to latency on the network.
FTP -- Utilizes a passive multi-stream TCP connection (data ports). If failure occurs, protocol attempts a passive TCP connection (both control and data are sent through main transfer port). Fast transfers over LAN, slow transfers over WAN.
HTTP -- Data sent via HTTP posts to HTTP Servlet (default web port 80), where the data is captured reconstructed on the servlet and forwarded to the FileCatalyst server. Slower transfers, but able to navigate through most firewalls.
AUTO -- Protocol will attempt to transfer via UDP. If connections cannot be established, failover cascades to other protocols in this order to attempt a successful connections:

  1. UDP
  2. Passive TCP
  3. Port TCP
  4. HTTP (if http servletLocation is configured)

Default value: UDP
Valid values: AUTO, UDP, FTP, HTTP

In FileCatalyst Webmail and Workflow, this parameter is set automatically

servletLocation

HTTP Servlet location. Used in conjuction with the Unlimited FTP Servelet to allow HTTP transfers.

Default value: ""
Valid values: Network address for the HTTL servlet (example: "http://192.168.1.100:8080/uftps/ftpservlet")

In FileCatalyst Webmail and Workflow, this parameter is set automatically

servletUploadMultiplier

Adjusts the size of data passed into each HTTP post (n * 2048 bytes)

Default value: 2
Valid values: 1-10 (may be servlet limit)

numFTPStreams

Specify the number of encoders (concurrent threads/blocks) to use for multi-stream TCP transfers.

Default value: 1
Valid values: 1-10

In FileCatalyst Webmail and Workflow, this parameter is set automatically

unitSize

The packet size in bytes. Set to 1024 for Windows, 1472 for Linux/MacOSX.
For high speed transfers (1+gbps), set to maximum size of network MTU - 28 byte (for headers):
Ex: for 9000 MTU, set to 8972

Default value: 1024
Valid values: integer value

In FileCatalyst Webmail and Workflow, this parameter is set automatically

blockSize

The block size in bytes

Default value: 4096000 (4 MB)
Valid values: integer value

In FileCatalyst Webmail and Workflow, this parameter is set automatically

numSenderThreads

Specify the number of Sender Threads (concurrent threads/blocks) to use for UDP transfers.

Default value: 3
Valid values: 1-75 (server-side limit applies)

In FileCatalyst Webmail and Workflow, this parameter is set automatically

numSenderSockets

Specify the number of UDP streams sender can utilize for UDP transfers. Allows for higher core utilization in sending data packets out.
Recommend increase for every ~2gbps on link.

Default value: 1

numReceiveSockets

Specify the number of UDP streams receiver can utilize for UDP transfers. Allows for higher core utilization in getting packets in.
Recommend increase for every ~5gbps on link.

Default value: 1

congestionControl

If set to true, applet will start with a slow transfer speed and ramp up to the maximum specified by bandwidth. This is best used if you do not know your linespeed, or must share network connections with other traffic.

Default value: true
Valid values: true, false

In FileCatalyst Webmail and Workflow, this parameter is set automatically

startRate

Specifies the rate in Kbps at which transfer will start before ramping up to target bandwidth.

Default value: 1000 (1 mbps connection)
Valid values: integer value
(normally < 1/2 maximum bandwidth, or speed of slowest expected connection)

In FileCatalyst Webmail and Workflow, this parameter is set automatically

congestionControlAggression

Specifies the UDP congestion control aggression between 1 and 10. 1 is least aggressive and 10 is most aggressive.

  • "2" will normally allow other TCP traffic on the network to take priority (speed will be throttled back by congestion control).
  • "5" default -- will let the UDP transfer compete for network bandwidth with other protocols (such as TCP).
  • ">7" will let the UDP transfer try and overpower all other network traffic.

Default value: 5
Valid values: 1-10;

In FileCatalyst Webmail and Workflow, this parameter is set automatically

congestionControlStrategy

Specifies the strategy used by UDP congestion control.

  • "0" RTT based congestion control (strategy used for FileCatalyst v2.x software).
  • "1" Packet-loss based congestion control.

Default value: 1
Valid values: 0,1;

numPacketProcessors

Number of threads used to process packets on the receiving side.
Recommended increasing for every 500mbps (up to # of processors) when using AES encryption to allow multi-core decryption. Download only.

Default value: 1

packetQueueDepth

Number of packets application keeps buffered before processing. Download only.

numBlockWriters

Number of block writers when saving the file to disk. Download only.

Default value: 1

writeBufferSizeKB

Size in KB of each write operation when saving the file to disk. Allows optimize writes to disk.

Default value: network block size

writeFileMode

Mode the application will use to write file to disk. RW will utilize OS buffers to attempt to speed transfer speeds, and is generally the fastest option. RWD and RWS forces the OS to update meta data for each write operation, while also forcing the write operation is written to disk (not buffers) before returning.

Default blank lets the FileCatalyst application choose the value based on operating system.

Default value: ""
Valid values: "", rw, rws, rwd

numBlockReaders

Number of block readers when reading the file from disk. Upload only.

Default value: 1

readBufferSizeKB

Size in KB of each read operation when loading the file from disk.

Default value: network block size

forceTCPmodeACKs

Force Acks to flow on the TCP channel (if UDP ACKs" are blocked by firewall)

Default value: false (basic DSL/Cable)
Valid values: true, false

2.c Transfer Content Parameters

These parameters can be pre-defined to list out the files/directories to transfer.

Transfer Content Parameters
files

Predefine file/directory list to upload. Use "delimiter" to seperate out multiple files.

Semi-colon (Windows) or colon (Linux, OS X, Solaris) delimited list of filenames if delimiter is not explicitly defined.

Default value: ""
Valid values: String value

localdir

Default browse directory (default is home directory)

Default value: ""
Valid values: String value - full path

remotedir

Path to which the uploaded files will be placed on the server (relative to ftp user home)

Default value: ""
Valid values: String value

In FileCatalyst Webmail and Workflow, this parameter is set automatically

autoUpload

Transfers will start immediately when added to the transfer queue

Default value: false
Valid values: true, false

maxfiles

allows a limit to be set for number of files added to the queue

Default value: "" (2147483647, or max int)
Valid values: Integer value

In FileCatalyst Webmail and Workflow, this parameter is set automatically

maxsize

allows a limit for individual file sizes to be set, in bytes

Default value: "" (9223372036854775807, or max long)
Valid values: long value

In FileCatalyst Webmail and Workflow, this parameter is set automatically

maxtotalsize

allows a limit for the total file size in cart to be set, in bytes

Default value: "" (9223372036854775807, or max long)
Valid values: long value

regex

allows a Java regular expression to be used to limit the file name / type that is allowed to be added to the queue

Default value: ""
Valid values: Regular expression (see below for examples)

In FileCatalyst Webmail and Workflow, this parameter is set automatically

limitUploadToFiles

restricts the user to only be able to select files for uploads and not directories

Default value: false
Valid values: true, false

2.d Transfer Features

Parameters provide multiple features to accelerate file transfers (compression, delta, etc) or to add robustness to the transfers (MD5 verification, preserve file structure, preserve file attributes, etc).

Transfer Features
incremental

When set to true, only files that have been modified will be transferred (default false)

Default value: false
Valid values: true, false

incrementalMode

Delta feature. This determines what the applet will do when it recognizes that an incremental transfer must occur (file has changed). Incremental mode can either overwrite the existing file, or create a new file using a numerical notation.

  • "0": transfer the entire file if the applet has detected that the file has been modified.
  • "1": transfer ONLY file delta (rsync).
  • "2": transfer the entire file, but do not overwrite the original file (format: file.#.txt).
  • "3": transfer ONLY file delta (rsync), but do not overwrite the original file (format: file.#.txt).

Default value: 0
Valid values: 0,1,2,3

verifyintegrity

MD5 verification of the file content on both sides of the transfer.

Default value: false
Valid values: true, false

verifyMode

Method used to perform file verfication.

Performing an MD5 on the entire file (after transfer is complete) is a more standard approach, but can result in the transfer appearing to pause, especially on high bandwidth transfers verifying large files are performed.

On-the-fly MD5 calculations are performed after each block (or file part) is saved to the disk. These are done while data is still being transferred on the network by other threads, resulting in a smoother network usage.

Concurrent MD5 offloads file verification to another connection, resulting in complete utilization of networking resources.

  • "0": Verification after file transfer
  • "1": On-the-fly verification
  • "2": Concurrent verification

Default value: 0
Valid values: 0, 1, 2

progressive

After transfer is complete, applet will check if file has grown on server since the start of the transfer. If the file has grown, the transfer will resume until the file size is set.

Default value: false
Valid values: true, false

autoresume

If the applet detects that a partial file already exists, the applet will attempt to resume the transfer rather than start the transfer from the beginning.

Default value: true
Valid values: true, false

compression

files will be compressed on the fly as they are transferred.

Default value: false
Valid values: true, false

compMethod

Determines the compression method requested.

  • Use Zip Deflater (0) or LMZA (1) for on-the-fly compression.

    Note: LMZA is VERY CPU and MEMORY intensive for the encoder.

Default value: 0
Valid values: 0-1

compLevel

Determines the compression level requested.

  • "1": offers the least compression but lowest CPU overhead
  • "9": offers the best compression but highest CPU overhead

Default value: 4
Valid values: 1-9

autoZip

Single archive transfer. If set to true, all files are first archived into a single zip file and sent off the network as one file transfer before being uncompressed at it's destination. Set this if the file contents include many small files (< 100k), as the work required to build up and tear down each data connection often exceeds the time of transfer for small files. The generated ZIP file must be 4GB or smaller, otherwise the auto-zip transfer will fail.

Default value: false
Valid values: true, false

autoUnzip

Automatically unzip contents when it has arrived at the destination. Setting this to false makes the applet transfer the zip content to it's destination, but leaves it archived.

Default value: true
Valid values: true, false

zipFilename

Specify a temporary file name for the zip archive to be called as it's being tranferred. Default will let the applet choose a temporary name.

Default value: ""
Valid values: String value

zipFileSizeLimit

This feature will limit the size of the zip archives to the specified value (in bytes) when possible. If a file set is too large to fit in a single zip of the specified size, multiple zip archives will be created. These zip archives are automatically extracted and cleaned up as they are transferred. At most 5 archives will be created at one time. Additional archives are generated only after the oldest file is deleted. This helps reduce disk space required.

Default value: ""
Valid values: long value

useTempName

File will be transferred with a .tmp extension, and renamed when transfer is complete

Default value: false
Valid values: true, false

preservePathStructure

Path info is maintained when files are transferred. If set to faluse, the path is not included and directories are flattened.

Default value: true
Valid values: true, false

In FileCatalyst Webmail and Workflow, this parameter is set automatically

keepFileAttributes

preserve the file modification date/time and UNIX file permissions if applicable

Default value: false Valid values: true, false

deletePartial

Partial files will be deleted when an transfer is cancelled.

Default value: false
Valid values: true, false

This flag also determines the applet behaviour to stop a transfer.

  • When set to TRUE, the stop button is labeled "CANCEL", and a redirect cancel is called.
  • When set to FALSE, the stop button is labeled "PAUSE", and a redirect pause is called.

If the seperatePauseCancel is set to true (both pause and cancel buttons are displayed), the value will be modified based on user action (pressing pause will set to false, pressing cancel will set to true).

confirmOverwrite

If this value is set to "true", FileCatalyst will prompt you if the file(s) already exist when a transfer is attempted. You will have the choice to overwrite if you want to. If it is set to false, FileCatalyst will always overwrite.

Default value: false
Valid values: true, false

autoDMG

Mac OS X platform. If set to true, on Mac OS X platform all files will be added to a single DMG file before upload, maintaining resource forks

Default value: false
Valid values: true, false

dmgFileName

When using AutoDMG, this specifies the filename for the generated DMG file that is transfered.

Default value: files.dmg
Valid values: String value

2.e GUI Behaviours and Presentation

Parameters provide multiple options to customize or modify existing behaviour of the applet.

GUI Behaviour and Presentation
width

the overall width of the applet itself in pixels (button size may vary)

Default value: 500

In FileCatalyst Webmail and Workflow, this parameter is set automatically

height

the overall height of the applet itself in pixels (button size may vary)

Default value: 300

In FileCatalyst Webmail and Workflow, this parameter is set automatically

background

Changes the background color of the applet.

Example:

  • "255,255,255" is white
  • "255,0,0" is red
  • "0,0,255" is blue
  • "0,0,0" is black

Default value: "255,255,255" (white)
Valid values: R,G,B notation in 0-255 decimal numeration (not HEX)

buttonTextColor

Changes the color of the applet text.

Example:

  • "255,255,255" is white
  • "255,0,0" is red
  • "0,0,255" is blue
  • "0,0,0" is black

Default value: "0,0,0" (white)
Valid values: R,G,B notation in 0-255 decimal numeration (not HEX)

buttonbackground

Changes the background color of the buttons.

Example:

  • "255,255,255" is white
  • "255,0,0" is red
  • "0,0,255" is blue
  • "0,0,0" is black

Default value: "220,220,220" (light grey)
Valid values: R,G,B notation in 0-255 decimal numeration (not HEX)

buttonTextColorOnMouseOver

Changes the text color of the buttons when the mouse is moving over.

Example:

  • "255,255,255" is white
  • "255,0,0" is red
  • "0,0,255" is blue
  • "0,0,0" is black

Default value: "0,0,0" (black)
Valid values: R,G,B notation in 0-255 decimal numeration (not HEX)

buttonColorOnMouseOver

Changes the button color when the mouse is moving over.

Example:

  • "255,255,255" is white
  • "255,0,0" is red
  • "0,0,255" is blue
  • "0,0,0" is black

Default value: "200,200,200" (grey)
Valid values: R,G,B notation in 0-255 decimal numeration (not HEX)

headerTextColor

Changes the header color.

Example:

  • "255,255,255" is white
  • "255,0,0" is red
  • "0,0,255" is blue
  • "0,0,0" is black

Default value: "0,0,0" (black)
Valid values: R,G,B notation in 0-255 decimal numeration (not HEX)

showDialogs

Toggle all GUI elements of the applet displayed. If set to false, the applet becomes hidden to the user. Can be used in conjunction of the JavaScript methods to produce a headless applet.

Default value: true
Valid values: true, false

embedProgress

Progress information will not be shown in a dialog but rather as an embedded part of the applet interface

Default value: false
Valid values: true, false

showPreview

A preview of each selected file will be shown on the applet GUI as files are browsed. This works well when the transfer content are images (ex: JPG, GIFF).

Default value: false
Valid values: true, false

hideLocal

The local browse pane will be hidden by default. Files can be transferred by using drag and drop.

Default value: false
Valid values: true, false

showBrowseSwitchButton

Toggles if the browse switch button is visible to the user. The button effectively sets the hideLocal functionality, so can be used in conjuction with hideLocal to lock the GUI for the user to either always browse local or never browse local (use drag and drop only).

Default value: true
Valid values: true, false

ProgBarGraphic

Progress bar icon available for logo placement.

Default value: ""
Valid values: Logo file name

showUploadPauseButton

If set to false, the Upload/Pause button will be hidden

Default value: true
Valid values: true, false

showStopCancelButton

If set to false, the Stop/Cancel button will be hidden

Default value: true
Valid values: true, false

showHelpButton

If set to false, the Help button will be hidden

Default value: true
Valid values: true, false

showRemoveFromQueueButton

If set to false, the "Remove from queue" (red X) button will be hidden

Default value: true Valid values: true, false

showAddFileToQueueButton

If false, hides the > button to add a file to the queue

Default value: true
Valid values: true, false

showAddAllFilesToQueueButton

If false, hides the >> button to add all files to the queue

Default value: true
Valid values: true, false

showRemFileFromQueueButton

If false, hides the < button to remove a file from the queue

Default value: true
Valid values: true, false

showRemAllFilesFromQueueButton

If false, hides the << button to remove all files from the queue

Default value: true
Valid values: true, false

allowExternalDragAndDrop

Toggles drag and drop functionality from the applet.

Default value: true
Valid values: true, false

labelPlay

Sets a caption for the Play/Upload button

Default value: ""
Valid values: String value

labelPause

Sets a caption for the Pause/cancel button

Default value: ""
Valid values: String value

labelSwitchToDDView

Sets a caption for the "Switch to Drag and Drop" button

Default value: ""
Valid values: String value

labelSwitchToBrowseView

Sets a caption for the "Switch to Browse" button

Default value: ""
Valid values: String value

labelRemoveFromQueue

Sets a caption for the "Remove from queue" button

Default value: ""
Valid values: String value

separatePauseCancel

Seperate out functionality of pause and cancel into seperate buttons. The double-rectangle pause will leave partial files on the system, while the square cancel button will clean up partially transfered files if possible.

Default value: false
Valid values: true, false

lookAndFeel

Select the look and feel for the applet.

Default value: Metal
Valid values: Metal, Basic

rememberBrowseLocation

successive calls to the browseLive() calls (javascript calls) will remember where the user went to when selecting files, and will reset the localDir value for each file selection. Default is to not remember and to always launch the file chooser based on the value configured in localDir.

Default value: false
Valid values: true | false

2.f PostURL and Browser Redirection

Parameters provide additional applet behaviour after a transfer has completed. This provide a side array of integration options into existing web portals.

PostURL and Browser Redirection
delimiter

Delimiter that will be used to separate the file paths in the post data

Semi-colon (Windows) or colon (Linux, OS X, Solaris) if not defined.

Default value: varies on platform

In FileCatalyst Webmail and Workflow, this parameter is set automatically

sendLogsToURL

Send applet logs as an HTTP Post to a specified URL.

Default value: ""
Valid values: Valid URL

In FileCatalyst Webmail and Workflow, this parameter is set automatically

postURL

URL where the applet performs the HTTP Post to send the list of uploaded files

Example of formatted data from the postURL, following transfer of 6 files from D drive:

d:\01-xlarge.mp3 (120 MB)
d:\tmp\smallfile_1.txt (10 KB)
d:\tmp\smallfile_2.txt (20 KB)
d:\tmp\smallfile_3.txt (30 KB)
d:\tmp\smallfile_4.txt (40 KB)
d:\tmp\smallfile_5.txt (50 KB)

The URL-encoded string before parsing and formatting (the raw POST) looks like this:

f=%2Fsmallfile_2.txt;%2Fsmallfile_3.txt;%2Fsmallfile_4.txt;
%2Fsmallfile_5.txt;%2Fsmallfile_1.txt;%2F01-xlarge.mp3&baseDir=D%3A%5C
&s=20720;31110;41440;51800;10360;121493091&totalSize=121648521
  • The “f=” variable signifies the start of the file list, with each “%2F” being a URL-encoded forward slash.
    (“/smallfile_2;/smallfile_3;/smallfile_4;/smallfile_5;/smallfile_1;/01-xlarge.mp3”)
  • After “&baseDir=” is the parent directory common to all files
    (“D:\”)
  • After “&s=” are the file sizes (in bytes) in the same order as file names:
    (“20720;31110;41440;51800;10360;121493091”)
  • After “&totalSize=” is the total size of the transfer, in bytes:
    (“121648521”)

Leave blank to disable postURL

Default value: ""
Valid values: Valid URL

In FileCatalyst Webmail and Workflow, this parameter is set automatically

filesParam

Descriptor indicating the beginning of the file list in the post URL.

Default value: f
Valid values: Valid String

In FileCatalyst Webmail and Workflow, this parameter is set automatically

addSkippedFilesToPost

Include files skipped due to incremental transfer being enabled

Default value: false
Valid values: true, false

In FileCatalyst Webmail and Workflow, this parameter is set automatically

autoRedirect

If set to false and callurlaftertransfer is set, the applet will only redirect when a button is clicked.

Default value: true
Valid values: true, false

In FileCatalyst Webmail and Workflow, this parameter is set automatically

callurlaftertransfer

applet will redirect to this URL after transfer is complete. The redirect URL also contains variables in a query string that may be used by the destination page.

Example of redirect: "http://localhost"
http://localhost/?totalSize=103600&duration=1&m=UDP

  • Total size of transfer apppended to URL (bytes):
    font size="2">?totalSize=103600
  • Transfer duration in seconds:
    &duration=1
  • Transfer mode selected:
    &m=UDP

Leave blank to disable

Default value: ""
Valid values: Valid URL

In FileCatalyst Webmail and Workflow, this parameter is set automatically

callurlaftertransfertarget

If redirecting, applet will redirect the target window

Default value: "_self"
Valid values: String value

In FileCatalyst Webmail and Workflow, this parameter is set automatically

transfererrorurl

Applet will redirect to this URL if a transfer error occurred

Default value: ""
Valid values: Valid URL

In FileCatalyst Webmail and Workflow, this parameter is set automatically

transfererrorurltarget

If redirecting, applet will redirect the target window

Default value: "_self"
Valid values: String value

In FileCatalyst Webmail and Workflow, this parameter is set automatically

transfercancelurl

Applet will redirect to this URL if a transfer cancel occurred

Default value: ""
Valid values: Valid URL

In FileCatalyst Webmail and Workflow, this parameter is set automatically

transfercancelurltarget

If redirecting, applet will redirect the target window

Default value: "_self"
Valid values: String value

In FileCatalyst Webmail and Workflow, this parameter is set automatically

transferpauseurl

Applet will redirect to this URL if a pause occurs

Default value: ""
Valid values: Valid URL

In FileCatalyst Webmail and Workflow, this parameter is set automatically

transferpauseurltarget

If redirecting, applet will redirect the target window

Default value: "_self"
Valid values: String value

In FileCatalyst Webmail and Workflow, this parameter is set automatically

othererrorurl

Applet will redirect to this URL if non connection or transfer related error occurs

Default value: ""
Valid values: Valid URL

In FileCatalyst Webmail and Workflow, this parameter is set automatically

othererrorurltarget

If redirecting, applet will redirect the target window

Default value: "_self"
Valid values: String value

In FileCatalyst Webmail and Workflow, this parameter is set automatically

callurlonload

Applet will execute this call once the applet has loaded. Useful for JavaScript integration, where errors are being seen by the integrating website due to calls to the applet before it has completely loaded.

An example can be seen in the live.html call, where a JavaScript function can be called to set a variable once loading is complete.

var callurlonload = "JavaScript:appletLoaded()";

Default value: ""
Valid values: Valid URL

callurlonloadtarget

Applet will redirect the target window

Default value: "_self"
Valid values: String value

2.g Debug Parameters

Parameters which provide additional debug information, and do not play a role with the behaviour or the look/feel of the application.

Debug Parameters
debug

Will output onto the Java console all parameters loaded up by the applet, including if a default value has been utilized or not. Can help debugging applets when parameters are not being set properly by the JavaScript. As this can cause excessive logging, it is recommended that it be turned off except when trying to troubleshoot applet issues.

Default value: false
Valid values: true, false

In FileCatalyst Webmail and Workflow, this parameter is set automatically

allParamsLoaded

Last variable parameter loaded in the JavaScript example page. If this value is set to true, but the Java console reports the variable as false, there is likely an error in the JavaScript which is preventing all params to be proprely loaded by the application.

Default value: true

3 Performance Tuning

Tuning your FileCatalyst Software for performance requires knowledge of your system, network and usage scenario.

There is no magic "optimize for all" button.

This section attempts to address a few common scenarios encountered by clients, and seeing where improvements can be made to allow greater throughput.

3.a Know Your System

Disk IO

Achieving high throughput transfers requires first and foremost that you have the physical ability to read/write to data at the speeds you are trying to transfer.

FileCatalyst HotFolder and Remote Server Admin now include features that allow you to test write speed of your system. Running a file IO test with those applications can help determine if your system itself can keep up with the amount of data being written. You may use the results to help tune FileCatalyst applets.

If the IO speed is not what you expected, verify the following:

  1. Ensure that the user home directory (or HotFolder location) is pointing to the correct path.
  2. If the mount point is a network drive (Samba, SAN), make sure that the networking infrastructure can support the capacity (i.e.: Gbps ISCSI mounted via a 100 Mbps switch)

Encryption and CPU Overhead

Certain features of the FileCatalyst Software are computationally intensive, and may bottleneck high speed transfers.

Encryption of the control channel alone does not impose significant overhead. However, setting the FileCatalyst Server to use encryption on the control + data channel does impose a performance (CPU) penalty.

A typical client or server machine can handle transfer speeds of 100 Mbps. However, hitting higher speeds (>500 Mbps) becomes difficult if the data channel is encrypted, with transfer starting to become CPU-bound. This may be verified by running an Activity Monitor (i.e. "Task Manage" on Windows or "top" on Linux) for your OS and checking the CPU usage while the transfer is occurring.

Depending on your workflow, a possible solution to overcome this would be to encrypting the data BEFORE transferring it over the wire, and decrypt it once the data has reached its destination.

Compressing files can also be CPU intensive. Under most circumstances, the gain in network throughput outweighs the CPU overhead to zip up files before transfer, especially when files are text based and can be highly compressed. However, compression should be avoided for most binary and media files, which are normally considered non-compressible. If compression AND AES on the data channel are utilized, high speed transfers are likely to be limited by the amount of computing power that can be assigned for each transfer.

3.b Know Your Bandwidth

FileCatalyst HotFolder has the ability to test the bandwidth between itself and the server. It is recommended that this test be run if you are unsure of the real network bandwidth. The results of the test may inform applet configuration.

If there is a network bottleneck between the client and the server, setting the bandwidth speed above the real network capacity may result in high level of dropped packets and slower overall throughput.

If the bandwidth is above 500 Mbps and the supporting disk IO can support these speeds, it is recommended that client application specify UDP sender streams to be 2 or greater. This allows more data to be pushed out of the system, and provides higher maximum speeds.

3.c Know Your Network Infrastructure

Once the bandwidth is known, other settings can now be adjusted to accommodate your specific network environment.

Congestion Control: Play nice with TCP, or give it the boot

Our UDP protocol has the ability to be very friendly to other traffic sharing the network resource. It also has the ability to push out any other traffic and try to attain maximum line speed at all costs. Both of these are valid scenarios, and it is important to understand the impact this can have on your transfers.

  1. For typical shared land-line networks (where other traffic is also present), UDP transfers should enable congestion control. This allows the protocol to react to packet loss (possibly due to other TCP traffic) in a friendly manner, and slow down the rate of transfer to allow multiple traffic sources to share the line.
  2. By default, the congestion control value is set to 2 (low aggressive rate), which reacts fairly quickly. When other traffic is detected, transfer rates should come down quickly and play nice. The start rate should also be set to be the minimum speed which you know the line is capable of handling.
  3. For dedicated land-line networks, customers often turn on the congestion control, but set the aggression much higher (5, or halfway), with the start rate at half the known link speed. This allows the file transfer to ramp up quickly, and maintain high rates should other traffic try to share the line. The higher the aggressive rate, the more the protocol will kick out any other TCP/UDP traffic on the line.
  4. For improved line speed, set the network upload/download to known link capacity, minimizing the software from exceeding the network and causing continuous peak/valley from the congestion control.
  5. For maximum line speed, disabling congestion control removes most overhead from the UDP protocol. However, if the software exceeds the link capacity, any dropped packets due to network slowdown are amplified, increasing the risk of all sender threads entering retransmission mode increases (slows down transfer rate temporarily).
  6. For satellite transfers, packet loss could simply be caused by an object passing quickly between the dish and the satellite, or momentary electrical/solar interference. For these types of connections, customers often turn off congestion control altogether, preferring to power through any temporary loss of signal and not risk slowing down the connection.

Compensate for Packet Loss

Performance can be further optimized if you know your levels of packet loss.

The FileCatalyst protocol performs error correction should some packets not arrive on first attempt. However, retransmission of data blocks is much slower and latency-sensitive than the original sending of data. To compensate for this slower retransmission period, our algorithm employs multiple sender threads to send out the data, increasing probability that at least one sender is able to send data at full rate while the other threads clean up any missing packets.

For high packet loss, it is recommended that more threads be deployed, each with a smaller block size. This greatly decreases the possibility that one particular transfer finds all its sender threads in "retransmission" mode, slowing the throughput down.

Compensate for High Latency

To compensate for high latency, it is recommended that block sizes (the amount of data each sender thread can manage at one time) be increased, or that the amount of threads be increased. Each of these allows more data to be in-flight, before a callback is initiated to see if packets must be resent.

There is a simple equation to know the minimum values required to saturate a particular line:

RTT * bandwidth <= numSenderThreads * blocksize

i.e. RTT is 250ms, and your line speed is 100000 Kbps. numSenderThreads * blocksize should be greater than 25000000 bytes, or roughly 25 MB. Setting numSenderThreads set to 3 (default), you need a minimal block size of 8333333 to keep the line always full, or a little over 8MB.

To compensate for potential packet loss, it is recommended that the thread value be increased by at least 1 thread to ensure that one sender thread is always transmitting a new block of data.

Increase OS TCP/UDP Buffers and Window Sizes

Most operating systems allow the administrator to configure TCP/UDP parameters. Increasing the number of packets that may be in-flight may result in fewer lost packets, thereby increasing performance.

See http://www.29west.com/docs/THPM/udp-buffer-sizing.html, sections 8.7, 8.8, 8.9

See knowledge base article: http://support.filecatalyst.com/index.php?_m=knowledge base&_a=viewarticle&kbarticleid=17

NOTE: This is considered a requirement for any transfers higher than 150-200 Mbps.

3.d Know Your Data

FileCatalyst products supports many tools to decrease the amount of data that is required to be sent across the network. The exact combination of features to utilize and increase file transfer performance depends on the nature of the data being sent across the network.

Archive Small Files Together

Each individual file transfer, regardless if the file is 50 GB or 1 kb, requires overhead to setup a network connection between the client and the server. For large files, this overhead has minimal impact, as the software will spend most of the time transferring data across the wire. For transfers of small individual files however, the reverse is true: the software can spend most of the time setting up and tearing down a connection, only to transfer a directory full of many small files.

It is recommended that if transfers consists of many small files, that compression be turned on and that the ‘compress files into single archive’ option be enabled. This can significantly increase performance, as now the application will spend all its time transferring data instead of managing ports and network connections. Enabling progressive transfers will also allow the application to start transferring even as the single archive is being built.

There are caveats to single archive transfers to keep in mind. Should a transfer be interrupted, auto-resume will does not work, and restarting the transfer will start the archive/transfer from the beginning. Also, delta file transfers (incremental) does not work.

It is therefore recommended to enable the "Max size" feature for the single archives. This will create several smaller archives and extract them as they are transferred. If the transfer is interrupted, generally some of the file will have already been transferred.

Compress Text Files

FileCatalyst clients allow files to be compressed before being sent out. Transferring of text files (logs, emails, etc) should be compressed before being transferred. The CPU overhead of compression is negligible compared to the bandwidth savings of compressing an ASCII based file. It is recommended that progressive transfers be set when compression is used. This allows the file to be sent over even as it is being compressed, reducing the overall transfer time.

Media files (audio, video, images) however do not compress easily, and often add intensive computing overhead to a transfer while giving no bandwidth savings. Ensure that non-compressible files are filtered out by using the file compression filter.

Delta File Transfer

If the same file is sent out more than once (perhaps with only a small modification to the file), enabling incremental delta transfers will significantly reduce the bandwidth involved in sending a file.

For large files, it is also recommended to turn on progressive transfers. This allows the file deltas to be transferred. even as they are being built on the server, reducing the time it takes for the file to be transferred.

If the same file contains entirely new data each time (example: logs truncated on daily basis, and uploaded using the same file name), it is recommended to disable incremental transfers and simply start to transfer the file as new (avoid the computation required to determine what is different in the file, as all of it has changed).

MD5 on-the-fly

MD5 calculations are able to be conducted as each chunk of data is being received. This allows the application to perform calculations in parallel with the data transfer, and skips the otherwise long MD5 done at the end of the file transfer.

3.e Know the Client Endpoint

Low bandwidth clients

For clients connecting with low bandwidth, ensure that the congestion control start rate is set below the maximum line speed. By default, the value for the start rate is 384kbps, or the speed of a slower residential DSL/Cable connection. If the congestion control rate is still higher than available bandwidth, the application will quickly flood the network connection, producing lost/dropped packets and forcing the sender threads to perform most of its transfer in retransmission mode (slower).

For very low bandwidth clients and low latency transfers, FTP mode can at times perform faster transfer rates than UDP.

Using our on-line comparison calculator, you can see if there is an advantage to using FTP rather then UDP. Note, however, that the values in the on-line calculator do not take high packet loss into account. TCP at high latency is extremely sensitive to even 0.1% packet loss, while our UDP protocol handles this much more gracefully.

High Load Servers (or lots of clients)

Transferring UDP at high speed is memory intensive. Using the example listed above, a 250ms transfer (LA to Hong Kong) at 100 000 Kbps requires a minimum of 25MB of data to be in-flight at any one time.

For a standard install of FileCatalyst Server the default memory allowed is 1GB. This means the system can likely accept around 40 concurrent connections at full speed before starting to run short of memory resources.

The key to getting the FileCatalyst Server to accept higher numbers of concurrent connections is to lower the memory footprint each connection can take on the system. The server has an option via the Server Remote Administration to optimize for concurrency. This will lower the maximum thread count and block size each client connection consumes, resulting in the server able to accept more clients at any one time.

If the maximum sender thread and maximum block size is set too low, limiting clients from connecting at full bandwidth, the following may be done:

Limiting Concurrent Connections

The FileCatalyst Server allows setting the maximum concurrent user connections to levels below what the license allows. This allows administrators the possibility of setting the memory footprint high for each client, but limiting the connections to ensure that the server does not itself start to run short of memory.

Increase Java Heap Size

The optimizations performed for high-load servers are based upon your existing available JAVA heap size. Increasing the memory footprint JAVA is allowed to consume will increase the number of threads or block sizes each transfer may consume.

Windows Service

For FileCatalyst Servers running as a service, these may be configured in fcconf.conf:

# Java Additional Parameters
#original: wrapper.java.additional.1=-XX:MaxDirectMemorySize=1G
wrapper.java.additional.1=-XX:MaxDirectMemorySize=2G

# Initial Java Heap Size (in MB)
#original: wrapper.java.initmemory=1024
wrapper.java.initmemory=2048

# Maximum Java Heap Size (in MB)
#original: wrapper.java.maxmemory=1024
wrapper.java.maxmemory=2048

Note: in 32-bit Java, can increase to 2GB. You cannot currently modify the heap size in the Windows executable found in the start menu.

Linux Command Line

For FileCatalyst Servers started at the command line, the starting scripts may be modified to use alternative memory sizes than the standard 1GB:

fc_server.sh

#original: java -Xms64M -Xmx1024M -XX:MaxDirectMemorySize=512M -jar FileCatalystServer.jar
-controlpanel & java -Xms256M -Xmx2048M -XX:MaxDirectMemorySize=1024M -jar FileCatalystServer.jar -controlpanel &
Linux Service

For FileCatalyst Servers running as a service in Linux, these may be set in <application directory>/conf/wrapper.conf

# Java Additional Parameters
#original: wrapper.java.additional.1=-XX:MaxDirectMemorySize=1G
wrapper.java.additional.1=-XX:MaxDirectMemorySize=2G

# Initial Java Heap Size (in MB)
#original: wrapper.java.initmemory=1024
wrapper.java.initmemory=2048

# Maximum Java Heap Size (in MB)
#original: wrapper.java.maxmemory=1024
wrapper.java.maxmemory=2048

NOTE: in 32-bit Java, can increase to 2GB; in 64-bit Java, can increase beyond 2GB

4 Firewall Considerations

4.a Configuring for the Control Connection

In all but the strictest of environments, the client side's firewall does NOT need to be configured. If your environment has outgoing TCP blocked (extremely rare), the IT department will need to allow outgoing TCP connections on port 21, or whatever other port has been set as the server listen port in the server configuration. You can test connection by clicking the "Test settings" button.

4.b Configuring for the Data Connections

Again, client-side firewall configuration is rare. In the "Transfer Options" tab of the Task Edit view, incoming UDP is set to port "0", which is an alias for "passive". In other words, HotFolder will tell FileCatalyst Server which port to use in order to make the data connection. The most common scenario requiring firewall configuration is UDP-based download. If the client-side is failing on UDP download, the next step is to specify a data port. In the "Transfer Options" tab, specify an arbitrary available port.

5 Regular Expressions

FileCatalyst utilizes Java regular expressions, allowing you to build complex file filters to select which file you want to include in a transfer.

For additonal documentation to how Java implements regular expressions, the following sources may be used:

  • http://java.sun.com/j2se/1.4.2/docs/api/java/util/regex/Pattern.html
  • http://www.sitepoint.com/article/java-regex-api-explained/

Example Usage

  1. Transfer only files that end with ".dat". Windows wildcard equivalent of "*.dat"
    .*\.dat$
    
    Explanation:
    .* --> any character, 0, 1, or multiple times
    \. --> followed by a period (you must escape dot character, or it will be treated as a wild card)
    dat --> the exact characters dat all lowercase
    $ --> must be at end of line
  2. Transfer sound files (.mp3 or .wav) with numbers in the title
    .*[0-9]+.*\.(mp3|wav)$
    
    Explanation:
    .* --> any character, 0, 1, or multiple times
    [0-9]+ --> is any numerical digit one or more times
    .* --> any character, 0, 1, or multiple times
    \. --> followed by a period (you must escape dot character, or it will be treated as a wild card)
    (mp3|wav) --> last characters may match exactly "mp3" OR "wav" -- all lowercase
    $ -->; must be at end of line
  3. Transfer everything except temporary files ending in ".tmp"
    .*\.[^(tmp)]$
    
    Explanation:
    .* --> any character, 0, 1, or multiple times
    \. --> followed by a period (you must escape dot character, or it will be treated as a wild card)
    [ --> Start of a block
    ^(tmp) --> exclude files with "tmp" in the name
    ] --> end the block
    $ --> must be at end of line

Note: Because JavaScript will try to interpret some characters, the following actions must be taken into consideration:

  • You must double up backspace characters (JavaScript inteprets a single backspace as an escape character)
  • convert any "+" signs as "%2B", as they will be removed by JavaScript (into spaces)
  • convert any "%" into "%25", as the JavaScript may not start up

6 Support

Browse our Knowledgebase for existing questions and solutions

Submit a Ticket for best response-time and proper tracking of individual issues.