As of FileCatalyst Workflow 4.7, you no longer need to bother your Admins for this and your Admins no longer need to take time to complete these requests.

All the Admin has to do is create a new Field in the FileCatalyst Workflow Solution called “ftpservers”. Add the list of FC/FTP Servers they wish to be available for selection by end users.

A user may then log into the Workflow application and as part of the new Job submission process, they can choose the server where the files will go.

Practice One Way

You need to have a plan. You need to know when you are able to release. What's the pass rate of your test cases? How many blocking issues do you have? Are you doing Agile Development or Waterfall? Do you care about metrics before you ship? These and many other questions are important, but you need to pick something and go with it. Use a process EVERY time in a consistent way. Don't just make stuff up as you go along. Have a plan. Know what you are doing.

Slow Down Your Swing

I know. You're in a rush and the software needs to get out the door…now, now, NOW. Who cares if it's Christmas tomorrow; customers need that update under the tree! Really? Is this really true? Probably not…but come hell or high water it's crunch time. No one can get away from the "Quality Triangle": Quality, Cost, or Speed: pick two. I believe slowing down may actually be a better choice in the end, despite what the bean counters say. A good process can be slower, but at least you know what direction you’re going and how to get there. Software Development should look effortless to an outside observer. Be an Ernie Els, not a Charles Barkley. Just sayin'.

Practice, Practice, Practice

Fancy talk for training. You're not going to get anywhere if your process is not used—and used by everyone—a lot. Have tools in place. Have workflows in place. Make people use them. Have dry runs for new hires…heck, have them for experienced people as well. Use this time to think things through. I know you're not going to follow #2 (or more accurately, you’ll be forced to ignore #2). Under the pressure of a release, all your practice and preparation will pay off. You'll just know what do to because you've built up the muscle memory by practicing the process.

Get Good Equipment

I once worked for a company that used Excel spreadsheets to track test results. Don't do this. You don't see Tiger Woods hitting the links with 14 bread sticks in his bag, so why on earth would you use a spreadsheet for tracking test results or bug reports. Get the right tools. Maybe—just maybe—the reason you don't hit the deadline all the time is because you have the wrong set of tools. Maybe the reason you're stuck in the office on Christmas Eve is because it takes a week to extract test results for the CEO's monthly report when it should take 5 minutes. There are 3rd party vendors out there creating ways to do it better than you. Tools. Get them. Your designers and testers will love you for it, and so will your nerves.

Develop a Routine

What's the cadence of your development cycle? Do you even have one or are you inconsistent with your releases? Can you visualize what happens in your organization the minute you have software requirements in your hand? This is about getting into a comfort zone when the development process starts. Much like your routine of arriving at the course, hitting some balls, and approaching the first tee, get a routine going so that everyone gets "in the zone" the moment the project starts.

Final Thoughts

If you're serious about Software Development, forget about meeting deadlines. I know…an absolutely shocking thing to say, but you're putting the cart before the horse if all you think about is a release date. Much like Golf, consistency is the key and without it you'll always find yourself slicing.

Properties of REST

Addressability

• An application is addressable if it exposes the interesting aspects of its data set as resources.
• An addressable application exposes a URI for every piece of information it might conceivably serve.

Statelessness

To eliminate state from a protocol is to eliminate a lot of failure conditions. The server never has to worry about the client timing out, because no interaction lasts longer than a single request. Clients never have to worry about performing actions in the wrong order, because of some state kept on the server. A RESTful service is "stateless" if the server never stores any application state. Hence, a stateless application is easy to scale and cache.

Application State

• Information about the path the client has taken through the application
• Lives on the client and is client specific. A web service only needs to care about a client's application state when actually making a request. Since this information is client specific, it needs to be included in the request.
• example: application or API key that is required for every web service request.

Resource State

• Resource state for all clients which lives and stays on the server.
• Sent to the client in the form of representations.

Connectedness

RESTful services representations are hypermedia documents. These are documents that contain not just data, but links to other resources. The server guides the client's path by serving “hypermedia”: links and forms inside hypertext representations. The server sends the client guidelines about which states are near the current one. The quality of having links is called “connectedness”. Resources should link to each other in their representations. Hence, why the human web is easy to use because it is well connected.

For instance, Amazon S3 is a RESTful web service that is addressable and stateless, but not connected. S3 never includes URIs.

Uniform Interface

A resource is anything that's important enough to be referenced by itself. “Resource” is about as vague as “thing”, so any kind of data or algorithm you want to expose can be a resource. If you might want to make a hyperlink to it then you should make it a resource. There are three types of resources:

• Predefined one-off resources, such as the service's home page.
• A large (possibly infinite) number of resources corresponding to individual items of data of the same type (database rows, objects in object-oriented systems).
• A large (probably infinite) number of resources corresponding to the possible output of an algorithm. For instance, the list from a search engine.

The URI is simply the name or address of a resource. If a piece of information does not have a URI, it is not a resource. Every URI designates exactly one resource. URI's should be descriptive and structured and by definition no two resources can be the same. If they were the same, you would only have one resource. However, at some moment in time two different resources may point to the same data. A resource may have one URI or many. URI are supposed to designate resources, not operations on the resources. This means it is never appropriate to put the names of operations in the URI. If you have an operation in the URI, this is probably indicative of an RPC-style service.

Hypermedia and RESTful Services

For the most part the Addressability, Statelessness, and Uniform Interface properties of most REST services are adhered to. It is the “connectedness” property that is the most abused.

The hypermedia part of RESTful Services is what contains the “connectedness” information for resources on the Web. On the Web, the life cycle of a single resource is more than the creation, updating, reading, or deleting of that resource. Metadata about the resource, the subset of the verbs it understands, and information about other resources we might want to interact with can also be part of the state returned with that resource. Links or “connectedness” is good in Web-based systems and yet it seems to be the most forgotten aspect of many RESTful services. These links act as state transitions and the application conversation is captured in terms of these states.

Basically one is describing programmatic contracts with the defined links. For instance, links on a normal Web page (accessed via a browser) constitute a contract for page traversals. The same is true of the programmatic Web. One can use the links to describe state transitions in programmatic Web services. By navigating these resources you change the application state by following those links. This is the definition of a state machine! These same links can lead to other resources which also have links, and so on.

Why are links important? Links describes Protocols. Links declare next valid steps. This is HATEOAS (Hypermedia as the Engine of Application State)!

Conclusion

Although REST services which do not offer links to interesting and related information can still be very useful, they, in my opinion are not 100% RESTful. Hence, do not forget to include links to your response payload and you will fulfill the “connectedness” REST principle and become 100% REST compliant.

10 Gbps transfers

We knew going into the show that we offer the only product capable of reaching 10 Gbps file transfer speeds for a single file. Other companies are reaching a sustainable speed of only 1/3 that, peaking at 1/2 only with multiple files. But knowing this didn't prepare us for the number of people who came to our booth specifically to ask about 10 Gbps transfers. After all, that is a huge pipe not necessarily available to even the world's largest enterprises. But the questions came; from non-technical stakeholders who just want to know they can transfer a single multi-terabyte file at high speed, all the way to CTOs with engineering backgrounds who had some great questions about the core protocol.

Suffice it to say, there is a need out there for 10 Gbps transfers, and we were delighted to be the only company able to provide it.

Mobile file transfer

We trailed a little bit (just a little!) behind some of our competitors in bringing a mobile app to the market. But that's OK, because we introduced a great one that caught people's attention. People working “in the field” have uploading as their primary use case (for example, journalists sending mobile-captured images and video for a breaking story) and we provide a super-streamlined experience. Visitors to the NAB booth sometimes had “do you have anything for mobile?” as their first question, but almost all of them had it at least as a “nice to have”.

That picture will probably change as cellular networks become even faster and see wider deployment of high-speed connectivity. One visitor to the booth noted, "My cell connection is faster than my home connection." This trend will have an obvious impact on the importance of fast file transfer from mobile devices, and FileCatalyst is right there to provide a solution.

On the mobile front—and again taking us a bit by surprise—at least half of the people who were seriously interested in mobile were also interested in what we have to offer in our SDK. While the app may be iOS-only (for now), the SDK contains APIs for both Objective-C and Android (for the Dalvik VM). For either of the world's two most popular smartphone/tablet platforms, you can already integrate FileCatalyst transfers within a mobile app, or create a new one that meets your needs.

Face-to-face Time

One of the great things about being at the NAB is the chance to meet face to face with existing clients. We get to share first-hand our upcoming and new features (expiring packages for Webmail and Workflow were a welcome new addition!) and more importantly we get to re-establish what brought us together in the first place. It's absolutely amazing to see what companies like Aframe are doing to bring a great experience to their users. And it's so motivating to see the genuine enthusiasm companies like Aberdeen Captioning have for what we do.

Most companies exhibit at tradeshows to promote new products or meet new prospective buyers, and that's true for FileCatalyst as well. But on a personal level, the most energizing part for me is the time we get to spend talking with existing clients.

Until the next NAB

It's a done deal that we'll be exhibiting at NAB 2013. But that's a year away! In the meantime, if you would like to meet with us in person, we have a few upcoming shows finalized:

• Interop Las Vegas: This one's coming right up, from May 8–10. You can schedule a meeting and get 25% off a conference pass (or a free Expo pass!) by visiting our FileCatalyst at Interop page.
• CANSEC: Also coming right up (May 30 & 31) in our HQ hometown, CANSEC is a defence and security showcase. Obviously, our main topic of conversation there will be security in data transfer. (request meeting)
• Interop New York: Later in the year we will be at the smaller of the two Interop shows, held in New York. The perfect opportunity for companies with easier access to New York to stop by and see what we have to offer! (request meeting)

With the release of File Catalyst Direct 3.0.1, Central will provide the ability to enable external access to its database of transfer information.

First, a little basic information: Central uses a Derby database to store its data. This means that whatever tool you are planning on using to access the database will need to be able to interact with Derby. In order to do this, the tool must be able to use the Apache Derby Client driver. My preferred client is a free Java based tool called SQuirrL SQL Client which includes the Apache drivers by default. This is the tool that will be used in this how-to. However, you can use any tool that supports the driver. You will just need to adjust your set up as appropriate for your chosen tool.

Connecting SQuirreL to FileCatalyst Central DB

The first step is to configure Central to provide access. Currently the configuration UI of Central does not allow for configuration of the database access, so we need to make the changes directly to the MAConfig.conf file which can be found in the install directory. In order to make these changes, it is best to shut down the Central service.

Once Central is shut down, open MAConfig.conf in a text editor and scroll down to the bottom. You are looking for the parameter:

• FCMonitoringAgent.config.allow.external.database.access This parameter determines whether or not the database will be accessible or not. If true, it is. If false it is not.

There are four other parameters that may also be set:

• FCMonitoringAgent.config.derby.drda.host This value is the address that remote hosts must use to connect to the database. It defaults to “localhost” if no value is set, preventing any non-local access to the database. This value can be any address that points to the central server. If you wish to allow anyone access, as long as their address points to the correct machine, the value must be set to “0.0.0.0”. In this case, the network server will listen on all interfaces.
• FCMonitoringAgent.config.derby.drda.portNumber This value is the port which clients use to connect to the server. The default value is “1527”. This is the standard port for Derby databases. You can use any port number here that is available on the machine.
• FCMonitoringAgent.config.external.database.username This value is the username used for remote access. The default value is “username”. Because this value is loaded in from the configuration file and is not encoded in any way, it is limited in the characters that it can accept. It is recommended that you use a simple string of alphanumeric values. When this information can be modified from the UI, it will be encoded and thus able to be a more complex string.
• FCMonitoringAgent.config.external.database.password This value obviously sets the password for the remote access and defaults to “password”. It has all the same issues as the username.

As with any Central configuration value, if the values are not present in maconfig.conf, they will be set to the defaults. Once you have your values set, you can restart Central and move to the next step in connecting to the Central database: configuring your client. For this article I will be using SQuirrL as the client, but the basics should be similar for any client that supports the Derby driver.

In SQuirrL we want to add an Alias. This is what the new alias dialog looks like:


As you can see, most of the values are pretty straight forward. The URL, however is a little complicated because it contains most of the information that is used to connect to the database.

jdbc:derby://<server>[:<port>]/<databaseName>[;<URL attribute>=<value>]

Broken down, it is less complicated than it looks. The part at the end between the square brackets we can ignore as it is just for additional attributes that we don’t use. The only values we need are the , and . The square brackets around the port value indicate that it is optional. If you leave it out, the connection will use the Derby default port of 1527. Do not use square brackets in the URL.

Here's one I prepared earlier:


As you can see, once the values are filled in, it doesn’t look nearly as complicated. Our server is set to localhost, our port to the default of 1527 and finally our database to the name of our database: MONITOR_DB. The Name at the top is just for recognition purposes, it can be anything you like.

The username and password should be the values that we configured earlier. In this case I am using the default values. I like using the auto logon feature of SQuirrL so I have selected that but I like to control when the connection happens so I did not check the Connect at Startup option.

If you like you can test your connection and then once it is connecting successfully, you can hit OK. At this point you should see the name you entered in the aliases list:


Double click on the alias and SQuirrL will open and Auto Logon to the database. This opens a tab that shows the contents of the database.



The only branch on the tree that we need to concern ourselves with is the MONITOR_DB branch. As mentioned above, this is the name of our database and this branch contains all of the tables with our data under TABLE. However, before we are able to access our data, we have one more step.

In Derby, the database assumes that you want to connect to a schema within the database that is the same as the username you have used to connect. In this case, it assumes that we want to connect to a schema within the database named USERNAME as that’s the username we used. Not only is this not the schema we want, it doesn’t even exist. While we can open the schema with the above viewer and examine the contents, we do not have the ability to make queries which is likely the reason you wanted to connect to the database in the first place.

In order to access the correct schema we need to switch to the SQL tab and enter the command:

SET SCHEMA MONITOR_DB

After we have entered this command we can now retrieve data from the database. Note that the external user is read only so only selects may be done.

Here is our current Schema:


]]> http://www.filecatalyst.com/connecting-to-the-filecatalyst-central-database/feed/ 0 http://www.filecatalyst.com/congestion-control-in-filecatalyst/ http://www.filecatalyst.com/congestion-control-in-filecatalyst/#comments Tue, 28 Feb 2012 19:45:06 +0000 Chris http://www.filecatalyst.com/?p=2843 Our customers often ask about our “congestion control” feature: what exactly is congestion control, why is it needed and how does it work in the context of FileCatalyst? Not easy questions to answer without getting overly technical, but this article will attempt to give high-level answers.

Congestion Control: General Overview

On an IP network such as the internet, there are millions of data streams that must co-exist at any given time. Those streams must not conflict with one another or the internet would come to a grinding halt and all applications that require internet access would cease to function properly. The reason these connections are able to coexist is because of congestion control.

Congestion control is a mechanism that will increase or decrease the speed of data transmission based on network conditions. Network conditions are monitored during a transmission, and if congestion is detected the transmission rate is decreased; otherwise it is increased. That's the general idea. However, the way congestion is detected, as well as how much the rate is increased or decreased is where different congestion control mechanisms differ.

Detecting Congestion

How is congestion detected? Data packets run through a series of routers along their path to their final destination. Each router is responsible for handling several unique data streams, and directing the packets from each stream to the proper destination. Packets arriving at the router awaiting processing are stored in queues or buffers. These queues are generally processed using a FIFO (first in first out) policy. This means simply that they are processed in the order they arrive.

These queues are limited in size. When packets cannot be processed fast enough by the router, the queue grows until it is full. At this point any new packets will be discarded, causing packet loss. In addition, when queues are full, the packets already in the queue are delayed because of the large number of packets that will be processed first. This causes an increase in the RTT (round trip time), which is the time it takes for a transmitted packet to make it to a receiver and a reply to make it back to the transmitter. Monitoring either packet loss or RTT over the duration of a file transfer are viable means of detecting congestion.

Congestion control is built into TCP, the protocol that accounts for the majority of all traffic on the internet. TCP determines that there is congestion by detecting packet loss as described above. The protocol itself will slow down a stream when a packet is lost, and speed up again when no more loss is detected. Various algorithms do exist within TCP to avoid congestion (i.e. Tahoe, Reno, Vegas, etc...) but these are outside the scope of this post.

The FileCatalyst Approach to Congestion Control

In FileCatalyst, the UDP protocol is used for data transmission instead of TCP. It therefore doesn't have congestion control at the protocol level. For that reason, FileCatalyst has application level congestion control. There are 2 modes of congestion control used in FileCatalyst, one based on RTT and one based on packet loss.

RTT-based

The first congestion control algorithm used is based on RTT. This works by establishing a baseline average RTT before the data starts to flow. Once the transmission begins, RTT is monitored continuously. While the RTT stays within a certain range of the baseline RTT, the speed of the transfer will be increased. Once the RTT begins to go above a certain range, the speed is decreased. How much the RTT is allowed to spike above the baseline average is controlled by the Congestion Control aggression setting FileCatalyst provides.

This type of congestion control works well on wireless or satellite links where there is packet loss from other sources besides congestion. TCP will slow down, for example, when a packet is lost due to interference or being too far from a cellular tower. When FileCatalyst is using the RTT based congestion control it ignores individual packet losses and focuses only on RTT. For these scenarios, FileCatalyst continues to maintain high speeds through the kinds of packet loss “hiccups” that would trigger decreases speeds under TCP.

Loss-based

There are circumstances in which this RTT based may not work properly. For example, when a router's queue is very small, the RTT may never spike when there is congestion. The RTT will remain low, and therefore FileCatalyst would continue to increase its transmission rate even when there is congestion present. For these scenarios, the only way to detect the congestion is using a packet loss approach. As outlined previously, packet loss may come from other sources besides congestion, so this mode is best used on terrestrial or land based networks where all packet loss is due to real congestion.

Loss-based congestion control mode reacts to packet loss by slowing down (just like TCP); however, FileCatalyst is not nearly as aggressive as TCP. The primary reason for using a UDP-based file transfer solution like FileCatalyst is because TCP is so aggressive in its congestion avoidance that it often ends up under-utilizing your link. The FileCatalyst loss-based congestion control algorithm was designed such that is it able to maximize link utilization while still avoiding congestion. Like the RTT-based congestion control, the loss-based mode can be tuned to be more or less aggressive. More aggressive settings allow for a higher percentage of packet loss before slowing down, while for more passive settings the opposite is true.

Conclusion

There is certainly a need for congestion control in any protocol that transfers data across the internet. It is built into TCP, but TCP's implementation is conservative to the point that file transfers are usually crippled as a result. UDP without congestion control is potentially just an illusion of speed, with packets being “blasted” out, lost, and re-sent. FileCatalyst's UDP-based transfer uses two different methods of congestion detection, allowing for effective management of transfer rates. This helps prevent excessive packet loss while transferring files at maximum speed.

]]> http://www.filecatalyst.com/congestion-control-in-filecatalyst/feed/ 0 http://www.filecatalyst.com/making-file-transfer-easier-compliant-and-more-secure/ http://www.filecatalyst.com/making-file-transfer-easier-compliant-and-more-secure/#comments Tue, 14 Feb 2012 19:34:40 +0000 Greg http://www.filecatalyst.com/?p=2801 We just added a new whitepaper into our Datasheets and Collaterals page, entitled Making File Transfer Easier, Compliant, and More Secure.

Though we are sponsors, the paper is written by Osterman Research. What this means is simply that the contents are the product of Osterman. Being that we are in the business of file transfer, we found ourselves agreeing with the findings and opinions and took the sponsorship opportunity to let readers be aware of FileCatalyst and the solutions we offer to some of the problems discussed.

]]> http://www.filecatalyst.com/making-file-transfer-easier-compliant-and-more-secure/feed/ 0 http://www.filecatalyst.com/filecatalyst-javascript-api-in-browser-file-transfer/ http://www.filecatalyst.com/filecatalyst-javascript-api-in-browser-file-transfer/#comments Tue, 31 Jan 2012 19:32:04 +0000 Greg http://www.filecatalyst.com/?p=2763 Introduction

The FileCatalyst JavaScript API offers the ability to implement an accelerated file transfer application from within a web browser. Using the API, you are able to customize the end-user's file transfer experience, integrating the file transfer into your site (in terms of both functionality and UI) in a way not necessarily possible with the default configuration alone.

The Quick Version

If you just want to see how the API can be called, here's an example using jQuery:

$('#uploadButton').click(function() {
  document.FileCatalyst.uploadLive();
});

It's that easy. The API documentation gives a rundown of each of the methods available via JavaScript. Here, we have bound a click listener to some element with the ID "uploadButton". When that element is clicked, the JavaScript triggers the upload function of the applet.

Requirements

End user: JavaScript enabled on their browsers, Java installed on their machines (for the desktop environment, this is the vast majority of users)

Developer: 2 files that come with FileCatalyst; plus 2 custom components:

1. Included: The applet script (ex. fcupload.js)
2. Included: The applet JAR file (referenced by the applet script) (ex. FileCatalystApplets.jar)
3. Custom: A webpage that will load the applet and provide controls (applets ship with sample pages to use as a starting point)
4. Custom: Additional script(s)

Our Longer Example: Uploader with buttons

In this simple example, we have decided that we want to reduce the full-blown UI down to three key buttons: “Add to Queue”, “Upload”, and “Cancel”. In the following code samples, three important tasks are accomplished: 1. the applet is added to the page; 2. the applet's default UI is hidden; 3. our three buttons are hooked up using the FileCatalyst API and the jQuery JavaScript library.

The Webpage

Let's have a look at the webpage. In this case we're just using static HTML.

<html>
  <head>
    <title>FileCatalyst Upload with JavaScript API</title>
    <script src="http://code.jquery.com/jquery-1.7.1.min.js"></script> <!-- load jQuery from wherever you prefer -->
    <script src="uploadApp.js"></script> <!-- our custom script that handles button events etc. -->
  </head>
  <body>
    <div class="wrapper">
      <h1>Upload Sample</h1>
      <input id="browseButton" type=button class="fcbutton" value="Add to Queue" disabled="disabled"/>
      <input id="uploadButton" type=button class="fcbutton" value="Upload" disabled="disabled"/>
      <input id="cancelButton" type=button class="fcbutton" value="Cancel" disabled="disabled"/>
      <div id="fc_upload">
        <script src="fcupload.js"></script> <!-- The upload applet will load here -->
      </div>
    </div>
  </body>
</html>

In the <head> section, we have included jQuery (which we will use for event binding) and then our custom script (which will contain the event binding and any other JS functionality we may wish for). jQuery is possibly overkill if your page is literally as simple as this example; however, many people are already using jQuery on their site.

Buttons are simply <input> elements with type "button". The buttons also have the class "fcbutton" for convenience in styling and scripting. They are disabled by default, and will be enabled upon applet load. This has the added benefit of keeping the buttons disabled for users without JavaScript enabled on their browsers.

Adding the applet to any webpage couldn't be easier. Below the buttons we simply load the applet's JavaScript file with a <script> or <embed> element (depending on browser). Once loaded, the applet appears in the DOM inside an <applet> node.

Hiding the applet

Next, we need to hide the applet's default UI. You can do this with pure CSS using visibility: hidden, but NOT with display: none (which will not allow the applet to load). A safer bet is to use the height and width parameters inside fcupload.js, however. Setting these to zero will work in some but not all browsers. The safest bet is to make the applet tiny (1X1) and stick it somewhere on the page that's out of the way.

Inside fcupload.js, find these lines and modify as shown:

// GUI Behaviours and Presentation
var width                       = "1";
var height                      = "1";
var background                  = "255,255,255"; //RGB value for white. You can make it the same color as the element it sits inside

// (later in file)
var callurlonload               = "javascript:appletLoaded()"; // this will trigger the JS "appletloaded" function

For additional insurance, you might have some CSS that hides the applet or takes it out of the document flow:

#fc_upload {
  position: absolute; /* remove from flow */
  visibility: hidden; /* optionally make it completely invisible */
}

Hooking Up the Buttons / Custom Scripts

This is the meat of the whole exercise. We are using jQuery for its simple syntax, but you can use whatever event-binding method you prefer (even inline onclick attributes). All we do in the script is bind a listener to each button. When the button is clicked, a JavaScript function from the FileCatalyst API (now available through the included applet) is fired.

Inside uploadApp.js (or whatever you have named your file):

function appletLoaded() {
  $('.fcbutton').removeAttr('disabled'); // the applet has loaded, so enable the buttons
}

/* bind the click events when the DOM is ready for scripting */

$(document).ready(function() {

  $('#browseButton').click(function() {
    document.FileCatalyst.browseLive(true);
  });

  $('#uploadButton').click(function() {
    document.FileCatalyst.uploadLive();
  });

  $('#cancelButton').click(function() {
    document.FileCatalyst.setCancelled();
  });
});

The appletLoaded function (referenced in fcupload.js modifications) removes the "disabled" attribute, enabling the buttons. The event handler listens for clicks on the buttons, and fires off a method call to the applet, using the available JavaScript functions.

Conclusion

Using a Java applet puts FileCatalyst right in the browser, with no installation required. Full acceleration and security options, without the end-user needing to configure anything is a big win for any organization that uses the web (and what organizations don't?). Our own API sits under our powerful Workflow and Webmail applications, but if you do not need a fully realized web portal out of the box, or are integrating FileCatalyst within your existing portal, the JavaScript API offers virtually every option you might need.

Transfer hook functions such as getTimeRemaining(), and getFilesSoFar() (along with a whopping 42 other useful hooks) allow you to capture information from an ongoing session. For the upload applet, functions such as the ones used in this post's example (along with another 20) allow you to invoke applet methods through JavaScript.

The potential is huge. Included in the FileCatalyst Upload Applet package is a sample of a web application that uses AJAX to fetch page content; the applet is never unloaded/destroyed (the way it would be when navigating to an entirely separate page), so you can have a transfer persisting while the user continues to explore the site. The sample uses a number of transfer hooks and upload applet methods, so look to it as a more sophisticated sample than the one provided in this exercise.

]]> http://www.filecatalyst.com/filecatalyst-javascript-api-in-browser-file-transfer/feed/ 0 http://www.filecatalyst.com/analysis-on-improving-throughput-part-1-disk-io/ http://www.filecatalyst.com/analysis-on-improving-throughput-part-1-disk-io/#comments Tue, 17 Jan 2012 21:34:54 +0000 Christian http://www.filecatalyst.com/?p=2750 In light of the release of FileCatalyst Direct v3.0, I thought I'd write a few articles about the road to achieving 10Gbps speeds. It seems to me the best place to start is with the endpoint of the transfer: the storage media. Why? Before we even think about the file transfer protocol, we have to be sure that our disks can keep up. It's very easy for people to forget how much of a bottleneck disk read & write IO can be.

Disk IO Tests

10Gbps (1250MB/s ) is fast. Especially for a single file stream. By comparison:

• Your average enterprise HDD SATA drive will read/write between 50-100MB/s (or 800Mbps)
• Your average SSD SATA drive will read data at 300MB/s, and write at 200MB/s (1.6 Gbps-2.4 Gbps)
• FiberChannel SAN arrays normally connect between 500MB/s to 1000MB/s (4Gbps or 8Gbps)

Getting FileCatalyst to run 10Gbps speeds (1250 MB/s) requires testing your hardware infrastructure. Having a fiber SAN may not be enough.

For our internal test environment, this meant adding a few RAID 0 arrays (8 SSD drives or 16 HDD drives) in order to achieve the desired speeds. Certainly not mentioning this as a recommendation for a production solution, but it was the simplest way to achieve the speeds we needed for testing.

To determine what your system is capable of providing, FileCatalyst includes tools within FileCatalyst Server, HotFolder, and Command Line Client. The tools are command-line scripts designed to emulate how the FileCatalyst software manages Disk IO, and can give you a good approximation of your system's potential before a single byte has transferred over the network.

Running Enhanced Disk IO Tests

ReadIO and Write IO tests have a similar syntax to run them. ReadIO is executed by the executing the application switch "-testReadIO", while the write io uses the switch "-testIO". Since most issues are found with write speeds (which tend to be slower than read), let's focus on write IO.

The Write IO test creates a file on your file system, and reports back how quickly the file was created by the application. How it creates that file is determined by the parameters you must fill in before it runs:

<code>C:\Users\ccharette\Desktop\applettests\CL>java -jar FileCatalystCL.jar -testIO
Entering Write TestIO. This will run a series of tests on the file system to 
attempt to discover the optimal values for # of writer threads and write block 
size for your system.The test is both IO intensive and CPU intensive. Please 
give the test adequate time to complete.

Please enter the drive/path you wish to test (ie:  C:/ or /mnt/data/ ):  c:\temp
  File to be written:  c:\temp/test.io

Please enter the size of file you wish write (in MB, default 500):  10000
  File size:  10000MB.

Please enter the timeout length (secs) per run (default 60 secs): 180
  Timeout:  180 seconds.

Please enter the number of runs to perform for each settings (default 5): 3
  Number of runs per iteration:  3

Test if buffer size used in writes to disk affect performance.
Please enter a buffer array to attempt (default:  '64,128,256,512,1024,2048,4096')
Size in KB, comma delimited: 4,16,64,256,1024,4096,16384
  4 Buffer Size values (KB):  4,16,64,256,1024,4096,16384.

Test if multiple writters offer performance benefit when saving a block to disk.
Please enter a writer thread array to attempt (default:  '1,2,4,6,8'): 1,2
  2 Thread values:  1,2.

How many files would you like to create concurrently for IO test (default 1)?
Note:  number of files will never exceed number of writer threads during tests.
  1 files will be created in test:  1.

Test using Direct IO when allocating buffer space by Java (default true):  true
  Use DirectIO = true.

Mode used to open up files (rw/rws/rwd -- default rw):  rw
  Mode used = rw.</code>

Things to note:

• It is better to create a file large enough that it represents your total data set expected for the transfer (ie: 9GB DVD ISO), or larger than the amount of memory the OS can utilize for file system buffers (see results below).
• Timeout should represent a valid length of time you expect can easily be reached when writing a single copy of the file.
• Specify # of runs with the average posted.
• Buffer size represents an optional switch on the Server & Clients which dictates how large each read/write IO should be from the JAVA code down to the file system. You should experiment with a few values here, as different disk configurations sometimes yield vastly different optimal results.
• The Server and Clients support multiple read and write threads per file.
• Keep the # of files to create concurrently to 1 if you are testing only a single file tranfer speed (single client to server endpoint). This is actually one of the hardest test cases to manage, as there are OS level locks which often task the CPU, limiting throughput you can get when writing to a single file. If you are looking to test 10 clients each utilizing 1Gbps connection, select multiple files (much higher file IO possible when multiple files are being saved at a time).
• Always select Direct IO, since this is what the FileCatalyst application uses.
• Select default "rw" mode, which takes advantage of OS level memory buffers if available.

Results

Machine 1a: Windows 7, single SSD drive	Machine 1b: Windows 7, single HDD drive
Tests run with the following parameters: file: c:/temp/test.io size: 10000000000 timeout: 180000 directIO: true file mode: rw Max # files to use: 1 # of THREADS |1 |2 Buffer size +=======+=======+ 4 |240 |150 16 |237 |150 64 |238 |150 256 |236 |152 1024 |237 |152 4096 |236 |149 16384 |236 |196	Tests run with the following parameters: file: E:/tmp/test.io size: 10000000000 timeout: 300000 directIO: true file mode: rw Max # files to use: 1 # of THREADS |1 |2 Buffer size +=======+=======+ 4 |89 |52 16 |92 |53 64 |92 |53 256 |92 |53 1024 |95 |54 4096 |100 |55 16384 |104 |79
Machine 2a: Ubuntu RAID 0, 8 x SSD, 10GB file	Machine 2b: Ubuntu RAID 0, 8 x SSD, 60GB file
Tests run with the following parameters: file: /opt/tmp//test.io size: 10000000000 timeout: 60000 directIO: true file mode: rw Max # files to use: 1 # of THREADS |1 |2 Buffer size +=======+=======+ 4 |1431 |1282 16 |1722 |1614 64 |2059 |1748 256 |2239 |1933 1024 |2095 |2050 4096 |2078 |2048 16384 |1841 |1720	Tests run with the following parameters: file: /opt/tmp//test.io size: 60000000000 timeout: 180000 directIO: true file mode: rw Max # files to use: 1 # of THREADS |1 |2 Buffer size +=======+=======+ 4 |1093 |893 16 |1177 |1131 64 |1400 |1184 256 |1387 |1324 1024 |1402 |1303 4096 |1293 |1294 16384 |1140 |1271

Observations: Note that neither of these machines benefitted from multiple writer threads, and that performance was higher when a single writer was involved.

Machine 1a: When writing to SSD, we can get 230+ MB/s (>1.8Gbps) of write speed when using 1 thread. Block sizes do not affect throughput.

Machine 1b: Same machine, but utilizing slower secondary HDD drive. When using slower disks, the software can only get a fraction of the bandwidth (in this case < 1Gbps). We do see marginal improvements the larger the block size used, so limiting the block size is not a good idea. By default, the FileCatalyst application will use the largest block size it can (determined by UDP block size).

Machine 2a: Can read/write at 2000+MB/s (>16Gbps) for 10GB files. We can also see a sweet spot of ~256KB write block, where smaller writes adversely affect performance (as to larger blocks). This system however has 48GB of RAM on it, so the numbers it provides me are actually above what I would expect the system to give me.

Machine 2b: Same test, but with 60GB file. Now we have realistic numbers which match the disk IO, giving us a system capable of sustaining 1350MB/s (10.5Gbps) write speed.

Configuration Values

On the Server & Client side, the following configuration options are therefore available to set (CLI arguments shown):

• numBlockWriters [#]
• writeBufferSizeKB [# KB]
• writeFileMode [rw/rwd/rws]
• numBlockReaders [#]
• readBufferSizeKB [# KB]

This are machine specific settings. To maximize performance, you need to run tests on both endpoints (client + server). On the server (if client connections are going to do both upload and downloads), you should run both read and write tests.

For both the FileCatalyst Server and HotFolder, these settings are configuration file changes that must be manually set (fcconf.conf for server, fchf.conf on HotFolder). For the CLI, these may be passed in as run-time arguments.

Conclusion

Knowing the limits of your system IO is the first required step in achieving high speed transfers. FileCatalyst v3.0 provides several tools to help both benchmarks those limits and tune the application to best take advantage of your system.

]]> http://www.filecatalyst.com/analysis-on-improving-throughput-part-1-disk-io/feed/ 0