7) Global File Transfer

Global File Transfer (GFT)

The Global File Transfer (GFT) plugin allows very large files to be transferred efficiently across long distances. You can expect significantly faster transfers vs standard SCP for those located outside the US and Canada. Utilizing a patented UDP-based file transfer technology, GFT overcomes the issue of slow file transfers caused by network impairments such as latency and packet loss. GFT will work on any internet connection.

To access GFT go to “Tools –> Global File Transfers…” from the Sabalcore ssClient menu.

Features of GFT

  • Designed to transfer gigabytes and terabytes of data over large distances (global)
  • Transfer large files hundreds of times faster than FTP over vast and varied global networks
  • Industry standard 256-bit AES encryption for secure file transfers
  • Checkpoint restart – resume transfer in the event the connection is interrupted

Prerequisites

Large files only. GFT is designed to transfer very large individual files. Small files (under 1/2 MB) will transfer slowly compared to standard SCP. Directories containing many small files must be zipped before transferring. The larger the file the better. See information about zipping files below.

Sufficient internet bandwidth. GFT could over take your local internet connection and affect other users on your network. Be sure you have adequate bandwidth on hand. See “Settings” for information about limiting the amount bandwidth GFT uses.

Firewalls

GFT should work without any modification to your PC’s local firewall. However, sometimes these ports could be blocked by a network firewall. Should you run into issues verify the ports are available. GFT requires the following ports:

  • Port 22 for TCP. This port is used for SSH connections which is used by the ssClient so it should already be open.
  • Port 7000 for TCP. This Port is secured over SSL and is also used as a communication channel.
  • Port range 6500-6999 for TCP and UDP. Both TCP and UDP need to be enabled. Note: Some firewalls and NAT devices require setting up 2 separate rules while others allow you to specify TCP and UDP. These ports are used to transfer the Data to and from.

Using GFT

To transfer a file select “Tools –> Global File Transfers…” from the Sabalcore ssClient menu. You will have to enter your sabalcore user ID and password at the start of each session. After a few moments, the directory structure will display. Note: this may take 5-30 seconds depending on how deep your file structure is and the number of files. Please wait until the directory structure is listed.

Zipping Files Prior to Transfer (required)

Small files must be archived(zipped) into one large file before transfer. It is not necessary to use compression. The compression process could take longer relative to the time it takes to perform the transfer.

PeaZip – Compressing and Zipping Files using PeaZip tool

PeaZip is a graphical interface to 7zip. PeaZIP is available to create archives.

  1. Start a Remote Graphics session.
  2. Select “PeaZip” from the Sabalcore launcher menu.
  3. The default node specification is fine for general archiving and compression. Adjust the number of cores based on your workload.
  4. If this is your first time using PeaZip, select the “Home” bookmark on the left to jump to your home directory.
  5. Find the folder(s) you wish to zip and select it.
  6. Select “Add” from the menu bar at the top.
  7. Under output, we recommend “ZIP”, “Fastest, Fast, or Normal”, and “Single volume”. There is usually not much benefit in using very high compression because the time it takes to compress the file can be used to transfer it instead.
  8. Click “OK”.
  9. When finished, be sure to exit the interactive job. Peazip will not exit the job when it’s finished. You must manually exit the job.
p7Zip – Compressing and Zipping Files using 7zip (command line)

7zip is command line only. We recommend contacting Sabalcore Support if your a unfamiliar. Support will create a zip.pbs template that you can easily modify for your needs.

First, start an interactive job with 6 or more cores. 7zip will utilize all the cores. Using more cores may speed up the compression. Load the 7zip module “module load p7zip”.

To compress a folder:

7za a my_archive.7z my_folder

Install 7zip on your PC. Use it to extract the archive.

You can use the “&& exit” option to automatically exit the interactive job when the zip command completes. For example:

7za a my_archive.7z my_folder && exit

GFT Options

GFT options can be found in the bottom right corner. Click “Options” button to open the options menu.

Option settings will be saved and loaded the next time GFT is used.

Auto resume partial transfers (enabled by default): GFT will auto resume partial transfers in the event there is a connection interruption.

Target Transfer Speed (in kB/s): Default is maximum (leave blank). GFT will transfer at the maximum available bandwidth. Enter a transfer speed in kB/s (kilo Bytes) to limit the amount of bandwidth GFT will use on your network. For example, to limit the transfer speed to 10Mbps, enter 1250. GFT will attempt to transfer at the target transfer speed.

GFT could over take your local internet connection and affect other users on your network. Set a target transfer speed to avoid saturating your network.

Preserve Last Modification Time: Default disabled. Enable to perverse the file’s last modification time.