From Documentation
Jump to: navigation, search
Globus
Description: Compute Canada portal for intuitive, fast, secure and reliable data transfer.
SHARCNET Package information: see Globus software page in web portal
Full list of SHARCNET supported software


Introduction

The Compute Canada Globus instance is an intuitive portal that provides fast, secure and reliable data transfers with the goal of making big data portable. Most Compute Canada storage systems are defined as endpoints for transfers and user machines can be added as endpoints. Most SHARCNET file systems can be found via the computcanada#sharcnet-dtn1 endpoint. General information regarding the Compute Canada Globus implementation can be found at the Compute Canada Globus Wiki page, and more general information about Globus can be found at the Globus site.

Usage

Login and select Endpoints

Rather than using this utility from a SHARCNET system, Globus is accessed via its Compute Canada Portal. For up-to-date details and instructions for signing into the Compute Canada Globus Portal with your Compute Canada Data Base (CCDB) credentials visit the Compute Canada Globus Wiki.

Once that you are signed into the Globus Portal you will need to select and activate the computecanada#sharcnet-dtn1 Endpoint (using your SHARCNET credentials) to access your files on SHARCNET systems (See image below).

To access the storage on the new Compute Canada clusters, graham and cedar, select endpoints computecanada#cedar-dtn and computecanada#graham-dtn.

Ccglobus dtn1.png

For information about managing Globus Endpoints and transferring from a personal computer see the Personal Computers section of the Compute Canada Globus Wiki page.

Example Transfer

Once that two endpoints have been activated transferring files with the Globus Portal is simply a metter of selecting a file or folder from one end point and moving it to the selected Path at the other Endpoint. In the image below the user is sending the "batch" folder from the Path "/~/Downloads/batch_context-master" on their local computer ("jad_lem" Endpoint) to their scratch folder at Orca via the Data Transfer Node (Endpoint = "computecanada#sharcnet-dtn1" and Path ="/scratch/orca/scratch/jdesjard").

Ccglobus local to scratch.png

General Notes

computecanada#sharcnet-dtn1 Endpoint folder layout

By default when you activate your computecanada#sharcnet-dtn1 Enpoint the Path will point to your /home directory ("/~/" or /home/username"). This will provide access to your global access /home folder on SHARCNET systems. Similarly, in order to access your globa access /work folder you can change the computecanada#sharcnet-dtn1 Endpoint Path to "/work/username" (this also applies to the "/freezer" global access file system).

For the non-global /scratch file systems the computecanada#sharcnet-dtn1 Endpoint Path needs to specify the desired systems. Available /scratch Paths are:

  • /scratch/angel/scratch/username
  • /scratch/dusky/username
  • /scratch/goblin/username
  • /scratch/iqaluk/username
  • /scratch/monk/username
  • /scratch/orca/scratch/username
  • /scratch/redfin/username
  • /scratch/saw/scratch/username


TAR file efficiency

As with all data transfer methods it is important to recognize the difference in transfer rates between sending a few large files and sending many small files. For example, the difference between transferring 1TB in the form of a single file and transferring the same 1TB in the form of 100,000 small files can be substantial. If you intend on sending a large number of small files it is important to consider encapsulating the files using utilities such as TAR and DAR. This can substantially improve the completion time of your transfer and contribute to the overall performance of the resource.

References

Compute Canada Globus Portal

Globus main page.

Compute Canada Globus Wiki