(this page is still needed to provide instructions about accessing data on our legacy systems)
|Line 1:||Line 1:|
|Line 5:||Line 4:|
Revision as of 13:51, 12 November 2019
|Description: Compute Canada portal for fast, secure and reliable data transfer.|
|SHARCNET Package information: see Globus software page in web portal|
|Full list of SHARCNET supported software|
Note - Please refer the Globus page on Compute Canada documentation wiki for general instructions on how to use globus. The purpose of this page is to provide some specific on how to use globus to access the storage of our legacy systems.
The Compute Canada Globus instance is a portal that provides fast, secure and reliable data transfers. Most Compute Canada storage systems are defined as endpoints for transfers and user machines can be added as endpoints. Most SHARCNET file systems can be found via the computcanada#sharcnet-dtn1 endpoint. General information regarding the Compute Canada Globus implementation can be found at the Compute Canada Globus Wiki page, and more general information about Globus can be found at the Globus site.
Login and select Endpoints
Rather than using this utility from a SHARCNET system, Globus is accessed via its Compute Canada Portal. For up-to-date details and instructions for signing into the Compute Canada Globus Portal with your Compute Canada Data Base (CCDB) credentials visit the Compute Canada Globus Wiki.
Once that you are signed into the Globus Portal you will need to select and activate the
computecanada#sharcnet-dtn1 Endpoint (using your SHARCNET credentials) to access your files on SHARCNET systems (See image below).
To access the storage on the new Compute Canada clusters, graham and cedar, select endpoints
For information about managing Globus Endpoints and transferring from a personal computer see the Personal Computers section of the Compute Canada Globus Wiki page.
Once that two endpoints have been activated transferring files with the Globus Portal is simply a metter of selecting a file or folder from one end point and moving it to the selected Path at the other Endpoint. In the image below the user is sending the "batch" folder from the Path "/~/Downloads/batch_context-master" on their local computer ("jad_lem" Endpoint) to their scratch folder at Orca via the Data Transfer Node (Endpoint = "computecanada#sharcnet-dtn1" and Path ="/scratch/orca/scratch/jdesjard").
computecanada#sharcnet-dtn1 Endpoint folder layout
By default when you activate your computecanada#sharcnet-dtn1 Enpoint the Path will point to your /home directory ("/~/" or /home/username"). This will provide access to your global access /home folder on SHARCNET systems. Similarly, in order to access your globa access /work folder you can change the computecanada#sharcnet-dtn1 Endpoint Path to "/work/username" (this also applies to the "/freezer" global access file system).
For the non-global /scratch file systems the computecanada#sharcnet-dtn1 Endpoint Path needs to specify the desired systems. Available /scratch Paths are:
TAR file efficiency
As with all data transfer methods it is important to recognize the difference in transfer rates between sending a few large files and sending many small files. For example, the difference between transferring 1TB in the form of a single file and transferring the same 1TB in the form of 100,000 small files can be substantial. If you intend on sending a large number of small files it is important to consider encapsulating the files using utilities such as TAR and DAR. This can substantially improve the completion time of your transfer and contribute to the overall performance of the resource.