(→Run Parallel Test)
(→Using A Script to Submit Batch Jobs)
|Line 184:||Line 184:|
sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulation$i.fsp
sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulation$i.fsp
Revision as of 15:18, 25 November 2014
|Description: Providing photonic and optoelectronic TCAD device simulation and photonic integrated circuit design products|
|SHARCNET Package information: see FDTD software page in web portal|
|Full list of SHARCNET supported software|
- 1 Introduction
- 2 Version Selection
- 3 Examples
- 4 General Notes
- 5 References
As of February 2013, SHARCNET no long holds the FDTD Solutions (also known as front end or GUI) licence. Users who need to use FDTD Solutions for their model construction and data analysis must obtain their own licences.
Users who previously used CAD - the Unix port of FDTD Solution GUI for model construction and data analysis - must now either possess a licence (a FLEX managed licence) that allows them to use CAD on SHARCNET systems, or use FDTD Solutions on their own computers.
Thanks to the kind donation of FDTD Engine licences by Lumerical solutions, Inc., SHARCNET will continue to support the serial farming and parallel processing of FDTD simulations on SHARCNET systems. Note the FDTD engines are the only components that are supported. We no longer have the licence for the frontend FDTD Solutions.
In general, one conducts FDTD simulations in following steps:
- Constructing model, preparing data (.fsp files), this is typically done on one's local computers, not SHARCNET clusters;
- Bringing data files to SHARCNET;
- Running simulations on SHARCNET using FDTD engines;
- Getting results back and performing analysis and modifications on local computers
It is assumed that users prepare their simulation files on their own computers with FDTD Solutions and then carry on large scale simulations--being large number of cases or large size of problems that are not feasible on their computers--on SHARCNET systems through serial farming or parallel processing. Typically this will involve transferring simulation files to SHARCNET and copying computed results back to their local computers for analysis.
To transfer data between one's computers and SHARCNET systems, one may use the common secure file transfer client software such as SSH Secure File Transfer Client, WinSCP, putty-sftp, etc. Users are referred to the tutorials on using these tools.
module load fdtd/8.7.3
Before running Lumerical for the first time, remove the following config file, if it exists:
cd ~/.config/Lumerical rm -f "FDTD Solutions.ini"
or rename it such as FDTD Solutions.ini.bak. Note the space in the filename between FDTD and Solutions.ini is required hence why quotes are being used. Removing it cleans the cache of any corrupted Lumerical license information from previous versions. Next recreate the file using a text editor by copying in the following text:
[roberpj@saw-login2:~] nano ~/.config/Lumerical/"FDTD Solutions.ini" [license] type=flex flexserver\email@example.com
The licence file can also be created with the following commands
echo [license] > "FDTD Solutions.ini" echo type=flex >> "FDTD Solutions.ini" echo "flexserver\firstname.lastname@example.org" >> "FDTD Solutions.ini"
Another option for creating or updating the FDTD Solutions.ini file is to load the lumerical module and then run the fdtd-config-license script. Specify license2.sharcnet.ca for the server, and specify 27015 for the port. The result will be the same as the previous stanza. If you want to use your own license, replace the contents of FDTD Solutions.ini with your own license information.
Submit Serial Job
o To demonstrate howto submit a serial job onto a sharcnet cluster do:
cp /opt/sharcnet/lumerical/8.7.3/examples/nanowire.fsp . module load fdtd/8.7.3 sqsub -q serial -r 1.5h -o ofile.%J fdtd-engine-mpich2nem nanowire.fsp
Submit Parallel Job
o To demonstrate howto run a parallel job use the paralleltest example:
cp -avf /opt/sharcnet/fdtd/8.7.3/examples/paralleltest.fsp .
o To submit a parallel job to the mpi queue on all clusters (except kraken) do:
module unload intel openmpi module load fdtd/8.7.3 intel/12.1.3 openmpi/intel/1.6.2 export LD_LIBRARY_PATH=/opt/sharcnet/openmpi/1.6.2/intel/lib:$LD_LIBRARY_PATH sqsub -r 1h -q mpi -n 4 --mpp 1g -o ofile.%J fdtd-engine-ompi-lcl paralleltest.fsp
o To submit a parallel job to the mpi queue on kraken do the following:
module unload intel openmpi module load fdtd/8.7.3 intel/12.1.3 openmpi/intel/1.6.2_1 export LD_LIBRARY_PATH=/opt/sharcnet/openmpi/1.6.2/intel/lib:$LD_LIBRARY_PATH sqsub -r 1h -q mpi -f myri -n 4 --mpp 1g -o ofile.%J fdtd-engine-ompi-lcl paralleltest.fsp
This section describes howto run fdte compute jobs from the command line useful for testing before submitting a production job into the queue.
Run Serial Test
o To run a serial (single core) interactive test on orca, vdi or iqaluk do:
ssh orca.sharcnet.ca; ssh orc-dev1 or ssh vdi-centos6.user.sharcnet.ca or ssh iqaluk.sharcnet.ca module load fdtd/8.7.3 fdtd-engine-mpich2nem nanowire.fsp
Run Parallel Test
o To run a parallel interactive test on orca, vdi-centos6 or iqaluk do:
ssh orca.sharcnet.ca; ssh orc-dev1 or ssh vdi-centos6.user.sharcnet.ca or ssh iqaluk.sharcnet.ca module unload intel openmpi module load fdtd/8.7.3 intel/12.1.3 openmpi/intel/1.6.2 export LD_LIBRARY_PATH=/opt/sharcnet/openmpi/1.6.2/intel/lib:$LD_LIBRARY_PATH mpirun -np 4 --bind-to-core fdtd-engine-ompi-lcl paralleltest.fsp
o To run a parallel interactive test on a sharcnet viz station (running fedora) do :
ssh viz10-uwo.sharcnet.ca module load fdtd/8.7.3 /usr/lib64/openmpi/bin/mpirun -np 8 -bind-to core fdtd-engine-ompi-lcl paralleltest.fsp
o To run a parallel interactive test on the sharcnet vdi machine do:
ssh vdi-centos6.user.sharcnet.ca module load fdtd/8.7.3 /usr/lib64/openmpi/bin/mpirun -np 8 -bind-to-core fdtd-engine-ompi-lcl paralleltest.fsp
Using FDTD Solutions GUI CAD
This section is for those who want to build simulation models and prepare data files using Lumerical graphic user interface on SHARCNET systems using their own licence.
Without your own licence for FDTD Solutions (also named CAD for Unix), you will be unable to use the GUI on SHARCNET systems. As of February 2013, SHARCNET no longer has the licence to the front end GUI (CAD for Linux). Users who previously used CAD - the Unix port of FDTD Solution GUI for model construction and data analysis - must now either possess a licence (a FLEX managed licence) that allows them to use CAD on SHARCNET systems, or use FDTD Solutions on their own computers. If you try to run CAD withoutyour own license, and error message box will pop up such as:
Error: Unable to check out a FlexNet license. The flexNet error code is: -18, which corresponds to the error: License server system does not support this feature. And so forth
If one pursues to use CAD on SHARCNET systems using their own licence, he/she must first contact the administrator for technical arrangements.
To run CAD remotely on a SHARCNET login or devel node from a Windows desktop, one needs a X server running on Windows. We recommend using Xming + Putty or Cygwin/X which are all free. Or if your connecting from a linux desktop then run the "ssh -X clustername" command. To establish a graphical capable session on a sharcnet Visualization machine, start a remote desktop by clicking the blue icon beside vdi-centos6 on https://www.sharcnet.ca/my/systems. Once connected do the following:
module load fdtd/8.7.3 cd /scratch/$USER cp /opt/sharcnet/fdtd/8.7.3/examples/paralleltest.fsp . CAD paralleltest.fsp
Using A Script to Submit Batch Jobs
Suppose you have a number of jobs you want to process at once, and you want to avoid typing job submission command multiple times, the simplest way is to put the commands in a shell script and run the shell script instead. To create a script, use an editor to create a file, say "runsim.sh", and put the following lines
#!/bin/sh sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulation1.fsp sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulation2.fsp ... ... sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulationN.fsp
and save it.
To make this shell script executable, use command
chmod +x runsim.sh
To run the simulations 1 to N, as specified in the script, simply execute the command at the command line
If you have a large number of jobs and each of the .fsp files follows some pattern, e.g. a case number in ascending consecutive integer, then, you may simplify the script using a loop. The following shows an example of a script "runsims.sh" that submits 100 jobs in a loop
#!/bin/sh n=100 i=1 while [ $i -le $n ]; do sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulation$i.fsp i=$((i+1)) done
One may also use a C/C++ like syntax for the above loop
#!/bin/sh n=100 for((i=1;i<=n;i++)); do sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulation$i.fsp done
Note the examples shown above are not just for serial jobs, it can be used for parallel jobs too. One only needs to replace the sqsub options with those relevant to parallel jobs.
Show Sharcnet License Usage
[roberpj@orc-login2:~] module load fdtd/8.7.3 [roberpj@hnd20:~] fdtdstat lmutil - Copyright (c) 1989-2013 Flexera Software LLC. All Rights Reserved. Flexible License Manager status on Thu 10/16/2014 16:08 License server status: email@example.com License file(s) on license2.sharcnet.ca: licenses/LicenseFile.lic_18mar2014.opt:licenses/start.lic: license2.sharcnet.ca: license server UP v11.10 Vendor daemon status (on license2): LUMERICL: UP v11.10 Feature usage info: Users of FDTD_Solutions_engine: (Total of 10 licenses issued; Total of 4 licenses in use) "FDTD_Solutions_engine" v1.0, vendor: LUMERICL floating license roberpj vdi-centos6.user.sharcnet.ca /dev/tty (v1.0) (license2.sharcnet.ca/27015 402), start Thu 10/16 16:08 roberpj vdi-centos6.user.sharcnet.ca /dev/tty (v1.0) (license2.sharcnet.ca/27015 302), start Thu 10/16 16:08 cxxen hnd17 /dev/tty (v1.0) (license2.sharcnet.ca/27015 101), start Thu 10/16 16:06 scdutk hnd11 /dev/tty (v1.0) (license2.sharcnet.ca/27015 201), start Thu 10/16 16:06 Users of start: (Total of 1 license issued; Total of 0 licenses in use)
o Lumerical Homepage
o Parallel Engine Issues
o Run FDTD in Single Processor Mode