From Documentation
Jump to: navigation, search
(Submitting A Parallel Job)
(Using A Script to Submit Batch Jobs)
Line 140: Line 140:
 
./runsim.sh
 
./runsim.sh
 
</pre>
 
</pre>
 +
 +
If you have a large number of jobs and each of the .fsp files follows some pattern, e.g. a case number in ascending consecutive integer, then, you may simplify the script using a loop. The following shows an example of a script "runsims.sh" that submits 100 jobs in a loop
 +
 +
<pre>
 +
#!/bin/sh
 +
n=100
 +
 +
i=1
 +
while [ $i -le $n ]; do
 +
    sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulation$i.fsp
 +
    i=$((i+1))
 +
done
 +
</pre>
 +
 +
Note the examples shown above are not just for serial jobs, it can be used for parallel jobs too. One only needs to replace the sqsub options with those relevant to parallel jobs.
  
 
==References==
 
==References==

Revision as of 13:32, 17 December 2013

LUMERICAL
Description: Providing photonic and optoelectronic TCAD device simulation and photonic integrated circuit design products
SHARCNET Package information: see LUMERICAL software page in web portal
Full list of SHARCNET supported software


Licensing Information

As of February 2013, SHARCNET no long holds the FDTD Solutions (also known as front end or GUI) licence. Users who need to use FDTD Solutions for their model construction and data analysis must obtain their own licences.

Users who previously used CAD - the Unix port of FDTD Solution GUI for model construction and data analysis - must now either possess a licence (a FLEX managed licence) that allows them to use CAD on SHARCNET systems, or use FDTD Solutions on their own computers.

Thanks to the kind donation of FDTD Engine licences by Lumerical solutions, Inc., SHARCNET will continue to support the serial farming and parallel processing of FDTD simulations on SHARCNET systems.

Version Changes

The latest release is 8.5.3 and this is the only version supported. The licences to the older versions have expired.

Note the FDTD engines are the only components that are supported. We no longer have the licence for the frontend FDTD Solutions.

The Workflow

In general, one conducts FDTD simulations in following steps:

  • Constructing model, preparing data (.fsp files), this is typically done on one's local computers, not SHARCNET clusters;
  • Bringing data files to SHARCNET;
  • Running simulations on SHARCNET using FDTD engines;
  • Getting results back and performing analysis and modifications on local computers

It is assumed that users prepare their simulation files on their own computers with FDTD Solutions and then carry on large scale simulations--being large number of cases or large size of problems that are not feasible on their computers--on SHARCNET systems through serial farming or parallel processing. Typically this will involve transferring simulation files to SHARCNET and copying computed results back to their local computers for analysis.

To transfer data between one's computers and SHARCNET systems, one may use the common secure file transfer client software such as SSH Secure File Transfer Client, WinSCP, putty-sftp, etc. Users are referred to the tutorials on using these tools.

Loading Lumerical and Support Library Modules

Before you can run Lumerical, you must load the Lumerical module:

module load lumerical

As well one must perform the following

module unload openmpi intel compile
module load gcc/4.3.4
module load openmpi/gcc/1.4.2

Before You Go The First Time

Before you run Lumerical simulations for the first time, remove the existing configuration file in your home directory, if any:

cd ~/.config
rm FDTD\ Solutions.ini

This cleans the cached or corrupted Lumerical licencing infomation for previous versions.

Using FDTD Solutions GUI CAD

This section is for those who want to build simulation models and prepare data files using Lumerical graphic user interface on SHARCNET systems using their own licence.

Without your own licence for FDTD Solutions (also named CAD for Unix), you will not be able to use it on SHARCNET systems.

As of February 2013, SHARCNET no longer has the licence to the front end GUI (CAD for Linux). Users who previously used CAD - the Unix port of FDTD Solution GUI for model construction and data analysis - must now either possess a licence (a FLEX managed licence) that allows them to use CAD on SHARCNET systems, or use FDTD Solutions on their own computers.

If one pursues to use CAD on SHARCNET systems using their own licence, he/she must first contact the administrator for technical arrangements.

To run CAD remotely on SHARCNET systems (login nodes or visualization systems) from a Windows desktop, one needs an X server running on Windows. We recommend using Xming + Putty or Cygwin/X. They are all free. Or if your connecting from a linux desktop then run the "ssh -X clustername" command for instance:

ssh -X tope.sharcnet.ca
module load lumerical
cd /scratch/$USER
cp /opt/sharcnet/lumerical/version/examples/paralleltest.fsp .
CAD paralleltest.fsp

Submitting A Serial Job

One must load necessary modules before submitting a job. Loading a module sets the environment variables needed for the running executables. The environment variables set will be inherited on the compute nodes automatically

module load lumerical
sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulation.fsp

Submitting A Parallel Job

Starting with version 7 the lumerical module must be loaded first before submitting a job or running the CAD command for instance as follows:

module load lumerical
sqsub -q mpi -r 1.5h -n num_procs -o output%J fdtd-engine-ompi-lcl parallel_simulation.fsp

For further details about Distributed simulation testing please refer to the documentation on Lumerical's web site which can be found here[1].

See also Lumerical's User Guide

Using A Script to Submit Batch Jobs

Suppose you have a number of jobs you want to process at once, and you want to avoid typing job submission command multiple times, the simplest way is to put the commands in a shell script and run the shell script instead. To create a script, use an editor to create a file, say "runsim.sh", and put the following lines

#!/bin/sh
sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulation1.fsp
sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulation2.fsp
... ...
sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulationN.fsp

and save it.

To make this shell script executable, use command

chmod +x runsim.sh

To run the simulations 1 to N, as specified in the script, simply execute the command at the command line

./runsim.sh

If you have a large number of jobs and each of the .fsp files follows some pattern, e.g. a case number in ascending consecutive integer, then, you may simplify the script using a loop. The following shows an example of a script "runsims.sh" that submits 100 jobs in a loop

#!/bin/sh
n=100

i=1
while [ $i -le $n ]; do
    sqsub -q serial -r 1.5h -o output%J fdtd-engine-mpich2nem simulation$i.fsp
    i=$((i+1))
done

Note the examples shown above are not just for serial jobs, it can be used for parallel jobs too. One only needs to replace the sqsub options with those relevant to parallel jobs.

References

o Lumerical Homepage
http://www.lumerical.com/