From Documentation
Jump to: navigation, search
(Submit to Mpi Queue)
(Example Job)
Line 66: Line 66:
 
cp -r /opt/sharcnet/abaqus/6.14.2/snsamples/heattransfermanifold_cavity.inp .
 
cp -r /opt/sharcnet/abaqus/6.14.2/snsamples/heattransfermanifold_cavity.inp .
 
cp -r /opt/sharcnet/abaqus/6.14.2/snsamples/heattransfermanifold_cavity_parallel.inp .
 
cp -r /opt/sharcnet/abaqus/6.14.2/snsamples/heattransfermanifold_cavity_parallel.inp .
rm -f ofile* job*
 
</pre>
 
 
<b>Version 6.13-3</b>
 
<pre>
 
ssh orca.sharcnet.ca
 
rm -rf /scratch/$USER/abaqus
 
mkdir /scratch/$USER/abaqus
 
module load abaqus/6.13.3
 
mkdir -p ~/samples/abaqus/6.13.3-samples
 
cd ~/samples/abaqus/6.13.3-samples
 
# cp  /opt/sharcnet/abaqus/6.13.3/6.13-3/samples/job_archive/samples.zip .
 
cp -r /opt/sharcnet/abaqus/6.13.3/snexamples/heattransfermanifold_cavity.inp .
 
cp -r /opt/sharcnet/abaqus/6.13.3/snexamples/heattransfermanifold_cavity_parallel.inp .
 
 
rm -f ofile* job*
 
rm -f ofile* job*
 
</pre>
 
</pre>

Revision as of 18:49, 26 March 2019

Note: Some of the information on this page is for our legacy systems only. The page is scheduled for an update to make it applicable to Graham.
ABAQUS
Description: Suite of programs that allow users to carry out Unitifed FEM (Finite Element Method).
SHARCNET Package information: see ABAQUS software page in web portal
Full list of SHARCNET supported software


Introduction

Sharcnet currently provides 16 free tokens (to be increased to 21 free standard tokens in November) which is sufficient to run one 16cpu threaded job (1x16tokens) or three 1cpu serial jobs (3x5 tokens). A single cpu "abaqus -cae" job run interactively on the vdi-centos6 machine uses 5 tokens similar to a serial job run in the queue, and requires 1 free cae token out of 2 freely available cae tokens (1 cae is reserved by a purchasing group).

Version Selection

Graham Cluster

Instruction for using the free sharcnet license on graham cluster (and the soon to be reinstalled orca cluster) will be posted here in early September once the cluster is up and running again.

Legacy Clusters

o Users from any Compute Canada institution can use the free 16 token pool provided by the sharcnet license server by running:

module load abaqus/6.14.2

o Users residing at Western campus (not remote calaborators) can use the free 20 token consolidated uwo license pool on graham or goblin (clusters located on Westerns campus) by running:

ssh dusky.sharcnet.ca
module load abaqus/6.14.2-uwo

If an abaqus job fails to get a license upon startup, it must be manually killed otherwise it will sit idle on the compute node until its "sqsub -r runtime" limit is reached, and the queue automatically kills it off. One can tell if a job is in an idle state by checking the "sqsub -o ofile.%J" output which will appear such as :

[roberpj@dus241:~/test-abaqus] cat ofile.93249
Abaqus JOB myjob
Abaqus 6.13-3
"standard" license request queued for the License Server.
Total time in queue: 0 seconds.
"standard" license request queued for the License Server.
Total time in queue: 30 seconds.
"standard" license request queued for the License Server.
Total time in queue: 60 seconds.
etc

Therefore kill the job by doing:

[roberpj@dus241:~/test-abaqus] sqkill 93249

Job Submission

Please see the following section which shows howto submit a serial, threaded or mpi job by example. Note that the mpp value must be set sufficiently high for a abaqus job to run. During a run which includes the scratch option a directory for temp files will be created as /scratch/$USER/abaqus/userid_job_8cpu_11247. If the job successfully completes the directory will be removed and all output files including the ofile can be found in the submit directory.

Example Job

As preparation to run an interactive (non-cae gui) job do the following steps.

Version 6.14-2

ssh orca.sharcnet.ca
rm -rf /scratch/$USER/abaqus
mkdir /scratch/$USER/abaqus
module load abaqus/6.14.2
mkdir -p ~/samples/abaqus/6.14.2-samples
cd ~/samples/abaqus/6.14.2-samples
cp -r /opt/sharcnet/abaqus/6.14.2/snsamples/heattransfermanifold_cavity.inp .
cp -r /opt/sharcnet/abaqus/6.14.2/snsamples/heattransfermanifold_cavity_parallel.inp .
rm -f ofile* job*

Submit to Serial Queue

sqsub -r 1h --mpp=6G -q serial -o ofile.%J abaqus job=job_1cpu interactive \
      input=heattransfermanifold_cavity.inp scratch=/scratch/$USER/abaqus

Abaqus/Standard will check out 5 tokens for this serial job to run.
Note that goblin does not have scratch therefore omit the scratch= switch.

To compile fortran files append the user option and fortran file as shown. The sharcnet environment default intel compiler module must be loaded prior to loading the abaqus module since the abaqus environment is hardcoded to use it in order for the user option to work in the queue.

module load intel/12.1.3
sqsub -r 1h --mpp=6G -q serial -o ofile.%J abaqus job=job_1cpu interactive \
      input=heattransfermanifold_cavity.inp scratch=/scratch/$USER/abaqus user=3DCrackC3D20.for

Submit to Threaded Queue

sqsub -r 1h --mpp=6G -q threaded -n 8 -o ofile.%J abaqus cpus=8 mp_mode=threads job=job_8cpu interactive \
                input=heattransfermanifold_cavity.inp scratch=/scratch/$USER/abaqus

Abaqus/Standard will check out 12 tokens for this 8core job to run.
Dusky needs only mpp=2G for this example to run.  Goblin and Orca require mpp=6G.

To compile fortran files append the user option and fortran file as shown here:

module load intel/12.1.3
sqsub -r 1h --mpp=6G -q threaded -n 4 -o ofile.%J abaqus cpus=4 mp_mode=threads job=job_4cpu interactive \
         input=heattransfermanifold_cavity.inp scratch=/scratch/$USER/abaqus user=3DCrackC3D20.for

Note that an intel compiler such as intel/12.1.3 must be loaded for abaqus to compile the fortran file.

Submit Parallel Jobs on Dusky with Slurm

Parallel abaqus jobs maybe run on one or multiple nodes by setting the paritition to threaded or mpi as shown in the below slurm script. For performance reasons the threaded partition is recommeded memory permitting. Once the script is configured it can be submitted to the schedular with command "sbatch slurmscript.sh" instead of the sqsub command.

[roberpj@gra-login3:~]$ cat slurmscript.sh
#!/bin/bash
#SBATCH --time=00-00:10           # (D-HH:MM)
#SBATCH --output=ofileM.%J        # ofileM.jobid
#SBATCH --ntasks=4                # number cpu cores
#SBATCH --mem-per-cpu=2G          # memory per cpu core
#SBATCH --partition=threaded      # threaded or mpi
rm -rf testM*
mkdir -p /scratch/$USER/abaqus
module load abaqus/2019-sn
cpu=1
srun hostname -s > hostlist
mp_host_list="("
for i in $(cat hostlist)
do
mp_host_list="${mp_host_list}('$i',$cpu),"
done
mp_host_list=`echo ${mp_host_list} | sed -e "s/,$/,)/"`
rm -f ~/abaqus_v6.env
echo "mp_host_list=${mp_host_list}" >> ~/abaqus_v6.env
abaqus env=mp_host_list cpus=$SLURM_NTASKS mp_mode=mpi standard_parallel=all job=testM \
  interactive input=heattransfermanifold_cavity_parallel.inp scratch=/scratch/$USER/abaqus

General Notes

Running CAE GUI On A Viz or Vdi Workstation

First install tigervnc client on your desktop, as described in https://www.sharcnet.ca/help/index.php/Remote_Graphical_Connections#vncviewer_.28Windows.2C_MacOS.2C_and_Fedora.29 then connect to a sharcnet vizualization workstation where abaqus is installed ( see the Availability Table on the main abaqus software page https://www.sharcnet.ca/my/software/show/101). Currently vdi-centos6.user.sharcnet.ca is the best workstation to use, therefore enter its full hostname when requested by tigervnc. You may need to retry connecting a couple of times, but eventually a remove desktop should appear which you can log into with your sharcnet username and password. Once thats done open a terminal window and startup abaqus with the following two steps:

1) connect to vdi-centos6.user.sharcnet.ca with tigervnc
2) run the following two commands ...
module load intel/12.1.3 abaqus/6.13.3
abaqus cae

o 17 Feb 2015: Currently abaqus will not start on viz10-uwo unless "LD_PRELOAD abaqus cae" is specified, this however slows down performance.

Running CAE GUI On Goblin Login Node

While not recommended (due to runtime and memory limits that exist on all login nodes) it is possible to run abaqus in graphical mode on goblins login node by connecting using either xming, xwin32 or 'ssh -Y' and then using the -mesa switch (to avoid the fatal pixmap error) as follows:

ssh -Y goblin.sharcnet.ca
module load abaqus/6.11.2
abaqus cae -mesa

Running Jobs from the Command Line

This provides and example of running an abaqus job from the command (no-gui, no-cae) on a sharcnet visualization workstation such as vdi-centos6.user.sharcnet.ca. The first task is to make sure the scratch directory exists. Vdi-centos6 does not have its own local scratch, instead it provides scratch from some legacy clusters. At one time all legacy clusters were available on vdi-centos6 however at the time of this writing only scratch from orca is available as shown. If you have questions finding a usable scratch on vdi-centos6 please open a ticket.

To run a job with one cpus do the following:

module load abaqus/6.14.2
abaqus job=job1 interactive  input=heattransfermanifold_cavity.inp scratch=/scratch/orca/scratch/$USER/abaqus

To run the same job from the command line with two cpus and also compile a fortran file do the following:

rm -f job*
module load intel/12.1.3 abaqus/6.14.2
abaqus cpus=2 mp_mode=threads job=job1 interactive input=heattransfermanifold_cavity.inp  scratch=/work/$USER/abaqus user=exa_blast_vdload.f

If you want to quit abaqus while it is still running, give the CTRL-c keys a press and wait about 20 seconds. In extreme situations if abaqus will not die then one can kill off all their processes including login terminal connections by logging into vdi-centos6 from a second terminal then run pkill -u $USER which will kill all of your processes including Abaqus and log you out.

Check License Status

Load the abaqus module and run tne following commands. Note that 1 CAE & 19 standard tokens are reserved for a research group who purchased them and these will always appear in use. To purchase a CAE or standard compute licenses (to ensure access) open a ticket, pro-rated pricing will be explained if during the middle of the license year commencing each mid January.

All Features

abaqusstat

Cae Feature

[roberpj@vdi-centos6:~] abaqusstat | grep cae  
Users of cae:  (Total of 3 licenses issued;  Total of 1 license in use)
  "cae" v61.8, vendor: ABAQUSLM

Standard Feature

[roberpj@vdi-centos6:~] abaqusstat | grep standard
Users of standard:  (Total of 40 licenses issued;  Total of 38 licenses in use)
  "standard" v61.8, vendor: ABAQUSLM

Show Abaqus Command Line Options

abaqus -help

Mapping from Tokens to CPUs for Abaqus Standard/Explicit

CPUS   1   2  4   8  12  16  24  32  64  128
TOKENS 5   6  8  12  14  16  19  21  28   38

See also https://deviceanalytics.com/abaqus-token-calculator

Example of Token Consumption with Consequtively Run Jobs

1) abaqus cae -mesa
Abaqus License Manager checked out the following license(s):
"cae" release 6.11 from license3.sharcnet.ca
<1 out of 2 licenses remain available>.

2) abaqus cae -mesa
Abaqus License Manager checked out the following license(s):
"cae" release 6.11 from license3.sharcnet.ca
<0 out of 2 licenses remain available>.

3) PARALLEL JOB USING 2CPU
sqsub -r 1h --nompirun --mpp=6G -q mpi -n 2 -N 1 -o ofile.%J
abaqus cpus=2 mp_mode=mpi interactive input=heattransfermanifold_cavity.inp job=job_par3
Abaqus/Standard checked out 6 tokens.
<29 out of 35 licenses remain available>.

4) ADDITIONAL PARALLEL JOB USING 4CPU
sqsub -r 1h --nompirun --mpp=6G -q mpi -n 4 -N 1 -o ofile.%J
abaqus cpus=4 mp_mode=mpi interactive input=heattransfermanifold_cavity.inp job=job_par3
Abaqus/Standard checked out 8 tokens.
<21 out of 35 licenses remain available>.

5) SUMMARY
[roberpj@viz10-uwo:~] abaqusstat | grep abaqus
Users of abaqus:  (Total of 35 licenses issued;  Total of 16 licenses in use)
[roberpj@viz10-uwo:~] abaqusstat | grep standard
Users of standard:  (Total of 35 licenses issued;  Total of 16 licenses in use)

ISSUE1: Text File Corruption

Unusual behaviour of a simulation that uses a text based input file format and previously ran can sometimes be traced back to text file corruption. Ascii mode (not binary mode) should be used when transferring text files (such as journal files) from a windows machine to SHARCNET otherwise control-M (DOS end of line markers) or control-Z (DOS end of file marker) codes may become invisibly embedded. If such "corrupted" files are used for running jobs (in the queue) unexpected crashes may occur dumping a variety of error messages into the output ofile(s). For instance UDF files might fail to load and compile resulting in an error message such as Error: yourUDFfile.c: line 1: syntax erro followed by hundreds of messages such as Error: chip-exc: function XYZ not found. Or you might see a error messages such as Error: eval: unbound variable or invalid integer and so forth.

The presence of problematic DOS codes can be confirmed by opening your file (typically your journal file) on SHARCNET into a text editor such as vi or nano then looking for a message such as [ Read 6 lines (Converted from DOS format) ] or [dos].

To "uncorrupt" the file back to a unix compatible format run one of the following commands:

dos2unix myfile
     --- OR  ---
sed -i '/\r//g' myfile

then close the file and finally reopen it to check such dos conversion codes no longer exist. <nowiki>Insert non-formatted text hereInsert non-formatted text here</nowiki>

ISSUE2: Cluster job fails immediately

If a job is submitted to the queue but then fails when it starts to run, the mpp value is probably too low. However as shown in the following two examples the output in the ofile may indicated different messages. The first example failed with mpp=16G but ran with mpp=18G. The second example failed with an mpp=4G but ran with mpp=6G.

Example1

[gb241:/work/username/test1] sqsub -r 1h --mpp=16G -q threaded -n 8 -o ofile.%J
abaqus job=job_par input=Job-1.inp cpus=8 mp_mode=threads interactive scratch=/scratch/$USER/abaqus
submitted as jobid 669290

[gb241:/work/username/test1] tail -f ofile.669290.gb240
Abaqus 6.11-2
Begin Analysis Input File Processor
Mon 19 Aug 2013 01:25:19 PM EDT
Run pre.exe
Abaqus License Manager checked out the following licenses:
Abaqus/Standard checked out 12 tokens.
<11 out of 35 licenses remain available>.
Mon 19 Aug 2013 01:25:23 PM EDT
Abaqus Error: Analysis Input File Processor exited with an error.
Abaqus/Analysis exited with errors

Example2

[roberpj@gb241:~/samples/abaqus/6.11-2_sample_threaded] sqsub -r 1h --mpp=4G
-q threaded -n 4 -o ofile.%J abaqus cpus=4 mp_mode=threads job=job_8cpu interactive
input=heattransfermanifold_cavity.inp scratch=/scratch/$USER/abaqus

[roberpj@gb241:~/samples/abaqus/6.11-2_sample_threaded] cat ofile.789204.gb240
Abaqus JOB job_8cpu
Abaqus 6.13-3
Abaqus License Manager checked out the following licenses:
Abaqus/Standard checked out 8 tokens.
<3 out of 35 licenses remain available>.
Begin Analysis Input File Processor
Tue 14 Jan 2014 06:27:12 PM EST
Run pre
Tue 14 Jan 2014 06:27:14 PM EST
Abaqus Error: The executable pre
aborted with system error "Illegal memory reference" (signal 11).
Please check the .dat, .msg, and .sta files for error messages if the files
exist.  If there are no error messages and you cannot resolve the problem,
please run the command "abaqus job=support information=support" to report and
save your system information.  Use the same command to run Abaqus that you
used when the problem occurred.  Please contact your local Abaqus support
office and send them the input file, the file support.log which you just
created, the executable name, and the error code.
Abaqus/Analysis exited with errors

[root@gb44:~] cat /var/log/messages| grep ABQ
Jan 14 18:27:14 gb44 kernel: pre[5742]: segfault at 133333330 ip 00007ff819be7d57
sp 00007fffa03bb170 error 6 in libABQSMAAspSupport.so[7ff819b10000+356000]

. .

References

o Abaqus 6.13-3
https://www.sharcnet.ca/Software/Abaqus/6.13.3/index.html

o Abaqus 6.11-2
https://www.sharcnet.ca/Software/Abaqus/6.11.2/index.html

o SIMULIA Academic Portfolio
http://www.3ds.com/products/simulia/academics/

o SIMULIA Abaqus Portfolio
http://www.3ds.com/products/simulia/portfolio/abaqus/abaqus-portfolio/