|Target usage: General purpose with a focus on large-memory parallel applications|
|System information: see redfin system page in web portal|
|System status: see redfin status page|
|Real time system data: see Ganglia monitoring page|
|Full list of SHARCNET systems|
|CPU Model||AMD Opteron 2.1GHz (6172)||AMD Opteron 2.1GHz (6172)|
|Memory/node||98 GB||196 GB|
|Interconnect||QDR InfiniBand||QDR InfiniBand|
|/scratch Storage||10 TB||10 TB|
|OS||CentOS 6.3||CentOS 6.3|
For system notice/history, please visit the Redfin system page in the SHARCNET web portal.
System Access and User Environment
[isaac@red-admin:~] ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 127036 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 8192 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) unlimited cpu time (seconds, -t) 3600 max user processes (-u) 100 virtual memory (kbytes, -v) 1000000 file locks (-x) unlimited
Redfin is a contributed system, open to all SHARCNET users, but with higher priority for groups which have donated the system.
It is very similar to Orca (based on 12-core AMD Opteron Magny Cours CPUs, 2 per node, 24 core per node total, with a 4X QDR Infiniband network) but has more memory per core. There are 24 nodes in the system, for a total of 576 cores.
The first 14 nodes of the system (nodes 1 to 14) have 98GB of memory per node (approximately 4GB/core) while the remaining 10 (nodes 15-24) have double the memory, 196GB (approximately 8GB/core).
Jobs submitted by regular users can only indicate a runtime limit of 4 hours or less ( sqsub -r 4h ... ) to be eligible to run on red[1-14] (as opposed to the standard runtime limit of 7 days). As a consequence of this, any job submitted to the mpi queue requesting more than 240 core, or the equivalent per-core amount of memory (196*10 = 1960G total) has to indicate a maximum runtime of 4h or less to be eligible to start.
Note that some of the nodes on the cluster are reserved for the contributing group, and cannot run normal user's jobs.
Redfin has 10TB of local nfs storage which is provided as the /scratch filesystem to each node.
Each redfin node has 57GB of temporary storage provided at /tmp . This should only be accessed when necessary, by running jobs. Please contact us if you think you need to use it rather than /scratch or /work .
System-specific Usage Concerns
OpenMPI "Cannot allocate memory"
If your MPI job uses a significant amount of memory and is communication intensive you may see your job fail with a message similar to the following:
[red7][[42101,1],161][../../../../../openmpi-1.4.2/ompi/mca/btl/openib/connect/btl_openib_connect_oob.c:464:qp_create_one] error creating qp errno says Cannot allocate memory
In this case you should submit your job such that OpenMPI is told "unpin" memory. This requires passing "--nompirun" to sqsub and specifying the direct path to mpirun and a configuration change. For example, to run "my_mpi_program.x" as a 48-way job for 20 hours one would submit the job like:
sqsub -q mpi --nompirun -o out.%J -e out.%J --mpp=1G -n 48 -r 20h mpirun -mca mpi_leave_pinned 0 ./my_mpi_program.x