... | ... | @@ -37,18 +37,22 @@ By default, the Intel compiler suite is installed, but not added to the PATH var |
|
|
On virtual machines within openstack there is one issue:
|
|
|
Apparently the Intel compiler option -xHost (automated architecture dependent vectorization) can cause segmentation faults, at least within ALF. Possible alternatives are -axCORE-AVX2 (vectorization upto AVX2 [ymm-register]) and -axCORE-AVX512 (using the 512 [zmm-]register) both work fine, to the best of our knowledge.
|
|
|
|
|
|
Jobs may be compiled from the Julia login node, `ssh username@julia.uni-wuerzburg.de`, or in the OpenStack.
|
|
|
|
|
|
Interactive sessions are also available from the login node. To get one use `srun --pty bash`.
|
|
|
|
|
|
It is a good idea to check first if there are any resources available for interactive sessions by running `sinfo` and seeing if there are any nodes in the "idle" state. If your desired partition is not available (see below for partitions) there may be nodes in other partitions that are currently idle and you can use for testing. Specify a partition in the interactive session by using `srun --pty -p <partition name> bash`.
|
|
|
|
|
|
## Job submission
|
|
|
To submit jobs via SLURM, use `ssh username@julia.uni-wuerzburg.de` and then in the directory of your job script use `sbatch <job_script>`.
|
|
|
|
|
|
There is currently not an interactive session option (or compilers available). Use the OpenStack for testing and compilation, or alternatively for compilation from the login node you can submit a bash job through SLURM.
|
|
|
|
|
|
## Sample Job Script
|
|
|
For a serial or "embarrassingly parallel" job:
|
|
|
For a generic job (serial, embarrassingly parallel, or parallel, depending on options chosen):
|
|
|
|
|
|
```
|
|
|
#!/bin/bash
|
|
|
|
|
|
#SBATCH -J EmbParrJob
|
|
|
#SBATCH -J GenJob
|
|
|
#SBATCH -N 1
|
|
|
#SBATCH -t 24:00:00
|
|
|
#SBATCH --ntasks-per-node 32
|
... | ... | @@ -61,12 +65,17 @@ source /usr/local/etc/intel_mpi.sh |
|
|
export I_MPI_FABRICS=shm:ofi
|
|
|
export I_MPI_PMI_LIBRARY=/usr/lib/x86_64-linux-gnu/libpmi.so.0
|
|
|
|
|
|
./embParrScript.sh <arguments>
|
|
|
./serialScript.sh <arguments>
|
|
|
# or alternative for MPI
|
|
|
srun ./prog.out <arguments>
|
|
|
```
|
|
|
|
|
|
This will set up one node (N) with 32 cores (ntasks-per-node * cpus-per-task), which is in the category "xxl" for Julia, and allow the job to run for 24 hours at the most. The source (export) will be the default Debian installation ("Ubuntu" is an alternative option), and the working directory (workdir) is set so that the executable and input files can be found (`/home/<user>` is currently the default working directory.) Output files will be placed in the working directory as well. The executable will be able to make use of all the cores requested in the node.
|
|
|
This will set up one node (N) with 32 cores (ntasks-per-node * cpus-per-task), and allow the job to run for 24 hours at the most. Depending on the cores requested and availability of nodes, the job could end up on a "standard" node or an "ib" node, because the partition is not specified (more on partitions below). The source (export) will be the default Debian installation ("Ubuntu" is an alternative option), and the working directory (workdir) is set so that the executable and input files can be found (`/home/<user>` is currently the default working directory.) Output files will be placed in the working directory as well. The executable will be able to make use of all the cores requested in the node.
|
|
|
|
|
|
## Partitions
|
|
|
There are four partitions in the SLURM portion of the cluster: "standard*", "ib", "fat", and "gpu". A partition can be specified in a job script by adding an `#SBATCH -p <partition name>` line. The standard*, ib, and gpu partitions each consist of nodes with 32 threads and 384GB of memory available per node. If no partition is specified, jobs will end up either in the standard* or the ib partition (if no standard* nodes are available).
|
|
|
|
|
|
The ib partition nodes include additionally FDR 56 Gbit/s Mellanox infiniband interconnect on each node, and the gpu partition nodes include additionally two NVIDIA Tesla P100 graphics cards on each node. The fat partition nodes each have 2TB of memory and can handle 144 threads.
|
|
|
|
|
|
## `srun` vs. `mpirun` vs. `mpiexec`
|
|
|
For not quite embarrassingly parallel jobs, see here
|
... | ... | |