Welcome to an informal and unofficial Julia wiki
Feel free to contribute anything new you discover or update anything that no longer applies/is deprecated.
Get a Julia account and official information at:
The official documentation can be found here:
To check out a machine, compile code, and test performance, go to:
The log-in "Domain" is "julia."
- Set up an ssh key pair with your local machine.
- Launch an instance with this key pair selected.
- Assign a floating IP to the instance (under "Actions").
- Log in to the machine using
ssh -i <key_name> debian@<instance_floating_ip>(use debian or ubuntu, depending on the OS for the machine).
Note that this machine will be reserved for the user (even if shutdown) until "Delete Instance" is selected from the "Action" menu for the machine.
To preserve an environment with the desired installations and files on the hard disk (even after a machine is deleted), you can select "Create Snapshot" from the "Action" menu of a currently running machine. Once the snapshot is saved, any new machine can be quickly booted up with the desired installation and hard disk environment by selecting "Instance Snapshot" as the source from the "Select Boot Source" menu under "Source" when launching a new instance, and then selecting the desired snapshot. Note that saving a snapshot may take some time, but booting one (once saved) will be fast--the OpenStack is optimized for this.
The rest of the wiki consists mainly of information for operating the SLURM scheduled portion of Julia.
To transfer files, use the domain julia-storage.uni-wuerzburg.de
Jobs may be compiled from the Julia login node,
By default, the Intel compiler suite is installed, but not added to the PATH variable on the new julia login nodes. This is readily done by executing
source /usr/local/etc/intel_mpi.sh. For ALF you can chose the Intel enviroment (instead of SuperMUC or Jurece ...) and it should work just fine.
On virtual machines within openstack there is one issue:
Apparently the Intel compiler option -xHost (automated architecture dependent vectorization) can cause segmentation faults, at least within ALF. Possible alternatives are -axCORE-AVX2 (vectorization upto AVX2 [ymm-register]) and -axCORE-AVX512 (using the 512 [zmm-]register) both work fine, to the best of our knowledge.
Program performance may be tested in the OpenStack machines, but interactive sessions are also available from the login node. To get one use
srun --pty bash.
It is a good idea to check first if there are any resources available for interactive sessions by running
sinfo and seeing if there are any nodes in the "idle" state. If your desired partition is not available (see below for partitions) there may be nodes in other partitions that are currently idle and you can use for testing. Specify a partition in the interactive session by using
srun --pty -p <partition name> bash.
To submit jobs via SLURM, use
ssh firstname.lastname@example.org and then in the directory of your job script use
Sample Job Script
For a generic job (serial, embarrassingly parallel, or parallel, depending on options chosen):
#!/bin/bash #SBATCH -J GenJob #SBATCH -N 1 #SBATCH -t 24:00:00 #SBATCH --ntasks-per-node 32 #SBATCH --cpus-per-task 1 #SBATCH --export Debian #SBATCH --workdir /home/<user>/<launch_directory> # the following 3 lines sets srun compatible with the intel mpi library source /usr/local/etc/intel_mpi.sh export I_MPI_FABRICS=shm:ofi export I_MPI_PMI_LIBRARY=/usr/lib/x86_64-linux-gnu/libpmi.so.0 ./jobScript.sh <arguments> # or alternative for MPI srun ./prog.out <arguments>
This will set up one node (N) with 32 cores (ntasks-per-node * cpus-per-task), and allow the job to run for 24 hours at the most. Depending on the cores requested and availability of nodes, the job could end up on a "standard" node or an "ib" node, because the partition is not specified (more on partitions below). The source (export) will be the default Debian installation ("Ubuntu" is an alternative option), and the working directory (workdir) is set so that the executable and input files can be found (
/home/<user> is currently the default working directory.) Output files will be placed in the working directory as well. The executable will be able to make use of all the cores requested in the node.
There are four partitions in the SLURM portion of the cluster: "standard*", "ib", "fat", and "gpu". A partition can be specified in a job script by adding an
#SBATCH -p <partition name> line. The standard*, ib, and gpu partitions each consist of nodes with 32 threads and 384GB of memory available per node. If no partition is specified, jobs will end up either in the standard* or the ib partition (if no standard* nodes are available).
The ib partition nodes include additionally FDR 56 Gbit/s Mellanox infiniband interconnect on each node, and the gpu partition nodes include additionally two NVIDIA Tesla P100 graphics cards on each node. The fat partition nodes each have 2TB of memory and can handle 144 threads.
For not quite embarrassingly parallel jobs, see here https://docs.computecanada.ca/wiki/Advanced_MPI_scheduling#Why_srun_instead_of_mpiexec_or_mpirun.3F
Cluster Info from the Login Node
To get information on the nodes available use
To get the specifications for a specific node on the sinfo list use
scontrol show node <nodename>.
To check on jobs you've submitted use
To cancel a job use