-Y is needed to open windows from the Niagara command-line onto your local X server.
To run on Niagara's compute nodes, you must submit a batch job.
If you cannot log in, be sure first to check the [https://docs.scinet.utoronto.ca System Status] on this site's front page.
== Your various directories ==
By virtue of your access to Niagara you are granted storage space on the system. There are several directories available to you, each indicated by an associated environment variable
=== home and scratch ===
You have a home and scratch directory on the system, whose locations are of the form
$HOME=/home/g/groupname/myccusername
$SCRATCH=/scratch/g/groupname/myccusername
where groupname is the name of your PI's group, and myccusername is your CC username. For example:
$PROJECT=/project/g/groupname/myccusername
$ARCHIVE=/archive/g/groupname/myccusername
NOTE: Currently archive space is available only via [https://docs.scinet.utoronto.ca/index.php/HPSS HPSS], and is not accessible on the Niagara login, compute, or datamover nodes.
'''''IMPORTANT: Future-proof your scripts'''''
When writing your scripts, use the environment variables ($HOME, $SCRATCH, $PROJECT, $ARCHIVE) instead of the actual paths! The paths may change in the future.
=== Storage and quotas ===
You should familiarize yourself with the [[Data_management_at_Niagara#Purpose_of_each_file_system | various file systems]], what purpose they serve, and how to properly use them. This table summarizes the various file systems. See the [[Data_management_at_Niagara | Data management at Niagara]] page for more details.
{| class="wikitable"
! location
!colspan="2"| quota
!align="right"| block size
! expiration time
! backed up
! on login nodes
! on compute nodes
|-
| $HOME
|colspan="2"| 100 GB per user
|align="right"| 1 MB
|
| yes
| yes
| read-only
|-
|rowspan="2"| $SCRATCH
|colspan="2"| 25 TB per user
|align="right" rowspan="2" | 16 MB
|rowspan="2"| 2 months
|rowspan="2"| no
|rowspan="2"| yes
|rowspan="2"| yes
|-
|align="right"|50-500TB per group
|align="right"|[[Data_management#Quotas_and_purging | depending on group size]]
|-
| $PROJECT
|colspan="2"| by group allocation
|align="right"| 16 MB
|
| yes
| yes
| yes
|-
| $ARCHIVE
|colspan="2"| by group allocation
|align="right"|
|
| dual-copy
| no
| no
|-
| $BBUFFER
|colspan="2"| 10 TB per user
|align="right"| 1 MB
| very short
| no
| yes
| yes
|}
=== Moving data to Niagara ===
If you need to move data to Niagara for analysis, or when you need to move data off of Niagara, use the following guidelines:
* If your data is less than 10GB, move the data using the login nodes.
* If your data is greater than 10GB, move the data using the datamover nodes nia-datamover1.scinet.utoronto.ca and nia-datamover2.scinet.utoronto.ca .
Details of how to use the datamover nodes can be found on the [[Data_management_at_Niagara#Moving_data | Data management at Niagara]] page.
= Loading software modules =
You have two options for running code on Niagara: use existing software, or [[Niagara_Quickstart#Compiling_on_Niagara:_Example | compile your own]]. This section focuses on the former.
Other than essentials, all installed software is made available [[Using_modules | using module commands]]. These modules set environment variables (PATH, etc.), allowing multiple, conflicting versions of a given package to be available. A detailed explanation of the module system can be [[Using_modules | found on the modules page]].
Common module subcommands are:
module load <module-name>: use particular softwaremodule purge: remove currently loaded modulesmodule spider (or module spider <module-name>): list available software packagesmodule avail: list loadable software packagesmodule list: list loaded modulesml is the same as module list, and ml is the same as module load .
== Software stacks: NiaEnv and CCEnv ==
On Niagara, there are two available software stacks:
A [[Modules_specific_to_Niagara | Niagara software stack]] tuned and compiled for this machine. This stack is available by default, but if not, can be reloaded with
module load NiaEnvThe standard [[Available software|Alliance software stack]] which is available on Alliance's other clusters (including [[Graham]], [[Cedar]], [[Narval]], and [[Beluga]]):
module load CCEnv arch/avx512
Or, if you want the same default modules loaded as on Cedar, Graham, and Beluga, then do
module load CCEnv arch/avx512 StdEnv/2018.3
intel → intel/2018.2. It is usually better to be explicit about the versions, for future reproducibility.
Modules sometimes require other modules to be loaded first.
Solve these dependencies by using module spider.
= Available compilers and interpreters =
* For most compiled software, one should use the Intel compilers (icc for C, icpc for C++, and ifort for Fortran). Loading an intel module makes these available.
* The GNU compiler suite (gcc, g++, gfortran) is also available, if you load one of the gcc modules.
* Open source interpreted, interactive software is also available:
** [[Python]]
** [[R]]
** Julia
** Octave
Please visit the [[Python]] or [[R]] page for details on using these tools. For information on running MATLAB applications on Niagara, visit [[MATLAB| this page]].
= Using Commercial Software =
May I use commercial software on Niagara?
* Possibly, but you have to bring your own license for it. You can connect to an external license server using [https://docs.scinet.utoronto.ca/index.php/SSH_Tunneling ssh tunneling].
* SciNet and Alliance have an extremely large and broad user base of thousands of users, so we cannot provide licenses for everyone's favorite software.
* Thus, the only commercial software installed on Niagara is software that can benefit everyone: compilers, math libraries and debuggers.
* That means no [[MATLAB]], Gaussian, IDL,
* Open source alternatives like Octave, [[Python]], and [[R]] are available.
* We are happy to help you to install commercial software for which you have a license.
* In some cases, if you have a license, you can use software in the Alliance stack.
The list of commercial software which is installed on Niagara, for which you will need a license to use, can be found on the [https://docs.scinet.utoronto.ca/index.php/Commercial_software commercial software page].
= Compiling on Niagara: Example =
Suppose one want to compile an application from two c source files, main.c and module.c, which use the Gnu Scientific Library (GSL). This is an example of how this would be done:
Rule of thumb: tests should run no more than a couple of minutes, taking at most about 1-2GB of memory, and use no more than a couple of cores.
You can run the ddt debugger on the login nodes after module load ddt.
Short tests that do not fit on a login node, or for which you need a dedicated node, request an interactive debug job with the debug command:
where N is the number of nodes, If N=1, this gives an interactive session one 1 hour, when N=4 (the maximum), it gives you 30 minutes.
Finally, if your debugjob process takes more than 1 hour, you can request an interactive job from the regular queue using the salloc command. Note, however, that this may take some time to run, since it will be part of the regular queue, and will be run when the scheduler decides.
here N is again the number of nodes, and M is the number of hours you wish the job to run.
If you need to use graphics while testing your code through salloc, e.g. when using a debugger such as DDT or DDD, you have the following options, please visit the [[Testing_With_Graphics | Testing with graphics]] page.
Niagara uses SLURM as its job scheduler.
You submit jobs from a login node by passing a script to the sbatch command:
This puts the job in the queue. It will run on the compute nodes in due course.
In most cases, you will want to submit from your $SCRATCH directory, so that the output of your compute job can be written out (as mentioned above, $HOME is read-only on the compute nodes).
Jobs will run under their group's RRG allocation, or, if the group has none, under a RAS allocation (previously called 'default' allocation).
Some example job scripts can be found below.
Scheduling is by node, so in multiples of 40-cores.
Your job's maximum walltime is 24 hours.
Jobs must write to your scratch or project directory (home is read-only on compute nodes).
Compute nodes have no internet access.
[[Data_management_at_Niagara#Moving_data | Move your data]] to Niagara before you submit your job.
All job resource requests on Niagara are scheduled as a multiple of '''nodes'''.
Whatever your requests to the scheduler, it will always be translated into a multiple of nodes allocated to your job.
Memory requests to the scheduler are of no use. Your job always gets N x 202GB of RAM, where N is the number of nodes and 202GB is the amount of memory on the node.
If you run serial jobs you must still use all 40 cores on the node. Visit the [https://docs.scinet.utoronto.ca/index.php/Running_Serial_Jobs_on_Niagara serial jobs] page for examples of how to do this.
Since there are 40 cores per node, your job should use N x 40 cores. If you do not, we will contact you to help you optimize your workflow. Or you can [mailto:support@scinet.utoronto.ca contact us] to get assistance.
#SBATCH go to SLURM.mpi_job)mpi_example application (SLURM will inform mpirun or srun on how many processes to run).
#SBATCH go to SLURM.
* sbatch reads these lines as a job request (which it gives the name openmp_ex) .
* In this case, SLURM looks for one node with 40 cores to be run inside one task, for 1 hour.
* Once it found such a node, it runs the script:
** Change to the submission directory;
** Loads modules (must be done again in the submission script on Niagara);
** Sets an environment variable to set the number of threads to 40 (no hyperthreading in this example);
** Runs the appl_openmp_ex application.
* To use hyperthreading, just change --cpus-per-task=40 to --cpus-per-task=80.
== Monitoring queued jobs ==
Once the job is incorporated into the queue, there are some command you can use to monitor its progress.
squeue or sqc (a caching version of squeue) to show the job queue (squeue -u $USER for just your jobs);
qsum shows a summary of qudue by user
squeue -j JOBID to get information on a specific job
(alternatively, scontrol show job JOBID, which is more verbose).
squeue --start -j JOBID to get an estimate for when a job will run; these tend not to be very accurate predictions.
scancel -i JOBID to cancel the job.
jobperf JOBID to get an instantaneous view of the cpu and memory usage of the nodes of the job while it is running.
sacct to get information on your recent jobs.