Usage hints and rules

Everyone with a valid HZB Intranet ID can use the High Performance Computing Cluster dirac with the help of the access computer dirac-meister.

A special home directory is created once at the first login with ssh.

Rules

You should obey the follwing rules which enhance the Rules of usage:

  1. All users are treated equally.
  2. All jobs are maintained by the Open Grid Scheduler/Grid Engine.
  3. You should use the ressources CPU, RAM and mass storage on /home polite and frugal.
  4. Jobs with short computation time and smaller core number usage compared to those with a long computation time are preferred.
  5. On the compute nodes only batch jobs can be run.
  6. Please use for interactive Jobs the Linux servers dinux9, dinux8, dinux7 and dinux6.
  7. Jobs in the queue all.q will be terminated one hour after start. It is a queue for test jobs only.
  8. With the queues inter, internode and magny jobs may run up to 168 hours.
  9. The queue max is restricted. It may run jobs up to 672 hours upon special request.
  10. Due to operational reasons, all jobs in inter, internode and magny may be terminated after 24 hours wall clock time.
  11. The jobs should store their context regular and often (checkpointing).
  12. Every job may use 64 (48 core nodes) or 128 (64 core nodes) GB RAM on each node maximum.
  13. Jobs with its own parallelization or multithreading have to be started with the parallel environment smp with the maximal planned number of used threads or processes.
  14. In the job script there must be a statement (-l vf=size) about the maximal memory usage. If it is missing, the job will never start.
  15. The home directory /home should not be greater than 100 GB. Exceeding storage space has to be freed as soon as possible.
  16. The users with usage exceeding 500 GB in the home directory are prevented from starting new jobs until the usage falls below 400 GB.
  17. Temporary data may stored locally at the computing nodes in /tmp and cluster-wide in /mnt/fhgfs. The data there will be there even after a reboot.
  18. User files have to be stored below /mnt/fhgfs/HZB-ID and must be owned by HZB-ID.
  19. If an account has been deleted, all corresponding files below /mnt/fhgfs/ will be deleted, too.
  20. If the file system /mnt/fhgfs/ is 90% full, all users which occupy more than 1 TB will be urgently requested to reduce their file space usage.
  21. If the file system /mnt/fhgfs/ is 95% full, files will be deleted without further notice until the value drops below 95%. Oldest files from the user which occupies the most files will be deleted first.
  22. Any user may be disabled immediately, when he uses excessive space in /mnt/fhgfs/ (e.g. more than 10 TB)

How to avoid disk I/O bottlenecks and speed up programs

Many simulation programs produce big temporary files. These may put a heavy burden on cluster performance.

You may avoid these problems if you keep them on local storage devices.

Do not keep intermediate files in /home, it uses the slower network!

Jobs with heavy I/O constraints which are run on a single node should use a working directory in /tmp. All other jobs should use a working directory in /mnt/fhgfs. This uses the fast parallel Fraunhofer filesystem BeeGFS with Infiniband RDMA.

Using /mnt/fhgfs begins with the creation of an own directory like mkdir /mnt/fhgfs/$USER once. The shell variable $USER contains your HZB-ID. For a job working directory then create a subdirectory and put job files there, finally cd to that directory and submit the job file to the grid engine.

When the job has been done, you may analyse its output on dirac-meister, dinux6, dinux7 or dinux8.

Please copy only final results back to /home, nothing more. Please stay below 100 GB disk usage at /home.

A typical job shell using /tmp on a node does it like

  1. create a new working directory in /tmp
  2. copy necessary input files to that directory, or specify them with a full path
  3. cd to the working directory
  4. run the application
  5. when done copy results, but no intermediate files, back to /home or /mnt/fhgfs
  6. delete the working directory

For an example please look at /opt/Orca/mpijob.sh.

 

How to copy big data to your local PC

If you have simulation results and want to have a closer look, you may feel comfortable with your local PC and plan to copy data files.

When these files are just some Gigabyte, go ahead.

If it comes to more than 10 GBytes - think twice before copying files! You may not move more than 40 to 50 MBytes/s, as the limiting backbone gigabit ethernet throttles the flow. And you may and will annoy other users, as this medium is shared by all users!

The best solution is to look at these data on an interactive server like dinux6 to dinux9. Especially /mnt/fhgfs is suited well for that purpose.

If the program you're using may not be installed under SuSE Linux, then copy with care: Do not use cp, scp or rsync to copy more than 10 GBytes without speed limits!

If you are familiar with rsync, use the option

--bwlimit=3m

to limit the tranfer to three MBytes/s.

For scp this could be achieved with

-l 24000

to limit the transfer to 24000 KBits/s ~ 3 MBytes/s .

With these limits the transfer of a gigabyte takes roughly 6 minutes. Best do slow copy over night, not at work hours.