Hardware and user environment

Common features

The cluster has one control and access computer dirac-master, a file server dirac-mester and currently 32 computing nodes from node1 to node35 (node4 to node6 are out of order). The nodes node30, node32, node35 and node9 are accessible named dinux6, dinux7, dinux8 and dinux9 for interactive calculations. They are not maintained by the cluster batch system Open Grid Scheduler/Grid Engine.

All computing nodes (except node1) have four CPU sockets. The number of cores in these CPUs was changed with every enhancement of the cluster:


node   CPU   AMD   cores/ cores/  RAM     CPU    node names
count  count type   CPU    node            freq.
---------------------------------------------------------------------------------------------------
 1      2    7551 32/64   64/128   256 GB 2.0 GHz node1
 7      4    6172    12       48    64 GB 2.1 GHz node23-node29
 4      4    6276    16       64   128 GB 2.3 GHz node1-node3, node30 (dinux6)
 1      4    6344     6       24   128 GB 2.6 GHz dirac-meister
 5      4    6376    16       64   128 GB 2.3 GHz node31, node33-node35, node11
13      4    6376    16       64   256 GB 2.3 GHz node9, node10, node12-node22  
 1      4    6376    16       64   512 GB 2.3 GHz node32 (dinux7)
 1      2   E5530     2       16    12 GB 2.4 GHz dirac-mester (Intel Xeon-CPU)

All nodes, dirac-meister and dirac-mester are interconnected with 4*10 GBit/s Infiniband with an IB switch. This special, low-latency network provides fast inter process communication and ultra fast NFS. The two 1 GBit/s ethernet ports at each node are for maintainance and backup only. Another ethernet port provides remote configraution and maintainance.

With IPoIB (Internet Protocol over Infiniband) every node may communicate with the DotHill 3300 RAID array (Fibre Channel 8 GBit/s) with 48 TB mass storage capacity.

All active devices are maintained remotely by web. For the initial comfiguration a KVM switch may used. All components are installed in two cabinets. The uninterruptible power supply with additional diesel power provides the 24/365 operation mode. The cluster needs about 32 kW energy plus the necessary cooling power.

The benchmark SPECfp_base2000 calculates 1176 for the 1.8GHz nodes, and 1924 for the 2.4 GHz nodes. The 1.9 GHz Magny Cours nodes show 1744 units. The 2.1 GHz Magny Cours nodes show 1961 units. The values for the newer nodes were not calculated.

 

Home directories at dirac-meister: /home

The cluster uses special home directories below /home. For all users there is 8 TB space on it available.

If you login for the first time at dirac-meister, your home directory below /home will be created and filled with the right initialisation files.

Mass storage space will be provided by a limited fair use policy: Use as much as you need, but don´t get into conflict with other users. If you occupy more than 500 GB, you won´t be able to work until your usage will reach 400 GB again. The files in /home will be backed up weekly.

/home may be accessed from every computing node. This is the place for your programs and data which will be used by the batch system. From dirac-meister only you may access the home directories of the central HZB Linux environment /net/home and all other directories of this environment (all below /net or /chmi). dirac-meister:/home is accessible form all HZB Linux systems with the path /net/hpc-home.

 

How to access dirac-meister:/home from Windows systems

Please create once a soft link at your Linux home directory, e.g. at dinux6:

cd; mkdir hpc; cd hpc; ln -s /hpc-home.

You may than connect wit the file share

\\home\Your_HZB_ID\hpc\hpc-home

at your Windows system.

Please replace Your_HZB_ID with your personal HZB ID three letter account code.

 

/tmp and /mnt/fhgfs: Temporary and persistent file storage

Every computing node has a persistent file system /tmp with at least 240 GB space on each node.

The BeeGFS (former known as FhGFS) directory named /mnt/fhgfs may be used from every computing node, dirac-meister, dirac-mester, dinux7 and dinux6. There are max. 18 TB available. The ultra fast, low latency technology Infiniband RMDA will be used between the computing nodes and with dirac-meister. Data transfer rates may exceed 300 MByte/s. The files there will not be backed up.

 

Features of the CPUs at the cluster (interactive servers included)

Type                                         Clock          L2-      Cores    Cores    ADP TDP
AMD Opteron                                                 Cache    at CPU   at the        
                                                            per core          cluster
                                             [GHz]         [ MiB]    [number] [number] [W] [W] 
----------------------------------------------------------------------------------------------
6344 Abu Dhabi Piledriver (dirac-meister)    2.4/---/3.2    2/2     12       24            115 
 865 Egypt                (1st delivery)     1.8/---/---    1        2       40             95	
8378 Shanghai             (1st a. 2nd Ext.)  2.4/---/---    0,5      4        0         75 115	
6168 Magny Cours Lisbon   (3rd Extension)    2.4/---/---    0,5     12       96         80 115
6172 Magny Cours Lisbon   (4th to 6th Ext.)  2.1/---/---    0,5     12      336         80 115	
6276 Interlagos Valencia  (7th a. 8th Ext.)  2.3/2.6/3.2    2/2     16      320            115
6376 Abu Dhabi Piledriver (9th to 12th Ext.) 2.3/2.6/3.2    2/2     16      896            115
7551 Epyc Rizen (13th Extension) 2,0/2,55/3.0 64 32/64 128 155 180