# baro - 20260919 # HWLAB setup =========== 1) Lab overview (Aula E/Room E): "CL#" == cluster # == work station # (# == 1-9) "old" is a cluster as used in the previous editions of the course _______________________________________________________________ | [old] [CL6][sw3][CL5] [CL4][sw2][CL3] [CL2][sw1][CL1] | | | |_ |_| marvin | | | | |____[CL7][sw4][CL8]___[CL9][sw5]______ 2) Work station: ________ ________ [_master_] [_master_] [__node1_] ________ [__node1_] ___[__node2_]__[_switch_]__[__node2_]___ 3) Switch VLAN partitioning: Cisco C1200-24P-4X: 24 Gigabit Ethernet Ports, 4 SFP+ 10G Transceiver Sockets. | | | |<-- cluster Left -->|<-- cluster Right -->| | Ge 1-12 | Ge 13-24 | ____________________________________________________________ | _ _ _ _ _ _ _ _ _ _ _ _ | MGMT | [#][#][#][_][_][_] [_][_][_][#][#][#] | | _ _ _ _ _ _ _ _ _ _ _ _ _ _ | HPC | [#][#][#][#][_][_] [_][_][#][#][#][#] [_][_][_][#] | |____________________________________________________________| UPLINK (Te4) VLAN configuration: Each switch has two 12-port blocks (left/right), one per cluster. Each 12-port block is split into: - 6 ports for management VLAN (top row) - 6 ports for HPC VLAN (bottom row) VLAN ports mapping: AAAAAA CCCCCC BBBBBB DDDDDD A -> mgmt-left C -> mgmt-right B -> hpc-left D -> hpc-right 4) Micro-server port map: Supermicro SuperServer E200-9B. ______________________________________________ | IPMI | | DP ____ ___ ___ ___ | | (====| |BMC| |eno||eno| _____ | | |___| |_3_||_4_| \:::/ | | HDMI ____ ___ ___ ___ ___ _____ DB9 | | _ [----] |___||___| |eno||eno| \:::/ | |(_) |___||___| |_1_||_2_| VGA | |______________________________________________| power USB USB LAN LAN inlet 2.0 3.0 Display connections: - master DP port -> DP cable -> 24" display DP port - node 1 DP port -> DP/VGA adapter -> VGA cable -> 21" display VGA port - node 2 HDMI port -> HDMI headless dongle Notes: - BMC, IPMI, SP, iLO, iDRAC, refer to the same thing: a remote management interface that provides one or multiple protocols to remotely handle the power status, sensors, inventory of the machine - ethernet cables have a latch on the RJ45 connector, be careful when plugging/unplugging - BE CAREFUL when unplugging the DisplayPort-VGA adapter on node #1: it has a locking mechanism, push the "button" on top to avoid any damage 5) Wall network sockets: 1 2 3 4 ___________________ | | | _ _ _ _ | | [#] [#] [#] [#] | | | |___________________| socket #1: left cluster, master, BMC (white cable) socket #2: left cluster, master, eno1/LAN1 (red cable) socket #3: right cluster, master, eno1/LAN1 (red cable) socket #4: right cluster, master, BMC (white cable) 6) marvin: Marvin is a short-depth server installed in the rack at the back of the lab. Marvin is the default gateway to the SISSA network and the Internet, and acts as the frontier firewall for the lab. It provides DHCP, TFTP, NFS, DNS, and web services to the lab. These services are offered on the clusters' "external network". IP addresses: 172.16.0.1 ("external network", /16) 192.168.0.1 ("management network", /16) 147.122.52.42 ("SISSA network") 7) Switches: sw1(c24-1) cluster #1 and cluster #2 sw2(c24-2) cluster #3 and cluster #4 sw3(c24-3) cluster #5 and cluster #6 sw4(c24-4) cluster #7 and cluster #8 sw5(c24-5) cluster #9 VLAN configuration: all switches are partitioned in 4 VLANs, and each cluster will be assigned 2 of these VLANs each cluster will use 1 VLAN reserved to the network dedicated to the remote management of the nodes (BMCs) each cluster will use 1 VLAN reserved to the internal private network dedicated to HPC each VLAN has 6 ports (in a row) *** each switch has an additional link reserved for management on SFP port (far right of each switch) which should NOT be unplugged or reused. On the 24-ports switches: - the leftmost 12 ports belong to the cluster on the left - the rightmost 12 ports belong to the cluster on the right Of each block of 12 ports: - the 6 ports at the top belong to the management network of the cluster assigned to that 12-port block (1x green cable for the master, 2x blue cables for the nodes) - the 6 ports at the bottom belong to the private HPC network of the cluster assigned to that 12-port block (4x grey cables) Top row (odd-numbered ports): - Management VLAN (green+blue cables, 3x) (vlan mgmt-left: 1,3,5,7,9,11) (vlan mgmt-right: 13,15,17,19,21,23) Bottom row (even-numbered ports): - HPC VLAN (grey cables, 4x) (vlan hpc-left: 2,4,6,8,10,12) (vlan hpc-right: 14,16,18,20,22,24) 8) Wall ports (from left to right): 1st: white cable remote management link of the master node (BMC), leftmost cluster 2nd: red cable "external" connection of the master node (LAN1/eno1), leftmost cluster 3rd: red cable "external" connection of the master node (LAN1/eno1), rightmost cluster 4th: white cable remote management link of the master node (BMC), rightmost cluster 9) Cluster network address space: external network (172.16.0.0/16) IP, default gateway, DNS and hostname are assigned by DHCP. All these parameters can also be statically set using the same IPs. Be sure to use "PEERDNS=no" in case you want to override these assigned values. NODE INTERFACE IP/MASK HOSTNAME master eno1 172.16..1/16 master.hwlab management network (192.168.0.0/20) (BMCs, IPMI) NODE INTERFACE IP HOSTNAME master eno2 192.168.1.1 master.mgmt node #1 bmc 192.168.2.1 c01.mgmt node #2 bmc 192.168.2.2 c02.mgmt internal network (192.168.16.0/20) NODE INTERFACE IP/MASK HOSTNAME master eno3 192.168.16.1 master.hpc node1 eno1 192.168.17.1 c01.hpc node2 eno1 192.168.17.2 c02.hpc [master eno4 N/A bonding/LACP exercise] 10) Remote SSH access to your master node If you want to connect your laptop directly to the local switch, assign to your laptop an unused IP in the private network of your cluster and then ssh to the IP address that you assigned to your master node in that VLAN. OR, if you have DHCP running on the master node, try to assign dynamically an IP to your laptop using your DHCP configuration. You can optionally connect to your master node from the MHPC wifi network using it's "external" name: ssh root@clXm.hwlab Notes: network interfaces names/labels and their position on the back of the server: master BMC (left, on top of USB 2.0 ports) eno1 ("LAN 1") lower, left eno2 ("LAN 2") lower, right eno3 ("LAN 3") upper, left eno4 ("LAN 4") upper, right internal DNS setup: forwarders: 172.16.0.1 AlmaLinux ISO image download: At Exercise 2: "6. Copy CentOS Distribution Data" and Exercise 7: "Ansible Playbooks", "Working with conditionals" do NOT use the external URLs provided and be sure to download the ISO image in a location with enough storage space to contain it (e.g. /scratch): wget -P /scratch http://172.16.0.3/repo/iso/alma/AlmaLinux-8.10-x86_64-dvd.iso If the server at 172.16.0.3 is unreachable, as fallback use 172.16.0.1 instead. You can verify the integrity of the file by running sha256sum /scratch/AlmaLinux-8.10-x86_64-dvd.iso You should get this: 463fa92155b886e31627f6713e1c2824343762245a914715ffd6f2efc300b7a1 If it differs, the iso image is likely incomplete or corrupted. make "-j" option: look at the number of processors in /proc/cpuinfo or using lscpu on the micro servers do not exceed "4": make -j 4 In the "Exercise 8" at "1.2.1. Download the necessary prerequisites" libcgroup-utils should read libcgroup-tools console buffer scrollback: In case you cannot scroll back the output buffer of your console (Shift+PageUp / PageDown), work inside "tmux", then you can use its own scroll buffer: Ctrl+b then [ enter scroll mode Arrow keys / PageUp / PageDown navigate the buffer q exit scroll mode (quit) Updated URLs: Python 3.14.7 URL: https://www.python.org/ftp/python/3.14.7/Python-3.14.7.tar.xz LMOD URL: https://github.com/TACC/Lmod/archive/refs/tags/9.4.2.tar.gz https://github.com/TACC/Lmod/archive/refs/tags/9.3.3.tar.gz https://github.com/TACC/Lmod/archive/refs/tags/9.2.10.tar.gz https://github.com/TACC/Lmod/archive/refs/tags/9.1.2.tar.gz https://github.com/TACC/Lmod/archive/refs/tags/9.0.6.tar.gz https://github.com/TACC/Lmod/archive/refs/tags/8.7.67.tar.gz GCC URL: https://ftp.gnu.org/gnu/gcc/gcc-16.2.0/gcc-16.2.0.tar.xz https://ftp.gnu.org/gnu/gcc/gcc-15.3.0/gcc-15.3.0.tar.xz https://ftp.gnu.org/gnu/gcc/gcc-15.2.0/gcc-15.2.0.tar.xz https://ftp.gnu.org/gnu/gcc/gcc-14.4.0/gcc-14.4.0.tar.xz https://ftp.gnu.org/gnu/gcc/gcc-13.5.0/gcc-13.5.0.tar.xz CMAKE URL: # requires openssl>=3.0.0 # https://github.com/Kitware/CMake/releases/download/v4.4.3/cmake-4.4.3.tar.gz # requires openssl>=3.0.0 # https://github.com/Kitware/CMake/releases/download/v4.3.5/cmake-4.3.5.tar.gz https://github.com/Kitware/CMake/releases/download/v4.2.8/cmake-4.2.8.tar.gz https://github.com/Kitware/CMake/releases/download/v4.1.5/cmake-4.1.5.tar.gz https://github.com/Kitware/CMake/releases/download/v4.0.7/cmake-4.0.7.tar.gz https://github.com/Kitware/CMake/releases/download/v3.31.12/cmake-3.31.12.tar.gz LLVM URL: https://github.com/llvm/llvm-project/archive/refs/tags/llvmorg-23.1.1.tar.gz OPENMPI URL: https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.11.tar.bz2 https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.8.tar.bz2 Older versions are provided for reference only. Expected build times (approximate, on lab machines - Pentium N3710 4c@1.60GHz/2.56GHz): Package ./configure make -j4 make install ---------------------------------------------------------- Python 3.14.7 ~1 min ~15 min < 1 min CMake 4.2.8 ~23 min ~23 min < 30 s Lmod 9.4.2 ~10 s ~10 s < 30 s gcc 16.2.0 (full; bs) ~10 s 7-8 hours < 2 min gcc 16.2.0 (c,c++; !bs) ~10 s ~70 min < 1 min llvm 23.1.1 (+clang) ~2 min 8-9 hours < 3 min Times are approximate and depend on machine load. GCC and LLVM/CLANG are not expected to be fully built during the exercise. The GCC build process can be accelerated (~1h10') by adding these configuration options: --disable-multilib --disable-bootstrap --enable-languages=c,c++ where: --disable-multilib: Stops the build of libraries for other architectures (like 32-bit compatibility on a 64-bit system), compiling only for the host machine. --disable-bootstrap: Skips the multi-stage process where GCC compiles itself multiple times, sacrificing compiler optimization checks for speed. --enable-languages=c,c++: Disables compilers for other languages: Fortran (gfortran) Ada (GNAT) Go (gccgo) D (gdc) Objective-C and Objective-C++ Modula-2 (gm2) Rust (gccrs) COBOL (gcobol) ALGOL 68 (ga68) Java (gcj) (removed in recent versions) While these settings are appropriate for this lab environment, they are not recommended for production. The LLVM/CLANG build process can be accelerated by adding these configuration options: -DCMAKE_BUILD_TYPE=Release -DLLVM_BUILD_TESTS=OFF -DLLVM_TARGETS_TO_BUILD="X86" #EOF