Introduction to the Wisteria/BDEC-01
Characteristics of the system
The Wisteria/BDEC-01 is a supercomputer system operated by the Information Technology Center, The University of Tokyo. Wisteria/BDEC-01 is a hybrid system including two groups of compute nodes: Simulation Nodes (Odyssey) and Data/Learning Nodes (Aquarius). Simulation nodes group (Odyssey) is consists of 7,680 compute nodes equipped with Fujitsu Limited's FUJITSU Processor A64FX, same CPU as the world's most powerful supercomputer, Fugaku. Data/Learning nodes group (Aquarius) consists of 45 compute nodes with Intel 3rd Generation Xeon Scalable Processors (code name: Ice Lake) and NVIDIA A100 Tensor Core GPUs.
Hardware configuration
Overall configuration
Item | Wisteria-O (Odyssey) | Wisteria-A (Aquarius) | |
---|---|---|---|
Total theoretical calculating performance | 25.9 PFLOPS | 7.2 PFLOPS | |
Total number of nodes | 7,680 | 45 | |
Total main storage capacity | 240.0 TiB | 36.5 TiB | |
Network topology | 6 dimensions mesh / Torus | Full-bisection Fat Tree | |
Shared file system | System name | FEFS(Fujitsu Exabyte File System) | |
Server (OSS) | DDN SFA7990XE | ||
Number of servers (OSS) | 16 | ||
Storage capacity | 25.8 PB | ||
Storage data transmission speed | 504 GB/s | ||
Fast file system | System name | FEFS(Fujitsu Exabyte File System) | |
Server (OSS) | DDN SFA400NVXE | ||
Number of servers (OSS) | 16 | ||
Storage capacity | 1.0 PB | ||
Storage data transmission speed | 1.0 TB/s |
Node configuration
Item | Wisteria-O (Odyssey) | Wisteria-A (Aquarius) | |
---|---|---|---|
Machine name | FUJITSU Supercomputer PRIMEHPC FX1000 |
FUJITSU Server PRIMERGY GX2570 M6 | |
CPU | Processor name | A64FX | Intel Xeon Platinum 8360Y (code name: Ice Lake) |
Number of processors (number of cores) | 1 (48 + Assitant cores 2 or 4) | 2 (36 + 36) | |
Frequency | 2.2 GHz | 2.4 GHz | |
Theoretical calculating performance | 3.3792 TFLOPS | 5.53 TFLOPS | |
Memory | 32 GiB | 512 GiB | |
Memory bandwidth | 1,024 GB/s | 409.6 GB/s | |
GPU | Processor name | - | NVIDIA A100 |
Number of SM (unit) | 108 | ||
Memory (unit) | 40 GB | ||
Memory bandwidth (unit) | 1,555 GB/s | ||
Theoretical calculating performance (unit) | 19.5 TFLOPS | ||
Number of unit | 8 | ||
CPU-GPU connection | PCI Express Gen4 x 16 lanes (32 GB/s each direction per lane) |
||
GPU-GPU connection | NVLink x 12 (25 GB/s each direction per one) |
||
InterConnect | Tofu InterConnect-D | InfiniBand HDR(200 Gbps) x 4 |
Software configuration
Item | Wisteria-O (Odyssey) | Wisteria-A (Aquarius) |
---|---|---|
OS | Red Hat Enterprise Linux 8 | |
Compiler | GNU Compiler | |
Fujitsu Compiler (Fortran77/90/95/2003/2008, C, C++) | Intel Compiler (Fortran77/90/95/2003/2008, C, C++) NVIDIA HPC SDK (Fortran77/90/95/2003/2008, C, C++, OpenACC 2.7) NVIDIA CUDA SDK (CUDA C, CUDA C++) |
|
Message passing library | Fujitsu MPI | Intel MPI, Open MPI |
Library | SuperLU, SuperLU MT, SuperLU DIST, METIS, MT-METIS, ParMETIS, Scotch, PT-Scotch, PETSc, Trillinos, FFTW, GNU Scientific Library, NetCDF, Parallel netCDF, HDF5, Parallel HDF5, CMake, Miniconda, Xabclib, ppOpen-HPC, MassiveThreads, Boost C++, mpiJava | |
Fujitsu Library (BLAS, CBLAS, LAPACK, ScaLAPACK) | IntelLibrary (MKL) (BLAS, CBLAS, LAPACK, ScaLAPACK), cuBLAS, cuSPARSE, cuFFT, MAGMA, cuDNN, NCCL | |
Applications | OpenFOAM, ABINIT-MP, PHASE, FrontFlow/blue, FrontISTR, REVOCAP-Coupler, REVOCAP-Refiner, OpenMX, MODYLAS, GROMACS, BLAST, R packages, bioconductor, BioPerl, BioRuby, BWA, GATK, SAMtools, Quantum ESPRESSO, Xcrypt, ROOT, Geant4, LAMMPS, NWChem, DeepVariant, Paraview, Vislt, POV-Ray, TensorFlow, Chainer, PyTorch, Keras, Horovod, MXNet | |
- | HyperWorks, MATLAB, CP2K, Theano | |
Free software | Python, Julia, emacs, vim, gawk, git, gnuplot, make, screen, sed, tar, etc. | |
- | Globus Toolkit, Gfarm, FUSE | |
Container virtualization | singularity |