Introduction to the Wisteria/BDEC-01
Characteristics of the system
The Wisteria/BDEC-01 is a supercomputer system operated by the Information Technology Center, The University of Tokyo. Wisteria/BDEC-01 is a hybrid system including two groups of compute nodes: Simulation Nodes (Odyssey) and Data/Learning Nodes (Aquarius). Simulation nodes group (Odyssey) is consists of 7,680 compute nodes equipped with Fujitsu Limited's FUJITSU Processor A64FX, same CPU as the world's most powerful supercomputer, Fugaku. Data/Learning nodes group (Aquarius) consists of 45 compute nodes with Intel 3rd Generation Xeon Scalable Processors (code name: Ice Lake) and NVIDIA A100 Tensor Core GPUs.
Hardware configuration
Overall configuration
| Item | Wisteria-O (Odyssey) | Wisteria-A (Aquarius) | |
|---|---|---|---|
| Total theoretical calculating performance | 25.9 PFLOPS | 7.2 PFLOPS | |
| Total number of nodes | 7,680 | 45 | |
| Total main storage capacity | 240.0 TiB | 36.5 TiB | |
| Network topology | 6 dimensions mesh / Torus | Full-bisection Fat Tree | |
| Shared file system | System name | FEFS（Fujitsu Exabyte File System） | |
| Server (OSS) | DDN SFA7990XE | ||
| Number of servers (OSS) | 16 | ||
| Storage capacity | 25.8 PB | ||
| Storage data transmission speed | 504 GB/s | ||
| Fast file system | System name | FEFS（Fujitsu Exabyte File System） | |
| Server (OSS) | DDN SFA400NVXE | ||
| Number of servers (OSS) | 16 | ||
| Storage capacity | 1.0 PB | ||
| Storage data transmission speed | 1.0 TB/s | ||
Node configuration
| Item | Wisteria-O (Odyssey) | Wisteria-A (Aquarius) | |
|---|---|---|---|
| Machine name | FUJITSU Supercomputer PRIMEHPC FX1000 |
FUJITSU Server PRIMERGY GX2570 M6 | |
| CPU | Processor name | A64FX | Intel Xeon Platinum 8360Y (code name: Ice Lake) |
| Number of processors (number of cores) | 1 (48 + Assitant cores 2 or 4) | 2 (36 + 36) | |
| Frequency | 2.2 GHz | 2.4 GHz | |
| Theoretical calculating performance | 3.3792 TFLOPS | 5.53 TFLOPS | |
| Memory | 32 GiB | 512 GiB | |
| Memory bandwidth | 1,024 GB/s | 409.6 GB/s | |
| GPU | Processor name | - | NVIDIA A100 |
| Number of SM (unit) | 108 | ||
| Memory (unit) | 40 GB | ||
| Memory bandwidth (unit) | 1,555 GB/s | ||
| Theoretical calculating performance (unit) | 19.5 TFLOPS | ||
| Number of unit | 8 | ||
| CPU-GPU connection | PCI Express Gen4 x 16 lanes (32GB/s per side) |
||
| GPU connection | NVLink x 12 (25GB/s per side) |
||
| InterConnect | Tofu InterConnect-D | InfiniBand HDR(200Gbps) x 4 | |
Software configuration
| Item | Wisteria-O (Odyssey) | Wisteria-A (Aquarius) |
|---|---|---|
| OS | Red Hat Enterprise Linux 8 | |
| Compiler | GNU Compiler | |
| Fujitsu Compiler (Fortran77/90/95/2003/2008, C, C++) | Intel Compiler (Fortran77/90/95/2003/2008, C, C++) NVIDIA HPC SDK (Fortran77/90/95/2003/2008, C, C++, OpenACC 2.7) NVIDIA CUDA SDK (CUDA C, CUDA C++) |
|
| Message communication Library | Fujitsu MPI | Intel MPI, Open MPI |
| Library | SuperLU, SuperLU MT, SuperLU DIST, METIS, MT-METIS, ParMETIS, Scotch, PT-Scotch, PETSc, Trillinos, FFTW, GNU Scientific Library, NetCDF, Parallel netCDF, HDF5, Parallel HDF5, CMake, Miniconda, Xabclib, ppOpen-HPC, MassiveThreads, Boost C++, mpiJava | |
| Fujitsu Library (BLAS, CBLAS, LAPACK, ScaLAPACK) | IntelLibrary (MKL) (BLAS, CBLAS, LAPACK, ScaLAPACK), cuBLAS, cuSPARSE, cuFFT, MAGMA, cuDNN, NCCL | |
| Applications | OpenFOAM, ABINIT-MP, PHASE, FrontFlow/blue, FrontISTR, REVOCAP-Coupler, REVOCAP-Refiner, OpenMX, MODYLAS, GROMACS, BLAST, R packages, bioconductor, BioPerl, BioRuby, BWA, GATK, SAMtools, Quantum ESPRESSO, Xcrypt, ROOT, Geant4, LAMMPS, CP2K, NWChem, DeepVariant, Paraview, Vislt, POV-Ray, TensorFlow, Chainer, PyTorch, Keras, Horovod, MXNet, MATLAB | |
| - | Theano | |
| Free software | autoconf, automake, bash, bzip2, cvs, emacs, findutils, gawk, gdb, make, grep, gnuplot, gzip, less, m4, python, perl, ruby, screen, sed, subversion, tar, tcsh, tcl, vim, zsh, git etc. | |
| - | Globus Toolkit, Gfarm, FUSE | |
| Container virtualization | singularity | |