Introduction to the Wisteria/BDEC-01
Characteristics of the system
The Wisteria/BDEC-01 is a supercomputer system operated by the Information Technology Center, The University of Tokyo. Wisteria/BDEC-01 is a hybrid system including two groups of compute nodes: Simulation Nodes (Odyssey) and Data/Learning Nodes (Aquarius). Simulation nodes group (Odyssey) is consists of 7,680 compute nodes equipped with Fujitsu Limited's FUJITSU Processor A64FX, same CPU as the world's most powerful supercomputer, Fugaku. Data/Learning nodes group (Aquarius) consists of 45 compute nodes with Intel 3rd Generation Xeon Scalable Processors (code name: Ice Lake) and NVIDIA A100 Tensor Core GPUs.
Hardware configuration
Overall configuration
Item | Wisteria-O (Odyssey) | Wisteria-A (Aquarius) | |
---|---|---|---|
Total theoretical calculating performance | 25.9 PFLOPS | 7.2 PFLOPS | |
Total number of nodes | 7,680 | 45 | |
Total main storage capacity | 240.0 TiB | 36.5 TiB | |
Network topology | 6 dimensions mesh / Torus | Full-bisection Fat Tree | |
Shared file system | System name | FEFS(Fujitsu Exabyte File System) | |
Server (OSS) | DDN SFA7990XE | ||
Number of servers (OSS) | 16 | ||
Storage capacity | 25.8 PB | ||
Storage data transmission speed | 504 GB/s | ||
Fast file system | System name | FEFS(Fujitsu Exabyte File System) | |
Server (OSS) | DDN SFA400NVXE | ||
Number of servers (OSS) | 16 | ||
Storage capacity | 1.0 PB | ||
Storage data transmission speed | 1.0 TB/s |
Node configuration
Item | Wisteria-O (Odyssey) | Wisteria-A (Aquarius) | |
---|---|---|---|
Machine name | FUJITSU Supercomputer PRIMEHPC FX1000 |
FUJITSU Server PRIMERGY GX2570 M6 | |
CPU | Processor name | A64FX | Intel Xeon Platinum 8360Y (code name: Ice Lake) |
Number of processors (number of cores) | 1 (48 + Assitant cores 2 or 4) | 2 (36 + 36) | |
Frequency | 2.2 GHz | 2.4 GHz | |
Theoretical calculating performance | 3.3792 TFLOPS | 5.53 TFLOPS | |
Memory | 32 GiB | 512 GiB | |
Memory bandwidth | 1,024 GB/s | 409.6 GB/s | |
GPU | Processor name | - | NVIDIA A100 |
Number of SM (unit) | 108 | ||
Memory (unit) | 40 GB | ||
Memory bandwidth (unit) | 1,555 GB/s | ||
Theoretical calculating performance (unit) | 19.5 TFLOPS | ||
Number of unit | 8 | ||
CPU-GPU connection | PCI Express Gen4 x 16 lanes (32GB/s per side) |
||
GPU connection | NVLink x 12 (25GB/s per side) |
||
InterConnect | Tofu InterConnect-D | InfiniBand HDR(200Gbps) x 4 |
Software configuration
Item | Wisteria-O (Odyssey) | Wisteria-A (Aquarius) |
---|---|---|
OS | Red Hat Enterprise Linux 8 | |
Compiler | GNU Compiler | |
Fujitsu Compiler (Fortran77/90/95/2003/2008, C, C++) | Intel Compiler (Fortran77/90/95/2003/2008, C, C++) NVIDIA HPC SDK (Fortran77/90/95/2003/2008, C, C++, OpenACC 2.7) NVIDIA CUDA SDK (CUDA C, CUDA C++) |
|
Message communication Library | Fujitsu MPI | Intel MPI, Open MPI |
Library | SuperLU, SuperLU MT, SuperLU DIST, METIS, MT-METIS, ParMETIS, Scotch, PT-Scotch, PETSc, Trillinos, FFTW, GNU Scientific Library, NetCDF, Parallel netCDF, HDF5, Parallel HDF5, CMake, Miniconda, Xabclib, ppOpen-HPC, MassiveThreads, Boost C++, mpiJava | |
Fujitsu Library (BLAS, CBLAS, LAPACK, ScaLAPACK) | IntelLibrary (MKL) (BLAS, CBLAS, LAPACK, ScaLAPACK), cuBLAS, cuSPARSE, cuFFT, MAGMA, cuDNN, NCCL | |
Applications | OpenFOAM, ABINIT-MP, PHASE, FrontFlow/blue, FrontISTR, REVOCAP-Coupler, REVOCAP-Refiner, OpenMX, MODYLAS, GROMACS, BLAST, R packages, bioconductor, BioPerl, BioRuby, BWA, GATK, SAMtools, Quantum ESPRESSO, Xcrypt, ROOT, Geant4, LAMMPS, CP2K, NWChem, DeepVariant, Paraview, Vislt, POV-Ray, TensorFlow, Chainer, PyTorch, Keras, Horovod, MXNet, MATLAB | |
- | Theano | |
Free software | autoconf, automake, bash, bzip2, cvs, emacs, findutils, gawk, gdb, make, grep, gnuplot, gzip, less, m4, python, perl, ruby, screen, sed, subversion, tar, tcsh, tcl, vim, zsh, git etc. | |
- | Globus Toolkit, Gfarm, FUSE | |
Container virtualization | singularity |