Supercomputing Division, Information Technology Center, The University Tokyo

Introduction to the Wisteria/BDEC-01

Characteristics of the system

The Wisteria/BDEC-01 is a supercomputer system operated by the Information Technology Center, The University of Tokyo. Wisteria/BDEC-01 is a hybrid system including two groups of compute nodes: Simulation Nodes (Odyssey) and Data/Learning Nodes (Aquarius). Simulation nodes group (Odyssey) is consists of 7,680 compute nodes equipped with Fujitsu Limited's FUJITSU Processor A64FX, same CPU as the world's most powerful supercomputer, Fugaku. Data/Learning nodes group (Aquarius) consists of 45 compute nodes with Intel 3rd Generation Xeon Scalable Processors (code name: Ice Lake) and NVIDIA A100 Tensor Core GPUs.

System Diagram

Hardware configuration

Overall configuration

Item Wisteria-O (Odyssey) Wisteria-A (Aquarius)
Total theoretical calculating performance 25.9 PFLOPS 7.2 PFLOPS
Total number of nodes 7,680 45
Total main storage capacity 240.0 TiB 36.5 TiB
Network topology 6 dimensions mesh / Torus Full-bisection Fat Tree
Shared file system System name FEFS(Fujitsu Exabyte File System)
Server (OSS) DDN SFA7990XE
Number of servers (OSS) 16
Storage capacity 25.8 PB
Storage data transmission speed 504 GB/s
Fast file system System name FEFS(Fujitsu Exabyte File System)
Server (OSS) DDN SFA400NVXE
Number of servers (OSS) 16
Storage capacity 1.0 PB
Storage data transmission speed 1.0 TB/s

Node configuration

Item Wisteria-O (Odyssey) Wisteria-A (Aquarius)
Machine name FUJITSU Supercomputer
PRIMEHPC FX1000
FUJITSU Server PRIMERGY GX2570 M6
CPU Processor name A64FX Intel Xeon Platinum 8360Y
(code name: Ice Lake)
Number of processors (number of cores) 1 (48 + Assitant cores 2 or 4) 2 (36 + 36)
Frequency 2.2 GHz 2.4 GHz
Theoretical calculating performance 3.3792 TFLOPS 5.53 TFLOPS
Memory 32 GiB 512 GiB
Memory bandwidth 1,024 GB/s 409.6 GB/s
GPU Processor name - NVIDIA A100
Number of SM (unit) 108
Memory (unit) 40 GB
Memory bandwidth (unit) 1,555 GB/s
Theoretical calculating performance (unit) 19.5 TFLOPS
Number of unit 8
CPU-GPU connection PCI Express Gen4 x 16 lanes
(32 GB/s each direction per lane)
GPU-GPU connection NVLink x 12
(25 GB/s each direction per one)
InterConnect Tofu InterConnect-D InfiniBand HDR(200 Gbps) x 4

Software configuration

Item Wisteria-O (Odyssey) Wisteria-A (Aquarius)
OS Red Hat Enterprise Linux 8
Compiler GNU Compiler
Fujitsu Compiler (Fortran77/90/95/2003/2008, C, C++) Intel Compiler
(Fortran77/90/95/2003/2008, C, C++)
NVIDIA HPC SDK
(Fortran77/90/95/2003/2008, C, C++, OpenACC 2.7)
NVIDIA CUDA SDK
(CUDA C, CUDA C++)
Message passing library Fujitsu MPI Intel MPI, Open MPI
Library SuperLU, SuperLU MT, SuperLU DIST, METIS, MT-METIS, ParMETIS, Scotch, PT-Scotch, PETSc, Trillinos, FFTW, GNU Scientific Library, NetCDF, Parallel netCDF, HDF5, Parallel HDF5, CMake, Miniconda, Xabclib, ppOpen-HPC, MassiveThreads, Boost C++, mpiJava
Fujitsu Library (BLAS, CBLAS, LAPACK, ScaLAPACK) IntelLibrary (MKL) (BLAS, CBLAS, LAPACK, ScaLAPACK), cuBLAS, cuSPARSE, cuFFT, MAGMA, cuDNN, NCCL
Applications OpenFOAM, ABINIT-MP, PHASE, FrontFlow/blue, FrontISTR, REVOCAP-Coupler, REVOCAP-Refiner, OpenMX, MODYLAS, GROMACS, BLAST, R packages, bioconductor, BioPerl, BioRuby, BWA, GATK, SAMtools, Quantum ESPRESSO, Xcrypt, ROOT, Geant4, LAMMPS, NWChem, DeepVariant, Paraview, Vislt, POV-Ray, TensorFlow, Chainer, PyTorch, Keras, Horovod, MXNet
- HyperWorks, MATLAB, CP2K, Theano
Free software autoconf, automake, bash, bzip2, cvs, emacs, findutils, gawk, gdb, make, grep, gnuplot, gzip, less, m4, python, perl, ruby, screen, sed, subversion, tar, tcsh, tcl, vim, zsh, git, etc.
- Globus Toolkit, Gfarm, FUSE
Container virtualization singularity