Supercomputing Division, Information Technology Center, The University Tokyo

Introduction to the Oakforest-PACS Supercomputer System

Characteristics of the system

The Oakforest-PACS system is a supercomputer system for shared use from the Joint Center for Advanced High Performance Computing (JCAHPC), operated jointly by the Information Technology Center, the University of Tokyo and the Center for Computational Sciences, University of Tsukuba. It is the largest massively parallel cluster supercomputer in Japan, equipped with the next-generation Intel Xeon Phi processor, an ultra-high performance many-core processor produced by U.S. Intel Corporation, and 8,208 compute nodes mounted with Intel Omni-Path Architecture.

System Diagram

Hardware configuration

Overall configuration

Item Specifications
Total theoretical calculating performance 25.004 PFlops
Total number of nodes 8208
Total main storage capacity 897 TByte
Network topology Full-bisection Fat Tree
Parallel file system System name Lustre file system
Server (OSS) DDN SFA14KE
Number of servers (OSS) 10
Storage capacity 26 PB
Storage data transmission speed 500 GB/sec
Fast file cache systems Server DDN IME14K
Number of servers 25
Capacity 940 TB
Storage data transmission speed 1,560 GB/sec

Node configuration

Item Specifications
Machine name Fujitsu PRIMERGY CX1640 M1
CPU Processor name Intel® Xeon Phi ™ 7250
(code name: Knights Landing)
Number of processors (number of cores) 1 (68)
Frequency 1.4 GHz
Theoretical calculating performance 3.0464 TFlops
Memory 96 GB(DDR4)+ 16 GB(MCDRAM)
Interconnections Intel ® Omni-Path network
(100 Gbps)

Software configuration

Item Configuration
OS Red Hat Enterprise Linux 7, CentOS 7
Compiler GNU Compiler
Intel Compiler(Fortran77/90/95/2003/2008, C, C++)
Message communication library Intel MPI, Intel Omni-Path Fabric Software
Library Intel’s Math Kernel Library (MKL), BLAS, LAPACK, ScaLAPACK, FFTW, GNU Scientific Library, NetCDF, Parallel netCDF, Xabclib, ppOpen-HPC, ppOpen-AT, MassiveThreads, SuperLU, SuperLU MT, SuperLU DIST, METIS, MT-METIS, ParMETIS, Scotch, PT-Scotch, PETSc and Boost
Applications mpijava, omnicompiler, OpenFOAM, ABINIT-MP, PHASE, FrontFlow/blue, FrontISTR, REVOCAP, OpenMX, xTAPP, AkaiKKR, MODYLAS, ALPS, feram, GROMACS, BLAST, R, bioconductor, BioPerl and BioRuby
Free software autoconf, automake, bash, bzip2, cvs, emacs, findutils, gawk, gdb, make, grep, gnuplot, gzip, less, m4, python,perl, ruby, screen, sed, subversion, tar, tcsh, tcl, vim, zsh, cmake, HDF5 and git etc.