Supercomputing Division, Information Technology Center, The University Tokyo

Oakforest-PACS FAQ

Application for use

I would like to apply for the use of the Oakforest-PACS system. Could you tell me the start date for application and other details?
Application for use is accepted at any time as long as there are computational resources available. Please fill in the necessary fields in the application form for use of the Oakforest-PACS system, affix your seal, and submit it to the Research Support Team, Information Strategy Group, Information Systems Department, The University of Tokyo.
Are there any points I should check (take note of) before applying for and using the Oakforest-PACS system?
Various manuals are available on the application method, method of use, and other information pertaining to the Oakforest-PACS system. Please refer to these for details about the application method and service contents. (Manuals and guides provided by the manufacturers are only available to users.)

Courses of use for Oakforest-PACS
Overview of Oakforest-PACS tokens
Cost of use for Oakforest-PACS(Applicable from April 1, 2020)
Cost of use by month for Oakforest-PACS(Applicable from April 1, 2020)

(Explanations are provided on the overview of how to use the Oakforest-PACS system, etc.
For details on how to use the system, please refer to the guides and manuals provided in the User Support Portal.)

How is the quota for disk space within a group set?
In the disk space within a group, directories have been prepared for each individual user registered in the group (in the following example, /work/gz00/z3000*), as well as for shared use by all the users within the group (in the following example, /work/gz00/share). A quota is set for the users and the group for all the directories.
In the case where eight nodes are applied for, the quota for users and the group is both 8TB respectively.

Group course
Tokens
(69,120 node-hours allocated per 8 nodes applied for)
Disk space for the group
(8TB allocated per 8 nodes applied for)
/work/gz00/share
/work/gz00/z30000
/work/gz00/z30001
/work/gz00/z30002
It is stipulated that each group must have a group administrator. What kind of administration does the group administrator carry out?
For the Oakforest-PACS system, the number of tokens and nodes that are used to execute batch jobs is allocated to the entire group. However, the group administrator has the authority to set and change the maximum amount of these computational resources allocated to each user. Note that it is not possible to change the maximum number of tokens or nodes in excess of the maximum number that the group has been allocated with.
Functions of the group administrator
・Change the maximum number of tokens allocated to users within the group (Default value: Maximum number allocated to the group)
・Change the maximum number of nodes allocated to users within the group (Default value: Maximum number allocated to the group)
Is it possible to add users from a personal (group) course to my own group? In such cases, will new user numbers be assigned?
It is possible to add users to the group. However, when adding existing users of the Center to the group, such users are registered as additional users of the group based on the user numbers that they are currently using. Hence, new user numbers will not be assigned.
When adding users from a personal (group) course to my own group, how will the resources allocated to users in the group (tokens, disk storage) be handled?
For new users who are added and registered to the group, approval for the use of tokens available for the group will be granted, and a user directory will be created in the space allocated to users within the group.
When adding user “z40000” to the group

Group course
Tokens
(69,120 node-hours allocated per 8 nodes applied for)
z40000 can use the tokens
Disk space within the group
(8TB allocated per 8 nodes applied for)
/work/gz00/share
/work/gz00/z30000
/work/gz00/z30001
/work/gz00/z30002
/work/gz00/z40000 Directory created for z40000’s use
When deleting user “z40000” from the group
Group course
Tokens
(69,120 node-hours allocated per 8 nodes applied for)
z40000 cannot use the tokens
Disk space within the group
(8TB allocated per 8 nodes applied for)
/work/gz00/share
/work/gz00/z30000
/work/gz00/z30001
/work/gz00/z30002
/work/gz00/z40000 Directory for z40000’s use deleted
Is it possible to register for multiple groups?
Yes, it is. The representative of the group to be added should submit the application for use.
I was a group user, but use by the group I was registered will be cancelled. I would like to continue using the supercomputer system. What should I do?
Please submit a new application for use under the group course or personal course before the period of use of the current group you are registered with expires. Please note that if you submit an application for use under the group course or personal course after the period of use expires, you will not be able to retain the same user number and keep the files that you had been using.
Is it possible to add users for a group course at any time?
Yes, it is. Fill in the necessary fields in the application form for changes for the Oakforest-PACS system, and submit it to the Research Support Team, Information Strategy Group, Information Systems Department, The University of Tokyo.
Is there a maximum number of users who can be registered under the group course?
There are no limits to the number of users who can apply (be registered) for the group course. However, regardless of the number of users, the number of tokens that are allocated to the group is fixed. Hence, this fixed number of tokens will be used (consumed) by all the users registered in the group.
There are descriptions about the number of nodes applied for, and the maximum number of nodes. Could you tell me the difference between these?
Under the Oakforest-PACS system, it is possible to use up to the maximum number of nodes in excess of the number of nodes applied for. However, when executing a batch job in excess of the number of nodes applied for, the number of tokens consumed will increase.

▲ Return to the top of the FAQ for the Oakforest-PACS system

Token

Could you tell me about the tokens?
For explanations and FAQ about tokens, please refer to the webpage about “Tokens.”

▲ Return to the top of the FAQ for the Oakforest-PACS system

Overall system, service contents

What types of uses does educational use refer to?
The Information Technology Center, the University of Tokyo (hereafter, “the Center) provides supercomputing resources for practices in graduate and undergraduate classes.
There have been moves to use supercomputers as a practice tool for the use of methods applied to a wide range of research fields, including climate, fluid analysis, structural analysis, molecular science, nanotechnology, and aerospace. Supercomputers are already being used for practices related to structural analysis, Earth science, and fluid-related subjects in specialized undergraduate courses as well as graduate classes.
The use of supercomputers in education is anticipated to contribute to human resource development and the expansion of use of such systems. Hence, the Center accepts applications, at any time, for use in lectures and workshops (including intensive lectures) conducted by teaching faculties or teachers at graduate schools, undergraduate schools, and vocational institutes.
Use for educational purposes is not limited to use within the University of Tokyo. Applications for use outside of the University of Tokyo are also accepted. For details, please refer to “Educational Use.”

Available resources
Oakforest-PACS
Maximum execution time per job of 15 minutes, and maximum of 16 nodes (1,088 cores)
Is it possible to change the login shell?
Changes can be made using the “chsh” command. The standard login shell is set to “bash.”

▲ Return to the top of the FAQ for the Oakforest-PACS system

Compilers

How do I check for version updates for the compilers?
Information on version updates for compilers is provided on the User Support Portal. Please check the portal for more details.
Are debug profilers, etc. available?
Intel VTune Amplifier XE, Allinea DDT, etc. are available. For details, please refer to the User’s Guide for the Oakforest-PACS system and other documents available in the User Support Portal (-Documents)

▲ Return to the top of the FAQ for the Oakforest-PACS system

Libraries

Could you tell me about the versions of the libraries and applications provided?
The versions of the libraries and applications provided by the Center are listed as follows (as of December 2017). It is possible to obtain the version information using the “module” command. For more information on how to use the “module” command, please refer to the User’s Guide.
Library Version
Boost
FFTW
GNU Scientific Library
Intel library (MKL)
MassiveThreads
METIS
MT-METIS
NetCDF
Parallel netCDF
ParMETIS
PETSc
ppOpen-APPL/AMR-FDM
ppOpen-APPL/BEM
ppOpen-APPL/BEM-AT
ppOpen-APPL/DEM-util
ppOpen-APPL/FDM
ppOpen-APPL/FDM-AT
ppOpen-APPL/FEM
ppOpen-APPL/FVM
ppOpen-MATH/MP
ppOpen-MATH/VIS
ppOpen-AT
PT-Scotch
Scotch
SuperLU
SuperLU DIST
SuperLU MT
Xabclib
1.53
3.3.5
2.3
2017 update4
0.97
5.1.0
0.6.0
4.4.1
1.7.0
4.0.3
3.7.5
0.3.0
0.4.0
0.1.0
1.0.0
0.3.1
1.0.0
1.0.1
0.3.0
1.0.0
0.2.0
1.0.0
6.0.4
6.0.4
5.2.1
5.1.0
3.1
1.03

Application Versio
ABINIT-MP
AkaiKKR
ALPS
bioconductor
BioPerl
BioRuby
BLAST
feram
FrontFlow/blue
FrontISTR
GROMACS
MODYLAS
mpijava
omnicompiler
OpenFOAM
OpenMX
PHASE
R
REVOCAP
xTAPP
open1_rev5
cpa2002v009c
2.2.b4
3.4
1.6.924
1.5.1
2.4.0
0.26.00
8.1
4.5
5.1.4
1.0.4
1.2.7_ojdk1.8
1.1.1
3.0.1
3.8
2015.01.01
3.3.1
2.1
rc-150401
I would like to use the library (xxxx). Could you tell me how to use it?
Please refer to the User’s Guide for the Oakforest-PACS system and other documents available in the User Support Portal (- Browse documents).

▲ Return to the top of the FAQ for the Oakforest-PACS system

Applications

I would like to use the application (xxxx) provided. Could you tell me how to use it?
Please refer to the User’s Guide for the Oakforest-PACS system and other documents available in the User Support Portal (- Browse documents).
Can I use software that I have bought separately?
You may install and use your own software in the user directory after checking the license and other details.

▲ Return to the top of the FAQ for the Oakforest-PACS system

Job administration system

What are the types of job classes available?
Please refer to “Job Classes.”
Are there any commands I can use to check the job classes that I can submit jobs to?
You can check this using the “pjstat--rsc" command. Details about the “pjstat” command are provided in the User’s Guide for the Oakforest-PACS system (- 5.8. Display job status) available in the User Support Portal.
What is the maximum number of jobs that can be submitted, and the maximum number of jobs that can be executed simultaneously? (Applicable from April 1, 2018)
The number of jobs differs depending on the course used (personal or group course). Please refer to the following table.
Personal course
Per unit applied for
Group Course
User unit Group unit
Maximum number of jobs that can be executed simultaneously 16 No limit 16 when the number of nodes applied for is 8
20 when the number of nodes applied for is 16
Thereafter, add 4 jobs per 16 nodes

128 when the number of nodes applied for is 8
160 when the number of nodes applied for is 16
Thereafter, add 32 jobs per 16 nodes
Maximum number of submissions 16 No limit Maximum number that can be executed simultaneously × 8
128 for 8 nodes
160 for 16 nodes

Equivalent to the maximum number of jobs that can be executed simultaneously
Maximum number of nodes
(Job unit)
2,048 2,048 2,048

You can check the maximum number of jobs that can be executed simultaneously and the maximum number of jobs that can be submitted using the following command.

$ pjstat --limit
PROJECT     ACCEPT     RUN     BULK_RUN     NODE
gXXX         7/128    0/128      0/2048    0/2048
pXXXXX        0/16     0/16       0/256    0/2048
Could you tell me how to start up a batch job and an interactive job?
A batch job is submitted using the “pjsub” command. For details, please refer to the User’s Guide for the Oakforest-PACS System (- 5.4. Batch job execution, 5.5. Interactive job execution) available in the User Support Portal.
Could you tell me how to delete a job?
The “pjdel” command is used to delete a job. For details, please refer to the User’s Guide for the Oakforest-PACS System (- 5.9. Job cancellation) available in the User Support Portal.
Could you tell me how to check the execution status, etc. of a job?
The “pjstat” command is used to check the execution status, etc. of a job. For details, please refer to the User’s Guide for the Oakforest-PACS System (- 5.8. Display job status) available in the User Support Portal.
Job execution does not start.
Please check the following points.

1. There are cases where the system is waiting to secure the computational resources required for the job awaiting execution. You can check the overall usage status of the system using the “pjstat--nodeuse” command.

$ pjstat --nodeuse
RSCGRP                                                  Ratio Used/Total
regular-cache                 *****************--------   65%  2513/3846
regular-flat                  ***********************--   90%  3464/3846
debug-cache/interactive-cache -------------------------    0%     0/ 250
debug-flat/interactive-flat   *************------------   52%   129/ 250
prepost                       -------------------------    0%     0/  12

2. As job scheduling is carried out through FIFO operations, there are cases where the system is waiting to finish a job submitted earlier.

3. The time required for computation specified for a job awaiting execution may sometimes be longer than the time remaining until service suspension. You can check the time remaining until service suspension using the “pjstat” command.

4. There are cases where the system has reached its maximum limit on the number of jobs that can be executed simultaneously. Please check if other jobs are being executed, or if another user from the same group is executing the job. You can use the “pjstat --limit" command to check the number of jobs that are being executed within the group, and the maximum number of jobs that can be executed simultaneously.

5. A system problem may have occurred. Please inquire via the webpage for consultation and inquiries on system use.

▲ Return to the top of the FAQ for the Oakforest-PACS system

User Support Portal/Key registration

What can I do through the User Support Portal?
The User Support Portal allows users to register the public keys that are used to login to the Oakforest-PACS system, and to browse through the User’s Guide, manuals provided by the manufacturer, and information about version updates for compilers, etc. Some of the group administrator functions can also be carried out through the User Support Portal. For details, please refer to the User’s Guide for Oakforest-PACS Group Course Project Administrators.
I tried to register the public key through the User Support Portal, but could not login (authentication failed).
There are some points to note (shown in red outside the box) with regard to the input of the character string displayed in the default password field in the notification of user registration (hard copy) sent out to you. Please check these points and try again.
If you do not know your password, please contact the reception (Email address for inquiries).
I have changed the permissions in the .ssh directory, and cannot login anymore. What should I do?
Please contact us by providing the necessary information via the webpage for consultation and inquiries on system use. You will not be able to login if there are any errors in the permissions for the directory or files in question, or in the saved key. Hence, please take great care when editing the information. (Please back up any files, etc. beforehand, and check if you can connect from other terminals before logout.)

▲ Return to the top of the FAQ for the Oakforest-PACS system