FLY vers. 3.1
User Guide
U. Becciani V. Antonuccio-DeLogu and M. Comparato
Email: ,
Introduction
FLY is a tree N-body parallel code for three-dimensional self-gravitating collisionless systems evolution. It is based on the tree Barnes-Hut algorithm and periodical boundary conditions are implemented by means of the Ewald summation technique.
Each particle evolves following the laws of Newtonian physics, and the differential equations are integrated using the numerical Leapfrog integration scheme. All the particles, having the same mass, must be included in a cubic box having fixed dimensions.
The supported cosmological models are: Standard Cold Dark Matter (Omega_tot=1), Lambda Models (with Cold Dark Matter), Open Models (Omega_tot<1).
The code was originally developed on CRAY T3E system using the logically SHared MEMory access routines (SHMEM). This release uses the MPI-2 standard.
This is a guide to allow the user to install FLY and create the executables to run a simulation. For a detailed discussion on the FLY features, please refer to the FLY Reference Guide, or send an e-mail message to the authors.
The system evolution, generally starts from a time-step equal to 0 and executes up to a final time-step as indicated in the MAX STEP parameter in the FLY_3.1/bin/dyn_pars file or at the end of the programmed output (in redshift values) FLY_3.1/bin/out_32.tab. The evolution is done by executing several steps (hereafter FLY job): each of them evolving the system for several time-steps, as indicated in the NUM. STEP parameter in FLY_3.1/bin/stat_pars file.
The input file of each FLY job, will be /tmp/FLY/posvel_0 (IBOD_FILE + CURR.STEP). CURR.STEP (in the FLY_3.1/bin/dyn_pars file) gives the current time-step starting point of the FLY job. The FLY job will produce the intermediate output file /tmp/FLY/posvel_10 (IBOD_FILE + (CURR.STEP+NUM STEP)), and update the CURR.STEP to this new value (CURR.STEP=CURR.STEP+NUM STEP), allowing the exact execution of the next FLY job.
FLY produces also a little header file for each intermediate file, like /tmp/FLY/posvel_10_headcontaining the current redshift, the current time-step and the DT used in the last executed time-step. This information can be reused to reset the dyn_pars file to re-start the simulation from a previous time-step.
NOTE: To run this version of FLY all the working directories must be shared among all the processors that execute FLY.
1) Download FLY
FLY is freely available at the address. Download the FLY.tar.gz and copy the FLY.tar.gz file in a directory (i.e. your HOME directory) in the system where you will run. Make you sure that a subdirectory with the same name FLY_3.1 is not present, to avoid conflicts.
1.1) Unpacking FLY
Give the following commands: gunzip FLY.tar.gz and tar -xvf FLY.tar
The tar command will create the following directories
./FLY_3.1(FLY installation directory)
./FLY_3.1/bin(Executable programs and input parameters files)
./FLY_3.1/src(Source code and assistant program)
./FLY_3.1/src/tcl(TCL program: graphic interface)
./FLY_3.1/Testcase(Testcase files)
./FLY_3.1/Ewald(Code for Ewald table generation)
./FLY_3.1/FLY_ug.doc (This file)
./FLY_3.1/FLY_ref_guide.ps (Reference Guide version 2.1.)
The Reference Guide version 2.1 can be used, with minor changes reported in this document, to setup all the parameters to run a simulation.
2) Setting the box for a simulation
Fix the box dimension (BOX SIZE) in arbitrary unit (generally in Mpc).
3) Setting the number of particles of the simulation
Fix the number of particles (N. BODIES) you like to simulate. This must be a multiple of the processor number you will use.
4) Creating the initial condition file
Create a file containing the initial positions and velocities of all the particles. FLY does not contain the program for the initial condition generation, but it is possible to use any program the user likes (e.g. COSMICS by E. Bertschinger), provided the input file as described below. The input file must contain all the positions and after all the velocities of all the particles in double C format (8 byte for each component).
FLY uses a C routine to read and write the checkpoint particles data file. The initial file input must be produced using the C language, like in the following example.
void write_b_bin(pos,vel,cfilename,nlong)
FILE *f_b_bin;
double *pos,*vel;
signed int *nlong;
char *cfilename;
{
f_b_bin = fopen(cfilename,"r+");
fwrite(pos,sizeof(double),*nlong,f_b_bin);
fwrite(vel,sizeof(double),*nlong,f_b_bin);
fclose(f_b_bin);
}
where in C language, pos and vel must be defined with the following structure pos(1:nbodies,3) and vel(1:nbodies,3) in order to maintain the same memory map distribution of the Fortran language.
Anyhow the Fortran data format pos(3,1:nbodies) and vel(3,1:nbodies) must be the input data format of FLY.
It can be often convenient to create a /tmp/FLY directory (typically a temporary massive area) to put the initial file posvel_0.
The initial condition file must be produced without control characters: it must have an unblocked file structure with "stdio" style buffering, compatible with the C fwrite and fread functions.
Otherwise an ASCII input file can be given with the following data format:
REAL(KIND=8),DIMENSION(3,1:nbodies) :: pos,vel
....generate the pos and vel arrays...
OPEN(UNIT=12,FILE='posvel_0',STATUS='NEW')
DO i=1,nbodies
write(12,1000) pos(1:3,i)
ENDDO
DO i=1,nbodies
write(12,1000) vel(1:3,i)
ENDDO
CLOSE(UNIT=12)
1000FORMAT(3(1X,F20.10))
5) Creating the input parameter files
You can use the graphical interface to create all input parameters files, if the system you use has the Tcl/Tk installed.
prompt> cd ./FLY_3.1/src/tcl
prompt> wish fly_3.1.tcl
The graphical interface will guide the user to create all the following files.
Otherwise you can use the line command assistant program, on a Linux System, to create the input parameters files. The assistant program can be created, compiling the prepare.F90 file in the source code directory.
prompt> cd ./FLY_3.1/src
prompt> ./assistant_linux
The assistant program is structured with several sections and, where applicable, it gives a default value.
The user must fix a Working Directory where all the input parameter files will be stored and an Executable Directory. The assistant can not create these directories but the user must create them before starting the assistant.
The following files (5.1 – 5.6) must be created in the working directory. We will use the assistant program to discuss the content of each file.
At the beginning, the assistant asks to set the Working Directory where all the following input parameter files will be created. This directory must be created. The default value is indicated in square brackets
Setting the Working Directory
Please set the working directory. The default value is ../bin :
Working Directory [../bin/] :
The user must indicate the absolute path of the Working Directory.
The Testcase directory included in the FLY package, gives an example of all the input files.
5.1) Creating the ./FLY_3.1/bin/fly_fnames file
This file must have the name above indicated. It contains the filenames (including the path) of all the files that will be created. If this file does not exist, the assistant program will create it, and asks for the filenames: the default filenames will be as follow. If this file already exists, the assistant will use this filenames as default values.
Starting generation of ../bin/fly_fnames file...
Do you want to create this file (Y/N) : y
All NOT-ABSOLUTE filename paths will be ../bin/ :
Please give stat_pars filename [stat_pars] :
Please give dyn_pars filename [dyn_pars] :
Please give ew_grid filename [ew_grid] :
Please give ew_tab filename [ew_tab] :
Please give out32.tab filename [out32.tab] :
Please give ql.tab filename [ql.tab] :
At the end the file will be created, containing the filenames that the user gives.
STAT_PARS =stat_pars
DYN_PARS =dyn_pars
EW_GRID =ew_grid
EW_TAB =ew_tab
OUT32_TAB =out32.tab
QL_TAB =ql.tab
In the following we will use the default filenames.
5.2) Creating the stat_pars file
This file has the following structure:
HEADER =try_sim
NUM. STEP= 10
MAX_TIME = 10800
DELTA T. = 0.00450
DT VAR. =T
OPEN PAR.= 0.80
SOFT PAR.= 0.01000
QUADRUP. =T
OMEGA.CDM= 0.7000
OMEGA.HDM= 0. 0
OMEGA.LAM= 0.3000
HUB_CONST= 0.650000
N. BODIES= 2097152
MASS BODY= 0.00120
IBOD_FILE=/tmp/FLY/posvel_
IBOD_TYPE=B
OBOD_FILE=/tmp/FLY/out_
OBOD_TYPE=B
QLK_FILE =/tmp/FLY/qlk_
This file contains some information on the simulations that must be never changed during the system evolution. The assistant asks for the following:
5.2.1) Identification header of simulation (max 40 char) :
Type an HEADER string to identify the simulation.
5.2.2) Number of step of each FLY job [10]:
It is the number of time-steps that each FLY job will perform.
5.2.3) Maximum single CPU time (sec) for each run [10800] :
It is the maximum CPU time allowed for each job. This value must be lower or equal to the single CPU time as specified for the submitted job. If this value is greater than 0 the NUM. STEP parameter is not considered and the number of time-step cycles for a single job, is automatically computed by FLY using this reference number.
5.2.4) Delta T integrator time-step :
It is the DELTA T. integrator time-step used in the system evolution (format F20.15). For cosmological simulations a possible choice can be found in Efstathiou et al. (ApJS, vol. 57, p. 241, 1985). Give here a safety value.
5.2.5) Use FLY Variable dt ? (T/F) [T] :
FLY also uses an adaptive Delta T integrator determination automatically computed in dt_comp.F subroutine.
As a default FLY adopt a criterion based on the evaluation of the maximum acceleration among all the particles:
where soft, maxi=1,N|ai| are the
Users that want to use this adaptive Delta T integrator must answer T to this question. If the user answersF to this question, FLY will use the DELTA T. integrator the user given in the previous question.
5.2.6) Opening Angle Parameter (> 0.1) [0.8] :
The OPEN PAR (format F4.2) is used to open or close the cells during the tree walking phase to build the interaction list for each particle.
5.2.7) Softening length :
The SOFT PAR value (format F20.15) is crucial in determining the amount of "hardening" of the gravitational interaction, ultimately the amount of small-scale substructure which is not destroyed during the gravitational evolution of the system.
5.2.8) Use quadrupole momentum ? (T/F) [T] :
This is the QUADRUP. value. If QUADRUP.=T (true) the quadrupole momentum is used in the particle-cell
interaction to compute the force components.
5.2.9) Omega CDM parameter :
It is the OMEGA.CDM value (format F20.15).
5.2.10) Omega HDM parameter :
It is the OMEGA.HDM value (format F20.15).
5.2.11) Omega LAM parameter :
It is the OMEGA.LAM value (format F20.15).
Note: the equations of motion in the present version can be solved only for globally CLOSED models (OMEGA.CDM+OMEGA.HDM+OMEGA.LAM=1).
5.2.12) Hubble constant [0.65] :
It is the Hubble constant value (format F10.6).
5.2.13) Number of processors of the parallel run [8]:
It must be the number of processor you want to use during the FLY simulation. This value must be greater than 0. In order to obtain the best performances we recommend using a number of processors equal to a power of two.
5.2.14) Number of particles of the simulation:
Give the N. BODIES value.
5.2.15) Mass of each particle:
Give the MASS BODY value (format F20.15).
5.2.16) Root-name of the checkpoint file of FLY (max 256 char) [/tmp/FLY/posvel_] :
This is the root-name of the filename (IBOD_FILE) including the path of the initial condition (see sect. 4). The root-name must not contain the final number (like 0) that is automatically appended by FLY.
5.2.17) Checkpoint file type of FLY: B (binary) or A (ASCII) [B] :
The user must specify B for binary or A for Ascii data format for the IBOD_FILE data type.
5.2.18) Root-name of the Programmed Output file of FLY (max 256 char) [/tmp/FLY/out_] :
This is the filename (OBOD_FILE) including the path of the files containing the programmed output position and velocity (max 256 char). It must not contain the final number that is automatically appended by FLY code. These output files are the scientific data output created by FLY. The redshift value when these output files will be created, are listed in ./FLY_3.1/bin/out32.tab file. The user must use a different filenames than the previous one (5.2.16 par.).
5.2.19) Pogrammed Output file type of FLY: B (binary) or A (ASCII) [B] :
The user must specify B for binary or A for Ascii data format for the OBOD_FILE data type.
5.2.19.1) Binary output data file
The data output is generated in binary format with a C procedure, as an unblocked file structure with "stdio" style.
void c_write_f32(pos,vel, cfilename, nlong)
double *pos,*vel;
signed int *nlong;
char *cfilename;
FILE *f32;
{
f32 = fopen(cfilename,"wb")
fwrite(pos,sizeof(double),*nlong,f32);
fwrite(vel,sizeof(double),*nlong,f32);
fclose(f32);
}
5.2.19.2) Ascii output data file
FLY can generate an ASCII output file as follows:
REAL(KIND=8),DIMENSION(3,1:nbodies) :: pos,vel
CHARACTER(LEN=132):: fileout
....generate the fileout name as root_name//time_step (i.e. /tmp/FLY/out_10)
OPEN(UNIT=32,FILE=fileout,STATUS='NEW')
DO i=1,nbodies
write(32,1000) pos(1:3,i)
ENDDO
DO i=1,nbodies
write(32,1000) vel(1:3,i)
ENDDO
CLOSE(UNIT=32)
1000FORMAT(3(1X,F20.10))
5.2.20) Root-name of the ASCII quick-look file of FLY (max 256 char) [/tmp/FLY/qlk_] :
Give the root-name of the file containing the initial position and velocity (max 256 char). The root-name must not contain the final number that is automatically appended by FLY code (the same number, a redshift value, of the files in 5.2.18 par.). We must use a different filename than the previous files (5.2.16 and 5.2.18 par.).
These output files are not more automatically produced by FLY. The ASCII files can be created using random particles positions and velocity, using an external program. The randomized table ./FLY_3.1/bin/ql.tabis automatically created by the FLY assistant.
The redshift values, when these output files will be created, are listed in ./FLY_3.1/bin/out32.tab file.
During the system simulation an header file (suffix _hea, i.e. /tmp/FLY/qlk_hea) will be automatically created.
The qlk_hea file has a format like the following:
header_line
ASCII
time
262144
17 /gpfs/temp/ube/data262k/qlk_60.0000
10 /gpfs/temp/ube/data262k/qlk_50.0000
68 /gpfs/temp/ube/data262k/qlk_20.0000
The value 262144 is the number of data points included in the quick-look file, the filename suffix ( _60.0000) is the corresponding redshift value when the output is produced, and the number before the filenames (17, 10, 68) is the number of time-step that FLY has run from the previous step. The qlk files must be produced by an external user program.
5.3) Creating the dyn_pars file
This file contains information on the simulations that can be changed during the system evolution, and has the following structure:
CURR.REDS= 50.00
CURR.STEP= 0
MAX STEP= 100
LIV. GROU= 9
BODY GRO.= 16
BAL. PAR = 0.95
GR. PAR = 0.50
SORT_LEV.= 3
BOX SIZE = 50.0000
X MIN VER= 0.000000
Y MIN VER= 0.000000
Z MIN VER= 0.000000
This file is re-read at the begin of each time-step during a FLY job. This file is automatically updated by FLY at the end of each FLY job: the CURR. STEP and the CURR.TIME are updated at the final time-step that the FLY job reaches.
5.3.1) Redshift initial value :
Give the CURR.TIME: in the present cosmological version this is the redshift parameter at the current time-step.
5.3.2) Initial time-step value (0 - 999) [0]:
Give the initial time-step value CURR.STEP. We strongly recommend starting from 0 as initial time step. In this case FLY automatically checks for all the particle positions and re-arrange these inside the cubic region.
If you give a different time-step value you are responsible to verify that all the particle positions are inside the box region, otherwise an error will occur at the run-time.
5.3.3) Final time-step value (1 - 999) :
Give the final time-step value MAX STEP. When a FLY job reaches this value the simulation is stopped. FLY stops the simulation when MAX STEP is reached or the final redshift listed in out_32.tab is reached (see 5.4 par.)
5.3.4) Critical level for grouping value [9] :
Give the tree level threshold LIV. GROU where the grouping cell can be started to form, being LIV. GROU=0 the root level. We recommend using a six level for a 2-million-particles simulation up to 9 levelsfor 16-million-particles.
5.3.5) Particles max value in grouping cells [16] :
Give the BODY GROU parameter value. It is the maximum number of particles that a grouping cell can contain. We recommend to use a value from 8 up to 32.
5.3.6) Load Balance Parameter (0.01 - 1.0) [0.95] :
Give the BAL. PAR. value. It is used to balance the load in the processor. This value is re-computed during the run, to allow a perfectly load balancement. This number is the percentage of the local particles that must be computed by the local processor. The remaining portion is computed by the first available processor. We recommend to use the default value 0.95.
5.3.7) Group Balance Parameter (0.01 - 1.0) [0.50] :
Give the GR. PAR. value. It is used to balance the load of particles groups, in the processor. This value is re-computed during the run, to allow a perfectly load balancement. This number is the percentage of the local group cells that must be computed by the local processor. The remaining portion is computed by the first available processor. We recommend to use the default value 0.95.
5.3.8) Size of the box value (BOX SIZE) : 50.0
Give the box dimension BOX SIZE, in arbitrary unit (data format F9.4) (generally in Mpc).
5.3.9) X-Y-Z coordinates of the position of the lower-left corner in the box.
Position X :
Position Y :
Position Z :
It gives the coordinates of the lower-left corner of the cubic region, in arbitrary units (data format F10.6).
5.3.10) NOTES
The parameters GROUP FL. and BOX COMP. are for future usages.