Go to the first, previous, next, last section, table of contents.


Running LSP

LSP can be run on either single- or multiple-processor machines. The multiple-processor version is initiated by using the MULTI_PROCESS compiler directive (see section MULTI_PROCESS). An input file must be present in order for LSP to run (see section Input Variables) and a command file may be present (see section Command File), but is optional.

Single-Processor Machines

The single-processor version of LSP will run on Unix, Windows, and Mac OS X workstations. At the command line or shell prompt, enter

lsp [-r] [-ra] [-n N] [-s] input.lsp

where `input.lsp' is the user-supplied input file. To run in background with redirection of terminal output, one can use `lsp [-opt] input.lsp >&log&' (assuming a C shell), where `log' is the name of the file containing the redirected output.

The optional `-r' flag is used to restart a previous run from a restart dump, which will have the name `restart.dat' or `restart.alt'. This file is read after the input file is processed. The restart file contains probe history, particle and field data from the previous run. If the rename_restart_flag is ON (see section rename_restart_flag (flag)), then the most recent restart file may have the `.alt' extension (this can be determined by looking at the file dates). In this case, the file with the `.dat' extension can be removed from the simulation directory before restarting. The code will then attempt a restart from the `.alt' file. If this restart fails, the `.dat' file can be used as a backup. Alternatively, the `-ra' option has the same effect as the `-r' flag, except that the restart is performed from the `restart.alt' file instead of the `restart.dat' file. This is a more elegant way to restart from the `.alt' file than moving or deleting the `.dat' file.

Probe history data reside in the restart file and are written to the history file `history.p4' when the restart run begins, so this file need not be preserved between restarts. The user ordinarily increases one of the time_limit parameters on the input file prior to restarting a simulation, unless the previous run was stopped before reaching this limit. However, use of the number_of_steps parameter on a restart run will cause the simulation to execute exactly that number of additional timesteps, unless the time_limit is reached first.

The `-n N' sequence, where N is an unsigned integer, will cause the code to run that number of timesteps, regardless of what is specified on the input file. This may be useful for checking the simulation setup without directly changing the input file.

The `-s' option is used to perform initialization only and stops the run immediately regardless of any time limits in the input file. This may also be useful for checking the simulation setup before submitting a batch run, for example.

Multiple-Processor Machines

The multiple-processor version of LSP uses the MPI message-passing library (`http://www-unix.mcs.anl.gov/mpi/'). This version can make use of decomposition of the simulation space into separate domains in order to distribute the work-load among multiple processes and thereby achieve faster running times.

The multiple-processor version of LSP produces a separate restart file for each region, named `restart1.dat', `restart2.dat', .... All of these files must be present for the restart to work correctly. If the rename_restart_flag is ON (see section rename_restart_flag (flag)), then the most recent restart files may have the `.alt' extension (this can be determined by looking at the file dates). In this case, the files with the `.dat' extension can be moved out of the current run directory before restarting. The code will then attempt a restart from the `.alt' files. If this restart fails, the `.dat' files can be used as a backup by restoring them to the run directory. The most common reason for restart failure is a corrupted or incomplete restart file due to time-limit termination while the file was being written. A better way to restart a simulation from the `.alt' files than moving or deleting the `.dat' files is to use the `-ra' option on the command line instead of the `-r' flag.

Workstation Network

On a workstation network with the MPICH version of MPI installed (`http://www-unix.mcs.anl.gov/mpi/mpich/'), start a multiple-processor version of LSP using the p4 (no relation to the P4 postprocessor, section P4 Postprocessor) communication device as follows:

lsp -p4pg pgroup [-opt] input.lsp

where the file `pgroup' specifies a p4 processor group, such as

local 0
cerebro 1 /usr1/run7/lsp
achilles 1 /usr1/run7/lsp

which specifies that three processes are to be used (see section number_of_processes (integer)), one on the local computer, one on computer "cerebro" (which may also be the local computer), and one on computer "achilles". The path to the executable on each computer must be supplied. The path also specifies the run directory, so the executable, or a link to it, must be in the run directory.

DEC Cluster

On the DEC Unix cluster, start a multiple-processor version of LSP with:

dmpirun -np NP lsp [-opt] input.lsp

where `NP' is the number of processes to use. This number must be the same as the number_of_processes parameter in the input file (see section Control Input).

Intel Teraflop

On the Intel Teraflop (ASCI Red), start LSP interactively on `NP' processors using

yod -sz NP lsp [-opt] input.lsp >&log&

An NQS (Network Queueing System) job for `NP' processors can be submitted to the `QUEUE' queue on the Teraflop for a time `HH:MM:SS' (hours:mins:secs) using

qsub -q QUEUE -lP NP -lT HH:MM:SS -o log.lsp script.lsp

where `script.lsp' is a shell-script file. An example of this file follows.

#!/bin/sh
date
cd $QSUB_WORKDIR
/cougar/bin/yod lsp [-opt] input.lsp

where QSUB_WORKDIR is a shell variable which resolves to the working directory from which the qsub command is issued.

ASCIQ

On the ASCIQ system, start LSP interactively using

prun -n NP lsp [-opt] input.lsp >&log&

where `NP' is the number of processors.

An LSF (Load Sharing Facility) job for `NP' processors on `ND' nodes can be submitted to the `QUEUE' queue on the ASCIQ for a time `MMM' (min) using

bsub -q QUEUE -n NP -o log.lsp -W MMM prun -N ND lsp [-opt] input.lsp

Each node consists of 4 processors; therefore, a multiple of 4 must be used for `NP' when submitting a job beyond 4.

IBM-SP2

On the IBM-SP2, start LSP interactively using

lsp -procs NP [-opt] input.lsp >&log&

where `NP' is the number of processors.

Generally, the IP switch will be used for interactive jobs. At the command prompt, type setenv MP_EUILIB ip.

A LoadLeveler batch job can be submitted using

llsubmit script.lsp

where `script.lsp' is a control file such as

#@ job_name        = run1
#@ initialdir      = /scratch2/mydir/
#@ notification    = always
#@ notify_user     = myname@company.com
#@ error           = lsp.$(Cluster).err
#@ output          = lsp.$(Cluster).out
#@ job_type        = parallel
#@ requirements    = (Adapter == "hps_user")
#@ min_processors  = NP
#@ max_processors  = NP
#@ environment     = MP_EUILIB=us;MP_INFOLEVEL=3;MP_LABELIO=yes
#@ checkpoint      = no
#@ wall_clock_limit= HH:MM:SS
#@ account_no      = ACCOUNT_NUMBER
#@ queue

set nodes  = `echo $LOADL_PROCESSOR_LIST`
cat ${nodes}
set runid = "run1"
cp -f $HOME/mydir/lsp .
cp -f $HOME/mydir/runs/${runid} .
lsp ${runid}
tar cvf ${runid}.tar ${runid} *.dat *.out *.err
compress -f ${runid}.tar
hpsscp ${runid}.tar.Z hpss:/s/hpss/myname/${runid}.tar.Z
echo "Job completed"

where `NP' is the number of processors, `HH:MM:SS' (hours:mins:secs) is the wall-time and `ACCOUNT_NUMBER' is the user's account.

Startup Messages

When LSP begins running, several lines of data relating to when and where it was compiled, the compiler directives used, the input file name, and the start time are generated. An example is:

Compiled Sat Nov 14 12:24:57 MST 1998 on achilles

Compiler flags: -O4 -std1 -warnprotos

Code options defined by user: -DSTATIC_FIELDS -DMULTI_PROCESS

Code options defined at compile-time:
    STATIC_FIELDS
    MULTI_PROCESS

Coordinate system used:
    CARTESIAN

Input data file: input.lsp

Start time = Sat Nov 14 12:27:56 1998

The "code options" listed are those specified by the user at compilation time (see section Compiling LSP). These options are also referred to as "compiler directives".

Messages Generated By Errors in Input File

When starting an LSP run, a variety of error checking takes place to diagnose possible mistakes in the choice of compiler directives and the specification of input data. These errors will usually cause a printed message in standard output and prevent the simulation from continuing. However, in some cases, a warning message may occur without stopping the code from running. The user should be alert for this type of occurrence.

Input Parameter Errors

Most of the data appearing in the input file for a simulation is syntax dependent. Interpretation by the LSP code is sensitive to the spelling and order of input parameters. At present, the code issues an error message and exits immediately upon finding items in the input text that cannot be identified.

Boundary Errors

The user is responsible for simulation set-up. However, LSP is able to check that at least the boundaries of each domain (and therefore the entire simulation space) are well defined. This means that some physical boundary condition is defined everywhere at the boundaries. If any cells along an outer boundary are found not to be covered by one of the conditions necessary for simulation fidelity, appropriate error messages are printed which locate the portion of the grid containing the fault, and the simulation is aborted. An example is:

P0: boundary check error at ZMIN,
    x-range 0.00e+00:1.00e+00, y-range 0.00e+00:0.00e+00

 *** Error(s) on domain boundary detected ***
     Domain boundary = ZMIN
     Check input data for completeness

which indicates that the domain under control of process 0 has been found to have an undefined boundary at the lower side in the z-coordinate. The most common cause for this kind of error is a "hole" in the simulation boundary which can be patched with a conducting surface.

Command File

While running a simulation, LSP checks periodically for a file named `command' in the directory from which it is started. This file may contain one of the following strings, which cause the code to perform certain actions at the current timestep:

dump: Write all particle, scalar, field, and diagnostic dumps.
stop: Write a restart dump and stop the run.
abort: Stop the run with no restart dump.
rdump: Write a restart dump and continue.
report: Print domain statistics to gauge load balance.
particles: Print particle statistics by species.
balance: Rebalance the computational load among processes.

Under either Windows, Unix or Mac OS X, one can create the file `command' containing the word `stop' with the command echo stop > command.


Go to the first, previous, next, last section, table of contents.