flex gen cases
1)define and generate cases (options, pathnames and output dirs)
2)prepare for run (enforce conditions of existence and define execution files and instructions)
3)run (generate output) depends on system ()
3bis) set reference output
4)read generated output
5)compare to reference
Generates standard flexpart run cases.
The cases are derived from reference options dir and pathnames file (working examples by default from the current repository head /release? decide on the reference FIXME).
(currently focuses on ecmwf winds: pathanmes can be edited at a later stage to accomodate different winds with the same options IMPLEMENT)
files
- export*: copy define* to a different directory to build the run cases there.
- define_<runs>: Generates flexpart run cases (groups depending on _<runs>)
- (pathanmes could be edited here)
- loop_on_pathnames*.sh: for a fived options-pathnames pair, create a run case (adding an executable and an output directory)
- run_all_sl*: execute the run cases generating a set of flexpart outputs
- read_joblist*.sh: verify that the runs finished properly.
input:
options dir:
default located repo in the same directory
i.e. ../flexpart/options/
pathname: ../flexpart/pathnames
destinantion dir:
default "../flex_cases" alongside in the same directory level
it is created every time. it it already exists from a previous run it will stop
(add --force option ENHANCE)
history
1-
1bis-postprocessing: eg change format Legras/Fast or prepare CTBTO XML descriptor
eg
/Users/ignacio/todo/doing/flexpart/2018-05-23/loop_on_ls.sh
2- run
/Users/ignacio/todo/doing/flexpart/2018-04-03/run_generated_tests.sh
/Users/ignacio/todo/doing/flexpart/2018-04-04/run_generated_tests_serial.sh
/Users/ignacio/todo/doing/flexpart/2018-04-04/run_in_njord.sh
3bis-
/Users/ignacio/todo/doing/flexpart/2018-04-04/generate2
4,5- subsequent task
/Users/ignacio/Desktop/flexpart/read_cases/logbook
repo flex_gen_input cloned to ~/repos
how to run
example
./export_flexgen_0.sh
generates ../flex_cases
containing
Makefile
define_default_run_0.sh*
loop_on_pathnames_laptop.sh*
run_all_sl_laptop*
this by itself or a s a copy can generate a list of flexpart run cases
(only default FIXME)
just running
make or
define_default_run_0.sh
generates:
pathnames
output
options
./loop_on_pathnames_laptop.sh*
generates a file (e.g. batch_dry_pathnames*.sl) that associates the executable with that patnames in a FLEXPART_run_string
(the executable is by default FLEXPART_8d70e43)
in can be configured with exporting variable $FLEXPARTvar
now all batch_dry_pathnames*.sl
can be run by
./run_all_sl_*
the stdout of flexpart is tee'd to stdoutbatch_dry_pathnames*.sl
the output* directory(es) is(are) ready to be read by flex_read
running in SLURM
1) generate the cases
2) export the files together with the run generating scripts
3) run the run generator
specify the executable and other parameters needed by SLURM
4) run the run runner: IT HAS TO BE IN THE SAME DIR AS ABOvE OTHERWISE SLURM MAY BE CONFUSED
2018-08-22
decouple:
** gen_options_*.sh
input:
base options directory (preferrably a working one)
output:
derived options directories defining the corresponding test or use cases
e.g.:
gen_options_default.sh creates one run (copies the reference)
gen_options_group1_gridout.sh (gridded output)
gen_options_group2.sh (backward runs)
generate group 1 (gridded output) of test cases from default options in flexpart local repository
once options are created, corresponding output dir and pathnames are needed (mutually consistent)
On the other hand, a well formed, operational pathnames needs in addition
- wind field files
- a corresponding AVAILABLE file referring to those winds
** gen_pathnames_*.sh
input:
- base pathnames (template -- sets wind dirs and available file path)
- set of options (output from gen_options_*.sh) located in the current directory
output: corresponding set of pathnames
e.g.:
gen_pathnames_default.sh creates pathnames and output in local dir from repos default
gen_pathnames.sh loop on list of option dirs and generate corresponding pathnames
** gen_cases_*.sh
calls gen_options_*.sh and gen_pathnames_*.sh
input: input from gen_options_*.sh and gen_pathnames_*.sh combined
output:
- output from gen_options_*.sh and gen_pathnames_*.sh combined
- output dirs (in the local directory by default)
e.g.
gen_cases_default.sh
gen_cases.sh
** gen_runs_*.sh
gen_batch_jobs_*.sh
cretes cases and defines a running string for each case
the running string is part of a file that depends on the system
can be executed directly in the command line or using a workload manager (LoadLeveler, SLURM, etc.)
e.g.
gen_runs.sh
** gen_batch_jobs_*.sh
gen_batch_jobs_cl.sh
gen_batch_jobs_sl.sh
needs some pathanmes to exist in the directory
** run_batch_cl.sh run runs
depends on how the previous batch scripts have been created
run_batch_cl.sh
run_batch_sl.sh to be copied
** check_runs*
verifies that the runs have finished without errors