README 6.79 KB
Newer Older
Ignacio Pisso's avatar
Ignacio Pisso committed
1
2
flex gen cases

Ignacio Pisso's avatar
Ignacio Pisso committed
3
4
5
6
7
8
9
10
11
12
13
1)define and generate cases (options, pathnames and output dirs)
2)prepare for run (enforce conditions of existence and define execution files and instructions)
3)run (generate output) depends on system ()
3bis) set reference output 
4)read generated output
5)compare to reference

Generates standard flexpart run cases.
The cases are derived from reference options dir and pathnames file (working examples by default from the current repository head /release? decide on the reference FIXME).

(currently focuses on ecmwf winds: pathanmes can be edited at a later stage to accomodate different winds with the same options IMPLEMENT)
Ignacio Pisso's avatar
Ignacio Pisso committed
14
files 
15
16
- export*: copy define* to a different directory to build the run cases there.

Ignacio Pisso's avatar
Ignacio Pisso committed
17
18
- define_<runs>: Generates flexpart run cases (groups depending on _<runs>) 
- (pathanmes could be edited here)
19

Ignacio Pisso's avatar
Ignacio Pisso committed
20
21
22
23
24
25
26
27
28
- loop_on_pathnames*.sh: for a fived options-pathnames pair, create a run case (adding an executable and an output directory)
- run_all_sl*: execute the run cases generating a set of flexpart outputs
- read_joblist*.sh: verify that the runs finished properly.

input:
 options dir: 
   default located repo in the same directory 
   i.e. ../flexpart/options/
 pathname: ../flexpart/pathnames
29

Ignacio Pisso's avatar
Ignacio Pisso committed
30
31
32
33
34
35
 destinantion dir: 
  default "../flex_cases" alongside in the same directory level
  it is created every time. it it already exists from a previous run it will stop 
  (add --force option ENHANCE)

history
Ignacio Pisso's avatar
Ignacio Pisso committed
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
1-
1bis-postprocessing: eg change format Legras/Fast or prepare CTBTO XML descriptor
eg 
/Users/ignacio/todo/doing/flexpart/2018-05-23/loop_on_ls.sh

2- run
/Users/ignacio/todo/doing/flexpart/2018-04-03/run_generated_tests.sh
/Users/ignacio/todo/doing/flexpart/2018-04-04/run_generated_tests_serial.sh
/Users/ignacio/todo/doing/flexpart/2018-04-04/run_in_njord.sh

3bis-
/Users/ignacio/todo/doing/flexpart/2018-04-04/generate2
4,5- subsequent task
/Users/ignacio/Desktop/flexpart/read_cases/logbook

Ignacio Pisso's avatar
Ignacio Pisso committed
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
repo flex_gen_input cloned to ~/repos

how to run
example

./export_flexgen_0.sh

generates ../flex_cases 

containing 

Makefile
define_default_run_0.sh*
loop_on_pathnames_laptop.sh*
run_all_sl_laptop*

this by itself or a s a copy can generate a list of flexpart run cases
(only default FIXME)

just running 
make or
define_default_run_0.sh

generates: 
pathnames 
output
options

./loop_on_pathnames_laptop.sh*
generates a file (e.g. batch_dry_pathnames*.sl) that associates the executable with that patnames in a FLEXPART_run_string
(the executable is by default FLEXPART_8d70e43)
in can be configured with exporting variable $FLEXPARTvar

now all batch_dry_pathnames*.sl
can be run by

./run_all_sl_*

the stdout of flexpart is tee'd to stdoutbatch_dry_pathnames*.sl

the output* directory(es) is(are) ready to be read by flex_read

93
94
95
96
97
98
99
100
running in SLURM
1) generate the cases
2) export the files together with the run generating scripts
3) run the run generator 
    specify the executable and other parameters needed by SLURM
4) run the run runner: IT HAS TO BE IN THE SAME DIR AS ABOvE OTHERWISE SLURM MAY BE CONFUSED


Ignacio Pisso's avatar
Ignacio Pisso committed
101
102
103
2018-08-22
decouple:

104
105
106
107
108
109
110
111
112
113
114
** gen_options_*.sh
input: 
  base options directory (preferrably a working one)
output: 
  derived options directories defining the corresponding test or use cases  
e.g.:
 gen_options_default.sh creates one run (copies the reference)
 gen_options_group1_gridout.sh (gridded output)
 gen_options_group2.sh (backward runs)
   generate group 1 (gridded output) of test cases from default options in flexpart local repository

Ignacio Pisso's avatar
Ignacio Pisso committed
115

116
117
118
119
once options are created, corresponding output dir and pathnames are needed (mutually consistent)
On the other hand, a well formed, operational pathnames needs in addition 
- wind field files
- a corresponding AVAILABLE file referring to those winds 
Ignacio Pisso's avatar
Ignacio Pisso committed
120

121
** gen_pathnames_*.sh
Ignacio Pisso's avatar
Ignacio Pisso committed
122
123
input: 
- base pathnames (template -- sets wind dirs and available file path)
124
- set of options (output from gen_options_*.sh) located in the current directory 
Ignacio Pisso's avatar
Ignacio Pisso committed
125
output: corresponding set of pathnames
126
127
128
e.g.: 
 gen_pathnames_default.sh  creates pathnames and output in local dir from repos default  
 gen_pathnames.sh loop on list of option dirs and generate corresponding pathnames
Ignacio Pisso's avatar
Ignacio Pisso committed
129

130
** gen_cases_*.sh
Ignacio Pisso's avatar
Ignacio Pisso committed
131
132
133
134
135
calls gen_options_*.sh and gen_pathnames_*.sh
input: input from gen_options_*.sh and gen_pathnames_*.sh combined 
output: 
- output from gen_options_*.sh and gen_pathnames_*.sh combined 
- output dirs (in the local directory by default) 
136
137
138
e.g. 
 gen_cases_default.sh 
 gen_cases.sh
Ignacio Pisso's avatar
Ignacio Pisso committed
139

140
141
** gen_runs_*.sh
   gen_batch_jobs_*.sh 
Ignacio Pisso's avatar
Ignacio Pisso committed
142
143
cretes cases and defines a running string for each case
the running string is part of a file that depends on the system
144
145
146
can be executed directly in the command line or using a workload manager (LoadLeveler, SLURM, etc.) 
e.g. 
gen_runs.sh 
Ignacio Pisso's avatar
Ignacio Pisso committed
147

148
149
150
151
** gen_batch_jobs_*.sh
gen_batch_jobs_cl.sh
gen_batch_jobs_sl.sh
needs some pathanmes to exist in the directory
Ignacio Pisso's avatar
Ignacio Pisso committed
152

153
154
155
156
** run_batch_cl.sh run runs
depends on how the previous batch scripts have been created
run_batch_cl.sh
run_batch_sl.sh to be copied 
Ignacio Pisso's avatar
Ignacio Pisso committed
157

158
159
** check_runs*
verifies that the runs have finished without errors
Ignacio Pisso's avatar
Ignacio Pisso committed
160
161
162



163
164
165
166
167
168
169
170
171
172
173
174
 from scratch
./gen_options.sh 
./gen_options.sh define_options_1.sh
# now we have 9 options
./gen_pathnames.sh
# now we have corresponding pathnames and output
./gen_batch_jobs_cl.sh
# command line batch files
or 
./gen_batch_jobs_cl.sh $fp_exec $args
./gen_batch_jobs_cl.sh FLEXPART -i

Ignacio Pisso's avatar
Ignacio Pisso committed
175

176
./export_flexgen.sh exportA-I _[A-I]
Ignacio Pisso's avatar
Ignacio Pisso committed
177

178
179
./gen_options_all.sh ; ./gen_pathnames.sh ; 
WARNING: gen_options_all.sh on preexisting dirs creates a options subdir! FIXME
Ignacio Pisso's avatar
Ignacio Pisso committed
180
181


182
183
184
185
186
187
188
in laptop:
./gen_batch_jobs_cl.sh
run: run_batch_cl.sh

in server with SLURM 
gen_batch_jobs_SLURM.sh
run_batch_SLURM.sh
Ignacio Pisso's avatar
Ignacio Pisso committed
189
190


Ignacio Pisso's avatar
Ignacio Pisso committed
191
./export_flexgen.sh exportA-I _[A-I]
Ignacio Pisso's avatar
Ignacio Pisso committed
192

Ignacio Pisso's avatar
Ignacio Pisso committed
193
194
195
196
197
198
199
200
201
202
203
1 
 A fwd bin
 B fwd ncd
2
 C bwd
3
 D part
./export_flexgen.sh exportD _D
./gen_options_all.sh ; ./gen_pathnames.sh ; ./gen_batch_jobs_SLURM.sh
4
 E unit ind fwd and bwd
Ignacio Pisso's avatar
Ignacio Pisso committed
204
205
206
 
./export_flexgen.sh exportE _E
cd exportE
Ignacio Pisso's avatar
Ignacio Pisso committed
207
208
./gen_options_all.sh ; ./gen_pathnames.sh ; ./gen_batch_jobs_SLURM.sh

209
210
211
gen_batch_jobs_cl.sh FLEXPART -i


Ignacio Pisso's avatar
Ignacio Pisso committed
212
213
214
215
216
217
218
219
220
221
5
 F species
6
 G nested 
7
 H domainfill
8
 I init cond
9 
 J CBL
Ignacio Pisso's avatar
Ignacio Pisso committed
222
223
224



Ignacio Pisso's avatar
Ignacio Pisso committed
225
226
227
./export_flexgen.sh exportG _G
cd exportG
./gen_options_all.sh ; ./gen_pathnames.sh ; ./gen_batch_jobs_SLURM.sh
Ignacio Pisso's avatar
Ignacio Pisso committed
228

229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254

FLEXPARTa FLEXPARTb executables  
generate 2 different case sets for comparison of the output


export
 ./export_flexgen.sh casesA_2 _A 
 ./export_flexgen.sh casesA_1 _A 

cd
  cd casesA_2

generate set
 ./gen_options_all.sh
 ./gen_pathnames.sh

generate batch jobs
 ./gen_batch_jobs_cl.sh FLEXPARTa -i

../run_batch_cl.sh


 ./export_flexgen.sh default_fail _default 

./export_flexgen.sh exportA_fail _A

255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
2018-09-07

./export_flexgen.sh generate_cases _*

#./export_flexgen.sh ~repos/flexpart/generate_cases _*

#./export_flexgen.sh ~repos/flexpart/generate_cases  _[A-I]

#options_template_default=./options
# relative path to be included in the distribution
options_template_default=../options


./export_flexgen.sh ~/repos/flexpart/standard_cases  _[A-I]

270