Commit 1a2c9511 authored by Antoine Berchet's avatar Antoine Berchet
Browse files

Merge branch 'groups_monitor' into 'devel'

Groups monitor

See merge request satinv/cif!161
parents e2b0900d 2cfc978b
......@@ -10,13 +10,17 @@ config_file=`echo "$(cd "$(dirname "$config_file")"; pwd)/$(basename "$config_fi
pycif_root_dir=`echo "$(cd "$pycif_root_dir"; pwd)"`
# Fetch workdir to mount it in the container for saving outputs
workdir=`python3 -c \
"from pycif.utils.yml import ordered_load;
workdir=`python3 -W ignore -c \
"
from pycif.utils.yml import ordered_load;
with open('$config_file', 'r') as f:
config = ordered_load(f)
print('AAAAAAAAAAAA')
print(config['workdir']); "`
workdir=`echo $workdir | awk -F "AAAAAAAAAAAA" '{print $2}'`
mkdir -p $workdir
# Run the configuration into the container and writing outputs to workdir
......@@ -24,9 +28,9 @@ docker run -it -v $config_file:/config/config.yml \
-v $pycif_root_dir:/tmp/CIF/ \
-v $workdir:/workdir/ \
$extra_volumns \
--entrypoint /bin/bash pycif/pycif-tm5:0.2
pycif/pycif-tm5:0.2
# --entrypoint /bin/bash pycif/pycif-tm5:0.2
# --entrypoint /bin/bash pycif/pycif-ubuntu:0.1
# pycif/pycif-tm5:latest
......@@ -13,13 +13,13 @@ dirout=/home/chimereges/aberchet/debugchimere/pytest/
export PYCIF_DATATEST=/home/chimereges/PYCIF_TEST_DATA/
# Set up the platform to be run at LSCE, otherwise, use default docker parameters
#export PYCIF_PLATFORM=LSCE
export PYCIF_PLATFORM=LSCE
###
# select a subset of tests to run by using the tags ("mark")
#mark="(dummy and article and inversion and not adjtltest and not uncertainties) or (fwd and ref_config)"
mark="(dummy and article and inversion and not adjtltest and uncertainties) or (fwd and ref_config)"
#mark="(dummy and article and inversion and not adjtltest and uncertainties) or (fwd and ref_config)"
#mark="(fwd and ref_config)"
#mark="test_in_ci and dummy"
mark="test_in_ci and dummy"
#mark="test_in_ci and chimere"
#mark="chimere and argfrsd and fwd"
#mark="tm5 and test_in_ci and fwd"
......
......@@ -542,6 +542,7 @@ def build_rst_from_plugins(app):
".. toctree::",
" :maxdepth: 3",
"",
" ../plugin_description",
" ../dependencies"
] + [
" {}/index".format(Plugin.plugin_types[plg_type][0][1:])
......
......@@ -17,9 +17,9 @@ Requirements
To contribute to the documentation the following softwares need to be install on your system:
* `Sphinx <https://www.sphinx-doc.org/en/master/>`__ : automatic generation of html pages from rst files
* `Graphviz <https://www.graphviz.org/>`__ : embed graphics in the documentation
* pycif : the CIF python package needs to be properly installed on your system to automatically build from modules
* `Sphinx <https://www.sphinx-doc.org/en/master/>`__: automatic generation of html pages from rst files
* `Graphviz <https://www.graphviz.org/>`__: embed graphics in the documentation
* :doc:`pycif<installation>`: the CIF python package needs to be properly installed on your system to automatically build from modules
Compiling commands
==================
......@@ -34,6 +34,13 @@ To do so, type:
Then, you can simply open html files generated in :bash:`CIF_ROOT/docs/build/html/` in any web browser.
It is recommended to re-compile the documentation from scratch time to time:
.. code-block:: bash
cd CIF_ROOT/docs
make clean
Adding new files in the documentation
-------------------------------------
......@@ -43,21 +50,22 @@ Tables of contents are generally placed in the :bash:`index.rst` file in each su
Therein, the file name should be added below the command :bash:`.. toctree::` alongside existing files
(the rst extension is not mandatory in that command).
When documentating a new plugin in pyCIF, please find a template of the corresponding documentation file below:
Automatic construction from codes
---------------------------------
.. code-block:: rst
The automatic documentation from codes is available :doc:`here </documentation/plugins/index>`
#####################
My awesome plugin
#####################
.. automodule:: pycif.plugins.the_type_of_plugin.my_awesome_plugin
Class of plugins
================
Type of plugins can be described by a full description, giving e.g., information about the expected structure, functions, etc.
Examples are available in :bash:`pycif/plugins/modes/__init__.py`.
Any help to improve these descriptions is welcome.
This allows the documentation to build the documentation automatically from the code of the plugin (see below).
Automatic construction from codes
---------------------------------
Individual plugins
==================
The documentation is designed to include information in the pycif plugins directly and automatically.
To allow this feature to work properly, the :bash:`__init__.py` file of the plugin being documented should include:
......@@ -72,10 +80,16 @@ To allow this feature to work properly, the :bash:`__init__.py` file of the plug
I can use rst syntax in there, which will automatically compile in the documentation
"""
the result will be inserted in a `Description` section of the corresponding entry of the documentation
the result will be inserted in a `Description` section of the corresponding entry of the documentation.
.. note::
Please note that if you need sub-sections in the description section,
you should use the dot :bash:`..........` to underline sub-titles.
* a dictionary :bash:`input_arguments`: this dictionary is both used to define default values and document arguments
that are needed in the configuration file; for each entry, you can add a :bash:`"doc"` entry, including a string describing the corresponding argument:
that are needed in the configuration file; for each entry, you can add a :bash:`"doc"` entry,
including a string describing the corresponding argument:
.. code-block:: python
......@@ -87,13 +101,47 @@ To allow this feature to work properly, the :bash:`__init__.py` file of the plug
}
}
Some plugins may require sub-paragraphs to be properly defined. This can be done as follows:
.. code-block:: python
input_arguments = {
"some_argument": {
"doc": "some description",
"default": None if mandatory else a default value,
"accepted": a python type that the argument should fit
},
"some_subparagraph": {
"doc": "Documentation of the paragraph",
"default": None (must be None here),
"optional": True or False,
"structure": {
"sub-arguments1": {
"doc": "some description",
"default": None if mandatory else a default value,
"accepted": a python type that the argument should fit
},
"sub-arguments2": {
"doc": "some description",
"default": None if mandatory else a default value,
"accepted": a python type that the argument should fit
},
"some_subsubparagraph": {
...
}
}
}
}
Proposing tutorials
-------------------
Tutorials are critical for new users and developers to start working with the CIF efficiently.
They should encompass as many situations as possible.
Experienced users/developers are encouraged to share their experience and explain how to tackle some specific difficulties they met in using the system.
Experienced users/developers are encouraged to share their experience
and explain how to tackle some specific difficulties they met in using the system.
Rules
=====
......
......@@ -13,8 +13,8 @@ This version does not match any "official/standard" version since it is maintain
misc
meteo
emissions
ci
bc
inicond
bouncond
timesteps
processes
verticalparam
......
......@@ -8,6 +8,7 @@ General input and output structures in the CIF
monitor
controlvect
obsvect
......
......@@ -21,6 +21,7 @@ The observations are ordered along an index, which is simply the ID number of ea
The basic information are:
++++++++++++++++++++++++++
:date:
date at which the observation begins.
The date must be a datetime object.
......
##############################
Plugins: what are they?
##############################
......@@ -3,7 +3,11 @@ Yaml configuration
pyCIF can be entirely set up with a
`Yaml <http://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html>`__
file such as in the example below. The basic idea of Yaml
file such as in the example below.
The Yaml configuration file structure is used to define, initialize and run the building blocks of pyCIF:
:doc:`Plugins <plugins/index>`
The basic idea of Yaml
syntax is that you can easily define a tree structure using ':' and
indentation, which would be automatically interpreted by Python.
......
......@@ -13,9 +13,20 @@ Welcome to the Community Inversion Framework!
The Community Inversion Framework is governed by the CeCILL-C license under French law.
Please consult the reference text `here <https://cecill.info/licences/Licence_CeCILL-C_V1-en.html>`__ for further detail.
The license grants full rights for the users to use, modify and redistribute the original version of the CIF,
conditional to the obligation to make their modifications available to the community and to properly acknowledge the original authors of the code.
conditional to the obligation to make their modifications available to the community
and to properly acknowledge the original authors of the code.
.. admonition:: Use, aknowledgement and citation
The Community Inversion Framework has been designed by a community of scientists who agreed to openly share their developments.
The maintenance and further developments are made possible through continued efforts by the core team of developper,
with the support of their respective funding agencies.
Any use of the Community Inversion Framework should then be fairly acknowledged.
Users are required to establish contact with `the team of developers <help@community-inversion.eu>`__ to determine an appropriate level of acknowledgement
through co-authorship and relevant citations.
.. _VERIFY: http://verify.lsce.ipsl.fr
.. toctree::
......
......@@ -2,4 +2,7 @@
Publications by developers
##########################
- Thanwerdas, J., Saunois, M., Berchet, A., Pison, I., Vaughn, B. H., Michel, S. E., and Bousquet, P.:
Variational inverse modelling within the Community Inversion Framework to assimilate δ13C(CH4) and CH4:
a case study with model LMDz-SACS, Geosci. Model Dev. Discuss. [preprint],
https://doi.org/10.5194/gmd-2021-106, in review, 2021.
......@@ -2,4 +2,9 @@
Publications by users
#####################
- Thanwerdas, J., Saunois, M., Pison, I., Hauglustaine, D., Berchet, A., Baier, B., Sweeney, C., and Bousquet, P.:
How do Cl concentrations matter for simulating CH4, δ13C(CH4) and estimating CH4 budget through
atmospheric inversions?, Atmos. Chem. Phys. Discuss. [preprint],
https://doi.org/10.5194/acp-2021-950, in review, 2021.
Check what has been done in the :bash:`workdir`:
===================================================
Checking the input files:
-------------------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Checking the output files:
--------------------------
XXX part about checking what is done by the pre-proc:
where are the files used by the simu, where are the outputs, etc
The observations plus their simulated equivalents are in
:bash:`$WORKIDR/obsoperator/fwd_0000/obsvect/concs/SPEC/monitor.nc`
with SPEC the species available in the observations.
In this simple case, check that the columns i, j, level, sim, tstep, tstep\_glo and dtstep are filled.
Elaborate the yaml for the CIF
==================================
.. important::
**How to use the** :doc:`cheat-sheet</documentation/dependencies>` **for plugins**
In the following, plugins have te be used and provided specifications.
The arguments can be found in the documentation of each plugin.
To make access to the plugins easier, the cheat-sheet shows them sorted by **type**:
the various types are the left-most (e.g. chemistry, controlvect, fields).
For each type, available plugins are listed with the **name, version** of each displayed.
Note that stating the name and version of a plugin is mandatory, whereas stating its type not always necessary.
.. contents::
:local:
Section for PyCIF parameters:
-----------------------------
.. container:: toggle
.. container:: header
Show/Hide Code
.. yml-block:: /yaml_examples/chimere/config_fwd_ref_chimere.yml
:keys: verbose, logfile, workdir, datei, datef
In this section of the yaml, it is possible to define :doc:`anchors</documentation/paths>` to be used in the rest of the file.
.. include:: ../common/mode.rst
.. include:: ../common/obsoperator.rst
.. include:: ../common/controlvect.rst
Model (:bash:`model`)
---------------------
Here, it is the :doc:`plugin for CHIMERE</documentation/plugins/models/chimere>`.
The usual user's choices for running a CHIMERE simulation (see :doc:`CHIMERE documentation</documentation/doc-models/chimere/index>`)
are either in the mandatory arguments or in the optional arguments, for which defalut values are specified.
Be sure to check all the mandatory AND OPTIONAL arguments to fully set up the simulation as wanted.
It must be consistent with e.g. the chemistry (see step 2 and i) and domain (see step 2 and h).
.. container:: toggle
.. container:: header
Show/Hide Code
.. yml-block:: /yaml_examples/chimere/config_fwd_ref_chimere.yml
:keys: model
The requirements for CHIMERE are :bash:`domain` (h), :bash:`chemistry` (i) and a set of components
(corresponding to the inputs of CHIMERE itself) to be detailed in
:bash:`datavect` (j): :bash:`meteo` (I), :bash:`fluxes` (II),
:bash:`biofluxes` (III) , :bash:`latcond`, :bash:`topcond` and :bash:`inicond` (IV)
Observation Vector (:bash:`obsvect`)
-------------------------------------
[to be updated?] To avoid useless runs, the current version of the CIF only runs
a simulation up to the point where observations are available.
The standard :doc:`obsvect</documentation/plugins/obsvects/index>`
must therefore be initialized. See XXXXXXXX for how to provide quick-dummy observation data.
.. container:: toggle
.. container:: header
Show/Hide Code
.. yml-block:: /yaml_examples/chimere/config_fwd_ref_chimere.yml
:keys: obsvect
Its requirements are :bash:`datavect` (j) and the :bash:`model` (e).
.. Not required for a simulation without comparison to observations.
.. include:: ../common/platform.rst
Domain (:bash:`domain`)
-------------------------------------
Specify :doc:`a domain for CHIMERE</documentation/plugins/domains/chimere>`
(see also the :doc:`cheat-sheet</documentation/dependencies>`) **consistently**
with the pre-computed input files (see step 2). The files defining the domain
can be stored directly in directory :bash:`repgrid` or symbolic links can be used.
.. container:: toggle
.. container:: header
Show/Hide Code
.. code-block:: yaml
#####################################################################
#####################################################################
domain :
plugin:
name : CHIMERE
version : std
repgrid: a_path_for_CHIMERE_COORD_definition_files/
domid : MYDOMAIN
nlev: 20
p1: 997
pmax: 200
pressure_unit: hPa
Chemistry (:bash:`chemistry`)
-------------------------------------
The only possibility so far is for :doc:`photolysis with tabulated Js</documentation/dependencies>`,
the chemical scheme being pre-computed (see step 2).
.. container:: toggle
.. container:: header
Show/Hide Code
.. code-block:: yaml
#####################################################################
#####################################################################
chemistry :
plugin:
name: CHIMERE
version: gasJtab
schemeid: name.chemistry
dir_precomp: the_path_for_the_directory_of_which_chemical_scheme_named_above_is_a_subdir/
Data vector (:bash:`datavect`)
-------------------------------------
So far, there is only the standard :bash:`datavect`.
.. container:: toggle
.. container:: header
Show/Hide Code
.. code-block:: yaml
#####################################################################
#####################################################################
datavect :
plugin:
name: stanbard
version: std
It contains ingredients, which list the input data for the model (e.g. emission fluxes)
and for the comparison to observations (e.g. concentration data) ,
which :bash:`controlvect, obsoperator` and :bash:`obsvect` will use for building the set-up to run.
For the first forward simulation, its components are the :doc:`requirements of the model's plugin</documentation/dependencies>`
(e) which are not already taken care of in the previous sections of the yaml
(i.e. excluding :bash:`domain` and :bash:`chemistry`) plus a component for the (dummy) observation data.
The names of the components must be names recognized by the model's plugin
so that it is able to fetch its inputs. For CHIMERE, for example,
the plugin expects :bash:`fluxes` to provide the information on the input emissions
and :bash:`concs` or :bash:`satellite` to provide the information on the concentrations
or columns to be compared to the simulated concentrations or columns. XXXXX mettre quelque part liste des components possibles par modele?XXXXXX
For each of its components, :bash:`datavect` expects a minimum of three pieces of information:
i) a mandatory :bash:`dir`, for the directory where the data relevant to this component is available
ii) a mandatory :bash:`file` which gives either a fixed name of data file or a general format for a set of files
(with generic year, month, day, hour, etc).
iii) an optional :bash:`varname`, which gives the name(s) of the variable(s) to read in the original data,
if the names of the variables in the files are not the same as the names used in CHIMERE.
If no varname is provided, the CIF assumes that the name of the parameter,
i.e. the same name as in CHIMERE is to be used (see example in II.).
It may be necessary to also indicate in the component the plugin(s) to use to deal with the raw data files.
In this case, two optional pieces of information may be useful (depending on the plugin, see examples below):
iv) the :bash:`file_freq`, which indicates the time period covered by each data file.
v) the :bash:`comp_type`, to distinguish the various boundary conditions (lateral, top, initial).
Note that these up to five arguments are linked to datavect and not to a given plugin
(see detail :doc:`here </documentation/plugins/datavects/standard>`):
If a plugin has to be used by a component to deal with the provided files,
its **name, version AND type** must be sepcified, as well as its arguments.
.. Technically, the model's plugin uses the key-words used for components to fetch its inputs.
A key-word can therefore direct to a plugin: the key-word is the name of the plugin
and the component takes exactly the same form as the plugins in the previous sections of the yaml.
A key-word can also direct to a transformation as is the case for the observation data with
:bash:`concs` XXXX????????XXXXXX XXXlien for advanced users vers create transform?XXX.
CHIMERE usual inputs
....................
The components dealing with the required inputs of the model direct to plugins which are able to deal with
:doc:`the inputs required by CHIMERE</documentation/doc-models/chimere/generalinfo>`:
:doc:`meteo</documentation/doc-models/chimere/meteo>` (I) for the meteorological inputs (see step 2),
:doc:`emission fluxes</documentation/doc-models/chimere/emissions>` (II and III),
:doc:`boundary conditions</documentation/doc-models/chimere/bouncond>` (IV)
and :doc:`initial conditions</documentation/doc-models/chimere/inicond>` (IV).
Therefore, the name of these components are the same as the names of the matching plugins.
I. Since the CIF is so far able to use only standard pre-computed
:doc:`METEO.nc files</documentation/doc-models/chimere/meteo>`, the specifications
for the :bash:`meteo` component are very simple:
i) the minimum information :bash:`dir` and :bash:`file` which direct to ready-made METEO.nc files
ii) the :doc:`the plugin for CHIMERE's meteo</documentation/plugins/meteos/chimere_meteo>`
(see also :doc:`cheat-sheet</documentation/dependencies>`) which deals with these files
iii) the :bash:`file_freq` XXX used by the plugin or elsewhere?XXX.
.. container:: toggle
.. container:: header
Show/Hide Code
.. yml-block:: /yaml_examples/chimere/config_fwd_ref_chimere.yml
:keys: datavect/components/meteo
In this case, the names of the :doc:`METEO.nc files match the format used by
CHIMERE</documentation/doc-models/chimere/meteo>`.
It is also possible to vary the template e.g. :bash:`file: METEO_some_etiket.%Y%m%d%H.X.nc`
II. If pre-computed files are also available for fluxes, biofluxes, inicond, latcond and topcond,
they can be specified in the same simple way (see example in II for fluxes).
.. note::
the pre-computed files must be consistent together and with the domain (h),
the PyCIF parameters (a) of the simulation and the choices made for the model (e),
particularly with the optional argument :bash:`periods` (default 1D = 24 hours).
In the same manner as for the meteorology, it it simple to specify the use of pre-computed
:doc:`AEMISSIONS.nc files</documentation/doc-models/chimere/emissions>`
with ready-made :bash:`dir` and :bash:`file` dealt with by the
:doc:`plugin for CHIMERE's fluxes</documentation/plugins/fluxes/chimere>`, with its type,
name and version taken from the :doc:`cheat-sheet</documentation/dependencies>`:
.. container:: toggle
.. container:: header
Show/Hide Code
.. code-block:: yaml
#####################################################################
#####################################################################
datavect:
plugin:
name: standard
version: std
components:
meteo:
dir: directory_containing_METEO.YYYYMMDDHH.*.nc_files
file: METEO.%Y%m%d%H.X.nc
plugin:
name: CHIMERE
version: std
type: meteo
file_freq: XH
fluxes:
dir: directory_containing_AEMISSIONS.YYYYMMDDHH.*.nc_files
file: AEMISSIONS.%Y%m%d%H.X.nc
plugin:
name: CHIMERE
version: AEMISSIONS
file_freq: XH
In this case, the CIF expects to find
:doc:`AEMISSIONS.nc files formatted exactly as CHIMERE uses them</documentation/doc-models/chimere/emissions>`
and containing all the species listed in :doc:`ANTHROPIC</documentation/doc-models/chimere/chemistry>`.
It is also possible to specify different input sources for the various emitted species
which do not require a sub-hourly interpolation.
These species must be listed as :bash:`parameters` of :bash:`fluxes`
and their names must match the names in :doc:`ANTHROPIC</documentation/doc-models/chimere/chemistry>`.
For each parameter, the specifications of the component plugin are inherited by default XXXX?XXX
but it is possible to individualize everything, including the plugin to use,
as shown in :doc:`the tutorial for more elaborated inputs</usertutos/elaborated-inputs/chimere/index>`.
III. The :bash:`biofluxes` specifications follow the same principles as :bash:`fluxes`.
If no fluxes with a sub-hourly interpolation are required (:bash:`useemisb` is False in :bash:`model`),
this component can be omitted. If it is omitted while :bash:`useemisb` is True,
an error is raised XXX CHECK THAT STILL TRUEXXXX.
In the same manner as for AEMISSIONS, it is simple to specify the use of
pre-computed :doc:`BEMISSIONS.nc files</documentation/doc-models/chimere/emissions>`,
making use of the :doc:`plugin for CHIMERE's fluxes</documentation/plugins/fluxes/chimere>`: