Commit d39d0d01 authored by Antoine Berchet's avatar Antoine Berchet
Browse files

Merge branch 'devel' into 'LSCE'

Warning when CHIMERE periods are not compatible with simulation window;...

See merge request satinv/cif!168
parents 0837e7c7 90242fef
#QSUB -s /bin/tcsh
#PBS -q longp
#PBS -l nodes=1:ppn=8
#PBS -q xlongp
#PBS -l nodes=1:ppn=10
python -m pycif /home/users/aberchet/CIF/config_files/tuto_chimere/config_chimere_argonaut_n2o_sat_inv_corr.yml
#python -m pycif /homen/users/aberchet/CIF/config_files/tuto_chimere/config_chimere_EUROCOM_satOMI.yml
python -m pycif /home/users/aberchet/CIF/config_files/RECAPP/config_chimere_fwd_EUROCOM_CO2_AB_6mois_TNO_2008_1.yml
#QSUB -s /bin/tcsh
#PBS -q xlongp
#PBS -l nodes=1:ppn=10
#PBS -N RECAPP_2008_1
python -m pycif /home/users/aberchet/CIF/config_files/RECAPP/config_chimere_fwd_EUROCOM_CO2_AB_6mois_TNO_2008_1.yml
......@@ -606,6 +606,7 @@ def build_rst_from_plugins(app):
# Loop over modules in the sub-type
package_subindex = []
package_subtitles = []
for mod in pkgutil.walk_packages(import_subpackage.__path__,
prefix=import_subpackage.__name__ + "."):
if not mod.ispkg:
......@@ -644,9 +645,11 @@ def build_rst_from_plugins(app):
# Append name for plugin type index
package_subindex.append(loc_mod.__name__.split(".")[-1])
package_subtitles.append(getattr(loc_mod, "_fullname", loc_mod._name))
# Sort names
package_subindex.sort()
package_subindex = [
x for _,x in sorted(zip(package_subtitles, package_subindex))]
# Write the plugin type index
if subtype == "":
......
......@@ -27,7 +27,7 @@ The :bash:`datavect` paragraph of your working yaml should look like that:
.. container:: header
Show/Hide Code
Example with CHIMERE
.. code-block:: yaml
:linenos:
......@@ -49,11 +49,13 @@ The :bash:`datavect` paragraph of your working yaml should look like that:
file: some_file
Do the following to make it work with the template flux:
1. follow the initial steps in :doc:`the flux template documentation page</documentation/plugins/datastreams/fluxes/flux_plugin_template>`
to initialize your new plugin and register it.
It includes copying the template folder to a new path and changing the variables
:bash:`_name`,:bash:`_fullname` and :bash:`_version` in the file :bash:`__init__.py`
:bash:`_name`, :bash:`_fullname` and :bash:`_version` in the file :bash:`__init__.py`
2. update your Yaml to use the template flux (renamed with your preference). It should now look like that:
......@@ -95,95 +97,256 @@ Include the following information:
- data format (temporal and horizontal resolution, names and shape of the data files)
- any specific treatment that prevents the plugin from working with another type of files.
Build and check the documentation
=================================
Before going further, please compile the documentation and check that your new plugin
appears in the list of datastreams plugins :doc:`here</documentation/plugins/datastreams/index>`.
Also check that the documentation of your new plugin is satisfactory.
To compile the documentation, use the command:
.. code-block:: bash
cd $CIF_root/docs
make html
Further details can be found :doc:`here</contrib_doc>`.
Updating functions and data to implement your flux data
=======================================================
Your new plugin will need functions to be coded to work.
fetch
------
The :bash:`fetch` function determines what files and corresponding dates are available
for running the present case.
The structure of the :bash:`fetch` function is shown here: :ref:`datastreams-fetch-funtions`.
Please read carefully all explanations therein before starting implementing your case.
By default, the :bash:`fetch` function will use the arguments :bash:`dir` and :bash:`file` in your yaml.
Make sure to update your yaml accordingly:
.. container:: toggle
.. container:: header
Show/Hide Code
.. code-block:: yaml
:linenos:
datavect:
plugin:
name: standard
version: std
components:
flux:
parameters:
CO2:
plugin:
name: your_new_name
type: flux
version: your_version
dir: path_to_data
file: file_name
Depending on how you implement your data stream, extra parameters may be needed.
Please document them on-the-fly in the :bash:`input_arguments` variable in :bash:`__init__.py`.
One classical parameter is :bash:`file_freq`, which gives the frequency of the input files
(independently to the simulation to be computed).
Once implemented, re-run your test case.
You can check that everything went as expected by checking:
1. in the folder :bash:`$workdir/datavect/flux/your_species/`, links to original data files should be initialized
2. it is possible to check that the list of dates and files is initialized as expected. To do so, use the option
:bash:`dump_debug` in the :bash:`datavect` paragraph in the yaml
(see details :doc:`here</documentation/plugins/datavects/standard>`).
It will dump the list of dates and files in a file named :bash:`$workdir/datavect/flux.your_species.txt`
get_domain (optional)
---------------------
A datastream plugin needs to be described by a domain to be processed in pyCIF.
There are three valid approaches to associate a valid domain to your flux data.
The two first one are given for information, but the third one is
the one to be preferred in most cases:
1. fetch it from another object in the set-up. This is relevant when the domain
should be exactly the same as the one of another Plugin in your configuration.
For instance, if you are implementing a flux plugin dedicated to a model,
you will expect it to have exactly the same domain as the model.
To ensure that your flux plugin fetch the domain from the present set-up,
it is possible to define a so-called :doc:`requirement </documentation/dependencies>`.
This is done be adding the following lines to the :bash:`__init__.py` file
.. code-block:: python
requirements = {
"domain": {"name": "CHIMERE", "version": "std", "empty": False},
}
In that case, the flux will expect a CHIMERE domain to be defined, otherwise pycif
will return an exception
2. directly define the domain in the yaml as a sub-paragraph.
This will look like that:
.. container:: toggle
.. container:: header
Show/Hide Code
.. code-block:: yaml
:linenos:
datavect:
plugin:
name: standard
version: std
components:
flux:
parameters:
CO2:
plugin:
name: your_new_name
type: flux
version: your_version
dir: path_to_data
file: file_name
domain:
plugin:
name: my_domain_name
version: my_domain_version
some_extra_parameters: grub
Such an approach is not necessarily recommended as it forces the user to properly
configure his/her Yaml file to make the case working properly.
.. warning::
If this path is chosen please document the usage very carefully.
3. Using the function :bash:`get_domain` to define the domain dynamically, based
on input files, or with fixed parameters.
The structure of the :bash:`get_domain` function is shown here: :ref:`datastreams-get_domain-funtions`.
Please read carefully all explanations therein before starting implementing your case.
Once implemented, re-run your test case.
The implementation of the correct domain will have an impact on the native resolution
used to randomly generate fluxes (remember that the :bash:`read` function still
comes from the template and thus generate random fluxes for the corresponding domain).
Therefore, pycif will automatically reproject the fluxes from the implemented domain to
your model's domain.
One can check that the implemented domain is correct by:
1. check that the flux files generated for your model seem to follow the native resolution of
your data
2. it is possible to dump intermediate data during the computation of pycif.
To do so, activate the option :bash:`save_debug` in the :bash:`obsoperator`:
.. container:: toggle
.. container:: header
#####################################
Show/Hide Code
.. code-block:: yaml
:linenos:
obsoperator:
plugin:
name: standard
version: std
save_debug: True
0. .. include:: ../newBCdata/knownplugin.rst
When activated this option dumps intermediate states in :bash:`$workdir/obsoperator/$run_id/transform_debug/`.
One has to find the ID of the :bash:`regrid` transform reprojecting the native fluxes to your model's domain.
This information can be found in :bash:`$workdir/obsoperator/transform_description.txt`.
Once the transform ID retrieved, go to the folder :bash:`$workdir/obsoperator/$run_id/transform_debug/$transform_ID`.
The directory tree below that folder can be complex, go to the deepest level.
You should find two netCDF files, one for the inputs, one for the outputs.
In the outputs, you should find the native resolution, in the output, the projected one.
1. In directory :bash:`plugins/fluxes`, copy the directory containing the template
for a flux plugin :bash:`flux_plugin_template` in the directory for your new plugin.
read
-----
.. include:: ../newBCdata/register.rst
The :bash:`read` function simply reads data for a list of dates and files as deduced from the
:bash:`read` function.
The expected structure for the :bash:`fetch` function is shown here: :ref:`datastreams-read-funtions`.
This function is rather straighforward to implement.
Be sure to have the following structure in outputs:
2. Modify the yaml file to use the new plugin: the minimum input arguments are
:bash:`dir`, :bash:`file`, :bash:`varname` and :bash:`unit_conversion`.
The default space and time interpolations will be applied
(see XXXX doc sur premiere simu directe avec exmeple yaml quand mise a jourXXXXX).
.. code-block:: python
.. code-block:: yaml
output_data.shape = (ndate, nlevel, nlat, nlon)
output_dates = start_date_of_each_interval
components:
fluxes:
plugin:
name: fluxes
version: template
type: fluxes
dir: dir_with_original_files/
file: file_with_new_fluxes_to_use_as_inputs
varname: NAMEORIG
unit_conversion:
scale: 1.
return xr.DataArray(
output_data,
coords={"time": output_dates},
dims=("time", "lev", "lat", "lon"),
)
3. .. include:: ../newBCdata/devplugin.rst
Similarly to the :bash:`get_domain` function, it is possible to check that
the :bash:`read` function is properly implemented by using the option :bash:`save_debug`
and checking that the input fluxes are correct.
XXXXXXX what about the input arguements? Ils demandent une partie dediee!?XXXXXXXXXX
.. warning::
4. Document the new plugin:
a) .. include:: ../newBCdata/readme.rst
It is likely that the fluxes in your native data stream don't have the same unit
as the one expected by your model.
To convert the unit properly, add the :bash:`unit_conversion` paragraph to your Yaml file:
.. container:: toggle
b) create the rst file that contains the automatic documentation in docs/source/documentation/plugins/fluxes/. Please provide it with a self-explaining name. Example for the template: file fluxes_template.rst reads
.. container:: header
.. code-block:: rest
Show/Hide Code
.. role:: bash(code)
:language: bash
.. code-block:: yaml
:linenos:
###########################
Template plugin for fluxes
###########################
datavect:
plugin:
name: standard
version: std
components:
flux:
parameters:
CO2:
plugin:
name: your_new_name
type: flux
version: your_version
dir: path_to_data
file: file_name
unit_conversion:
scale: ${scaling factor to apply}
.. automodule:: pycif.plugins.datastreams.fluxes.flux_plugin_template
write (optional)
-----------------
c) add the reference to the rst file in docs/source/documentation/plugins/fluxes/index.rst:
This function is optional and is necessary only when called by other plugins.
One probably does not need to bother about it at the moment...
.. code-block:: rest
#####################
Fluxes :bash:`fluxes`
#####################
Available Fluxes
=========================
The following :bash:`fluxes` are implemented in pyCIF:
.. toctree::
:maxdepth: 3
fluxes_template
dummy_nc
dummy_txt
edgar_v5
flexpart
chimere
lmdz_bin
lmdz_sflx
d) built the documentation (:bash:`make html` in docs/) and check that the link to the new plugin appears in the documentation at file:///your_path/cif/docs/build/html/documentation/plugins/index.html and that the section "doc" of the input arguments is correctly displayed at file:///your_path/cif/docs/build/html/documentation/plugins/fluxes/the_new_plugin.html
......@@ -59,13 +59,15 @@ Download the CIF image
**********************
The CIF Docker image is stored on `Docker Hub <https://hub.docker.com/>`__.
The image is publicly available and can be downloaded using the command:
.. code-block:: bash
docker pull pycif/pycif-ubuntu:0.1
Other images are available for specific usage of the CIF, especially for some CTMs.
Corresponding images can be found `here <https://hub.docker.com/u/pycif>`__.
********************************
Running the CIF inside the image
********************************
......
......@@ -9,6 +9,7 @@ General input and output structures in the CIF
monitor
controlvect
obsvect
others
......
###################
Other input data
###################
.. role:: bash(code)
:language: bash
.. role:: raw-math(raw)
:format: latex html
Inputs other than observations need to be provided to pyCIF.
It includes, e.g., meteorological fields constraining CTMs, flux data to define the control vector, etc.
All these data can be provided with their native format without further modification specific to pyCIF.
However, pyCIF cannot "guess" a data format.
Therefore, the input data must be one of those already implemented in the CIF.
The list of available data streams integrated in pyCIF is given :doc:`here </documentation/plugins/datastreams/index>`.
If one needs to use data whose format is not included in pyCIF, there are two options:
- manually modify the input files to fit one of the existing input formats
- integrate the new format to pyCIF. To do so, please follow the :doc:`tutorial here </devtutos/newfluxdata/newfluxdata>`
......@@ -5,7 +5,7 @@ pyCIF can be entirely set up with a
`Yaml <http://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html>`__
file such as in the example below.
The Yaml configuration file structure is used to define, initialize and run the building blocks of pyCIF:
:doc:`Plugins <plugins/index>`
:doc:`Plugins <plugins/index>`.
The basic idea of Yaml
syntax is that you can easily define a tree structure using ':' and
......@@ -49,3 +49,4 @@ other plugins and dependencies.
Further examples about the Yaml syntax can be found
`here <http://sweetohm.net/article/introduction-yaml.en.html>`__.
The initialized dictionary is then loaded as :doc:`Plugins <plugins/index>` for later use by pyCIF.
......@@ -30,6 +30,7 @@ model:
plugin:
name: dummy
version: std
reload_H: !join [*outdir, /H_matrix.pickle]
save_H: true
obsvect:
dir_obsvect: !join [*outdir, /ref_obsvect]
......@@ -68,6 +69,20 @@ datavect:
flux:
parameters:
CH4:
bands_i:
- 0
- 3
- 6
- 9
- 12
bands_j:
- 0
- 3
- 6
- 9
- 12
- 15
- 18
err: 1
errtype: max
flx_formula:
......@@ -91,7 +106,7 @@ datavect:
evalmin: 0
landsea: false
sigma: 500
hresol: hpixels
hresol: ibands
nlev: 1
plugin:
name: dummy
......
rootdir: &rootdir /tmp/CIF/
outdir: !join [*rootdir, /.tox/py38/tmp]
outdir: &outdir !join [*rootdir, /.tox/py38/tmp]
verbose: 2
logfile: pycif.logtest
workdir: !join [*outdir, /inversion_long_bands_4dvar_M1QN3]
workdir: !join [*outdir, /inversion_long_bands_4dvar_M1QN3]
datei: 2010-01-01
datef: 2010-01-05 00:00:00
mode:
minimizer:
df1: 0.5
epsg: 0.0002
maxiter: 10
nsim: 10
maxiter: 5
nsim: 5
plugin:
name: M1QN3
version: std
......@@ -19,7 +19,6 @@ mode:
name: gausscost
version: std
reload_from_previous: true
montecarlo: 10
plugin:
name: 4dvar
version: std
......@@ -38,14 +37,14 @@ model:
chemistry:
acspecies:
CH4: null
file_pg: !join [*rootdir, /model_sources/dummy_gauss/Pasquill-Gifford.txt]
file_pg: !join [*rootdir, /model_sources/dummy_gauss/Pasquill-Gifford.txt]
plugin:
name: dummy
version: std
reload_H: !join [*outdir, /H_matrix.pickle]
reload_H: !join [*outdir, /H_matrix.pickle]
save_H: true
obsvect:
dir_obsvect: !join [*outdir, /ref_obsvect]
dir_obsvect: !join [*outdir, /ref_obsvect]
dump_type: nc
plugin:
name: standard
......@@ -66,7 +65,7 @@ datavect:
seed: true
seed_id: 5
zmax: 100
fluxes:
flux:
parameters:
CH4:
bands_i:
......@@ -102,7 +101,6 @@ datavect:
square: null
variable: zlon
hcorrelations:
dircorrel: !join [*outdir, /test_integration_inversion_dum4/datavect]
dump_hcorr: true
evalmin: 0
landsea: false
......@@ -111,7 +109,7 @@ datavect:
nlev: 1
plugin:
name: dummy
type: fluxes
type: flux
version: txt
tcorrelations:
sigma_t: 12
......
rootdir: &rootdir /tmp/CIF/
outdir: !join [*rootdir, /.tox/py38/tmp]
outdir: &outdir !join [*rootdir, /.tox/py38/tmp]
verbose: 2
logfile: pycif.logtest
workdir: !join [*outdir, /inversion_long_bands_4dvar_congrad]
workdir: !join [*outdir, /inversion_long_bands_4dvar_congrad]
datei: 2010-01-01
datef: 2010-01-05 00:00:00
mode:
minimizer:
df1: 0.5
epsg: 0.0002
maxiter: 10
nsim: 10
maxiter: 5
nsim: 5
plugin:
name: congrad
version: std
......@@ -38,14 +38,14 @@ model:
chemistry:
acspecies:
CH4: null
file_pg: !join [*rootdir, /model_sources/dummy_gauss/Pasquill-Gifford.txt]
file_pg: !join [*rootdir, /model_sources/dummy_gauss/Pasquill-Gifford.txt]
plugin:
name: dummy
version: std
reload_H: !join [*outdir, /H_matrix.pickle]
reload_H: !join [*outdir, /H_matrix.pickle]
save_H: true
obsvect:
dir_obsvect: !join [*outdir, /ref_obsvect]
dir_obsvect: !join [*outdir, /ref_obsvect]
dump_type: nc
plugin:
name: standard
......@@ -66,7 +66,7 @@ datavect:
seed: true
seed_id: 5
zmax: 100
fluxes:
flux:
parameters:
CH4:
bands_i:
......@@ -102,7 +102,6 @@ datavect:
square: null
variable: zlon
hcorrelations:
dircorrel: !join [*outdir, /test_integration_inversion_dum9/datavect]
dump_hcorr: true
evalmin: 0
landsea: false
......@@ -111,7 +110,7 @@ datavect:
nlev: 1
plugin:
name: dummy
type: fluxes
type: flux
version: txt
tcorrelations:
sigma_t: 12
......
rootdir: &rootdir /tmp/CIF/
outdir: !join [*rootdir, /.tox/py38/tmp]
outdir: &outdir !join [*rootdir, /.tox/py38/tmp]
verbose: 2
logfile: pycif.logtest
workdir: !join [*outdir, /inversion_long_bands_ensrf_]
workdir: !join [*outdir, /inversion_long_bands_ensrf_]
datei: 2010-01-01
datef: 2010-01-05 00:00:00