Commit 455efd6f authored by Isabelle Pison's avatar Isabelle Pison
Browse files

templates and dev tutos on plugins for BCs and fluxes

parent 5085c2fb
.. role:: bash(code)
:language: bash
Run pycif with this yaml: the new plugin will simply perform what is in the template i.e. print some instructions on what you have to do where. The following codes must be developped in the places matching the instructions - and checked. To check that each new code works as intended, run the CIF with the yaml using the new plugin and with the same yaml but using a known plugin with print statements. The scripts have to be developped in this order:
a) :bash:`fetch.py` to retrieve the required files. WARNING: you do not need a fetch.py script only if the available files are already provided with the same temporal resolution as the chosen CTM XXX CHECK with ANTOINE, dangerousXXX
b) :bash:`get_domain.py` to get the information on the original domain. WARNING: you do not need a get_domain.py script only if the available files are already provided on the same domain i.e the same horizontal and vertical grids as used by the chosen CTM XXX CHECK with ANTOINE, dangerousXXX
c) :bash:`read.py` to actually read the data.
d) :bash:`write.py` to write the data in the right format for the model XXX mais que pour CHIMERE pour les BCs??? Comment choisit-on sinon???? Ou pas utilise en fait?XXXXXXXX
.. Hints: look into the other plugins to get ideas on how to code -> donner infos plus precises? Pourquoi tracfile pas toujours OK / tracdir?
.. role:: bash(code)
:language: bash
Have a yaml file ready with a simulation that works with a known BC plugin. For the :bash:`obsoperator`, choose the option :bash:`onlyinit` so that only the inputs are computed, not the whole simulation.
.. code-block:: yaml
obsoperator:
plugin:
name: standard
version: std
onlyinit: True
......@@ -5,30 +5,13 @@ How to add a new type of data for boundary conditions to be processed by the CIF
.. role:: bash(code)
:language: bash
0. Have a yaml file ready with a simulation that works with a known BC plugin. For the :bash:`obsoperator`, choose the option :bash:`onlyinit` so that only the inputs are computed, not the whole simulation.
0. .. include:: knownplugin.rst
.. code-block:: yaml
obsoperator:
plugin:
name: standard
version: std
onlyinit: True
1. In directory :bash:`plugins/fields`, copy the directory containing the template for a BC plugin :bash:`bc_plugin_template` in the directory for your new plugin. Following the instructions for :doc:`adding and registrering a new plugin</devtutos/newplugin/newplugin>`, register your new plugin by providing the chosen name (and version) in :bash:`__init__.py` instead of the template's ones.
.. code-block:: python
1. In directory :bash:`plugins/fields`, copy the directory containing the template for a BC plugin :bash:`bc_plugin_template` in the directory for your new plugin.
from .get_domain import get_domain
from .fetch import fetch
from .read import read
from .write import write
_name = "BCs"
_version = "template"
.. include:: register.rst
2. Modify the yaml file to use the new plugin: the minimum input arguments are :bash:`dir`, :bash:`file`, :bash:`varname` and :bash:`unit_conversion`. The default space and time interpolations will be applied (see XXXX doc sur premiere simu directe avec exmeple yaml quand mise a jourXXXXX).
2. .. include:: usenewplugin.rst
.. code-block:: yaml
......@@ -46,15 +29,50 @@ How to add a new type of data for boundary conditions to be processed by the CIF
unit_conversion:
scale: 1.
3. Run pycif with this yaml: the new plugin will simply perform what is in the template i.e. print some instructions on what you have to do where. The following codes must be developped and checked. To check that each new code works as intended, run the CIF with the yaml using the new plugin and with the same yaml but using a known plugin with print statements. The scripts have to be developped in this order:
a) :bash:`fetch.py` to retrieve the required files. WARNING: you do not need a fetch.py script only if the available files are already provided with the same temporal resolution as the chosen CTM XXX CHECK with ANTOINE, dangerousXXX
3. .. include:: devplugin.rst
b) :bash:`get_domain.py` to get the information on the original domain. WARNING: you do not need a get_domain.py script only if the available files are already provided on the same domain i.e the same horizontal and vertical grids as used by the chosen CTM XXX CHECK with ANTOINE, dangerousXXX
XXXXXXX what about the input arguements? Ils demandent une partie dediee!?XXXXXXXXXX
c) :bash:`read.py` to actually read the data.
4. Document the new plugin:
d) :bash:`write.py` to write the data in the right format for the model XXX mais que pour CHIMERE pour les BCs??? Comment choisit-on sinon???? Ou pas utilise en fait?XXXXXXXX
a) .. include:: readme.rst
b) create the rst file that contains the automatic documentation in docs/source/documentation/plugins/fields/. Please provide it with a self-explaining name. Example for the template: file bc_template.rst reads
.. code-block:: rest
.. role:: bash(code)
:language: bash
########################
Template plugin for BCs
########################
.. automodule:: pycif.plugins.fields.bc_plugin_template
c) add the reference to the rst file in docs/source/documentation/plugins/fields/index.rst:
.. code-block:: rest
####################
Fields
####################
.. role:: bash(code)
:language: bash
Available Fields
=========================
The following :bash:`fields` are implemented in pyCIF:
.. toctree::
bc_template
chimere_icbc
d) built the documentation (:bash:`make html` in docs/) and check that the link to the new plugin appears in the documentation at file:///your_path/cif/docs/build/html/documentation/plugins/index.html and that the section "doc" of the input arguments is correctly displayed at file:///your_path/cif/docs/build/html/documentation/plugins/fields/the_new_plugin.html
.. Hints: look into the other plugins to get ideas on how to code -> donner infos plus precises? Pourquoi tracfile pas toujours OK / tracdir?
.. role:: bash(code)
:language: bash
write all relevant information on the plugin in the input argument README in :bash:`__init__.py`:
.. code-block:: python
from .get_domain import get_domain
from .fetch import fetch
from .read import read
from .write import write
_name = "new_plugin_s_name"
_version = "new_plugin_s_version"
input_arguments = {
"README": {
"doc": "Write here the README about the plugin - XXXX how to display it when the plugin is called?"
" or give link to the doc about this plugin?XXXX"
"relevant information: type of files treated, including format of names and shape of data, "
" time resolution, and any specific treatment that prevents the plugin from working "
" with another type of files.",
"default": False,
"accepted": bool
},
"dummy_arg": {
"doc": "document here the argument",
"default": "let's say it's not mandatory",
"accepted": str
},
}
.. role:: bash(code)
:language: bash
Following the instructions for :doc:`adding and registrering a new plugin</devtutos/newplugin/newplugin>`, register your new plugin by providing the chosen name (and version) in :bash:`__init__.py` instead of the template's ones.
.. code-block:: python
from .get_domain import get_domain
from .fetch import fetch
from .read import read
from .write import write
_name = "new_plugin_s_name"
_version = "new_plugin_s_version"
.. role:: bash(code)
:language: bash
Modify the yaml file to use the new plugin: the minimum input arguments are :bash:`dir`, :bash:`file`, :bash:`varname` and :bash:`unit_conversion`. The default space and time interpolations will be applied (see XXXX doc sur premiere simu directe avec exmeple yaml quand mise a jourXXXXX).
################################################
####################################################################################
How to add a new type of flux data to be processed by the CIF into a model's inputs
################################################
####################################################################################
.. role:: bash(code)
:language: bash
0. Have a yaml file ready with a simulation that works with a known flux plugin. For the :bash:`obsoperator`, choose the option :bash:`onlyinit` so that only the inputs are computed, not the whole simulation.
0. .. include:: ../newBCdata/knownplugin.rst
.. code-block:: yaml
obsoperator:
plugin:
name: standard
version: std
onlyinit: True
1. Following the instructions for :doc:`adding and registrering a new plugin</devtutos/newplugin/newplugin>`, add and register an empty plugin for the new type of flux data - the example deals with the INS data.
1. In directory :bash:`plugins/fluxes`, copy the directory containing the template for a flux plugin :bash:`flux_plugin_template` in the directory for your new plugin.
.. include:: register.rst
2. Modify the yaml file to use the new plugin: the minimum input arguments are :bash:`dir`, :bash:`file`, :bash:`varname` and :bash:`unit_conversion`. The default space and time interpolations will be applied (see XXXX doc sur premiere simu directe avec exmeple yaml quand mise a jourXXXXX).
.. code-block:: yaml
.. code-block:: bash
components:
fluxes:
plugin:
name: fluxes
version: template
type: fluxes
dir: dir_with_original_files/
file: file_with_new_fluxes_to_use_as_inputs
varname: NAMEORIG
unit_conversion:
scale: 1.
cd pycif/plugins/fluxes
mkdir ins
cd ins
touch __init__.py
3. .. include:: ../newBCdata/devplugin.rst
then, add name and version in __init__.py:
XXXXXXX what about the input arguements? Ils demandent une partie dediee!?XXXXXXXXXX
.. code-block:: python
4. Document the new plugin:
_name = "INS"
_version = "2012web"
a) .. include:: ../newBCdata/readme.rst
b) create the rst file that contains the automatic documentation in docs/source/documentation/plugins/fluxes/. Please provide it with a self-explaining name. Example for the template: file fluxes_template.rst reads
.. code-block:: rest
.. role:: bash(code)
:language: bash
###########################
Template plugin for fluxes
###########################
.. automodule:: pycif.plugins.fluxes.flux_plugin_template
c) add the reference to the rst file in docs/source/documentation/plugins/fluxes/index.rst:
.. code-block:: rest
####################
Fluxes :bash:`fluxes`
####################
Available Fluxes
=========================
The following :bash:`fluxes` are implemented in pyCIF:
.. toctree::
:maxdepth: 3
fluxes_template
dummy_nc
dummy_txt
edgar_v5
flexpart
chimere
lmdz_bin
lmdz_sflx
d) built the documentation (:bash:`make html` in docs/) and check that the link to the new plugin appears in the documentation at file:///your_path/cif/docs/build/html/documentation/plugins/index.html and that the section "doc" of the input arguments is correctly displayed at file:///your_path/cif/docs/build/html/documentation/plugins/fluxes/the_new_plugin.html
XXXXXXXX
.. role:: bash(code)
:language: bash
########################
Template plugin for BCs
########################
.. automodule:: pycif.plugins.fields.bc_plugin_template
......@@ -13,4 +13,6 @@ The following :bash:`fields` are implemented in pyCIF:
.. toctree::
chimere_icbc
bc_template
chimere_icbc
.. role:: bash(code)
:language: bash
###########################
Template plugin for fluxes
###########################
.. automodule:: pycif.plugins.fluxes.flux_plugin_template
......@@ -13,7 +13,8 @@ The following :bash:`fluxes` are implemented in pyCIF:
.. toctree::
:maxdepth: 3
fluxes_template
dummy_nc
dummy_txt
edgar_v5
......
......@@ -6,15 +6,18 @@ from .write import write
_name = "BCs"
_version = "template"
#requirements = {
# "domain": {
# "name": "CHIMERE",
# "version": "std",
# "empty": False,
# "any": False,
# },
#}
input_arguments = {
"README": {
"doc": "Write here the README about the plugin - XXXX how to display it when the plugin is called?"
" or give link to the doc about this plugin?XXXX"
"relevant information: type of files treated, including format of names and shape of data, "
" time resolution, and any specific treatment that prevents the plugin from working "
" with another type of files.",
"default": False,
"accepted": bool
},
"dummy_arg": {
"doc": "document here the argument",
"default": "let's say it's not mandatory",
......
from .fetch import fetch
from .get_domain import get_domain
from .read import read
from .write import write
_name = "GCP"
_version = "1x1"
input_arguments = {
"README": {
"doc": "README about the plugin - how to disply it when the plugin is called?"
" or give link to the doc about this plugin?",
"default": False,
"accepted": bool
},
}
import datetime
import glob
import os
import pandas as pd
import numpy as np
from pycif.utils import path
from logging import info
from .utils import find_valid_file
def fetch(ref_dir, ref_file, input_dates, target_dir, tracer=None, **kwargs):
if getattr(tracer, "README"):
print('Display here the information about the plugin =how?XXXX')
list_files = {}
list_dates = {}
for datei in input_dates:
tmp_files = []
tmp_dates = []
for dd in input_dates[datei]:
#print('Date to simulate:', dd)
files_orig, dates_orig = find_valid_file(ref_dir, ref_file, dd)
tmp_files.extend(files_orig)
tmp_dates.extend(dates_orig)
# Fetching
local_files = []
for f in tmp_files:
target_file = "{}/{}".format(target_dir, os.path.basename(f))
path.link(f, target_file)
local_files.append(target_file)
list_files[datei] = sorted(list(set(local_files)))
list_dates[datei] = sorted(list(set(tmp_dates)))
return list_files, list_dates
import datetime
import glob
import os
import numpy as np
import xarray as xr
from pycif.utils.classes.setup import Setup
import copy
def get_domain(ref_dir, ref_file, input_dates, target_dir, tracer=None):
# Looking for a reference file to read lon/lat in
list_file = glob.glob("{}/*nc".format(ref_dir))
domain_file = None
# Either a file is specified in the Yaml
if ref_file in list_file:
domain_file = "{}/{}".format(ref_dir, ref_file)
# Or loop over available file regarding file pattern
else:
for flx_file in list_file:
try:
date = datetime.datetime.strptime(
os.path.basename(flx_file), ref_file
)
domain_file = flx_file
break
except ValueError:
continue
if domain_file is None:
raise Exception(
"GCP domain could not be initialized as no file was found"
)
print('domain file for GCP fluxes:', domain_file)
nc = xr.open_dataset(domain_file, decode_times=False)
# Read lon/lat in domain_file
lon = nc["LON"]
lat = nc["LAT"]
# must be increasing
if lat[1] < lat[0]:
lat2 = np.flip(lat)
lat = copy.copy(lat2)
nlon = lon.size
nlat = lat.size
lon_min = lon.min()
lon_max = lon.max()
lat_min = lat.min()
lat_max = lat.max()
# Compute corners #WARNING: only valid for regular grid in lat/lon -> to generalize
dx = (lon[1] - lon[0]) / 2
dy = (lat[1] - lat[0]) / 2
lonc = np.linspace(lon_min - dx/2., lon_max + dx/2., nlon + 1)
latc = np.linspace(lat_min - dy/2., lat_max + dy/2., nlat + 1)
# Read vertical information in domain_file
# if relevant for emissions
# XXXXXXXXX comment voir si une dimension verticale est presente?
# else: dummy vertical
punit = "Pa"
nlevs = 1
sigma_a = np.array([0])
sigma_b = np.array([1])
# Initializes domain
setup = Setup.from_dict(
{
"domain": {
"plugin": {
"name": "dummy",
"version": "std",
"type": "domain",
},
"xmin": lon_min,
"xmax": lon_max,
"ymin": lat_min,
"ymax": lat_max,
"nlon": nlon,
"nlat": nlat,
"nlev": nlevs,
"sigma_a": sigma_a[:],
"sigma_b": sigma_b[:],
"pressure_unit": "Pa"
}
}
)
Setup.load_setup(setup, level=1)
zlon, zlat = np.meshgrid(lon, lat)
zlonc, zlatc = np.meshgrid(lonc, latc)
setup.domain.zlon = zlon
setup.domain.zlat = zlat
setup.domain.zlonc = zlonc
setup.domain.zlatc = zlatc
return setup.domain
import datetime
import glob
import os
import numpy as np
import xarray as xr
from logging import info
from pycif.utils.netcdf import readnc
def read(
self,
name,
tracdir,
tracfile,
varnames,
dates,
interpol_flx=False,
tracer=None,
model=None,
**kwargs
):
# tracfile can be a list of same length as dates
if type(tracfile) == str:
tracfile = [tracfile[:]]
if len(tracfile) != len(dates) \
and len(tracfile) > 1:
raise Exception(
"Try read GCP files from a list of dates and a "
"list of files, but not of same length:\n{}\n{}".format(
tracfile, dates
)
)
elif len(tracfile) == 1:
list_files = len(dates) * tracfile
else:
list_files = tracfile[:]
# Reading fluxes for periods within the simulation window
trcr_flx = []
for dd, dd_file in zip(dates, list_files):
file_flx = dd.strftime(dd_file)
fluxes = readnc(file_flx, [varnames])
#print('FFFFFFFFFFFFFFFFFF',fluxes.shape)
time = readnc(file_flx, ['TIME'])
lat = readnc(file_flx,['LAT'])
year_beg_file = int((dd_file.split('-')[0]).split('.')[-1])
index_dd = (dd.year - year_beg_file) * 12 + dd.month -1
trcr_flx.append(fluxes[index_dd, :, :])
if lat[1] < lat[0]:
xmod = xr.DataArray(
np.array(trcr_flx)[:, np.newaxis, ::-1,:],
coords={"time": dates},
dims=("time", "lev", "lat", "lon"),
)
else:
xmod = xr.DataArray(
np.array(trcr_flx)[:, np.newaxis, :,:],
coords={"time": dates},
dims=("time", "lev", "lat", "lon"),
)
return xmod
import datetime
import glob
import os
import calendar
import numpy as np
def find_valid_file(ref_dir, file_format, dd):
# Get all files and dates matching the file and format
#print('Here, basic listing of all available files')
list_files_orig = os.listdir(ref_dir)
list_files_avail = []
for f in list_files_orig:
try:
check = datetime.datetime.strptime(f, file_format)
list_files_avail.append(f)
except:
continue
#print('All availables files',list_files_avail)
list_files = []
list_dates = []