From 5e94b00db29b94192b4949087c084121e746b86c Mon Sep 17 00:00:00 2001
From: Paul Eckhardt <pe@nilu.no>
Date: Thu, 22 Aug 2024 21:38:29 +0200
Subject: [PATCH] add variables
---
docs/fileformat_netcdf/dimensions.rst | 17 +++
docs/fileformat_netcdf/metadata.rst | 17 +--
docs/fileformat_netcdf/variables.rst | 165 +++++++++++++++++++++++++-
3 files changed, 181 insertions(+), 18 deletions(-)
diff --git a/docs/fileformat_netcdf/dimensions.rst b/docs/fileformat_netcdf/dimensions.rst
index 819eb98..39ecfcf 100644
--- a/docs/fileformat_netcdf/dimensions.rst
+++ b/docs/fileformat_netcdf/dimensions.rst
@@ -10,7 +10,11 @@ The following dimensions are present in every EBAS NetCDF file:
* ``time``: time dimension for measurement data
* ``metadata_time``: time dimension for metadata
* ``tbnds``: addirional dimension for :ref:``time boundary variables<time_bnds>``
+ * For each data variable, there will be a dimension for the different flags
+ used in the flag variable.
+ * Additional dimensions for multidimensional variables
+.. _dim_time:
time
----
@@ -67,6 +71,8 @@ boundary variable ``metadata_time_bnds`` contains an additional dimension
validity period.
+.. _dim_additional:
+
Additional dimensions
=====================
@@ -101,4 +107,15 @@ Some parameters in EBAS have additional dimensions, the most prominent are:
- Location of measurements, used for auxiliary data. E.g. temperature, pressure or humidity measured at different points in the sample line.
E.g. temperature, Location=inlet; temperature, Location=instrument inlet; temperature, Location=instrument internal; temperature, Location=sheath air loop
+.. _dim_flag:
+
+Flag dimensions
+===============
+
+Each data variable is accompanied by a variable containing flag information.
+This flag variable has the same dimensions as the measurement variable, plus one
+additional dimension (``<variable name>_qc_flags``) for the flags which are
+associated with each single measurement.
+The length of this additional dimension is equal the maximum number of flags attached to a single measurement in the file.
+More information on this in :ref:`flag_variable`.
diff --git a/docs/fileformat_netcdf/metadata.rst b/docs/fileformat_netcdf/metadata.rst
index 3fad768..72a3e7a 100644
--- a/docs/fileformat_netcdf/metadata.rst
+++ b/docs/fileformat_netcdf/metadata.rst
@@ -78,8 +78,8 @@ Metadata will appear in three different places:
.. _time_dependent_metadata:
-Time depenendent metadata
-=========================
+Time depenendent EBAS metadata
+------------------------------
As mentioned above, some metadata in EBAS can change over time. For file formats like
the EBAS Nasa Ames file format, this means that a files need to be split on every change
@@ -89,21 +89,12 @@ One of the biggest technical advantage of the EBAS NetCDF fileformat is that tho
metadata changes can be stored in the file. Thus, all datasets (DOIs) in EBAS
can be written to a single NetCDF file.
-Implementation
---------------
-
Time dependent metadata are implemented as an additional variable for **each data
-variable**, named like the main variable name with a postfix ``_ebasmetadata``.
-
-The data type for the metadata variables is string and it contains a set of all ebas
-metadata for the specific time range as json encoded string.
+variable**, see :ref:`metadata_variable`
There is a dedicated time dimension for all metadata changes, called :ref:``metadata_time<dim_metadata_time>``.
This should not be confused with the ``time`` dimension used for the measurement data.
-The ``time`` dimension has usually a length of > 1000 (one for each measurment interval),
+The ``time`` dimension has usually a length of > 1000 (one for each measurement interval),
the ``metadata_time`` dimension has typically only a length of < 10, one for each change of metadata.
-In analogy to the time variable, the ``metadata_time`` variable has also a related boundary
-variable ``metadata_time_bnds``. See :ref:`time_bnds` for more information on the boundary variables for the time dimensions.
-
diff --git a/docs/fileformat_netcdf/variables.rst b/docs/fileformat_netcdf/variables.rst
index 17c9ac8..48db301 100644
--- a/docs/fileformat_netcdf/variables.rst
+++ b/docs/fileformat_netcdf/variables.rst
@@ -2,9 +2,164 @@
Variables
*********
- * time_bnds ? --> dimensions
- * metadatatime_bnds ? --> dimensions
+Dimension Variables
+===================
+
+Some special variables have already been mentioned before when describing dimensions:
+
+ * :ref:`time<dim_time>` and time_bnds for the measurement and time dimension
+ * :ref:`metadata_time<dim_metadata_time>` and metadata_time_bnds for the metadata time dimension
+ * :ref:`dim_additional` like Wavelength, D (particle diameter), Tower inlet height, etc.
+ * :ref:`dim_flag`
+
+
+Data variables
+==============
+
+Each data variable in EBAS is represented by three NetCDF variables:
+
+ * :ref:`measurement variable`: The main variable containing the measurement values. Dimensions: :ref:`dim_time`, optional: :ref:`dim_additional`
+ * :ref:`flag_variable`: an additional variable containing the flag information for the variable. Dimensions: same as measurements + :ref:`dim_flag`
+ * :ref:`metadata_variable`: a variable for changing metadata over time
+
+.. _measurement variable:
+
+Measurement variable
+--------------------
+
+This is the main variable, containing the measurement values for an EBAS variable.
+The name for measurement variables is described :ref:`here<variable names>`.
+
+The dimensions are:
+
+ * :ref`dim_time`
+ * optional: :ref:`dim_additional`
+
+The data type is ``double``.
+
+Variable Attributes
+^^^^^^^^^^^^^^^^^^^
+
+The variable attributes contain the full set of EBAS metadata (constant over time
+for the variable). But the most important attributes for understanding the content of the variable are:
+
+ * ``cell_methods`` and ``ebas_statistics`` give information about the statistical interpretation (e.g. arithmetic mean, percentile:15.87, ...)
+ * ``ebas_component``: Component name in EBAS
+ * ``ebas_matrix``: Matrix name in EBAS
+ * ``units`` and ``ebas_unit``: Unit of the measurements
+ * ``ancillary_variables``: reference to the associated :ref:`flag_variable` and :ref:`metadata_variable`
+
+
+.. _flag_variable:
+
+Flag variable
+-------------
+
+For each measurement variable a related flag variable is added which contains the flag information for the measurements.
+Flag variable are named like the measurement variable plus a postfix ``_qc``.
+
+The dimensions are:
+
+ * Same dimensions as the measurement variable
+ * An additional :ref:`flag dimension<dim_flag>` called ``<variable name>__qc_flags``
+
+The data type is ``int``.
+
+
+.. _metadata_variable:
+
+Metadata variable
+-----------------
+
+This additional variable implements :ref:`time_dependent_metadata`.
+It contains metadata which change over time. This variable is
+provided for each data variable.
+The metadata variables are named like the main variable with a postfix ``_ebasmetadata``.
+
+The dimensions are:
+
+ * :ref:`dim_metadata_time`
+ the length of the dimension is 1 for static metadata and n + 1 for each metadata change over time within the file
+
+
+The data type for the metadata variables is string and it contains a set of all ebas
+metadata for the specific time range as json encoded string.
+
+
+Distinguishing Measurement variables, Flag variables and Metadata variables
+---------------------------------------------------------------------------
+
+ * Only the measurement variable has EBAS metadtaa as variable attributes
+ * Only the measurement variable has a variable attributes ``_metadata_variable`` with references to the associated flag variable and metadata variable
+
+ * The name of flag variables always ends with ``_qc`` (same name as measurement variable + ``_qc``)
+ * The variable attribute ``standard_name`` is set to``status_flag``
+
+ * The name of metadata variables always ends with ``_ebasmetadata`` (same name as measurement variable + ``_ebasmetadata``)
+
+
+.. _variable names:
+
+Variable names
+--------------
+
+Flag variables and metadata variables always use the same name as the measurement variables
+plus a specific postfix (``_qc`` and ``_ebasmetadata`` respectively).
+
+The name for measurement variables is not straight forward, and may vary in
+different files depending on other variables in the file and related name conflicts.
+
+ # Generally, the variable name will be set to the ``ebas_component`` name
+ # If there are name conflicts (variables with an identical name) and one or
+ more of the following metadata distinguish the variables with identical names:
+
+ * matrix
+ * unit
+ * statistics
+ * additional dimensions
+
+ The respective element will be added to the name of all conflicting variables.
+
+Examples 1:
+^^^^^^^^^^^
+
+A file contains three variables with component name ``aerosol_light_scattering_coefficient_amean``,
+one with ``ebas_statistics=arithmetic mean``, one with ``ebas_statistics=percentile:15.87`` and one with ``ebas_statistics=percentile:84.13``
+
+Solution: The tree variable will be called:
+
+ * ``aerosol_light_scattering_coefficient_amean``
+ * ``aerosol_light_scattering_coefficient_prec1587`` and
+ * ``aerosol_light_scattering_coefficient_prec8413``.
+
+And each of those variables will be accompanied by their ``_qc`` and ``_ebasmetadata`` variable (total 9 variables).
+
+Example 2:
+^^^^^^^^^^
+
+A file contains ozone measurement both in the EBAS default unit 'ug/m3' and in a converted variable in 'nmol/mol'. Both are present as ``ebas_statistics=arithmetic mean``, ``ebas_statistics=min``, ``ebas_statistics=max`` and ``ebas_statistics=stddev``.
+
+Solution: The eight variables will be called:
+
+ * ``ozone_ug_per_m3_amean``
+ * ``ozone_ug_per_m3_min``
+ * ``ozone_ug_per_m3_max``
+ * ``ozone_ug_per_m3_stddev``
+ * ``ozone_nmol_per_mol_amean``
+ * ``ozone_nmol_per_mol_min``
+ * ``ozone_nmol_per_mol_max`` and
+ * ``ozone_nmol_per_mol_stddev``
+
+And each of those variables will be accompanied by their ``_qc`` and ``_ebasmetadata`` variable (total 24 variables).
+
+.. warning::
+ Never rely on variable names. As shown above, they might vary from file to file.
+
+ In order to find measurement variables in a file, use the variable attributes!
+
+ For example, search for a variable with the following combination of attributes:
+
+ * ``ebas_component=ozone``
+ * ``ebas_statistics=arithmetic mean``
+ * ``ebas_unit=nmol/mol``
- * general - amean, gc, ebasmetadata
- * statistics - precentiler, std etc.
- * converted units
--
GitLab