Skip to content
Snippets Groups Projects

Home

Last edited by Paul Eckhardt

EBAS IO

Introduction

ebas-io is a python package primarily for reading/writing NASA-Ames 1001 files with EBAS extensions (EBAS NASA-Ames).

EBAS IO provides a memory model for data files. As a primary import format, EBAS Nasa Ames is implemented, some other input formats for reading can be found in the Examples folder. Additional input formats can be implemented by users as needed.

For output, additional formats are supported (appart form Nasa Ames, NetCDF and CSV is implemented in the core module). Other output formats can be extended by users as needed.

Data read into the memory model can be used conveniently for assessments or model input.

Contact and Support

If you have any questions please contact ebas@nilu.no.

Just want to test it?

If you do not want to install the package on any of your machines yet, you can try the package in binder. With the link below, a virtual environment is set up for you in mybinder and you can use the Jupyter notebooks from the Example directory interactively and also try different things. Binder

Downloading the software

The python wheel is available in this git repository (dist folder; mind the two different versions for python 2 and 3) together with the example scripts. The idea behind the git repo ist to make it easier for users to contribute their software using ebas-io and make it available for other users.

a) You can clone the repository with git clone https://git.nilu.no/ebas/ebas-io.git

b) If you are not familiar with git, and don't know how to clone a repository, please proceed like this:

  • go to the Files section
  • on the right hand side above the file list you will find a drop down selector containing Download ...
  • select the archive format you like best and download a static copy of the whole repository
  • extract the downloaded archive

In either way, you should finally end up with a directory called ebas-io.

Installation

The ebas-io package can be installed using pip. See the pip documentation if pip does not work for you. The whl files can be found in the directory ebas-io/dist

pip install  ebas_io-4.3.19-py3-none-any.whl

You might need to use a certain installation of pip, according to the python version you plan to use (e.g. pip2, pip3 etc.). Please pay attention to the python version of the wheel (...-py2-... for python2, ...-py3-... for python3).

Please mind that you probably have to have administrator rights ('sudo pip ...' on linux/unix machines). Alternatively you may want to install the packages locally for your user (see the --user option in the pip documentation).

Known problems in python 3.5.: At least in python 3.5.6, problems have been reported using ebas-io. The problem seems to be related to a bug in pytz (during unpickling datetime objects). If you experience the following exception, please consider upgrading your python version: TypeError: an integer is required (got type str)

Examples

Example scripts and test data can be found in the repository in the directory ebas-io/Examples.

The most recent examples can be found in the Notebooks folder. Those notebooks can also be used hands-on in a Binder environment (Binder), which is the most convenient way to work with the examples.

The examples described here are plain python scripts which can be either used as is, or can be customised to your needs.

Most examples support the --help (or -h) option on the commandline which gives a short explanation of the script and the commandline arguments.

Reading EBAS files: ebas_read_example.py

This example shows the main functionality for reading EBAS NASA-Ames files. It demonstrates how to:

  • open a file
  • access and filter the time axes
  • access all metadata for each variable
  • access variable values and flags
  • read data without flags (valid values only)
  • look for specific variables in a file (based on metadata)
./ebas_read_example.py Testdata/DE0001R.20141225000000.20160114101135.uv_abs.ozone.air.1d.1h.DE03L_UBA_We_Mon_0301.DE03L_uv_abs.lev2.nas

The script reads the input file and writes a lot of data about the file as output. See the program code (and the comments within) how things work and compare it to the output. Please notice that the input file contains the ozone variable twice (in different units: ug/m3 and nmol/mol, this was generated for user convenience when the file was generated by the database). However, when reading the file, the nmol/mol varable gets automatically converted to ug/m3 as it would be stored in ebas. If you do not want this behaviour, you can specify the argument --no-unitconvert-read in order to avoid this conversion. In this case you will need to use the argument --ignore parameter as well, otherwise the parameter is rejected because of the wrong ebas unit.

./ebas_read_example.py Testdata/DE0001R.20141225000000.20160114101135.uv_abs.ozone.air.1d.1h.DE03L_UBA_We_Mon_0301.DE03L_uv_abs.lev2.nas --no-unitconvert-read --ignore parameter,template-check

Changed argument names in 4.0.0 Starting with version 4.0.0, some argument names have changed: --skip-unitconvert is now --no-unitconvert-read and --no-unitconvert-write depending whether the conversion on input or output should be controlled. The argument --ignore_parameter is now --ignore parameter and has been collected with other ignore arguments. See also --help for more info.

Now, the file is read as is, no unit conversion is done, and the ozone variables in both units can be accessed.

Generating EBAS files: ebas_genfile.py

This example shows how to generate a valid EBAS NASA-Ames files for data submission with a script. You can use this script as a template for programming your own data submission software, tailored to your dataflows. This script demonstrates how to:

  • set file global metadata
  • set the time axes of the file
  • define variables
  • set variable metadata
  • fill in variable values and flags
  • write the file
./ebas_genfile.py

will create an output file named NO0042G.20140101110000.20150213125421.passive_puf..air.11mo.3mo.NO01L_puf_42.NO01L_gc_ms.lev2.nas. Read the program code and discover how the file is generated.

More examples for generating EBAS

In the subdirectory ebas_genfile_templates we started to collect scripts for creating data submissions according to different specific submission templates (one for filter_absorbtion_photometer data, one for smps data, ...). This collection will hopefully grow over time, we will be happy about any contribution of scripts by users. We'd like to thank those who provided their scripts already!

Reformat an EBAS Nasa Ames File: ebas_reformat.py

This script reads an EBAS Nasa Ames File and writes it again. In between, the format is standardised automatically by the ebas-io module:

  • The order of the metadata elements is standardised
  • The occurrence of metadata elements in the file global metadata vs. varable metadata is optimized
  • Variables are sorted
  • Flag columns are optimised
  • Number of digits used for flags is minimised
  • Number format and missing values are optimised according to actual data
  • The filename is standardised
  • ...

The tool can be used to reformat badly formatted files generated manually or by other tools into a reproducible standard representation. The source file must be readable (not syntax errors)

./ebas_reformat.py Testdata/ugly_file

Produces a standardized output file in the current directory, the filename is according to the EBAS file name convention.

File format conversion: ebas_convert.py

This script converts from EBAS Nasa Ames to other export formats supported by EBAS (use ebas_convert.py --help for alist of formats). There are some additional features, like:

  • ignore certain errors in the source file
  • skip some variables
  • change the flag output style

When using --format NasaAmes, the script does about the same as the ebas_reformat example above.

./ebas_convert.py Testdata/*.nas --format CSV --createfiles

Converts all the .nas files in the Testdata directory to .csv format and creates the output files in the current directory.

Convert to other file formats: ebas_flatcsv.py

This shows that new export formats can easily be implemented by a user. One is not limited by the supported export formats in EBAS.

This example originates from a data user's request whether we could deliver EBAS data in a different file format. He wanted to have a simple, flat, highly redundant csv format for easy ingestion of the data into his software. We came up with this short script and he was able to convert the EBAS NASA-Ames files we delivered by himself. No new data format to maintain for us, full control of the interface for the data user.

./ebas_flatcsv.py Testdata/*.nas

All files in the Examples directory are read and the data are written to one flat csv file called 'output.csv'.

More Documentation

EBAS Metadata tags and the keys to be used in the IO object

The EbasMetadata class controls the syntax of the metadata elements used in a file and relates them to the key names used in the IO objects for accessing the respective metadata element.

As an example for the Station code: The EbasMetadata class defines that the tag used in the file is 'Station code' and that the metadata key 'station_code' is used in the IO object. Additionally all the basic syntax checks are also defined in the EbasMetadata class.

So, for knowing which EBAS metadata elements can be used at all, what's their tag in the NASA Ames file, and which key needs to be used in the IO object to set or retrieve the metadata element, the EbasMetdata object is the place to look for. Some examples of how to use that class can be found in an ipython Notebook under Examples/Doc/Notebooks/EbasMetadata.ipynb. For users who are less familiar with git and ipython notebooks we provide a static html version of the file.

Old Release Notes

Attention: changed behaviour of the EbasCommandline wrapper object! Starting with version 3.1.1, the EbasCommandline object does not automatically execute the main function. Now an explicit run() on the object is needed. Please consider this if you use the EbasCommandline wrapper in your scripts! Example:

EbasCommandline(
    ebas_convert,
    custom_args=['CONFIG', 'LOGGING', 'TIME_CRIT'],
    private_args=add_private_args,
    help_description='%(prog)s example for converting a NasaAmes datafile to '
    'different dataformats', version=__version__).run()

Attention: changed package structure as of version 3.0.13! Starting with version 3.0.13, all nilu internal dependencies (nilutility, AtmosPhys, fileformats) are now included in the ebas-io egg. This means, the installation is now much simpler. But: users having installed an earlier version should clean their python installation form this old extra packages before installing the new ebas-io package. This can be done as follows:

pip uninstall AtmosPhys
pip uninstall fileformats
pip uninstall nilutility
pip uninstall ebas-io

Comments

    Please register or sign in to add a comment.