ebas-io is a python package for reading/writing NASA-Ames 1001 files with EBAS extensions (EBAS NASA-Ames).
Attention: changed behaviour of the EbasCommandline wrapper object!
Starting with version 3.1.1, the EbasCommandline object does not automatically execute the main function. Now an explicit run() on the object is needed. Please consider this if you use the EbasCommandline wrapper in your scripts!
EbasCommandline(ebas_convert,custom_args=['CONFIG','LOGGING','TIME_CRIT'],private_args=add_private_args,help_description='%(prog)s example for converting a NasaAmes datafile to ''different dataformats',version=__version__).run()
Downloading the software
Starting with version 3.00.13, the python egg is available in this git repository (dist folder) together with the example scripts. The idea behind the git repo ist to make it easier for users to contribute their software using ebas-io and make it available for other users.
a) You can clone the repository with git clone https://git.nilu.no/ebas/ebas-io.git
b) If you are not familiar with git, and don't know how to clone a repository, please proceed like this:
on the right hand side above the file list you will find a drop down selector containing Download ...
select the archive format you like best and download a static copy of the whole repository
extract the downloaded archive
In either way, you should finally end up with a directory called ebas-io.
The ebas-io package can be installed the standard way with easy_install. See the setuptools documentation if easy_install does not work for you. The egg file can be found in the directory ebas-io/dist
Please mind that you probably have to have administrator rights ('sudo easy_install ...' on linux/unix machines). Alternatively you may want to install the packages locally for your user (see --prefix in the easy_install documentation).
Attention: changed package structure as of version 3.0.13!
Starting with version 3.0.13, all nilu internal dependencies (nilutility, AtmosPhys, fileformats) are now included in the ebas-io egg. This means, the installation is now much simpler.
But: users having installed an earlier version should clean their python installation form this old extra packages before installing the new ebas-io package. This can be done as follows:
The script reads the input file and writes a lot of data about the file as output. See the program code (and the comments within) how things work and compare it to the output.
Please notice that the input file contains the ozone variable twice (in different units: ug/m3 and nmol/mol, this was generated for user convenience when the file was generated by the database). However, when reading the file, the nmol/mol varable gets automatically converted to ug/m3 as it would be stored in ebas. If you do not want this behaviour, you can specify the argument --skip_unitconvert in order to avoid this conversion. In this case you will need to use the argument --ignore_parameter as well, otherwise the paramter is rejected because of the wrong ebas unit.
Now, the file is read as is, no unit conversion is done, and the ozone variables in both units can be accessed.
Generating EBAS files: ebas_genfile.py
This example shows how to generate a valid EBAS NASA-Ames files for data submission with a script. You can use this script as a template for programming your own data submission software, tailored to your dataflows. This script demonstrates how to:
set file global metadata
set the time axes of the file
set variable metadata
fill in variable values and flags
write the file
will create an output file named NO0042G.20140101110000.20150213125421.passive_puf..air.11mo.3mo.NO01L_puf_42.NO01L_gc_ms.lev2.nas. Read the program code and discover how the file is generated.
More examples for generating EBAS
In the subdirectory ebas_genfile_templates we started to collect scripts for creating data submissions according to different specific submission templates (one for filter_absorbtion_photometer data, one for smps data, ...). This collection will hopefully grow over time, we will be happy about any contribution of scripts by users. We'd like to thank those who provided their scripts already!
Reformat an EBAS Nasa Ames File: ebas_reformat.py
This script reads an EBAS Nasa Ames File and writes it again. In between, the format is standardised automatically by the ebas-io module:
The order of the metadata elements is standardised
The occurrence of metadata elements in the file global metadata vs. varable metadata is optimized
Variables are sorted
Flag columns are optimised
Number of digits used for flags is minimised
Number format and missing values are optimised according to actual data
The filename is standardised
The tool can be used to reformat badly formatted files generated manually or by other tools into a reproducible standard representation. The source file must be readable (not syntax errors)
Produces a standardized output file in the current directory, the filename is according to the EBAS file name convention.
File format conversion: ebas_convert.py
This script converts from EBAS Nasa Ames to other export formats supported by EBAS (use ebas_convert.py --help for alist of formats). There are some additional features, like:
ignore certain errors in the source file
skip some variables
change the flag output style
When using --format NasaAmes, the script does about the same as the ebas_reformat example above.
Converts all the .nas files in the Testdata directory to .csv format and creates the output files in the current directory.
Convert to other file formats: ebas_flatcsv.py
This shows that new export formats can easily be implemented by a user. One is not limited by the supported export formats in EBAS.
This example originates from a data user's request whether we could deliver EBAS data in a different file format. He wanted to have a simple, flat, highly redundant csv format for easy ingestion of the data into his software. We came up with this short script and he was able to convert the EBAS NASA-Ames files we delivered by himself. No new data format to maintain for us, full control of the interface for the data user.
All files in the Examples directory are read and the data are written to one flat csv file called 'output.csv'.
EBAS Metadata tags and the keys to be used in the IO object
The EbasMetadata class controls the syntax of the metadata elements used in a file and relates them to the key names used in the IO objects for accessing the respective metadata element.
As an example for the Station code: The EbasMetadata class defines that the tag used in the file is 'Station code' and that the metadata key 'station_code' is used in the IO object. Additionally all the basic syntax checks are also defined in the EbasMetadata class.
So, for knowing which EBAS metadata elements can be used at all, what's their tag in the NASA Ames file, and which key needs to be used in the IO object to set or retrieve the metadata element, the EbasMetdata object is the place to look for. Some examples of how to use that class can be found in an ipython Notebook under
Examples/Doc/Notebooks/EbasMetadata.ipynb. For users who are less familiar with git and ipython notebooks we provide a static html version of the file.