Skip to content
Snippets Groups Projects
Commit 0e1fdbf5 authored by Paul Eckhardt's avatar Paul Eckhardt
Browse files

add netcdf tags

parent bba1cd7a
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Examples/Doc/Notebooks/EbasMetadata.ipynb
+ 139
− 142
View file @ 0e1fdbf5
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
# Using the EbasMetadata object # Using the EbasMetadata object
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
The EbasMetadata object defines the syntax of the ebas.io file objects. It is used internally by the io.file objects for creating the output files and parsing input files. The EbasMetadata object defines the syntax of the ebas.io file objects. It is used internally by the io.file objects for creating the output files and parsing input files.
The user migtht, when creating an output file use the object for retrieving information about how to set a specific attribute in the file object in order to create a specifig metadata element in the output file. The user migtht, when creating an output file use the object for retrieving information about how to set a specific attribute in the file object in order to create a specifig metadata element in the output file.
Or vice versa, when reading an input file, know which attribute to access when the contents of a specific metadata element in the file is needed. Or vice versa, when reading an input file, know which attribute to access when the contents of a specific metadata element in the file is needed.
Please find below some examples how to list this kind of information interactively. Please find below some examples how to list this kind of information interactively.
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## General setup: imports and instanciation ## General setup: imports and instanciation
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
This part is needed for all the examples below. This part is needed for all the examples below.
Be aware to specify the datalevel if you want to see metadata that are only allowed for level 0. This is important if you want to write a script for level 0 data submissions. For some templates (e.g. mmps, nephelometer and filter_absorption_potometer), additional metadata are used which are only valid for data level 0. Be aware to specify the datalevel if you want to see metadata that are only allowed for level 0. This is important if you want to write a script for level 0 data submissions. For some templates (e.g. mmps, nephelometer and filter_absorption_potometer), additional metadata are used which are only valid for data level 0.
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
from ebas.io.ebasmetadata import EbasMetadata, EBAS_IOFORMAT_NASA_AMES from ebas.io.ebasmetadata import EbasMetadata, EBAS_IOFORMAT_NASA_AMES
meta = EbasMetadata('EBAS_1.1', EBAS_IOFORMAT_NASA_AMES, data_level='0') meta = EbasMetadata('EBAS_1.1', EBAS_IOFORMAT_NASA_AMES, data_level='0')
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## List all possible metadata elements ## List all possible metadata elements
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
This code sequence lists all possible metadata elements in EBAS. Each line contains the tag (the tag used in the Nasa Ames files) and the key (the key used in the io object's metadata). This code sequence lists all possible metadata elements in EBAS. Each line contains the Nasa Ames tag (the tag used in the Nasa Ames files), the NetCDF tag (attribute name used in NetCDF files) and the object key (the key used in the io object's metadata).
We only list metadata which are allowed either as global metadata (main) or as variable metadata (vname) We only list metadata which are allowed either as global metadata (main) or as variable metadata (vname)
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
print(f'{"Nasa Ames tag":45s} {"NetCDF attribute":35s} Object key')
print(f'{"-"*45} {"-"*35} ----------')
for elem in [x for x in meta.metadata if x['main'] or x['vname']]: for elem in [x for x in meta.metadata if x['main'] or x['vname']]:
if not 'renamed_tag' in elem: if not 'renamed_tag' in elem:
print("{:45s} {}".format(elem['tag']+":", elem['key'])) print(f'{elem["tag"]:45s} {elem["nc_tag"]:35s} {elem["key"]}')
``` ```
%% Output %% Output
Data definition: datadef Nasa Ames tag NetCDF attribute Object key
Data license: license --------------------------------------------- ----------------------------------- ----------
Citation: citation Data definition data_definition datadef
Set type code: type Data license data_license license
Timezone: timezone Citation citation citation
Timeref: timeref Set type code set_type_code type
File name: filename Timezone timezone timezone
Represents DOI: doi Timeref timeref timeref
Contains data from DOI: doi_list File name file_name filename
File creation: creation_time Represents DOI represents_doi doi
Export state: export_state Contains data from DOI contains_doi doi_list
Export filter: export_filter File creation file_creation creation_time
Startdate: startdate Export state export_state export_state
Revision date: revdate Export filter export_filter export_filter
Version: revision Startdate startdate startdate
Version description: revdesc Revision date revision_date revdate
Input dataset: input_dataset Version version revision
Software: software Version description version_description revdesc
Statistics: statistics Input dataset input_dataset input_dataset
Data level: datalevel Software software software
Period code: period Statistics statistics statistics
Resolution code: resolution Data level data_level datalevel
Sample duration: duration Period code period_code period
Orig. time res.: rescode_sample Resolution code resolution_code resolution
Station code: station_code Sample duration sample_duration duration
Platform code: platform_code Orig. time res. orig_time_res rescode_sample
Station name: station_name Station code station_code station_code
Station WDCA-ID: station_wdca_id Platform code platform_code platform_code
Station GAW-ID: station_gaw_id Station name station_name station_name
Station GAW-Name: station_gaw_name Station WDCA-ID station_wdca_id station_wdca_id
Station AIRS-ID: station_airs_id Station GAW-ID station_gaw_id station_gaw_id
Station other IDs: station_other_ids Station GAW-Name station_gaw_name station_gaw_name
Station state/province: station_state_code Station AIRS-ID station_airs_id station_airs_id
Station land use: station_landuse Station other IDs station_other_ids station_other_ids
Station setting: station_setting Station state/province station_state_province station_state_code
Station GAW type: station_gaw_type Station land use station_land_use station_landuse
Station WMO region: station_wmo_region Station setting station_setting station_setting
Station latitude: station_latitude Station GAW type station_gaw_type station_gaw_type
Station longitude: station_longitude Station WMO region station_wmo_region station_wmo_region
Station altitude: station_altitude Station latitude station_latitude station_latitude
Measurement latitude: mea_latitude Station longitude station_longitude station_longitude
Measurement longitude: mea_longitude Station altitude station_altitude station_altitude
Measurement altitude: mea_altitude Measurement latitude measuremenet_latitude mea_latitude
Measurement height: mea_height Measurement longitude measurement_longitude mea_longitude
Regime: regime Measurement altitude measurement_altitude mea_altitude
Component: comp_name Measurement height measurement_height mea_height
Unit: unit Regime regime regime
Matrix: matrix Component component comp_name
Laboratory code: lab_code Unit unit unit
Instrument type: instr_type Matrix matrix matrix
Instrument name: instr_name Laboratory code laboratory_code lab_code
Instrument manufacturer: instr_manufacturer Instrument type instrument_type instr_type
Instrument model: instr_model Instrument name instrument_name instr_name
Instrument serial number: instr_serialno Instrument manufacturer instrument_manufacturer instr_manufacturer
Sensor type: sensor_type Instrument model instrument_model instr_model
Analytical laboratory code: ana_lab_code Instrument serial number instrument_serial_number instr_serialno
Analytical measurement technique: ana_technique Sensor type sensor_type sensor_type
Analytical instrument name: ana_instr_name Analytical laboratory code analytical_laboratory_code ana_lab_code
Analytical instrument manufacturer: ana_instr_manufacturer Analytical measurement technique analytical_measurement_technique ana_technique
Analytical instrument model: ana_instr_model Analytical instrument name analytical_instrument_name ana_instr_name
Analytical instrument serial number: ana_instr_serialno Analytical instrument manufacturer analytical_instrument_manufacturer ana_instr_manufacturer
Ext. lab. code: ext_lab Analytical instrument model analytical_instrument_model ana_instr_model
Method ref: method Analytical instrument serial number analytical_instrument_serial_number ana_instr_serialno
Standard method: std_method Ext. lab. code ext_lab_code ext_lab
Calibration scale: cal_scale Method ref method_ref method
Calibration standard ID: cal_std_id Standard method standard_method std_method
Secondary standard ID: sec_std_id Calibration scale calibration_scale cal_scale
Inlet type: inlet_type Calibration standard ID calibration_standard_id cal_std_id
Inlet description: inlet_desc Secondary standard ID secondary_standard_id sec_std_id
Inlet tube material: inlet_tube_material Inlet type inlet_type inlet_type
Inlet tube outer diameter: inlet_tube_outerD Inlet description inlet_description inlet_desc
Inlet tube inner diameter: inlet_tube_innerD Inlet tube material inlet_tube_material inlet_tube_material
Inlet tube length: inlet_tube_length Inlet tube outer diameter inlet_tube_outer_diameter inlet_tube_outerD
Time from entry inlet line to entry of converter: time_inlet_to_converter Inlet tube inner diameter inlet_tube_inner_diameter inlet_tube_innerD
Duration of stay in converter or bypass line: time_converter_or_bypass_line Inlet tube length inlet_tube_length inlet_tube_length
Duration of stay in converter: time_stay_converter Time from entry inlet line to entry of converter time_from_entry_inlet_line_to_entry_of_converter time_inlet_to_converter
Converter temperature: converter_temp Duration of stay in converter or bypass line duration_of_stay_in_converter_or_bypass_line time_converter_or_bypass_line
Maintenance description: maintenance_desc Duration of stay in converter duration_of_stay_in_converter time_stay_converter
Flow rate: flow_rate Converter temperature converter_temperature converter_temp
Filter face velocity: filter_face_velocity Maintenance description maintenance_description maintenance_desc
Exposed filter area: filter_area Flow rate flow_rate flow_rate
Filter description: filter_descr Filter face velocity filter_face_velocity filter_face_velocity
Medium: medium Exposed filter area exposed_filter_area filter_area
Coating/Solution: coating_solution Filter description filter_description filter_descr
Filter prefiring: filter_prefiring Medium medium medium
Filter conditioning: filter_conditioning Coating/Solution coating_solution coating_solution
Filter type: filter_type Filter prefiring filter_prefiring filter_prefiring
Sample preparation: sample_prep Filter conditioning filter_conditioning filter_conditioning
Blank correction: blank_corr Filter type filter_type filter_type
Artifact correction: artifact_corr Sample preparation sample_preparation sample_prep
Artifact correction description: artifact_corr_desc Blank correction blank_correction blank_corr
Charring correction: charring_corr Artifact correction artifact_correction artifact_corr
Ozone correction: ozone_corr Artifact correction description artifact_correction_description artifact_corr_desc
Water vapor correction: watervapor_corr Charring correction charring_correction charring_corr
Zero/span check type: zero_span_type Ozone correction ozone_correction ozone_corr
Zero/span check interval: zero_span_interval Water vapor correction water_vapor_correction watervapor_corr
Humidity/temperature control: hum_temp_ctrl Zero/span check type zero_span_check_type zero_span_type
Humidity/temperature control description: hum_temp_ctrl_desc Zero/span check interval zero_span_check_interval zero_span_interval
Volume std. temperature: vol_std_temp Humidity/temperature control humidity_temperaure_control hum_temp_ctrl
Volume std. pressure: vol_std_pressure Humidity/temperature control description humidity_temperaure_control_description hum_temp_ctrl_desc
Detection limit: detection_limit Volume std. temperature volume_std_temperature vol_std_temp
Detection limit expl.: detection_limit_desc Volume std. pressure volume_std_pressure vol_std_pressure
Upper range limit: upper_range_limit Detection limit detection_limit detection_limit
Measurement uncertainty: uncertainty Detection limit expl. detection_limit_expl detection_limit_desc
Measurement uncertainty expl.: uncertainty_desc Upper range limit upper_range_limit upper_range_limit
Zero/negative values code: zero_negative Measurement uncertainty measurement_uncertainty uncertainty
Zero/negative values: zero_negative_desc Measurement uncertainty expl. measurement_uncertainty_expl uncertainty_desc
Absorption cross section: abs_cross_section Zero/negative values code zero_negative_values_code zero_negative
Mass absorption cross section: mass_abs_cross_section Zero/negative values zero_negative_values zero_negative_desc
Multi-scattering correction factor: multi_scattering_corr_fact Absorption cross section absorption_cross_section abs_cross_section
Maximum attenuation: max_attenuation Mass absorption cross section mass_absorption_cross_section mass_abs_cross_section
Leakage factor zeta: leakage_factor_zeta Multi-scattering correction factor multi_scattering_correction_factor multi_scattering_corr_fact
Compensation threshold attenuation 1: comp_thresh_atten1 Maximum attenuation maximum_attenuation max_attenuation
Compensation threshold attenuation 2: comp_thresh_atten2 Leakage factor zeta leakage_factor_zeta leakage_factor_zeta
Compensation parameter k min: comp_param_kmin Compensation threshold attenuation 1 compensation_threshold_attenuation_1 comp_thresh_atten1
Compensation parameter k max: comp_param_kmax Compensation threshold attenuation 2 compensation_threshold_attenuation_2 comp_thresh_atten2
Coincidence correction: coincidence_corr Compensation parameter k min compensation_parameter_k_min comp_param_kmin
Charge type: charge_type Compensation parameter k max compensation_parameter_k_max comp_param_kmax
Inlet diffusion loss data: inlet_diffusion_loss_data Coincidence correction coincidence_correction coincidence_corr
CPC default pulse width: cpc_default_pulse_width Charge type charge_type charge_type
QA measure ID: qm_id Inlet diffusion loss data inlet_diffusion_loss_data inlet_diffusion_loss_data
QA measure description: qm_desc CPC default pulse width cpc_default_pulse_width cpc_default_pulse_width
QA date: qa_date QA measure ID qa_measure_id qm_id
QA outcome: qa_outcome QA measure description qa_measure_description qm_desc
QA bias: qa_bias QA date qa_date qa_date
QA variability: qa_variability QA outcome qa_outcome qa_outcome
QA document name: qa_doc_name QA bias qa_bias qa_bias
QA document date: qa_doc_date QA variability qa_variability qa_variability
QA document URL: qa_doc_url QA document name qa_document_name qa_doc_name
Originator: originator QA document date qa_document_date qa_doc_date
Submitter: submitter QA document URL qa_document_url qa_doc_url
Acknowledgement: acknowledgements Originator originator originator
Comment: comment Submitter submitter submitter
Acknowledgement acknowledgement acknowledgements
Comment comment comment
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## List only metadata which are mandatory when writing files ## List only metadata which are mandatory when writing files
Here we restrict to _global metadata_ (```main```) which are _mandatory when writing_ (```& 2```) a file: Here we restrict to _global metadata_ (```main```) which are _mandatory when writing_ (```& 2```) a file:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
for elem in [x for x in meta.metadata if x['main'] & 2]: for elem in [x for x in meta.metadata if x['main'] & 2]:
if not 'renamed_tag' in elem: if not 'renamed_tag' in elem:
print("{:45s} {}".format(elem['tag']+":", elem['key'])) print("{:45s} {}".format(elem['tag']+":", elem['key']))
``` ```
%% Output %% Output
Data definition: datadef Data definition: datadef
Set type code: type Set type code: type
Timezone: timezone Timezone: timezone
File name: filename File name: filename
File creation: creation_time File creation: creation_time
Startdate: startdate Startdate: startdate
Revision date: revdate Revision date: revdate
Data level: datalevel Data level: datalevel
Period code: period Period code: period
Resolution code: resolution Resolution code: resolution
Station code: station_code Station code: station_code
Platform code: platform_code Platform code: platform_code
Regime: regime Regime: regime
Component: comp_name Component: comp_name
Matrix: matrix Matrix: matrix
Laboratory code: lab_code Laboratory code: lab_code
Instrument type: instr_type Instrument type: instr_type
Instrument name: instr_name Instrument name: instr_name
Method ref: method Method ref: method
Originator: originator Originator: originator
Submitter: submitter Submitter: submitter
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
#### specifications for cardinality #### specifications for cardinality
- main: specifies the cardinality of the element in the main header (NNCOM lines) - main: specifies the cardinality of the element in the main header (NNCOM lines)
Bitfield: Bitfield:
0 not allowed 0 not allowed
1 allowed but not mandatory 1 allowed but not mandatory
2 mandatory on export 2 mandatory on export
4 mandatory on import 4 mandatory on import
8 critical on import (if missing, exit after reading header) 8 critical on import (if missing, exit after reading header)
- vname: specifies the cardinality of the element in the vname line. - vname: specifies the cardinality of the element in the vname line.
Bitfield: Bitfield:
0 not allowed 0 not allowed
1 allowed but not mandatory 1 allowed but not mandatory
2 mandatory on export (not used) 2 mandatory on export (not used)
4 mandatory on import 4 mandatory on import
8 critical on import (if missing, exit after reading header) 8 critical on import (if missing, exit after reading header)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Finding the key to a tag ## Finding the key to a tag
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
If you see a metadata element in the NASA Ames file and want to access it after reading the file using the ebas.io module, you need to find the element's *key* when knowing the *tag*: If you see a metadata element in the NASA Ames file and want to access it after reading the file using the ebas.io module, you need to find the element's *key* when knowing the *tag*:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
meta.metadata_tags['Instrument manufacturer']['key'] meta.metadata_tags['Instrument manufacturer']['key']
``` ```
%% Output %% Output
'instr_manufacturer' 'instr_manufacturer'
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
meta.metadata_tags['Orig. time res.']['key'] meta.metadata_tags['Orig. time res.']['key']
``` ```
%% Output %% Output
'rescode_sample' 'rescode_sample'
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Finding the tag to a key ## Finding the tag to a key
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
The other way around, you might want to find the *tag* to a known *key*: The other way around, you might want to find the *tag* to a known *key*:
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
meta.metadata_keys['instr_manufacturer']['tag'] meta.metadata_keys['instr_manufacturer']['tag']
``` ```
%% Output %% Output
'Instrument manufacturer' 'Instrument manufacturer'
%% Cell type:code id: tags:
``` python
```
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment