Skip to content
Snippets Groups Projects
Commit 82295313 authored by Test Paul's avatar Test Paul
Browse files

ADDED smps genfile scripts, kindly provided by Athina Kalogridi and 

Stergios Vratolis from NCSR Demokritos
parent 837be2b4
No related branches found
Tags 3.0.13
No related merge requests found
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 1048 additions and 0 deletions
Examples/ebas_genfile_templates/ebas_genfile_smps/LEV1/Input_metadata_lev1.csv 0 → 100755
+ 12
− 0
View file @ 82295313
comp_name,pressure,temperature,relative_humidity,number of size bins,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution
comp_name,pressure,temperature,relative_humidity,number of size bins,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution,particle_number_size_distribution
unit,hPa,K,%,no unit,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3,1/cm3
matrix,instrument,instrument,instrument,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10,pm10
title,p_sys,T_sys,RH_sys,nBn,bin_01,bin_02,bin_03,bin_04,bin_05,bin_06,bin_07,bin_08,bin_09,bin_10,bin_11,bin_12,bin_13,bin_14,bin_15,bin_16,bin_17,bin_18,bin_19,bin_20,bin_21,bin_22,bin_23,bin_24,bin_25,bin_26,bin_27,bin_28,bin_29,bin_30,bin_31,bin_32,bin_33,bin_34,bin_35,bin_36,bin_37,bin_38,bin_39,bin_40,bin_41,bin_42,bin_43,bin_44,bin_45,bin_46,bin_47,bin_48,bin_49,bin_50,bin_51,bin_52,bin_53,bin_54,bin_55,bin_56,bin_57,bin_58,bin_59,bin_60,bin_61,bin_62,bin_63,bin_64,bin_65,bin_66,bin_67,bin_68,bin_69,bin_70,bin_71
statistics,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Uncertainty value,,,,,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10,10
Uncertainty unit,,,,,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%,%
Characteristictype,Location,Location,Location,None,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D,D
Value,instrument internal,instrument internal,instrument internal,None,10.068 nm,10.354 nm,10.95 nm,11.58 nm,12.248 nm,12.954 nm,13.702 nm,14.494 nm,15.332 nm,16.219 nm,17.158 nm,18.153 nm,19.206 nm,20.322 nm,21.504 nm,22.756 nm,24.083 nm,25.489 nm,26.98 nm,28.56 nm,30.236 nm,32.013 nm,33.889 nm,35.9 nm,38.024 nm,40.278 nm,42.673 nm,45.217 nm,47.92 nm,50.794 nm,53.851 nm,57.103 nm,60.565 nm,64.251 nm,68.18 nm,72.368 nm,76.912 nm,81.604 nm,86.699 nm,92.145 nm,97.971 nm,104.209 nm,110.895 nm,118.068 nm,125.77 nm,134.051 nm,142.964 nm,152.569 nm,162.934 nm,174.134 nm,186.254 nm,199.39 nm,213.65 nm,229.155 nm,246.042 nm,264.467 nm,284.605 nm,306.654 nm,330.837 nm,357.407 nm,387.031 nm,418.874 nm,454.447 nm,493.766 nm,537.277 nm,585.478 nm,638.923 nm,698.229 nm,764.082 nm,837.249 nm,876.858 nm
instrumenttype,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps,smps
0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75
Examples/ebas_genfile_templates/ebas_genfile_smps/LEV1/Level1_final.txt 0 → 100755
+ 289
− 0
View file @ 82295313
This diff is collapsed.
Examples/ebas_genfile_templates/ebas_genfile_smps/LEV1/ebas_genfile_smps_lev1.py 0 → 100755
+ 353
− 0
View file @ 82295313
#!/usr/bin/env python
# coding=utf-8
"""
$Id: ebas_genfile.py 1431 2016-10-27 14:25:12Z pe $
Example for creating an EBAS_1.1 NasaAmes datafile.
"""
from ebas.io.file import nasa_ames
from nilutility.datatypes import DataObject
from ebas.domain.time_period import estimate_period_code, \
estimate_resolution_code, estimate_sample_duration_code
import datetime
from ebas.io.ebasmetadata import DatasetCharacteristicList
import pandas as pd
import numpy
def testnan(val):
if val * 0. == 0:
return val
else:
return None
#INPUT DATA-VALUES
f = r'Level1_final.txt'
first_column=1 #(first column with variable to process)
last_column=75 # (last column to process)
index_flag_col=76 #(nb column with flag data)
tresol=5 #sample duration
df=pd.read_csv(f, delimiter=',', header=0, engine='python') #Specify nb_lines of header, delimiter etc
df = df.where((pd.notnull(df)), None) #Replace nan values with None
start_times_df=pd.to_datetime(df['DateTime'],format='%Y-%m-%d %H:%M:%S') #Specify format of datetime column to read. The column in called by the header name
names=list(df.columns.values)
numb_col=df.shape[1]
#INPUT METADATA_VARIABLES !!!! METADATA and INPUT FILES MUST HAVE THE SAME NUMBER OF COLUMNS AND IN THE SAME ORDER AND POSITION!!
f_metadata=r'Input_metadata_lev1.csv'
df_metadata=pd.read_csv(f_metadata, delimiter=',', header=0)
#OUTPUT FOLDER
destdir_out='./'
__version__ = '1.00.00'
def set_fileglobal_metadata(nas):
"""
Set file global metadata for the EbasNasaAmes file object
Parameters:
nas EbasNasaAmes file object
Returns:
None
"""
# All times reported to EBAS need to be in UTC!
# Setting the timezone here explicitly should remind you to check your data
nas.metadata.timezone = 'UTC'
nas.metadata.revdate = datetime.datetime(2017, 07, 20, 13, 23, 00)
nas.metadata.revision = '1.1a'
nas.metadata.revdesc = \
'initial revision, NRT SMPS_control_lev0 lev0 2 lev1 v.0.0_2'
nas.metadata.type = 'TU'
# Revision information
nas.metadata.startdate = datetime.datetime(2016, 01, 01, 00, 00, 00) #I am not sure about the time format
nas.metadata.datalevel = '1'
nas.metadata.period = '1y'
nas.metadata.resolution = '5mn'
nas.metadata.duration = '5mn'
nas.metadata.rescode_sample = '5mn'
# Data Originator Organisation
nas.metadata.org = DataObject(
OR_CODE='GR05L',
OR_NAME='NCSR Demokritos',
OR_ACRONYM='DEM', OR_UNIT='URL',
OR_ADDR_LINE1='Ag.Paraskevi', OR_ADDR_LINE2=None,
OR_ADDR_ZIP='15310', OR_ADDR_CITY='Athens', OR_ADDR_COUNTRY='Greece')
# Projects the data are associated to
nas.metadata.projects = ['GAW-WDCA', 'ACTRIS']
# Station metadata
nas.metadata.station_code = 'GR0101R'
nas.metadata.platform_code = 'GR0101S'
nas.metadata.station_name = u'Helmos Mountain'
nas.metadata.station_wdca_id = 'GAWAGR__HAC'
nas.metadata.station_gaw_id = 'HAC'
nas.metadata.station_gaw_name = u'Helmos Mountain'
nas.metadata.station_airs_id = '' # N/A
nas.metadata.station_other_ids = ''
# nas.metadata.station_state_code = # N/A
nas.metadata.station_state_code = ''
nas.metadata.station_landuse = 'gravel and Stone'
nas.metadata.station_setting = 'Mountain'
nas.metadata.station_gaw_type = 'R'
nas.metadata.station_wmo_region = 6
nas.metadata.station_latitude = 37.984265
nas.metadata.station_longitude = 22.196262
nas.metadata.station_altitude = 2340.0
nas.metadata.mea_latitude = 37.984265
nas.metadata.mea_longitute = 22.196262
nas.metadata.mea_altitude = 2340.0
nas.metadata.mea_height = 6
# More file global metadata, but those can be overridden per variable
# See set_variables for examples
nas.metadata.comp_name = 'particle_number_size_distribution'
nas.metadata.unit = '1/cm3'
nas.metadata.matrix = 'pm10'
nas.metadata.lab_code = 'GR05L'
nas.metadata.instr_type = 'smps' #MUST BE in LOWER CASE LETTERS!! OTHERWISE GIVES ERRORS
nas.metadata.instr_name = 'SMPS-TROPOS'
nas.metadata.instr_manufacturer ='TROPOS-TSI'
nas.metadata.instr_model = 'TROPOS-SMPS'
nas.metadata.instr_serialno = 'SMPS=TROPOS'
nas.metadata.method = 'GR05L__NRT_SMPS_lev1'
nas.std_method = 'SOP=Wiedensohler2012'
nas.metadata.inlet_type = 'Impactor direct'
nas.metadata.inlet_desc = 'PM10 impactor'
nas.metadata.hum_temp_ctrl = 'Nafion dryer'
nas.metadata.hum_temp_ctrl_desc = 'Humidity/temperature control description: sample dried to below 40% RH with nafion dryer'
nas.metadata.vol_std_temp = 273.15
nas.metadata.vol_std_pressure = 1013.25
nas.metadata.detection_limit=(0, '1/cm3')
nas.metadata.detection_limit_desc=' Determined only by instrument counting statistics, no detection limit flag used'
nas.metadata.uncertainty_desc='uncertainty range between instruments in intercomparison by Wiedensohler et al. 2012. (AMT)'
nas.metadata.zero_negative='Zero values possible'
nas.metadata.zero_negative_desc='Zero values may appear due to statistical variations at very low concentrations'
# Data Originators (PIs)
nas.metadata.originator = []
nas.metadata.originator.append(
DataObject(
PS_LAST_NAME='Eleftheriadis', PS_FIRST_NAME='Konstantinos', PS_EMAIL='elefther@ipta.demokritos.gr',
PS_ORG_NAME='NCSR Demokritos',
PS_ORG_ACR='DEM', PS_ORG_UNIT='ERL',
PS_ADDR_LINE1='Ag.Paraskevi', PS_ADDR_LINE2=None,
PS_ADDR_ZIP='15310', PS_ADDR_CITY='Athens',
PS_ADDR_COUNTRY='Greece'))
nas.metadata.originator.append(
DataObject(
PS_LAST_NAME=u'Vratolis', PS_FIRST_NAME='Stergios', PS_EMAIL='vratolis@ipta.demokritos.gr',
PS_ORG_NAME='NCSR Demokritos',
PS_ORG_ACR='DEM', PS_ORG_UNIT='ERL',
PS_ADDR_LINE1='Ag.Paraskevi', PS_ADDR_LINE2=None,
PS_ADDR_ZIP='15310', PS_ADDR_CITY='Athens',
PS_ADDR_COUNTRY='Greece'))
# Data Submitters (contact for data technical issues)
nas.metadata.submitter = []
nas.metadata.submitter.append(
DataObject(
PS_LAST_NAME=u'Gini', PS_FIRST_NAME='Maria', PS_EMAIL='gini@ipta.demokritos.gr',
PS_ORG_NAME='NCSR Demokritos',
PS_ORG_ACR='DEM', PS_ORG_UNIT='ERL',
PS_ADDR_LINE1='Ag.Paraskevi', PS_ADDR_LINE2=None,
PS_ADDR_ZIP='15310', PS_ADDR_CITY='Athens',
PS_ADDR_COUNTRY='Greece'))
#nas.metadata.qm_id=''
#nas.metadata.qm_doc_date=''
#nas.metadata.qm_doc_url=''
nas.metadata.acknowledgements='Request acknowledgment details from data originator'
nas.metadata.comment='none'
def set_time_axes(nas):
"""
Set the time axes and related metadata for the EbasNasaAmes file object.
Parameters:
nas EbasNasaAmes file object
Returns:
None
"""
# define start and end times for all samples
start_times_list=start_times_df.tolist()
start_times_dt_list=[start_times_list[i].to_pydatetime()for i in xrange(len(start_times_list))]
end_times_df=start_times_df+datetime.timedelta(minutes=tresol)
end_times_list=end_times_df.tolist()
end_times_dt_list=[end_times_list[i].to_pydatetime()for i in xrange(len(start_times_list))]
nas.sample_times=zip(start_times_dt_list,end_times_dt_list)
#nas.sample_times = \
# [(datetime.datetime(2008, 1, 1, 0, 0), datetime.datetime(2008, 1, 1, 1, 0)),
# (datetime.datetime(2008, 1, 1, 1, 0), datetime.datetime(2008, 1, 1, 2, 0)),
# (datetime.datetime(2008, 1, 1, 2, 0), datetime.datetime(2008, 1, 1, 3, 0)),
# (datetime.datetime(2008, 1, 1, 3, 0), datetime.datetime(2008, 1, 1, 4, 0)),
# (datetime.datetime(2008, 1, 1, 4, 0), datetime.datetime(2008, 1, 1, 5, 0)),
# (datetime.datetime(2008, 1, 1, 5, 0), datetime.datetime(2008, 1, 1, 6, 0)),
# (datetime.datetime(2008, 1, 1, 6, 0), datetime.datetime(2008, 1, 1, 7, 0)),
# (datetime.datetime(2008, 1, 1, 7, 0), datetime.datetime(2008, 1, 1, 8, 0))]
#
# Generate metadata that are related to the time axes:
#
# period code is an estimate of the current submissions period, so it should
# always be calculated from the actual time axes, like this:
nas.metadata.period = estimate_period_code(nas.sample_times[0][0],
nas.sample_times[-1][1])
# Sample duration can be set automatically
nas.metadata.duration = estimate_sample_duration_code(nas.sample_times)
# or set it hardcoded:
# nas.metadata.duration = '3mo'
# Resolution code can be set automatically
# But be aware that resolution code is an identifying metadata element.
# That means, several submissions of data (multiple years) will
# only be stored as the same dataset if the resolution code is the same
# for all submissions!
# That might be a problem for time series with varying resolution code
# (sometimes 2 months, sometimes 3 months, sometimes 9 weeks, ...). You
# might consider using a fixed resolution code for those time series.
# Automatic calculation (will work from ebas.io V.3.0.7):
nas.metadata.resolution = estimate_resolution_code(nas.sample_times)
# or set it hardcoded:
# nas.metadata.resolution = '3mo'
# It's a good practice to use Jan 1st of the year of the first sample
# endtime as the file reference date (zero point of time axes).
nas.metadata.reference_date = \
datetime.datetime(nas.sample_times[0][1].year, 1, 1)
def set_variables(nas):
"""
Set metadata and data for all variables for the EbasNasaAmes file object.
Parameters:
nas EbasNasaAmes file object
Returns:
None
"""
#x=14
x=first_column
while x < last_column+1:
col=df.take([x],axis=1)
values=col.values.T.tolist()[0]
values=numpy.array(values, dtype=numpy.float)
values=[ round(elem,6) for elem in values]
values=map(testnan, values)
col_metadata=df_metadata.take([x],axis=1)
col_flag=df.take([index_flag_col],axis=1).astype(int)
flags=col_flag.values.tolist()
metadata = DataObject()
metadata.comp_name = col_metadata[0:1].values.T.tolist()[0][0]
#Unit metadata
metadata.unit = col_metadata[1:2].values.T.tolist()[0][0]
if metadata.unit!=metadata.unit:
metadata.unit = None
else:
metadata.unit = metadata.unit
metadata.matrix=col_metadata[2:3].values.T.tolist()[0][0]
metadata.title=col_metadata[3:4].values.T.tolist()[0][0]
#Statistics metadata
metadata.statistics=col_metadata[4:5].values.T.tolist()[0][0]
if metadata.statistics!=metadata.statistics:
metadata.statistics = None
else:
metadata.statistics = metadata.statistics
#Uncertainity metadata
uncertainty_value=col_metadata[5:6].values.T.tolist()[0][0]
uncertainty_unit=col_metadata[6:7].values.T.tolist()[0][0]
if uncertainty_value!=uncertainty_value:
metadata.uncertainty = None
else:
metadata.uncertainty = (int(uncertainty_value), uncertainty_unit)
#Characteristics
if col_metadata[7:8].values.T.tolist()[0][0]=='None':
nas.variables.append(DataObject(values_=values, flags=flags, flagcol=True,
metadata=metadata))
else:
metadata.characteristics = DatasetCharacteristicList()
Characteristic_type=col_metadata[7:8].values.T.tolist()[0][0]
Value=col_metadata[8:9].values.T.tolist()[0][0]
Instrument_type=col_metadata[9:10].values.T.tolist()[0][0]
metadata.characteristics.add_parse(Characteristic_type, Value, Instrument_type, metadata.comp_name)
nas.variables.append(DataObject(values_=values, flags=flags, flagcol=True,
metadata=metadata))
#nas.variables.append(DataObject(values_=values, flags=flags, flagcol=True,
#metadata=metadata))
#
x+=1
def ebas_genfile():
"""
Main program for ebas_flatcsv
Created for lexical scoping.
Parameters:
None
Returns:
none
"""
# Create an EbasNasaAmes file object
nas = nasa_ames.EbasNasaAmes()
# Set file global metadata
set_fileglobal_metadata(nas)
# Set the time axes and related metadata
set_time_axes(nas)
# Set metadata and data for all variables
set_variables(nas)
# write the file:
nas.write(createfiles=True,destdir=destdir_out)
# createfiles=True
# Actually creates output files, else the output would go to STDOUT.
# You can also specify:
# destdir='path/to/directory'
# Specify a specific relative or absolute path to a directory the
# files should be written to
# flags=FLAGS_COMPRESS
# Compresses the file size by reducing flag columns.
# Flag columns will be less explicit and thus less intuitive for
# humans to read.
# flags=FLAGS_ALL
# Always generate one flag column per variable. Very intuitive to
# read, but increases filesize.
# The default for flags is: Generate one flag column per file if the
# flags are the same for all variables in the file. Else generate one
# flag column per variable.
# This is a trade-off between the advantages and disadvantages of the
# above mentioned approaches.
ebas_genfile()
Examples/ebas_genfile_templates/ebas_genfile_smps/LEV2/ebas_genfile_smps_lev2b.py 0 → 100755
+ 353
− 0
View file @ 82295313
#!/usr/bin/env python
# coding=utf-8
"""
$Id: ebas_genfile.py 1431 2016-10-27 14:25:12Z pe $
Example for creating an EBAS_1.1 NasaAmes datafile.
"""
from ebas.io.file import nasa_ames
from nilutility.datatypes import DataObject
from ebas.domain.time_period import estimate_period_code, \
estimate_resolution_code, estimate_sample_duration_code
import datetime
from ebas.io.ebasmetadata import DatasetCharacteristicList
import pandas as pd
import numpy
def testnan(val):
if val * 0. == 0:
return val
else:
return None
#INPUT DATA-VALUES
f = r'InputData_for_lev2.txt'
first_column=1 #(first column with variable to process)
last_column=217 # (last column to process)
index_flag_col=218 #(nb column with flag data)
tresol=60 #sample duration
df=pd.read_csv(f, delimiter=',', header=0, engine='python') #Specify nb_lines of header, delimiter etc
df = df.where((pd.notnull(df)), None) #Replace nan values with None
start_times_df=pd.to_datetime(df['DateTime'],format='%Y-%m-%d %H:%M:%S') #Specify format of datetime column to read. The column in called by the header name
names=list(df.columns.values)
numb_col=df.shape[1]
#INPUT METADATA_VARIABLES !!!! METADATA and INPUT FILES MUST HAVE THE SAME NUMBER OF COLUMNS AND IN THE SAME ORDER AND POSITION!!
f_metadata=r'metadata_level2.csv'
df_metadata=pd.read_csv(f_metadata, delimiter=',', header=0)
#OUTPUT FOLDER
destdir_out='.'
__version__ = '1.00.00'
def set_fileglobal_metadata(nas):
"""
Set file global metadata for the EbasNasaAmes file object
Parameters:
nas EbasNasaAmes file object
Returns:
None
"""
# All times reported to EBAS need to be in UTC!
# Setting the timezone here explicitly should remind you to check your data
nas.metadata.timezone = 'UTC'
nas.metadata.revdate = datetime.datetime(2017, 07, 20, 13, 23, 00)
nas.metadata.revision = '1.1a'
nas.metadata.revdesc = \
'initiol revision, SMPS_control_lev0 v.0.0_2'
nas.metadata.type = 'TU'
# Revision information
nas.metadata.startdate = datetime.datetime(2016, 01, 01, 00, 00, 00) #I am not sure about the time format
nas.metadata.datalevel = '2'
nas.metadata.period = '1y'
nas.metadata.resolution = '5mn'
nas.metadata.duration = '5mn'
nas.metadata.rescode_sample = '5mn'
# Data Originator Organisation
nas.metadata.org = DataObject(
OR_CODE='GR05L',
OR_NAME='NCSR Demokritos',
OR_ACRONYM='DEM', OR_UNIT='URL',
OR_ADDR_LINE1='Ag.Paraskevi', OR_ADDR_LINE2=None,
OR_ADDR_ZIP='15310', OR_ADDR_CITY='Athens', OR_ADDR_COUNTRY='Greece')
# Projects the data are associated to
nas.metadata.projects = ['GAW-WDCA', 'ACTRIS']
# Station metadata
nas.metadata.station_code = 'GR0101R'
nas.metadata.platform_code = 'GR0101S'
nas.metadata.station_name = u'Helmos Mountain'
nas.metadata.station_wdca_id = 'GAWAGR__HAC'
nas.metadata.station_gaw_id = 'HAC'
nas.metadata.station_gaw_name = u'Helmos Mountain'
nas.metadata.station_airs_id = '' # N/A
nas.metadata.station_other_ids = ''
# nas.metadata.station_state_code = # N/A
nas.metadata.station_state_code = ''
nas.metadata.station_landuse = 'gravel and stone'
nas.metadata.station_setting = 'Mountain'
nas.metadata.station_gaw_type = 'R'
nas.metadata.station_wmo_region = 6
nas.metadata.station_latitude = 37.984265
nas.metadata.station_longitude = 22.196262
nas.metadata.station_altitude = 2340.0
nas.metadata.mea_latitude = 37.984265
nas.metadata.mea_longitute = 22.196262
nas.metadata.mea_altitude = 2340.0
nas.metadata.mea_height = 6
# More file global metadata, but those can be overridden per variable
# See set_variables for examples
nas.metadata.comp_name = 'particle_number_size_distribution'
nas.metadata.unit = '1/cm3'
nas.metadata.matrix = 'pm10'
nas.metadata.lab_code = 'GR05L'
nas.metadata.instr_type = 'smps'
nas.metadata.instr_name = 'SMPS-TROPOS'
nas.metadata.instr_manufacturer ='TROPOS-TSI'
nas.metadata.instr_model = 'TROPOS-SMPS'
nas.metadata.instr_serialno = 'SMPS=TROPOS'
nas.metadata.method = 'GR05L__NRT_SMPS_lev2'
nas.std_method = 'SOP=Wiedensohler2012'
nas.metadata.inlet_type = 'Impactor direct'
nas.metadata.inlet_desc = 'PM10 impactor'
nas.metadata.hum_temp_ctrl = 'Nafion dryer'
nas.metadata.hum_temp_ctrl_desc = 'Humidity/temperature control description: sample dried to below 40% RH with nafion dryer'
nas.metadata.vol_std_temp = 273.15
nas.metadata.vol_std_pressure = 1013.25
nas.metadata.detection_limit=(0, '1/cm3')
nas.metadata.detection_limit_desc=' Determined only by instrument counting statistics, no detection limit flag used'
nas.metadata.uncertainty_desc='uncertainty range between instruments in intercomparison by Wiedensohler et al. 2012. (AMT)'
nas.metadata.zero_negative='Zero values possible'
nas.metadata.zero_negative_desc='Zero values may appear due to statistical variations at very low concentrations'
# Data Originators (PIs)
nas.metadata.originator = []
nas.metadata.originator.append(
DataObject(
PS_LAST_NAME='Eleftheriadis', PS_FIRST_NAME='Konstantinos', PS_EMAIL='elefther@ipta.demokritos.gr',
PS_ORG_NAME='NCSR Demokritos',
PS_ORG_ACR='DEM', PS_ORG_UNIT='ERL',
PS_ADDR_LINE1='Ag.Paraskevi', PS_ADDR_LINE2=None,
PS_ADDR_ZIP='15310', PS_ADDR_CITY='Athens',
PS_ADDR_COUNTRY='Greece'))
nas.metadata.originator.append(
DataObject(
PS_LAST_NAME=u'Vratolis', PS_FIRST_NAME='Stergios', PS_EMAIL='vratolis@ipta.demokritos.gr',
PS_ORG_NAME='NCSR Demokritos',
PS_ORG_ACR='DEM', PS_ORG_UNIT='ERL',
PS_ADDR_LINE1='Ag.Paraskevi', PS_ADDR_LINE2=None,
PS_ADDR_ZIP='15310', PS_ADDR_CITY='Athens',
PS_ADDR_COUNTRY='Greece'))
# Data Submitters (contact for data technical issues)
nas.metadata.submitter = []
nas.metadata.submitter.append(
DataObject(
PS_LAST_NAME=u'Gini', PS_FIRST_NAME='Maria', PS_EMAIL='gini@ipta.demokritos.gr',
PS_ORG_NAME='NCSR Demokritos',
PS_ORG_ACR='DEM', PS_ORG_UNIT='ERL',
PS_ADDR_LINE1='Ag.Paraskevi', PS_ADDR_LINE2=None,
PS_ADDR_ZIP='15310', PS_ADDR_CITY='Athens',
PS_ADDR_COUNTRY='Greece'))
#nas.metadata.qm_id=''
#nas.metadata.qm_doc_date=''
#nas.metadata.qm_doc_url=''
nas.metadata.acknowledgements='Request acknowledgment details from data originator'
nas.metadata.comment='none'
def set_time_axes(nas):
"""
Set the time axes and related metadata for the EbasNasaAmes file object.
Parameters:
nas EbasNasaAmes file object
Returns:
None
"""
# define start and end times for all samples
start_times_list=start_times_df.tolist()
start_times_dt_list=[start_times_list[i].to_pydatetime()for i in xrange(len(start_times_list))]
end_times_df=start_times_df+datetime.timedelta(minutes=tresol)
end_times_list=end_times_df.tolist()
end_times_dt_list=[end_times_list[i].to_pydatetime()for i in xrange(len(start_times_list))]
nas.sample_times=zip(start_times_dt_list,end_times_dt_list)
#nas.sample_times = \
# [(datetime.datetime(2008, 1, 1, 0, 0), datetime.datetime(2008, 1, 1, 1, 0)),
# (datetime.datetime(2008, 1, 1, 1, 0), datetime.datetime(2008, 1, 1, 2, 0)),
# (datetime.datetime(2008, 1, 1, 2, 0), datetime.datetime(2008, 1, 1, 3, 0)),
# (datetime.datetime(2008, 1, 1, 3, 0), datetime.datetime(2008, 1, 1, 4, 0)),
# (datetime.datetime(2008, 1, 1, 4, 0), datetime.datetime(2008, 1, 1, 5, 0)),
# (datetime.datetime(2008, 1, 1, 5, 0), datetime.datetime(2008, 1, 1, 6, 0)),
# (datetime.datetime(2008, 1, 1, 6, 0), datetime.datetime(2008, 1, 1, 7, 0)),
# (datetime.datetime(2008, 1, 1, 7, 0), datetime.datetime(2008, 1, 1, 8, 0))]
#
# Generate metadata that are related to the time axes:
#
# period code is an estimate of the current submissions period, so it should
# always be calculated from the actual time axes, like this:
nas.metadata.period = estimate_period_code(nas.sample_times[0][0],
nas.sample_times[-1][1])
# Sample duration can be set automatically
nas.metadata.duration = estimate_sample_duration_code(nas.sample_times)
# or set it hardcoded:
# nas.metadata.duration = '3mo'
# Resolution code can be set automatically
# But be aware that resolution code is an identifying metadata element.
# That means, several submissions of data (multiple years) will
# only be stored as the same dataset if the resolution code is the same
# for all submissions!
# That might be a problem for time series with varying resolution code
# (sometimes 2 months, sometimes 3 months, sometimes 9 weeks, ...). You
# might consider using a fixed resolution code for those time series.
# Automatic calculation (will work from ebas.io V.3.0.7):
nas.metadata.resolution = estimate_resolution_code(nas.sample_times)
# or set it hardcoded:
# nas.metadata.resolution = '3mo'
# It's a good practice to use Jan 1st of the year of the first sample
# endtime as the file reference date (zero point of time axes).
nas.metadata.reference_date = \
datetime.datetime(nas.sample_times[0][1].year, 1, 1)
def set_variables(nas):
"""
Set metadata and data for all variables for the EbasNasaAmes file object.
Parameters:
nas EbasNasaAmes file object
Returns:
None
"""
x=first_column
while x < last_column+1:
col=df.take([x],axis=1)
values=col.values.T.tolist()[0]
values=numpy.array(values, dtype=numpy.float)
values=[ round(elem,6) for elem in values]
values=map(testnan, values)
col_metadata=df_metadata.take([x],axis=1)
col_flag=df.take([index_flag_col],axis=1).astype(int)
flags=col_flag.values.tolist()
metadata = DataObject()
metadata.comp_name = col_metadata[0:1].values.T.tolist()[0][0]
#Unit metadata
metadata.unit = col_metadata[1:2].values.T.tolist()[0][0]
if metadata.unit!=metadata.unit:
metadata.unit = None
else:
metadata.unit = metadata.unit
metadata.matrix=col_metadata[2:3].values.T.tolist()[0][0]
metadata.title=col_metadata[3:4].values.T.tolist()[0][0]
#Statistics metadata
metadata.statistics=col_metadata[4:5].values.T.tolist()[0][0]
if metadata.statistics!=metadata.statistics:
metadata.statistics = None
else:
metadata.statistics = metadata.statistics
#Uncertainity metadata
uncertainty_value=col_metadata[5:6].values.T.tolist()[0][0]
uncertainty_unit=col_metadata[6:7].values.T.tolist()[0][0]
if uncertainty_value!=uncertainty_value:
metadata.uncertainty = None
else:
metadata.uncertainty = (int(uncertainty_value), uncertainty_unit)
#Characteristics
if col_metadata[7:8].values.T.tolist()[0][0]=='None':
nas.variables.append(DataObject(values_=values, flags=flags, flagcol=True,
metadata=metadata))
else:
metadata.characteristics = DatasetCharacteristicList()
Characteristic_type=col_metadata[7:8].values.T.tolist()[0][0]
Value=col_metadata[8:9].values.T.tolist()[0][0]
Instrument_type=col_metadata[9:10].values.T.tolist()[0][0]
metadata.characteristics.add_parse(Characteristic_type, Value, Instrument_type, metadata.comp_name)
nas.variables.append(DataObject(values_=values, flags=flags, flagcol=True,
metadata=metadata))
#nas.variables.append(DataObject(values_=values, flags=flags, flagcol=True,
#metadata=metadata))
#
x+=1
def ebas_genfile():
"""
Main program for ebas_flatcsv
Created for lexical scoping.
Parameters:
None
Returns:
none
"""
# Create an EbasNasaAmes file object
nas = nasa_ames.EbasNasaAmes()
# Set file global metadata
set_fileglobal_metadata(nas)
# Set the time axes and related metadata
set_time_axes(nas)
# Set metadata and data for all variables
set_variables(nas)
# write the file:
nas.write(createfiles=True,destdir=destdir_out)
# createfiles=True
# Actually creates output files, else the output would go to STDOUT.
# You can also specify:
# destdir='path/to/directory'
# Specify a specific relative or absolute path to a directory the
# files should be written to
# flags=FLAGS_COMPRESS
# Compresses the file size by reducing flag columns.
# Flag columns will be less explicit and thus less intuitive for
# humans to read.
# flags=FLAGS_ALL
# Always generate one flag column per variable. Very intuitive to
# read, but increases filesize.
# The default for flags is: Generate one flag column per file if the
# flags are the same for all variables in the file. Else generate one
# flag column per variable.
# This is a trade-off between the advantages and disadvantages of the
# above mentioned approaches.
ebas_genfile()
Examples/ebas_genfile_templates/ebas_genfile_smps/LEV2/metadata_level2.csv 0 → 100755
+ 13
− 0
View file @ 82295313
This diff is collapsed.
Examples/ebas_genfile_templates/ebas_genfile_smps/README.md 0 → 100644
+ 3
− 0
View file @ 82295313
# Generate EBAS smps files
Those scripts have been kindly provided by Athina Kalogridi and Stergios Vratolis from NCSR Demokritos, Greece.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment