Raven

Open source airquality system.
Main purpose is to simplify delivery of dataflows to EEA.

PLEASE NOTE

This branch (master) is under development and will frequently change.
Please use the v1.0 tag to install Raven.

Features

• Data import with csv files via GUI or API
• View and compare data module
• GUI for editing the following meta data:
  • Responsible authorities
  • Networks
  • Stations
  • Sampling Points
  • Process
  • Samples
  • Observation Capabilities
  • Assessment Regimes
  • Attainments
  • Exceedance Description
• Scaling, including calculate data (needed for NO-NO2-NOx)
• Validation module
• Verification module
• XML-generation via download or API for the following dataflows:
  • information on zones and agglomerations (data flow B)
  • information assessment methods (data flow D)
  • information assessment regimes (data flow C)
  • information on attainment of environmental compliance (data flow G)
  • primary validated data (data flow E1a)
  • primary Up-To-Date data (data flow E2a) NB! Only available through API
• Users/groups read/write access on network level

Requirements

The following is required to install and run the application:

• Postgres 11+ (https://www.postgresql.org/download/) (v9.5+ for upserts, v11+ for window functions)
• Postgis extension (https://postgis.net/install/)
• Python 3 (https://www.python.org/downloads/) (tested with v3.7)
• Yarn or NPM (if using Yarn you need to install Node.js first)
• WSGI complient server
• See requirements.txt for external python libraries used

Configuration

Add a file in the web folder called config.ini.
Look at config.example.ini to see what needs to be inside.

Database

Setup postgres database with the postgis extension.
Create new database. Connect to new created database. Import the schema.sql file from the db_scripts folder

Running the app

Install all required python libraries with:

pip install -r requirements.txt

Navigate to the web/client folder and build the gui with the command:

yarn install
yarn build

Run the python server with the the following command:
Linux

export FLASK_APP=web
flask run

Windows

$env:FLASK_APP = "web" 
flask run

```With cmd
FLASK_APP=web
flask run

See https://flask.palletsprojects.com/en/1.1.x/quickstart/ for more information.

For detailed installation steps see: "Steps to install on Windows.md" file

Development

Make sure you have all javascript packages installed by running yarn in the web/client folder.
Run yarn build once so the python server has the required files.
Start the web server as described in "running the app"
Start the client side server by running the command yarn serve

Linux web server

On the webserver we have to start with a configuration for apache2. Write a custom configuration file, /etc/apache2/sites-available/raven.conf.

The ServerName settings in raven.conf has to be registered as a CNAME, pointing to the A-record for the webserver, in the DNS-servers.

Make ssl-certs for https://someurl.com, and put the key and crt file into /etc/apache2/ssl.

Make a DocumentRoot directory under /var/www/html, call it raven. Give it sufficient permissions for apache2 to be able to read it.

Copy all of the files in the web folder to there.

Make a raven.wsgi file, to start the application. Here we source the virtual environment and starts the application.

Next is to make a virtual environment for python, and then installing all the modules from requirement.txt from that. psycopg2 is dependent on libpq-dev and python-dev.

IIS

Please follow this article to install the application on IIS: http://netdot.co/2015/03/09/flask-on-iis/

Make sure the Python folder has IIS_IUSRS rights.
Install all dependencies from requirements.txt globally or with virtualenv.
To do this globally run the following line in the python script folder:
pip install -r requirements.txt
It is highly recommended that you set up your server with https

Importing data

Importing data is done by uploading responisble_authorities, networks, stations, sampling_points, processes, samples, observing_capabilities and observations as csv files. See csv_examples.
Use Curl, Postman or similar tools for this.
Set Content-Type to "multipart/form-data" and use the form name "csv".
Example uploading networks with curl

curl -X POST -u usr:psw --form "csv=@path/to/networks.csv" https://someurl.com/imports/networks

If you upload a content with id's that exists, it will be updated.

There are dependencies between the csv files, so the import order is important.
Start with settings, users, responisble_authorities, networks, stations, sampling_points, processes, samples, observing_capabilities and observations

Security

This application uses basic authentication.
The default user:password combination is admin:admin
It is recommended to change the default password immediately.
You can do this with the user management module on Raven web site.

Settings

The namespace needs to be set for dataflow D and E.
Example of namespace is "NO.NILU.AQD"
To change the namespace with json payload, do the following:

Update namespace:

PAYLOAD {"namespace":"some_namespace"}
POST imports/settings

Responisble authorities

Create a csv file with the following headers. All headers must be present:

id,name,organisation,address,locator,postcode,email,phone,website,is_responsible_reporter

id: A unique self created id. (Required)
name: The name of the reporter (Required)
organisation: The name of the reporting organisation (Required)
address: The address of the reporting organisation (Required)
locator: City of the reporting organisation (Required)
postcode: Post code of the reporting organisation (Required)
email: Email to the reporter (Required)
phone: Phone number to the reporter (Required)
website: Url to the reporting organisation (Required)
is_responsible_reporter: One reporter needs to be set to true. This reporter will be used for AQD_ReportingHeader. (Required)

Insert or update one or multiple reporters:

POST imports/responsible_authorities

Get all imported reporters:

GET imports/responsible_authorities

Delete a specific reporter:

DELETE imports/responsible_authorities/{id}

Networks

Create a csv file with the following headers. All headers must be present:

id,name,responsible_authority_id,organisational,begin_position,end_position,aggregation_timezone

id: The EEA code for your network with prefix. eg NET_NO019A (Required)
name: A name for the network. Can be anything you want (Required)
responsible_authority_id: The id to the responsible authority. This id must already exist in the system (Required)
organisational: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/organisationallevel (Required)
begin_position: When the network started measuring. ie 2000-01-01T00:00:00+01:00 (Required)
end_position: When the network stopped measuring. ie 2000-01-01T00:00:00+01:00 (Optional)
aggregation_timezone: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/timezone (Required)

Insert or update one or multiple networks:

POST imports/networks

Get all imported networks:

GET imports/networks

Delete a specific network:

DELETE imports/networks/{id}

Stations

Create a csv file with the following headers. All headers must be present:

id,name,begin_position,end_position,network_id,municipality,eoi_code,national_station_code,latitude,longitude,epsg,altitude,mobile,area_classification,distance_junction,traffic_volume,heavy_duty_fraction,street_width,height_facades

id: The EEA code for your station with prefix. eg STA_NO0083A (Required)
name: A name for the station. Can be anything you want (Required)
begin_position: When the station started measuring. ie 2000-01-01T00:00:00+01:00 (Required)
end_position: When the station stopped measuring. ie 2000-01-01T00:00:00+01:00 (Optional)
network_id: The id to the network. This id must already exist in the system (Required)
municipality: The name of the municipality where the station is located. (Optional)
eoi_code: The EEA code for your station. (Required)
national_station_code: The national/local id for the station. (Required)
latitude: The latitude of the station. (Required)
longitude: The longitude of the station. (Required)
epsg: The epsg integer code of the station. eg 4326 (Required)
altitude: The altitude of the station. (Required)
mobile: Is the station mobile. True or false (Required)
area_classification: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/areaclassification (Required)
distance_junction: (Optional)
traffic_volume: (Optional)
heavy_duty_fraction: (Optional)
street_width: (Optional)
height_facades: (Optional)

Insert or update one or multiple stations:

POST imports/stations

Get all imported stations:

GET imports/stations

Delete a specific station:

DELETE imports/stations/{id}

Sampling Points

Create a csv file with the following headers. All headers must be present:

id,assessment_type,station_id,station_classification,main_emission_sources,traffic_emissions,heating_emissions,industrial_emissions,distance_source,begin_position,end_position,mobile

id: A unique id with prefix. Recommended format is stationid_componentid_uniqueid. ie SPO_NO0083A_9_1023 (Required)
assessment_type: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/assessmenttype (Required)
station_id: The id to the station. This id must already exist in the system (Required)
station_classification: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/stationclassification (Required)
main_emission_sources: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/emissionsource (Optional)
traffic_emissions: The traffic emissions. (Optional)
heating_emissions: The heating emissions. (Optional)
industrial_emissions: The industrial emissions. (Optional)
distance_source: The distance source. (Optional)
begin_position: When the station started measuring for pollutant. ie 2000-01-01T00:00:00+01:00 (Required)
end_position: When the station stopped measuring for pollutant. ie 2000-01-01T00:00:00+01:00 (Optional)
mobile: Is the sampling point mobile. True or false (Required)

Insert or update one or multiple sampling points:

POST imports/sampling_points

Get all imported sampling points:

GET imports/sampling_points

Delete a specific sampling point:

DELETE imports/sampling_points/{id}

Processes

Create a csv file with the following headers. All headers must be present:

id,responsible_authority_id,measurement_type,measurement_method,other_measurement_method,sampling_method,other_sampling_method,analytical_tech,other_analytical_tech,sampling_equipment,other_sampling_equipment,measurement_equipment,other_measurement_equipment,equiv_demonstration,equiv_demonstration_report,detection_limit,detection_limit_uom,uncertainty_estimate,documentation,qa_report,duration_number,duration_unit,cadence_number,cadence_unit

id: A unique id with prefix. Recommended format is stationid_componentid_uniqueid. ie SPP_NO0083A_9_1023 (Required)
responsible_authority_id: The id to the responsible authority. This id must already exist in the system (Required)
measurement_type: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/measurementtype (Required)
measurement_method: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/measurementmethod (Optional)
other_measurement_method: A text if measurement_method is set to other. (Optional)
sampling_method: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/samplingmethod (Optional)
other_sampling_method: A text if sampling_method is set to other. (Optional)
analytical_tech: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/analyticaltechnique (Optional)
other_analytical_tech: A text if analytical_tech is set to other. (Optional)
sampling_equipment: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/samplingequipment (Optional)
other_sampling_equipment: A text if sampling_equipment is set to other. (Optional)
measurement_equipment: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/measurementequipment (Optional)
other_measurement_equipment: A text if measurement_equipment is set to other. (Optional)
equiv_demonstration: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/equivalencedemonstrated (Optional)
equiv_demonstration_report: A text if equiv_demonstration is set to other. (Optional)
detection_limit: The detection limit (Optional)
detection_limit_uom: An EEA url. See http://dd.eionet.europa.eu/vocabulary/uom/concentration (Optional)
uncertainty_estimate: The uncertainty estimate. (Optional)
documentation: The documentation. (Optional)
qa_report: The QA report. (Optional)
duration_number: The duration number. (Required)
duration_unit: An EEA url. See http://dd.eionet.europa.eu/vocabulary/uom/time (Required)
cadence_number: The cadence number. (Required)
cadence_unit: An EEA url. See http://dd.eionet.europa.eu/vocabulary/uom/time (Required)

Insert or update one or multiple processes:

POST imports/processes

Get all imported processes:

GET imports/processes

Delete a specific process:

DELETE imports/processes/{id}

Samples

Create a csv file with the following headers. All headers must be present:

id,inlet_height,building_distance,kerb_distance

id: A unique id. Recommended format is stationid_componentid_uniqueid_counter. ie NO0083A_9_1023_1 (Required)
inlet_height: The inlet height in meters. (Required)
building_distance: The building distance in meters. (Optional)
kerb_distance: The kerb distance in meters. (Optional)

Insert or update one or multiple samples:

POST imports/samples

Get all imported samples:

GET imports/samples

Delete a specific sample:

DELETE imports/samples/{id}

Observing capabilities

Create a csv file with the following headers. All headers must be present:

id,begin_position,end_position,pollutant,sampling_point_id,sample_id,process_id,concentration,timestep

id: A unique id. Recommended format is stationid_componentid_uniqueid_counter. ie NO0083A_9_1023_1 (Required)
begin_position: When the stations started measuring with capability. ie 2000-01-01T00:00:00+01:00 (Required)
end_position: When the stations stopped measuring with capability. ie 2000-01-01T00:00:00+01:00 (Optional)
pollutant: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/pollutantn (Required)
sampling_point_id: The id to the sampling point. This id must already exist in the system (Required)
sample_id: The id to the sample. This id must already exist in the system (Required)
concentration: An EEA url. See http://dd.eionet.europa.eu/vocabulary/uom/concentration (Required)
timestep: An EEA url. See http://dd.eionet.europa.eu/vocabulary/aq/primaryObservation (Required)

Insert or update one or multiple observing capabilities:

POST imports/observing_capabilities

Get all imported observing capabilities:

GET imports/observing_capabilities

Delete a specific observing capability:

DELETE imports/observing_capabilities/{id}

Observations

This should be imported on a regular basis, ie hourly.
Create a csv file with the following headers. All headers must be present:

sampling_point_id,begin_position,end_position,value,verification_flag,validation_flag

sampling_point_id: The id to the sampling point. This id must already exist in the system (Required)
begin_position: When the measurement started. ie 2000-01-01T00:00:00+01:00 (Required)
end_position: When the measurement stopped. ie 2000-01-01T01:00:00+01:00 (Required)
value: The measured value. (Required)
verification_flag: The integer verification flag. See notation in http://dd.eionet.europa.eu/vocabulary/aq/observationverification (Required)
validation_flag: The integer validation flag. See notation in http://dd.eionet.europa.eu/vocabulary/aq/observationvalidity (Required)

Insert or update one or multiple observations:

POST imports/observations

Get all imported observations:

GET imports/observations

Delete a specific observation:

DELETE imports/observations/{sampling_point_id}/{end_position}

Dataflows

Dataflows do not require authentication.

D

To get dataflow D set the year and the timezone in the url. ie year=2000&tz=1

GET dataflows/d ?year={year}&tz={timezone:int}

E1A

To get dataflow E1A set the year and the timezone in the url. ie year=2000&tz=1

GET dataflows/e1a ?year={year}&tz={timezone:int}

E2A

To get dataflow E2A set the last_request in the url. ie 2000-01-01T00:00:00
This will get all imported/updated data after the set date.

GET dataflows/e2a ?last_request={YYYY-MM-DDTHH:mm:ss}