RAVEN Administration
RAVEN is an open source web based Air Quality data validation and e-reporting system, with the aim to control the flow, metadata inventory, the quality of the monitoring data and producing the XML files required for the Air Quality B-G, except from D1b (Information on the assessment methods - for models and objective estimation) and E1b (Information on primary validated assessment data – modelled).
The system is managed and developed by NILU
, with support from 4sFera
, on the behalf of the European Environmental Agency
.
Requirements
Python version 3.10.8
Node version 18.12.1
Postgres version 12+
Postgis extension
NPM
Clone repository from git
git clone https://git.nilu.no/raven/raven-administration
Setup the database
Run db scripts to create the database
- Create a postgres database, ie
ravendb
- Install Postgis (https://postgis.net/install/) and enable it on the database
CREATE EXTENSION postgis;
- Run the
sql\schema.sql
script - Run the
sql\data.sql
script - Run the
sql\pre_aggregates.sql
script - Run the
sql\use_in_public_api.sql
script - Run the
sql\meteo.sql
script
Set environment varables
Create a file called .env
in the root
folder and set the variables
API_PORT=5000
CLIENT_PORT=80
DB_URI = postgresql://dbuser:password@host:5432/database
JWT_ACCESS_TOKEN_EXPIRES_SECONDS = 3600
JWT_SECRET_KEY = make-up-a-secure-key
CONTAINER_NAME_API = raven-api
CONTAINER_NAME_CLIENT = raven-client
Hint: Use host.docker.internal if database is local. Ip 172.17.0.1 for Linux
Docker
Make sure you have Docker engine installed. (https://www.docker.com/)
Then build and run the docker-compose
file.
# build and run
docker-compose build
docker-compose up
# Access RAVEN at: http://localhost
Development
Create a virtual environment and activate it
# create
python -m venv venv
# activate on Windows
.\venv\Scripts\activate
# activate on Mac and Linux
source venv/bin/activate
In the api
folder install the required python packages
pip install -r requirements.txt
In the client
folder install the required js packages
npm install
Run Raven
# from inside the api folder start backend server
# on Windows
$env:FLASK_APP = "app.py"
flask run
# on Mac and Linux
export FLASK_APP=app.py
flask run
# from inside the client folder start the frontend
npm run dev
Pre aggregations
Aggregating data can be triggered manually in the raven app.
However, it is recommended to set up a schedule to trigger the aggregation on a daily basis.
The sql command that needs to run is select raven_refresh_aggregates()
One ways is to use the postgres extension pgagent
. This will enable a schedular within postgres.
Another way is to set up a cron
job in linux or a schtasks
in windows
The cron
folder has a python script that can be used together with a schedular.
It does require psycopg2-binary==2.9.5
, so make sure this is available for the scheduled task.
# Example of how to set up a scheduled task in Windows
schtasks /create /SC DAILY /TN raven-refresh-views /TR "<path_to_python> <path_to_raven>\cron\refresh_views.py" /ST 00:10