Scripts for the COMTESSA project

This repository contains the analysis scripts for the Comtessa project. You can find more information on the project under https://comtessa-turbulence.net.

The contained scripts analyze SO2 camera images from the Comtessa dispersion experiments. The images are post-processed using the pyplis library and many scripts build on top of pyplis. The post-processed images are used for analysis of puff dispersion.

Setup

Install pyplis using the imglist_flex branch on my or jonas' account (This branch contains some recent methods which are needed for the scripts...). For that you need to install pyplis from the console. See the instruction at the pyplis page.

I made a copy of my environment in environment.yml. This file contains the exact version number of all libraries installed with conda or pip in my environment. You probably don't need all of them. But if there are problems with some libraries, try to install the same version as in this file (or create your environment directly from this file). I used python 3.6.8 but newer version should work in principle.

Note: The pyplis library saw major API changed since the beginning of this repository and it generally does not support the comtessa file format although changes have been made to the base classes. Scripts from the 2017-campaign have to be run on the master_2017 branch of my pyplis fork.

I use to run all scripts from inside spyder to be able to change parameters during executions and inspect the results. Many scripts contain a sys.exit() at the end of the main part of the script followed by code which can be used to inspect different aspects of the results.

Main analysis scripts

Comtessa_prepare_data.py

• defines all data read-in and image processing
• method prepare_imglist loads fully processed SO2 camera images for other scripts in form of pyplis.ImgListLayered. This object contains overview tables of parameters (ImgListLayered.metaData) and post-processing (background, calibration, etc) can be flexibly modified during later analysis.
• when executed as main, allows for image inspection, creation of videos or saving of series of processed images

Comtessa_detection_imagewise.py

• based on prepare_imglist
• detection and tracking of puffs

Comtessa_dispersion.py

• Retrieval of 3D trajectories and statistical analysis

Extra scripts producing auxiliary/intermediate data

Average_darks_clears.py

• script to produce averaged dark and background images from the raw images
• select the time and images to be averaged inside the script

Create_sky_masks

• creates a binary mask based on a single background image (as created by previous script)

Setup_2018.py

• defines class Setup() and few geometric functions
• when executed as main(), an instance of the Setup class is initialized with data from a *landmarks.txt file
• ToDo: could be divided into a module and a script to keep it more tidy

Distortion_2019.py

• calibrates the cameras based on the checkerboard measurements in 2019. This script varies depending on the image acquisition.

Calibration

• currently no separate scripts
• at the end of Comtessa_prepare_data.py, there are some lines which can be used for manual calibration

Comtessa_DOAS_2018.py

• calibration based on additional DOAS measurements
• SO2 SCDs need to be provided (DOAS evaluation has to been done separately e.g. using the script for DOASIS)
• still under test

Modules

constants.py

• defines universal or camera specific constants used throughout all scripts

pyplis_helper.py

• convenience functions e.g. plotting and animations

pyplis_puffs.py

• based on pyplis imagelists
• detection and tracking of puffs within SO2 camera images
• saves a text files per puff

pyplis_dispersion.py

• based on the text files from pyplis_puffs
• Definition of the classes Turbulence, Puff and Puff_ensemble
• Handling of 3D merging
• Handling of statistical analysis

m_cv.py

• handling of computer vision methods
• camera calibration and distortion correction

Auxiliary File structure

In the directory DIR_AUX several files should be kept (or produced and saved there). They are loaded automatically from this directory

1. Clears --> DATE_STR --> *_clear.fts

2. Darks --> DATE_STR --> *_dark.fts

3. Masks --> DATE_STR --> *_mask.fts

4. Landmarks --> *landmarks.txt Contains image coordinates of landmarks (release, meteorological towers, etc). One file per day, manually created, release is crucial for tracking and pose calculation.

5. Puff_ids *puff_ids.csv Contains puff ids (1 for first puff of the day) and release time based on UV1. Can contain additionally time of first detection in images of other cameras to assess time synchronization One file per day, manually or automatically (regular releases) created

6. plume_background_model.txt Contains information of background model as required by pyplis One line per camera and date ToDo: should be extended to handle changes in FOV more flexible

7. Distortion --> Distortion_*.txt Contains camera calibration and distortion parameters

8. Syncronisation --> *sync_UV1.txt Contains time offsets compared to UV1 Manually created

9. *mass_Norbert.txt and *mass.txt Contains puff mass from notes and from analysis; could be merged into the *puff_ids.csv

10. Wind (optional for plotting)

11. Setup --> setup_*_xyz.csv contains positions and poses of cameras based on landmarks

12. Setup --> setup_*_xyz_corr.csv contains positions and poses of cameras based on distortion-corrected landmarks

Common parameters

Some common parameters which usually have to be defined in the beginning of the scripts

Typical workflow

1. Creation of dark images and background images. Script: Average_darks_clears.py

2. Creation of images masks. Script: Create_sky_masks.py

3. Manual insertion of background correction parameters (e.g. scaling rectangular) in plume_background_model.txt

4. Inspection of image data. Script: Comtessa_prepare_data.py

   • Improvement of background parameters
   • Identification of release start time and manual insertion in *_puff_ids.csv
   • Based on release start time, estimate time offsets and insert in *_sync_UV1.txt (can be set to 0 for preliminary analysis)
   • Identification of landmarks (can also be done on raw images) and manual insertion in *_landmarks.txt

5. Detection and tracking of puffs in the images of a single camera. Script: Comtessa_detection_imagewise.py

6. Creation of setup file (one per date). Script: Setup_2018.py (can be used for 2019)

7. Dispersion analysis. Script: Comtessa_dispersion.py

Extras:

• calibration is done with tailored scripts.
• Many small scripts which plot specific things or analyses some aspects like noise or time syncronisation

Further implementation work

on pyplis side

• time dependent mask
• automatic cell calibration for comtessa file format (merge ImgListFlex into ImgList)
• include distortion inside pyplis
• median during readin

on analysis side

• include 6 focal lengths in Setup
• interpolated background