Commit 2d2723f6 authored by Ignacio Pisso's avatar Ignacio Pisso

delete preproc

parent 60517b36
# README #
This directory contains programs to generate FLEXPART WINDFIELDS from ECMWF ERA-Interim data.
+ `python contains python scripts to extract and generate WINDFIELDS
+ `src contains Fortran source code to create CONVERT2 executable (used by the python scripts)
+ `grib_templates contains GRIB-API templates for running CONVERT2 program.
See first the installation instruction in `src and then install python scripts on your system.
Please report any problems.
This diff is collapsed.
#
# (C) Copyright 2014 UIO.
#
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
#
#
# Creation: July 2014 - Anne Fouilloux - University of Oslo
#
#
from gribapi import *
import traceback
import sys,os
##############################################################
# GRIB utilities
##############################################################
class GribTools:
'class for GRIB API with new methods'
def __init__(self,filename):
self.filename=filename
# get keyvalues for a given list of keynames
# a where statment can be given (list of key and list of values)
def getkeys(self,keynames,wherekeynames=[],wherekeyvalues=[]):
fileid=open(self.filename,'r')
return_list=[]
while 1:
gid_in = grib_new_from_file(fileid)
if gid_in is None: break
select=True
i=0
if len(wherekeynames) != len(wherekeyvalues): raise Exception("Give a value for each keyname!")
for wherekey in wherekeynames:
if not grib_is_defined(gid_in, wherekey): raise Exception("where Key was not defined")
select=select and (str(wherekeyvalues[i])==str(grib_get(gid_in, wherekey)))
i=i+1
if select:
llist = []
for key in keynames:
llist.extend([str(grib_get(gid_in, key))])
return_list.append(llist)
grib_release(gid_in)
fileid.close()
return return_list
# set keyvalues for a given list of keynames
# a where statment can be given (list of key and list of values)
# an input file must be given as an input for reading grib messages
# note that by default all messages are written out
# if you want to get only those meeting the where statement, use
# strict=true
def setkeys(self,fromfile,keynames,keyvalues, wherekeynames=[],wherekeyvalues=[], strict=False, filemode='w'):
fout=open(self.filename,filemode)
fin=open(fromfile)
while 1:
gid_in = grib_new_from_file(fin)
if gid_in is None: break
select=True
i=0
if len(wherekeynames) != len(wherekeyvalues): raise Exception("Give a value for each keyname!")
for wherekey in wherekeynames:
if not grib_is_defined(gid_in, wherekey): raise Exception("where Key was not defined")
select=select and (str(wherekeyvalues[i])==str(grib_get(gid_in, wherekey)))
i=i+1
if select:
i=0
for key in keynames:
grib_set(gid_in, key, keyvalues[i])
i=i+1
if strict:
if select:
grib_write(gid_in,fout)
else:
grib_write(gid_in,fout)
grib_release(gid_in)
fin.close()
fout.close()
# Add the content of a grib file but only messages
# corresponding to keys/values
# if selectWhere is False select fields that are different from keynames/keyvalues
def copy(self,filename_in, selectWhere=True, keynames=[], keyvalues=[],filemode='w'):
fin=open(filename_in)
fout=open(self.filename,filemode)
while 1:
gid_in = grib_new_from_file(fin)
if gid_in is None: break
select=True
i=0
if len(keynames) != len(keyvalues): raise Exception("Give a value for each keyname!")
for key in keynames:
if not grib_is_defined(gid_in, key): raise Exception("Key was not defined")
if selectWhere:
select=select and (str(keyvalues[i])==str(grib_get(gid_in, key)))
else:
select=select and (str(keyvalues[i])!=str(grib_get(gid_in, key)))
i=i+1
if select:
grib_write(gid_in,fout)
grib_release(gid_in)
fin.close()
fout.close()
# Create index from a list of files if it does not exist or read it
def index(self,index_keys=["mars"], index_file = "my.idx"):
print "index to be done"
self.iid = None
if (os.path.exists(index_file)):
self.iid = grib_index_read(index_file)
print "Use existing index file: %s "%(index_file)
else:
for file in self.filename:
print "Inputfile: %s "%(file)
if self.iid == None:
self.iid = grib_index_new_from_file(file,index_keys)
else:
grib_index_add_file(self.iid,file)
if self.iid != None:
grib_index_write(self.iid,index_file)
return self.iid
# README #
This documentation shows how to use these python scripts to extract ECMWF ERA-Interim data and generate WINDFIELDS for running FLEXPART.
### Overview ###
To run FLEXPART with ECMWF ERA-Interim data, you first need to retrieve ECMWF ERA-Interim GRIB fields and generate FLEXPART WINDFIELDS to run FLEXPART.
The two main programs are called respectively `getEIdata.py` and `prepareFLEXPART.py`.
To get the usage of these programs, use `-h` option:
`getEIdata.py -h`
Optional arguments are mentionned in squared brackets.
* `getEIdata.py`
This program allows you to download ECMWF ERA-Interim data from ECMWF using [ecmwfapi](https://software.ecmwf.int/wiki/display/WEBAPI/ECMWF+Web+API+Home).
It requires `ecmwfapi` python library (see Requirements below). Check with your local IT group as it may be already available.
{{{
Usage: getEIdata.py --start_date=YYYYMMDD [--end_date=YYYYMMDD] [--times=tt1/tt2/tt3] [--levels=nlevels]
[--area=north/west/south/east] [--outputdir=output_directory]
Options:
-h, --help show this help message and exit
--start_date=start_date
start date YYYYMMDD
--end_date=end_date end_date YYYYMMDD
--times=times times such as 00/12
--levels=levels number of vertical levels
--area=area area defined as north/west/south/east with default
90.0/-179.0/-90.0/180.0
--outputdir=outputdir
root directory for storing output files
}}}
* `prepareFLEXPART.py`
This program allow you to generate FLEXPART WINDFIELDS (inputs for FLEXPART). It requires python interface to grib_api and `CONVERT2` program (located in `src` directory with instruction on how to compile it). You also need to provide a namelist for CONVERT2 (see test_1).
`
Usage: prepareFLEXPART.py --start_date=YYYYMMDD [--end_date=YYYYMMDD] [--namelist=namelist_for_convert] [--inputdir=input_root_directory] [--outputdir=output_directory]
Options:
-h, --help show this help message and exit
--start_date=start_date
start date YYYYMMDD
--end_date=end_date end_date YYYYMMDD
--namelist=namelist namelist used for converting
--inputdir=inputdir root directory for reading input files
--outputdir=outputdir
root directory for storing output files
`
### Requirements ###
| Python Support |
|----------------------:|
| python | [http://www.python.org](http://www.python.org) | We have use [python Anaconda](https://store.continuum.io/cshop/anaconda/) for our testings
| python-numpy | [http://www.numpy.org/](http://www.numpy.org/) | Not necessary if you have installed python Anaconda
| ecmwfapi | [https://software.ecmwf.int/wiki/display/WEBAPI/ECMWF+Web+API+Home](https://software.ecmwf.int/wiki/display/WEBAPI/ECMWF+Web+API+Home) | You also need to install your API key (as explained in the documentation)
| Utilities |
|----------------------:|
| grib-api | [https://software.ecmwf.int/wiki/display/GRIB/Home](https://software.ecmwf.int/wiki/display/GRIB/Home) | Make sure you install GRIB-API with JPEG support and python GRIB-API.
|----------------------:|
| FLEXPART programs | to run these programs (prepareFLEXPART.py), you need to compile CONVERT2 program (located in preproc/src). See separate README file to get instructions on how to compile this code.
|----------------------:|
### Installation ###
* Environment
At UIO, Red Hat 6 Linux systems (64 bits) were used for testing. We use the [Module package](http://modules.sourceforge.net/) to set-up user environment.
* Getting the source code
In the directory of your choice:
`
git clone git@bitbucket.org:flexpart/flexpart.git
`
This command will create a subdirectory called flexpart: it contains the latest FLEXPART version.
Set then environment variable `FLEXPART_HOME`:
** Korn-shell or Bash users:
`
cd flexpart
export FLEXPART_HOME=$PWD
`
** C-shell users:
`
cd flexpart
setenv FLEXPART_HOME=$PWD
`
* Installation
Make sure your first generate `CONVERT2` program (see separate instructions in `preproc/src`).
Users need to be able to execute prepareFLEXPART.py and getEIdata.py so make sure they have the correct unix permissions:
`
cd preproc/python
chmod uog+rx getEIdata.py prepareFLEXPART.py
`
These two programs must be in the user PATH. At UIO this is done automatically when loading flexpart. If not, you would need to do the following:
** Korn-shell or Bash users:
`
export PATH=$FLEXPART_HOME/preproc/python:$PATH
`
** C-shell users:
`
setenv PATH $FLEXPART_HOME/preproc/python:$PATH
`
Where `$FLEXPART_HOME` is the directory where FLEXPART
* Testing your installation
First check that grib-api python interface is correctly installed on your platform:
`
python
>>> from gribapi import *
>>>
`
Use `CTRL-D` to quit python.
Then check that `ecmwfapi` is properly installed:
`
python
>>> from ecmwfapi import *
>>>
`
If the two previous tests were successful, you can run `tests/preproc` (See separate instructions in `tests/preproc`).
If any of these two tests fail, this probably means that either `ecmwfapi` or `grib-api` have not been installed properly.
Please report any problems.
### Installation FAQ ###
#!/usr/bin/env python
#
# (C) Copyright 2014 UIO.
#
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
#
# Creation: October 2014 - Anne Fouilloux - University of Oslo
#
import os
class UIOFiles:
'class to manipulate files'
def __init__(self,suffix):
# type of files to manipulate such as ['.grib', 'grb', 'grib1', 'grib2', 'grb1','grb2']
self.suffix=suffix
def listFiles(self,pathname):
''' list files (suffix previously given) within this directory. '''
# Get the absolute path of the pathname parameter
pathname = os.path.abspath(pathname)
# Get a list of files in pathname
filesInCurDir = os.listdir(pathname)
self.counter = 0
self.files = []
# Traverse through all files
for file in filesInCurDir:
curFile = os.path.join(pathname, file)
# Check if it's a normal file or directory
if os.path.isfile(curFile):
# Get the file extension
fileNoExt,curFileExtension = os.path.splitext(curFile)
# Check if the file has an extension of typical video files
if curFileExtension in self.suffix:
# We have got a file file! Increment the counter
self.counter += 1
# add this filename in the list
self.files.append(curFile)
else:
# We got a directory, enter into it for further processing
self.listFiles(curFile)
#!/usr/bin/env python
#
# (C) Copyright 2014 UIO.
#
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
#
# Creation: July 2014 - Anne Fouilloux - University of Oslo
#
# Get ERA-Interim GRIB fields from ECMWF to prepare WINDFIELDS for running
# FLEXPART examples.
#
# To run this example, you need to self-register at ECMWF data server and
# get an API key available from https://api.ecmwf.int/v1/key/
#
# Inputs parameters:
# start_date: starting date for retrieving ECMWF data
# end_date : end date for retrieving ECMWF data; set to start_date if not specified
# times : forecast or analysis time for MARS retrievals
# levels : number of levels; set to 60 by default (for ERA-interim)
# area : you can define your area with --area=north/west/south/east (values are in degrees)
# outputdir : location where you want to store ERA-Interim outputs
#
from ecmwfapi import ECMWFDataServer
import calendar
import shutil
import datetime
import time
import os
from string import strip
from optparse import OptionParser
from FlexpartTools import MARSretrieval, EIFlexpart, daterange, mkdir_p, silentremove, months_between
def main():
usage = """usage: %prog --start_date=YYYYMMDD [--end_date=YYYYMMDD] [--times=tt1/tt2/tt3] [--levels=nlevels]
[--area=north/west/south/east] [--outputdir=output_directory] """
parser = OptionParser(usage=usage)
parser.add_option("--start_date", dest="start_date",
help="start date YYYYMMDD", metavar="start_date" )
parser.add_option( "--end_date", dest="end_date",
help="end_date YYYYMMDD", metavar="end_date")
parser.add_option("--times", dest="times",
default="00/03/06/09/12/15/18/21", help="times such as 00/12", metavar="times")
parser.add_option("--levels", dest="levels",
default="60",help="number of vertical levels", metavar="levels")
parser.add_option("--area", dest="area",
default="90.0/-179.0/-90.0/180.0",help="area defined as north/west/south/east with default 90.0/-179.0/-90.0/180.0", metavar="area")
parser.add_option("--outputdir", dest="outputdir",
help="root directory for storing output files", metavar="outputdir")
(options, args) = parser.parse_args()
if not options.start_date:
parser.error("start date must be specified!")
else:
start_date=options.start_date
if not options.end_date:
end_date=start_date
else:
end_date=options.end_date
if not options.outputdir:
# if WORKDIR is defined, we will use it otherwise files
# will be stored in the current directory
outputdir=os.environ.get("WORKDIR",".")
else:
outputdir=options.outputdir
print "start date %s "%(start_date)
print "end date %s "%(end_date)
server = ECMWFDataServer()
# Retrieve ERA interim data for running flexpart
syear=int(start_date[:4])
smonth=int(start_date[4:6])
sday=int(start_date[6:])
start = datetime.date( year = syear, month = smonth, day = sday )
eyear=int(end_date[:4])
emonth=int(end_date[4:6])
eday=int(end_date[6:])
end = datetime.date( year = eyear, month = emonth, day = eday )
current_ym = ""
ir_date = start
retrieve="no"
for date in daterange( start, end ):
# if new year & month then we create a new directory to store output files
if date.strftime("%Y%m") != current_ym and current_ym != "":
retrieve="yes"
if date == end:
retrieve="yes"
if retrieve == "yes":
# we need to retrieve MARS data for this period (maximum one month)
flexpart = EIFlexpart()
dates= ir_date.strftime("%Y%m%d") + "/to/" + er_date.strftime("%Y%m%d")
current_outputdir = outputdir + "/" + ir_date.strftime("%Y") + '/' + ir_date.strftime("%m") + '/'
mkdir_p(current_outputdir)
print "retrieve " + dates + " in dir " + current_outputdir
flexpart.retrieve(server, dates, options.times, options.area, options.levels, current_outputdir)
ir_date = date
retrieve="no"
er_date = date
current_ym = date.strftime("%Y%m")
if __name__ == "__main__":
main()
#!/usr/bin/env python
#
# (C) Copyright 2014 UIO.
#
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
#
# Creation: October 2014 - Anne Fouilloux - University of Oslo
#
# Compute WINDFILEDS from ECMWF GRIB data
#
import calendar
import shutil
import datetime
import time
import os
from UIOTools import UIOFiles
from string import strip
from optparse import OptionParser
from GribTools import GribTools
from FlexpartTools import EIFlexpart, daterange
def main():
usage = """usage: %prog --start_date=YYYYMMDD [--end_date=YYYYMMDD] [--namelist=namelist_for_convert] [--inputdir=input_root_directory] [--outputdir=output_directory] """
parser = OptionParser(usage=usage)
parser.add_option("--start_date", dest="start_date",
help="start date YYYYMMDD", metavar="start_date" )
parser.add_option( "--end_date", dest="end_date",
help="end_date YYYYMMDD", metavar="end_date")
parser.add_option("--namelist", dest="namelist",
help="namelist used for converting", metavar="namelist")
parser.add_option("--inputdir", dest="inputdir",
help="root directory for reading input files", metavar="inputdir")
parser.add_option("--outputdir", dest="outputdir",
help="root directory for storing output files", metavar="outputdir")
(options, args) = parser.parse_args()
if not options.start_date:
parser.error("start date must be specified!")
else:
start_date=options.start_date
if not options.end_date:
end_date=start_date
else:
end_date=options.end_date
if not options.namelist:
namelist='fort.4'
else:
namelist=options.namelist
if not options.inputdir:
# if WORKDIR is defined, we will use it otherwise files
# will be stored in the current directory
inputdir=os.environ.get("WORKDIR",".")
else:
inputdir=options.inputdir
if not options.outputdir:
# if FLEXPART_WINDS is defined, we will use it otherwise files
# will be stored in the current directory
outputdur=os.environ.get("FLEXPART_WINDS",".")
else:
outputdir=options.outputdir
syear=int(start_date[:4])
smonth=int(start_date[4:6])
sday=int(start_date[6:])
start = datetime.date( year = syear, month = smonth, day = sday )
eyear=int(end_date[:4])
emonth=int(end_date[4:6])
eday=int(end_date[6:])
end = datetime.date( year = eyear, month = emonth, day = eday )
cyear = -1
cmont = -1
inputfiles=UIOFiles(['.grib', '.grb', '.grib1', '.grib2', '.grb1','.grb2'])
if (not os.path.exists('fort.4')):
fnamelist=open('fort.4','w')
shutil.copyfileobj(open(namelist,'r'), fnamelist)
fnamelist.close()
for date in daterange( start, end ):
# data retrieved by year/month
if cyear != date.year or cmonth != date.month:
print 'Prepare year : ' + str(date.year) + ' month : ', date.month
cyear = date.year
cmonth = date.month
# we will make the list of files from the root inputdir
if cmonth < 10:
inputfiles.listFiles(inputdir + '/'+str(cyear)+'/0'+str(date.month))
else:
inputfiles.listFiles(inputdir + '/'+str(cyear)+'/'+str(date.month))
flexpart = EIFlexpart()
flexpart.create(inputfiles, outputdir)
if __name__ == "__main__":
main()
# README #
This documentation shows how to compile `CONVERT2` program
### Overview ###
### Requirements ###
| Utilities |
|----------------------:|
| make | [http://www.gnu.org/software/make/](http://www.gnu.org/software/make/) |
| Third party libraries |
|----------------------:|
| EMOS | [https://software.ecmwf.int/wiki/display/EMOS/Emoslib](https://software.ecmwf.int/wiki/display/EMOS/Emoslib) | Make sure you install EMOS lib with 64 bits reals.
| grib-api | [https://software.ecmwf.int/wiki/display/GRIB/Home](https://software.ecmwf.int/wiki/display/GRIB/Home) | Make sure you install GRIB-API with JPEG support and python GRIB-API.
### Installation ###
* Environment
At UIO, Red Hat 6 Linux systems (64 bits) were used for testing. We use the [Module package](http://modules.sourceforge.net/) to set-up user environment.
* Getting the source code
If you still haven't got the source from bitbucket (Please note that it needs to be done once only).
In the directory of your choice:
`
git clone git@bitbucket.org:flexpart/flexpart.git
`
This command will create a subdirectory called flexpart: it contains the latest FLEXPART version.
Set then environment variable `FLEXPART_HOME`:
** Korn-shell or Bash users:
`
cd flexpart
export FLEXPART_HOME=$PWD
`
** C-shell users:
`
cd flexpart
setenv FLEXPART_HOME=$PWD
`
* Installation
There is a makefile example called `makefile` which you can use as a template. It uses GNU gfortran version 4.8.2. The version of your `gfortran`compiler can be tested with the following command:
`
gfortran --version
`
For using our `makefile`, you need to define the following environment variables:
Please note that these environment variables may be automatically defined on your system. Please check with your local IT administrator. For instance at UIO, these environment variables are defined by loading the corresponding modulefiles (`module load grib_api emos`).
+ `GRIB_API_FFLAGS`
+ `GRIB_API_LDFLAGS`
+ `EMOS_LDFLAGS`
For instance at UIO on sverdrup.uio.no:
`
> echo $GRIB_API_FFLAGS
-I/site/opt/grib_api/1.12.3_gnu/include
> echo $GRIB_API_LDFLAGS
-L/site/opt/grib_api/1.12.3_gnu/lib -lgrib_api_f90 -lgrib_api
> echo $EMOS_LDFLAGS
-L/site/opt/emos/000392_grib_api_1.12.3_gnu -lemos
`
If `EMOS` and `GRIB-API` have been installed properly and you have adjusted your makefile then you can compile `CONVERT2`:
`
cd preproc/src
gmake
`
If successful, a program called `CONVERT2` is created in the current directory. You now need to add this program in your `PATH`:
** Korn-shell or Bash users:
`
export PATH=$FLEXPART_HOME/preproc/src:$PATH
`
** C-shell users:
`
setenv PATH $FLEXPART_HOME/preproc/src:$PATH
`
Where `$FLEXPART_HOME` is the directory where FLEXPART
* Testing your installation
See separate instructions in `tests/preproc`.
Please report any problems.
### Installation FAQ ###
MODULE CONVERT_MOD
CONTAINS
END MODULE CONVERT_MOD
MODULE FTRAFO
CONTAINS
! Berechnung des Gradienten eines Skalars aus dem Feld des
! Skalars XMN im Phasenraum. Zurueckgegeben werden die Felder der
! Komponenten des horizontalen Gradienten XLAM,XPHI auf dem Gauss'schen Gitter.
! GWSAVE ist ein Hilfsfeld fuer die FFT
! P enthaelt die assoziierten Legendrepolynome, H deren Ableitung
! MLAT enthaelt die Anzahl der Gitterpunkte pro Breitenkreis
! MNAUF gibt die spektrale Aufloesung an,
! NI = Anzahl der Gauss'schen Gitterpunkte,
! NJ = Anzahl der Gauss'schen Breiten,
! NK = Anzahl der Niveaus
SUBROUTINE PHGRAD(XMN,XLAM,XPHI,GWSAVE,IFAX,P,H,MLAT, &
MNAUF,NI,NJ,NK)
USE PHTOGR
IMPLICIT NONE
INTEGER J,K,M,N,NI,NJ,NK,MNAUF,GGIND,LL,LLP,LLH,LLS,LLPS,LLHS
INTEGER MLAT(NJ),IFAX(10,NJ)
REAL UFOUC(0:MAXAUF),MUFOUC(0:MAXAUF)
REAL VFOUC(0:MAXAUF),MVFOUC(0:MAXAUF)
REAL XMN(0:(MNAUF+1)*(MNAUF+2)-1,NK)
REAL P(0:(MNAUF+3)*(MNAUF+4)/2,NJ/2)
REAL H(0:(MNAUF+2)*(MNAUF+3)/2)
REAL XLAM(NI,NK),XPHI(NI,NK)
REAL GWSAVE(8*NJ+15,NJ/2)
REAL ERAD
REAL SCR,SCI,ACR,ACI,MUSCR,MUSCI,MUACR,MUACI,RT,IT
ERAD = 6367470.0
GGIND=0
DO J = 1,NJ/2
CALL DPLGND(MNAUF,P(0,J),H)
DO K = 1,NK
LL=0