idlcr8ascii.pro 123 KB
Newer Older
Ian Boyd's avatar
Ian Boyd committed
1
;Main Program Version: idlcr8ascii.pro v4.0b23, 20190821
Ian Boyd's avatar
Ian Boyd committed
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
;  Written by Ian Boyd for the AVDC - iboyd@astro.umass.edu
;
;Sub-versions:
;  v3.01, 20081020 - If the HDF4 file is created using the HDF4.2r3 library, then extra information
;                    regarding the dimensions of the VAR_DEPEND values may be included in the file
;                    (which show up as extra datasets in the HDF_SD_FILEINFO call). A check for
;                    this is carried out and, if found, the information is excluded from the output.
;  v3.02, 20090311 - Improve HDF4.2r3 library checks by comparing the number of datasets recorded
;                    under DATA_VARIABLES, with the number returned by the HDF_SD_FILEINFO call;
;                    Change 'WARNING' to 'INFORMATION; Add procedure to handle 'INFORMATION' text
;                    output; Add 'INFORMATION' text if anything irregular found with the datasets;
;                    Change 'Data dimensions exceed 8' and 'Data type not allowable' ERROR messages
;                    to 'INFORMATION' messages. Make log file output to the same directory as the
;                    first HDF file to be read.
;  v3.03, 20091208 - Incorporate HDF_SD_ISCOORDVAR to differentiate between datasets and dimension
;                    variable names. Add check for allowable VAR_DATA_TYPE=STRING. Add all
;                    INFORMATION text messages to INFOTXT_OUTPUT_A procedure to avoid duplication
;                    of code that creates the messages; Add INFORMATION messages when reading HDF5
;                    files; Rename AVDC Button to FORMAT and replace keyword option /AVDC with
;                    /FORMAT. Improve checks on parameter and keyword inputs; Improve checks for
;                    non-standard (i.e. non-groundbased) HDF files.
;  v4.0b1, 20101122  - Adopt GEOMS metadata standard; Because FORMAT attribute(s) have been dropped
;                    can no longer use defined formatting values when outputing data to files;
;                    Default ASCII formatting adopted - scientific notation for REAL and DOUBLE,
;                    except for DATETIME and related values, which have formatting dependent on
;                    the data type; Numeric metadata values remain in their original data type
;                    when writing to session memory, meaning: variable attribute labels and values
;                    now written to separate structure variables sds.va_l and sds.va_v (previously
;                    sds.va), and structure has 2 dimensions (n_sds, n_atts) instead of 1
;                    (previously n_sds); The dataset is written to the initial n_atts index (i.e.
;                    sds[n_sds,0].data); Rename saved SDS datatype according to GEOMS rules
;                    (affects INT, LONG, and FLOAT); Add JDF_2_DATETIME and JULIAN_DATE routines
;                    to the program; Include coordinate variables if they and the non-coordinate
;                    variables add up to the number of saved DATA_VARIABLES. Add return error code
;                    option so that program returns to the calling program if an error is
;                    generated.
;  v4.0b2, 20110323  - If required reverse the order of VAR_SIZE and VAR_DEPEND values so that they
;                    match the order of the multi-dimensional dataset. Note that the HDF read programs
;                    automatically transpose the multi-dimensional datasets to conform to the IDL
;                    convention of fastest changing index being listed first (opposite for HDF).
;  v4.0b3, 20111014  - Add netCDF support - writes variable attributes to GEOMS standard attributes
;                    where possible, otherwise writes to VAR_NOTES
;  v4.0b4, 20111208  - Test for dimension ordering of the netCDF files, and reverse if required (same
;                    rules as for HDF). Create separate Dimension Ordering Routine
;  v4.0b5, 20120328  - Add options to convert HDF/netCDF files to other HDF/netCDF formats e.g. H4 to
;                    H5 etc. Requires IDLcr8HDF in the search path to do this.
;  v4.0b6, 20120426  - Bug fix when a netCDF file only contains datasets with single dimensions -
;                    VAR_SIZE information was not being extracted during the read process.
;  v4.0b7  20130114  - Add check for IDL being run in DEMO mode by using LMGR command. Replace PRINTF,-1
;                    statements (i.e. when dux[0] eq -1) with PRINT (o/w causes DEMO mode to stop). Also
;                    disable /LOG, /FORMAT and /DUMP keywords in DEMO mode.
;  v4.0b8  20140325  - Fix bug that meant VAR_SIZE and VAR_DEPEND values were not being automatically
;                    reversed if the array sizes were the same and VAR_DEPEND did not include a DATETIME
;                    value. Now defaults to the 'reverse' option in the TEST_DIM_ORDER routine if no
;                    other conditions are satisfied; Fix bug that caused crash when VAR_SIZE values were
;                    not of type string.
;  v4.0b9  20141110  - Fix bug that caused multi-dimensional array ordering to not be correctly identified
;                    if the first dataset to be checked in the file had the same number of elements in
;                    the array (e.g. was a set of Averaging Kernels).
;  v4.0b10 20150127  - Allow for VAR_FILL_VALUE values for string datasets to be written and saved as an
;                    (empty) string of the correct length. Ensure string datasets are all written and
;                    saved to the correct length (that of the longest string in the dataset).
;  v4.0b12 20151109  - Convert DATETIME (and related) max, min and fill values to LONG64 data type before
;                    determining the format for ASCII output to account for very large values.
;  v4.0b15 20160725  - Fix bug causing program to crash when the number of DATA_VARIABLES values is greater
;                    than the number of datasets in the file.
;  v4.0b19 20180218  - Fix bug that caused multi-dimensional array ordering to not be correctly identified
;                    if the datasets have the same VAR_SIZE i.e. rev_vd_vs stays as 2 through all the
Ian Boyd's avatar
Ian Boyd committed
70
71
72
73
74
75
;                    checks. Now rev_vd_vs changes to 1 (VAR_SIZE, VAR_DEPEND and dataset ordering is 
;                    reversed) if rev_vd_vs=2 after all the checks.
;  v4.0b20 20181116  - Fix bug that caused multi-dimensional array ordering to not be correctly written if
;                    not consistent through the file i.e. rev_vd_vs eq 3. Now rev_vd_vs changes to 1 
;                    (VAR_SIZE, VAR_DEPEND and dataset ordering is reversed), but information message 
;                    advising of the issue is still reported.
Ian Boyd's avatar
Ian Boyd committed
76
77
78
;  v4.0b21 20190514  - Fix bug that occurred when GEOMS variable attribute information is missing from an 
;                    HDF4 file but written to the heap structure based on the contents of the file. Previously
;                    the variable attribute label was not written correctly.
Ian Boyd's avatar
Ian Boyd committed
79
80
81
82
83
;  v4.0b22 20190806  - Provide support for reading netCDF4 files. Currently uses the HDF5 routines to do this
;                    so need to account for features unique to netCDF4 such as; the inclusion of dimension
;                    names; standard netCDF attribute names; empty attribute values that do not include a 
;                    null termination character (which causes an error in the H5A_Read routine). New 
;                    sub-routines include is_a_number_ascii, alpha_numeric_underscore, nc_dimension_chk,
Ian Boyd's avatar
Ian Boyd committed
84
85
;                    file_format_a.
;  v4.0b23 20190821  - Fixed bugs associated with H5 TAG_NAMES checks introduced in v4.0b22.                    
Ian Boyd's avatar
Ian Boyd committed
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204

PRO idlcr8ascii_common
;Procedure to define the data COMMON block WIDGET_WIN_A, containing common variables
;associated with the Graphical User Interface
; ----------
;  Written by Ian Boyd for the AVDC - iboyd@astro.umass.edu
;
;  History:
;    20061004: Introduced to IDLCR8ASCII - Version 2.0
;    20080302: 'cwidg' variable removed, and 'dux' array added to identify requested
;              output options - Version 3.0
;    20101122: Add rerr - Version 4.0b1
;
;
;  Input: Nil
;
;  Output: Nil
;
;  Called by: N/A
;
;  Subroutines Called: None

COMMON WIDGET_WIN_A,wtxt,lineno,base,o3,b3,dux,rerr

END ; Procedure idlcr8ascii_common



PRO intro_a_event, ev
;Procedure to define how Events from the Start-up Introduction Window are handled
; ----------
;Written by Ian Boyd for the AVDC - iboyd@astro.umass.edu
;
;  History:
;    20061004: Introduced to IDLCR8ASCII - Version 2.0
;    20080302: Code added to define the Pop-up Window and Log Output events added to
;              the Introduction Window - Version 3.0
;
;  Input: ev - Selected widget event structure
;
;  Output: o3 - Common string array defining the various options for program output
;
;  Called by: XMANAGER in INTRO_A
;
;  Subroutines Called: None

COMMON WIDGET_WIN_A

;The uservalue is retrieved from a widget when an event occurs
WIDGET_CONTROL,ev.id,GET_UVALUE=uv
IF (uv EQ 'C') OR (uv EQ 'P') OR (uv EQ 'idlcr8ascii.log') OR $
   (uv EQ 'H4') OR (uv EQ 'H5') OR (uv EQ 'NC') THEN BEGIN
  CASE 1 OF
    uv EQ 'C': BEGIN
                 uvi=1 & eadd=12L
               END
    uv EQ 'P': uvi=3
    uv EQ 'H4': BEGIN
                  uvi=5 & eadd=6L
                END
    uv EQ 'H5': BEGIN
                  uvi=6 & eadd=5L
                END
    uv EQ 'NC': BEGIN
                  uvi=7 & eadd=4L
                END
    ELSE: uvi=4
  ENDCASE
  IF o3[uvi] EQ uv THEN o3[uvi]='0' ELSE o3[uvi]=uv
  IF (uv EQ 'C') OR (uv EQ 'H4') OR (uv EQ 'H5') OR (uv EQ 'NC') THEN BEGIN
    IF (o3[1] EQ '0') AND (o3[5] EQ '0') AND (o3[6] EQ '0') AND (o3[7] EQ '0') THEN $
      WIDGET_CONTROL,ev.id+eadd,Sensitive=0 $
    ELSE BEGIN
      WIDGET_CONTROL,ev.id+eadd,Sensitive=1
      IF uv EQ 'C' THEN BEGIN
        WIDGET_CONTROL,ev.id+1,/Set_Button
        o3[3]='P'
      ENDIF
    ENDELSE
  ENDIF
ENDIF ELSE IF (uv EQ 'F') OR (uv EQ 'D') THEN o3[0]=uv $ ;Assign button event to a variable name
ELSE IF uv EQ 'Cont' THEN o3[0]='0' ;changes o3[0] from -1 to 0
IF (uv NE 'C') AND (uv NE 'P') AND (uv NE 'idlcr8ascii.log') AND (uv NE 'H4') AND $
   (uv NE 'H5') AND (uv NE 'NC') THEN WIDGET_CONTROL,ev.top,/DESTROY

END ;Intro_Event handler



PRO intro_a, intype
;Procedure which creates an Introduction Window at start-up when IDLCR8ASCII is called without parameters,
;or invalid parameters, or is called by IDL Virtual Machine. The user has a choice of continuing with
;the program after selecting input options, or stopping the program.
; ----------
;Written by Ian Boyd for the AVDC - iboyd@astro.umass.edu
;
;  History:
;    20061004: Introduced to IDLCR8ASCII - Version 2.0
;    20080302: Swapped the order of the option windows, so that the options indicated by the check
;              boxes appear above the option boxes that close the Introduction Window and continue
;              with the program; Added options to allow the user to send input/output log information
;              to a Pop-Up Window and/or a Log File - Version 3.0
;    20091208: Change AVDC button to FORMAT and keyword option from /AVDC to /FORMAT - Version 3.03
;    20101122: Add vertxt array. Modify Introduction text to indicate that program accepts 'GEOMS
;              Compliant' files, and also to show the new structure format - Version 4.0b1
;
;  Input: intype - Integer set to -2 or -3: -2 indicates normal state; -3 indicates that input
;                  parameters or keywords at the IDLCR8ASCII call are invalid.
;
;  Output: Nil
;
;  Called by: IDLCR8ASCII
;
;  Subroutines Called: INTRO_A_EVENT (via XMANAGER)

COMMON WIDGET_WIN_A

;procedure which provides an introduction message before starting the program.
nhdr=46 & errtxt=STRARR(nhdr)
Ian Boyd's avatar
Ian Boyd committed
205
vertxt=['idlcr8ascii-v4.0_Readme.pdf','v4.0b23 August 2019']
Ian Boyd's avatar
Ian Boyd committed
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
errtxt[1]='Welcome to IDLcr8ASCII.  This program reads GEOMS compliant HDF4, HDF5 and netCDF files and'
errtxt[2]='saves contents to either session memory, an output window (summary only) or to ASCII or formatted'
errtxt[3]='files (also refer to '+vertxt[0]+').'
errtxt[5]='Inputs to the program:'
errtxt[6]='  HDF/netCDF FILE(s) - GEOMS compliant HDF4, HDF5 or netCDF files.  Input can be by DIALOG_BOX'
errtxt[7]='    (IDL VM or full version of IDL), or by command line (full version of IDL only).'
errtxt[9]='Choice of output is made by selecting options in this DIALOG_BOX (IDL VM or full version'
errtxt[10]='  of IDL), or by ''Keywords'' at the command line input (full version only), as follows:'
errtxt[11]='  /F or /FORMAT - generates two output files (with .META and .DATA extensions).  The resulting'
errtxt[12]='    files can be used as inputs to GEOMS compliant write programs (IDLcr8HDF etc), OR'
errtxt[13]='  /D or /DUMP - generates two output files (with .META and .DATA extensions).  Data values'
errtxt[14]='    will be shown as indicated in the array format defined by VAR_SIZE, AND/OR'
errtxt[15]='  /C or /CATALOG - sends variable index, variable name, data format, and dimension size'
errtxt[16]='    information to an output window and/or log file, AND/OR'
errtxt[17]='  /P or /POPUP - sends log input/output information to a Pop-up Dialog Box, AND/OR'
errtxt[18]='  /L or /LOG - sends log input/output information to a file named idlcr8ascii.log'
errtxt[19]='  /H4 - outputs contents of the input file as an HDF4 file'
errtxt[20]='  /H5 - outputs contents of the input file as an HDF5 file'
Ian Boyd's avatar
Ian Boyd committed
224
225
errtxt[21]='  /NC3 - outputs contents of the input file as a netCDF3 file'
errtxt[23]='Example of command line input:  idlcr8ascii,DFSpec[,/F][,/D][,/C][,/P][,/L][,/H4][,/H5][,/NC3]'
Ian Boyd's avatar
Ian Boyd committed
226
227
228
229
errtxt[24]='  where ''DFSpec'' can either be a string array containing filenames or a scalar string containing a'
errtxt[25]='  filespec or a single file name.'
errtxt[26]='If running the full version of IDL, and input is a single HDF file, output can also be saved'
errtxt[27]='  to session memory.  This can be done by calling idlcr8ascii with the following command line:'
Ian Boyd's avatar
Ian Boyd committed
230
errtxt[28]='  idlcr8ascii,DFFile,GA,SDS[,/F][,/D][,/C][,/P][,/L][,/H4][,/H5][,/NC3].'
Ian Boyd's avatar
Ian Boyd committed
231
232
233
234
235
errtxt[29]='DFFile can either be the name of an input file or '''' (in which case a DIALOG_BOX will open'
errtxt[30]='  and prompt for the input filename).  ''GA'' is a returned string array containing the Global'
errtxt[31]='  Attributes, and ''SDS'' is a returned structure using pointers containing the Variable'
errtxt[32]='  Attribute Labels (sds.va_l), Variable Attribute Values (sds.va_v), and the Data (sds.data).
errtxt[34]='Contacts -'
Ian Boyd's avatar
Ian Boyd committed
236
237
238
239
240
241
242
errtxt[35]='  Ian Boyd, BC Scientific Consulting LLC (iboyd@astro.umass.edu)'
errtxt[36]='  26 Campbells Rd'
errtxt[37]='  Pine Hill, Dunedin 9010, New Zealand'
errtxt[39]='  Ann Mari Fjaeraa, EVDC Project Manager (amf@nilu.no)'
errtxt[40]='  Norwegian Institute for Air Research, Instituttveien 18'
errtxt[41]='  Postbox 100, N-2027 KJELLER, NORWAY'
errtxt[43]='EVDC Website: Tools and documentation available from http://evdc.esa.int/'
Ian Boyd's avatar
Ian Boyd committed
243
244
245
246
247
248
249
errtxt[45]='To continue, please choose from the options below.'
errtxt='        '+errtxt

;Set-up text display widget
demomode=LMGR(/DEMO) & optsens=1
IF intype EQ -3 THEN xtxt=' - Command Line Input Error' ELSE xtxt=''
base=WIDGET_BASE(Title='idlcr8ascii '+vertxt[1]+xtxt,Tlb_Frame_Attr=1,/Column) ;,Tab_Mode=1)
Ian Boyd's avatar
Ian Boyd committed
250
wtxt=WIDGET_TEXT(base,xsize=90,ysize=25,/Scroll)
Ian Boyd's avatar
Ian Boyd committed
251
252
253
254
255
256
257
258
259
260
261

tip='Left Mouse Click or Tab to entry and hit <Spacebar>'
base2=WIDGET_BASE(base,/Nonexclusive)
cattxt='Catalog - Generate a summary of the contents of the input file(s) and send to an output window or file'
poptext='Open a Pop-Up Window to display log input/output (default if ''Catalog'' option is chosen)'
logtext='Append log input/output to the file ''idlcr8ascii.log'' '
IF demomode THEN BEGIN
  logtext=logtext+'(No log or netCDF file output permitted in IDL DEMO Mode)' & optsens=0
ENDIF ELSE logtext=logtext+'(will create the file IF it doesn''t exist)'
h4text='Convert input file(s) to HDF4'
h5text='Convert input file(s) to HDF5'
Ian Boyd's avatar
Ian Boyd committed
262
263
nctext='Convert input file(s) to netCDF3'
fctext1='For the following 3 options, any existing files will be overwritten, IDLcr8HDF must be in the IDL Search Path,'
Ian Boyd's avatar
Ian Boyd committed
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
fctext2='and a valid Table Attribute Values (TAV) file must be in the same directory as the Input file(s).'
b1=WIDGET_BUTTON(base2,value=cattxt,uvalue='C',frame=3)
b2=WIDGET_BUTTON(base2,value=poptext,uvalue='P',frame=3)
b3=WIDGET_BUTTON(base2,value=logtext,uvalue='idlcr8ascii.log',sensitive=optsens,frame=3)

base3=WIDGET_LABEL(base)
WIDGET_CONTROL,base3,set_value=fctext1
base4=WIDGET_LABEL(base)
WIDGET_CONTROL,base4,set_value=fctext2

base5=WIDGET_BASE(base,/Nonexclusive,ROW=1)
b4=WIDGET_BUTTON(base5,value=h4text,uvalue='H4',frame=3)
b5=WIDGET_BUTTON(base5,value=h5text,uvalue='H5',frame=3)
b6=WIDGET_BUTTON(base5,value=nctext,uvalue='NC',sensitive=optsens,frame=3)

base6=WIDGET_BASE(base,/Row)
b7=WIDGET_BUTTON(base6,value='ASCII Formatted',uvalue='F',sensitive=optsens,frame=3) ;,Tooltip=Tip)
b8=WIDGET_BUTTON(base6,value='ASCII Dump',uvalue='D',sensitive=optsens,frame=3) ;,Tooltip=Tip)
b9=WIDGET_BUTTON(base6,value='Continue',uvalue='Cont',frame=3,Sensitive=0) ;,Tooltip=Tip)
b10=WIDGET_BUTTON(base6,value='Stop',uvalue='S',frame=3) ;,ToolTip=Tip)
WIDGET_CONTROL,base,/Realize
WIDGET_CONTROL,b1,/Input_Focus
FOR i=0,N_ELEMENTS(errtxt)-1 do $
  WIDGET_CONTROL,wtxt,set_value=errtxt[i],/Append
XMANAGER,'intro_a',base

END ;Intro_A



PRO idlcr8ascii_event, ev
;Procedure to close the pop-up logging window after user selects 'Finish'.
; ----------
;Written by Ian Boyd for the AVDC - iboyd@astro.umass.edu
;
;  History:
;    20061004: Introduced to IDLCR8ASCII - Version 2.0
;
;  Input: Selected widget event structure
;
;  Output: Nil
;
;  Called by: XMANAGER in IDLCR8ASCII and STOP_WITH_ERROR_A
;
;  Subroutines Called: None

WIDGET_CONTROL,ev.top,/DESTROY
RETALL & HEAP_GC

END ;Proc IDLcr8ASCII_Event



PRO stop_with_error_a, txt1, txt2, lu
;Procedure called when an error in the program is detected. An error message is displayed
;and the program stopped and reset. If necessary, open files are closed. The error message
;is displayed in one or more of the following: a Pop-up dialog window; the Pop-up logging
;window; the Output Logging window in the IDLDE.
; ----------
;Written by Ian Boyd for the AVDC - iboyd@astro.umass.edu
;
;  History:
;    20050729: Original IDLCR8ASCII Routine - Version 1.0
;    20061004: Set-up so that the error output is displayed in the output window dependent on the
;              method that IDLCR8ASCII is called. If txt1 is preceeded by 'D_' or is null, the
;              error output is to a Pop-up Dialog window. Otherwise output will be to a logging
;              window. Renamed from STOP_WITH_ERROR to STOP_WITH_ERROR_A to avoid a name conflict
;              with the equivalent procedure in IDLCR8HDF. Common variable definition WIDGET_WIN_A
;              added - Version 2.0
;    20080302: Added code which sends the error message to the IDLDE output window and/or an external
;              file (as determined by the dux array values) - Version 3.0
;    20101122: Allow routine to return to the calling program, rather than stop the application, if
;              a 'reterr' argument is included in the call to idlcr8ascii - Version 4.0b1

;
;  Inputs: txt1 - the initial text line of the error message. Generally contains the name of the routine
;                 where the error was detected.
;          txt2 - the second text line which generally describes the error state.
;          lu - Where applicable, the file unit that needs to be closed at the termination of the program,
;               otherwise set to -1.
;
;  Output: Nil
;
;  Called by: The routine in which the error was detected. The following routines call STOP_WITH_ERROR:
;             READ_HDF_SDS; IDLCR8ASCII
;
;  Subroutines Called: IDLCR8ASCII_EVENT (via XMANAGER)

COMMON WIDGET_WIN_A

IF lu NE -1L THEN FREE_LUN,lu

IF txt1 EQ '' THEN BEGIN ;<cancel> chosen on Intro box
  res=DIALOG_MESSAGE('IDLcr8ASCII Stopped!',/Information,Title='AVDC IDLcr8ASCII')
ENDIF ELSE BEGIN
  IF STRMID(txt1,0,2) EQ 'D_' THEN txtx=STRMID(txt1,2) ELSE txtx=txt1
  FOR i=dux[0],dux[1],dux[2] DO BEGIN
    IF i EQ -1 THEN BEGIN
      PRINT,'  ERROR in '+txtx & PRINT,'  '+txt2
      PRINT,'' & PRINT,'IDLcr8ASCII stopped - Program Ended on '+SYSTIME(0)
    ENDIF ELSE BEGIN
      PRINTF,i,'  ERROR in '+txtx & PRINTF,i,'  '+txt2
      PRINTF,i,'' & PRINTF,i,'IDLcr8ASCII stopped - Program Ended on '+SYSTIME(0)
    ENDELSE
  ENDFOR
  IF dux[1] GT -1 THEN FREE_LUN,dux[1]
  IF (STRMID(txt1,0,2) EQ 'D_') AND (rerr EQ 'NA') THEN BEGIN
    ;write error to DIALOG Box instead of Popup window
    errtxt2=STRARR(4)
    errtxt2[0]=STRMID(txt1,2) & errtxt2[1]=txt2 & errtxt2[3]='IDLcr8ASCII Stopped!'
    res=DIALOG_MESSAGE(errtxt2,/Error,Title='AVDC IDLcr8ASCII Error')
  ENDIF ELSE IF rerr EQ 'NA' THEN BEGIN ;write error to Popup window
    lineno=lineno+4L
    WIDGET_CONTROL,wtxt,set_value='    ERROR in '+txt1,/Append
    WIDGET_CONTROL,wtxt,set_value='    '+txt2,/Append
    WIDGET_CONTROL,wtxt,set_value='',/Append,Set_text_top_line=lineno
    WIDGET_CONTROL,wtxt,set_value='HDF file read stopped - hit <Finish> to close program',/Append
    WIDGET_CONTROL,b3,Sensitive=1,/Input_Focus
    XMANAGER,'stop_with_error_a',base,Event_Handler='idlcr8ascii_event'
  ENDIF
ENDELSE
HEAP_GC
IF rerr EQ 'NA' THEN RETALL ELSE rerr='Unable to read HDF or NC file - '+txtx+txt2

END ;Procedure Stop_With_Error_A



Ian Boyd's avatar
Ian Boyd committed
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
FUNCTION is_a_number_ascii, value
  ON_IOERROR, ConversionError
  IF STRTRIM(value,2) EQ '' THEN RETURN, 0B
  n=DOUBLE(value)
  RETURN, 1B
  ConversionError:
  RETURN, 0B
END



FUNCTION alpha_numeric_underscore, str_value
;Function that returns the input string with non-alphanumeric characters replaced with '_'
; ----------
;Written by Ian Boyd for the EVDC - iboyd@astro.umass.edu
;
;  History:
;    20190806: Introduced - Version 4.0b22
;
;  Input: str_value: string
;
;  Returns: string with non-alphanumeric values replaced with '_'
;
;  Called by: Read_HDF_SDS
;
ON_IOERROR, ConversionError
numb=BINDGEN(10)+48B ;0-9
ucb=BINDGEN(26)+65B  ;A-Z
lcb=BINDGEN(26)+97B  ;a-z
str_value=STRING(str_value)
new_str_value=str_value
n_char=STRLEN(str_value)
FOR i=0L,n_char-1L DO BEGIN
  byt_value=BYTE(STRMID(new_str_value,i,1))
  testn=(byt_value GE numb[0]) AND (byt_value LE numb[9])
  testu=(byt_value GE ucb[0]) AND (byt_value LE ucb[25])
  testl=(byt_value GE lcb[0]) AND (byt_value LE lcb[25])
  IF (~testn) AND (~testu) AND (~testl) THEN $
    STRPUT,new_str_value,'_',i
ENDFOR
RETURN, new_str_value
ConversionError:
RETURN, 'io_error'

END ;Function alpha_numeric_underscore



Ian Boyd's avatar
Ian Boyd committed
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
PRO infotxt_output_a, in0, in1
;Procedure called to report information relevant to the reading of the HDF file.
;This information can be reported to a Pop-up logging window, IDLDE output log window, and/or
;the log file. Code for this output was originally written directly in the affected procedures
; ----------
;Written by Ian Boyd for the AVDC - iboyd@astro.umass.edu
;
;  History:
;    20090311: Introduced to IDLCR8ASCII - Version 3.02
;    20091208: Added all the INFORMATION text messages to the routine, to avoid duplication
;              in the HDF4 and HDF5 read sections of READ_HDF_SDS - Version 3.03
;    20160213: Update information message when in0[0] EQ 13 - Version 4.0b13
;    20161130: Added in0[0] messages 18 and 19 to cover non-GEOMS metadata attributes issues
;              - Version 4.0b16
;
;  Inputs: in0 - Integer array containing input used to make correct text output
;          in1 - String array containing input used to make correct text output
;
;  Output: Nil
;
;  Called by: The routine in which the request for INFORMATION text was made.
;             The following routines call INFOTXT_OUTPUT_A: READ_HDF_SDS; OUTPUT_HDF_DATA;
;             IDLCR8ASCII
;
;  Subroutines Called: None

COMMON WIDGET_WIN_A

CASE 1 OF
  in0[0] EQ 0: BEGIN
      infotxt=STRARR(3)
      IF LONG(in1[0]) EQ 0L THEN BEGIN
        infotxt[0]='  INFORMATION: Global Attribute DATA_VARIABLES not found or has no values.'
Ian Boyd's avatar
Ian Boyd committed
473
        infotxt[1]='    Dataset listing order matches the order that the dataset is read in the GEOMS file.'
Ian Boyd's avatar
Ian Boyd committed
474
475
      ENDIF ELSE BEGIN
        infotxt[0]='  INFORMATION: Number of Global Attribute DATA_VARIABLES values ('+in1[0]+')'
Ian Boyd's avatar
Ian Boyd committed
476
        infotxt[1]='    does not match the number of datasets saved to the GEOMS file ('+in1[1]+')'
Ian Boyd's avatar
Ian Boyd committed
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
      ENDELSE
      infotxt[2]='    Number of datasets determined from '+in1[2]+' call'
    END
  in0[0] EQ 1: infotxt='  INFORMATION: '+in1[0]+' has '+in1[1]+' data dimensions'
  in0[0] EQ 2: infotxt='  INFORMATION: '+in1[0]+' dataset is not one of the allowable Data Types - '+in1[1]
  in0[0] EQ 3: BEGIN
      infotxt=STRARR(4)
      IF (STRPOS(STRUPCASE(in1[1]),STRUPCASE(in1[0])) NE -1) AND (in0[1] EQ 4) AND (in0[2] EQ 0) THEN $
        infotxt[0]='  INFORMATION: Dataset Name is truncated (File created with HDF4.2r1 library or earlier):' $
      ELSE infotxt[0]='  INFORMATION: Dataset Name does not match VAR_NAME value:'
      infotxt[1]='    SDS_NAME: '+in1[0]
      infotxt[2]='    VAR_NAME: '+in1[1]
      IF in0[2] EQ 0 THEN infotxt[3]='    Output uses VAR_NAME from Metadata' $
      ELSE infotxt[3]='    Output uses Dataset Name from the HDF file'
    END
  in0[0] EQ 4: BEGIN
      infotxt=STRARR(2)
      IF in0[1] EQ 0 THEN BEGIN
        infotxt[0]='  INFORMATION: Metadata label VAR_NAME not found for dataset:'
        infotxt[1]='    SDS_NAME: '+in1[0]
      ENDIF ELSE IF (in0[2] EQ 0) AND (in0[3] NE 0L) THEN BEGIN
        infotxt[0]='  INFORMATION: VAR_NAME value does not match any DATA_VARIABLES values:'
        infotxt[1]='    VAR_NAME: '+in1[1]
      ENDIF
    END
  in0[0] EQ 5: BEGIN
      infotxt=STRARR(2)
      infotxt[0]='  INFORMATION: No Variable Attributes returned after call to '+in1[1]
      infotxt[1]='    SDS_NAME: '+in1[0]
    END
  in0[0] EQ 6: BEGIN
      infotxt='  INFORMATION: Metadata Dataset listing order may not match the order of'
      infotxt=infotxt+' the Global Attribute DATA_VARIABLES Values'
    END
  in0[0] EQ 7: BEGIN
      infotxt='  INFORMATION: Can read one HDF or netCDF file into session memory.'
      infotxt=infotxt+' Only first file in the array will be read'
    END
  in0[0] EQ 8: BEGIN
      infotxt='  INFORMATION: '+in1[0]+' is a coordinate variable'
    END
  in0[0] EQ 9: BEGIN
      infotxt=STRARR(3)
      infotxt[0]='  INFORMATION: Calibrated data in dataset '+in1[0]
      infotxt[1]='    has been converted back to its original form using the formula:'
      infotxt[2]='    Orig_Data = Scale_Factor * (Cal_Data - Offset)'
    END
  in0[0] EQ 10: BEGIN
      infotxt=STRARR(2)
      infotxt[0]='  INFORMATION: VAR_DATA_TYPE='+in1[1]+' for Calibrated data in dataset '+in1[0]
      infotxt[1]='    has been changed to the original datatype of '+in1[2]
    END
  in0[0] EQ 11: BEGIN
      infotxt=STRARR(2)
      CASE 1 OF
        in0[1] EQ 20: BEGIN
            infotxt[0]='  INFORMATION: Order of multi-dimensional datasets changed to match the programming'
            infotxt[0]=infotxt[0]+' language convention:
            infotxt[1]='    IDL/Fortran - fastest changing dimension first (datasets transposed when'
            infotxt[1]=infotxt[1]+' reading/writing to the HDF file)'
          END
        in0[1] MOD 10 EQ 0: BEGIN
            infotxt[0]='  INFORMATION: Listing order of multi-dimensional VAR_DEPEND and VAR_SIZE values'
            infotxt[1]='    in the HDF file changed to match corresponding dataset dimension ordering'
          END
        ELSE: BEGIN
            infotxt[0]='  INFORMATION: Unable to determine the correct listing order of multi-dimensional'
            infotxt[1]='    VAR_SIZE and VAR_DEPEND values in the HDF file'
          END
      ENDCASE
    END
  in0[0] EQ 12: BEGIN
      infotxt=STRARR(2)
      infotxt[0]='  INFORMATION: Table Attribute Values file must be in the same directory as the input file'
      infotxt[1]='    if wanting to convert from one Data Format to another'
    END
  in0[0] EQ 13: BEGIN
      infotxt=STRARR(2)
      infotxt[0]='  INFORMATION: IDLcr8HDF called but it is not in the IDL Search Path or it encountered an'
      infotxt[1]='    unexpected error. If the latter please contact Ian Boyd at iboyd@astro.umass.edu'
    END
  in0[0] EQ 14: BEGIN
      infotxt=STRARR(2)
      infotxt[0]='  INFORMATION: /POPUP keyword cannot be used together with the ''reterr'' argument.'
      infotxt[1]='    The request for a POPUP window has been ignored'
    END
  in0[0] EQ 15: BEGIN
      infotxt=STRARR(2)
      infotxt[0]='  INFORMATION: Argument ''reterr'' must be of type string.' ;a returnable variable of type string.'
      infotxt[1]='    idlcr8ascii will stop normally if an error is encountered'
    END
  in0[0] EQ 16: BEGIN
      infotxt=STRARR(2)
      infotxt[0]='  INFORMATION: /LOG, /FORMAT, and /DUMP keywords cannot be used in IDL DEMO mode and will be ignored.'
      infotxt[1]='    To generate ASCII files please use the free IDL Virtual Machine or a licenced version of IDL'
    END
  in0[0] EQ 17: BEGIN
      infotxt=STRARR(2)
      infotxt[0]='  INFORMATION: NetCDF file create feature is disabled in IDL DEMO mode.'
      infotxt[1]='    To generate files please use the free IDL Virtual Machine or a licenced version of IDL'
    END
  in0[0] EQ 18: BEGIN
      infotxt='  INFORMATION: Value for Attribute Label '+in1[0]+' is an invalid Data Type'
    END
  in0[0] EQ 19: BEGIN
      infotxt='  INFORMATION: '+in1[0]+' is not a valid GEOMS Metadata Attribute'
    END
ENDCASE

dm=SIZE(infotxt,/N_Elements)
IF o3[3] EQ '' THEN lineno=lineno+dm

FOR n=0,dm-1 DO BEGIN
  IF o3[3] EQ '' THEN BEGIN
    IF n EQ dm THEN WIDGET_CONTROL,wtxt,set_value='',/Append $
    ELSE WIDGET_CONTROL,wtxt,set_value=infotxt[n],/Append,Set_text_top_line=lineno
  ENDIF
  FOR m=dux[0],dux[1],dux[2] DO $
    IF m EQ -1 THEN PRINT,infotxt[n] ELSE PRINTF,m,infotxt[n] ;write out to log(s)
ENDFOR

END ;Procedure InfoTxt_Output_A



FUNCTION jdf_2_datetime, jdf, MJD2000=mjd2000, SHORTISO8601=iso, LONGISO8601=isoms
;Computes the UT date/time from JDF/MJD2000.
; ----------
;   Bojan R. Bojkov
;   bojan.r.bojkov@nasa.gov
;   03/04/2004
;   03/11/2004 bug fix
;   05/27/2005 add ShortISO8601 and LongISO8601 switches,
;              and change seconds value to decimal seconds (Ian Boyd)
;
; References ----------
;   Explan. Supp. Astron. Almanac, p.604 (1992); through E. Celarier
;
; Caveats ----------
;   Valid for all YYYY >= -4712 (i.e. for all JD .ge. 0)
;   The true Gregorian calendar was only adopted on 15 October 1582,
;   so any dates before this are "virtual" Gregorian dates.
;
; Input ----------
;   jdf     : double precision value
;   mjd2000 : flag if input is MJD2000
;   iso     : flag if output is in ISO8601 (YYYYMMDDThhmmssZ)
;   isoms   : flag if output is in ISO8601ms (YYYYMMDDThhmmss.sssZ)
;
; Output ---------
;   floating point array [YYYY, MM, DD, hh, mn, ss.sss],
;   or ISO8601 string format
; External subroutines ---------
;   NONE

jdf=DOUBLE(jdf)

j0=2451544.5D
IF KEYWORD_SET(mjd2000) THEN jdhold=jdf+j0 ELSE jdhold=jdf
jdi=LONG(jdhold)
df=jdhold-jdi

;Determine hh, mm, ss, ms
hh=(df+0.5)*24.D
q=WHERE(hh GE 24.D)
IF q[0] NE -1 THEN BEGIN
  hh[q]=hh[q]-24.D
  jdi[q]=jdi[q]+1
ENDIF
t1=hh
hh=LONG(t1)
t2=(t1-hh)*60.D
mn=LONG(t2)
t3=(t2-mn)*60.D
ss=t3
;ss=LONG(t3)
;ms=LONG((t3-ss)*1.d3)

;Determine YYYY, MM, DD
t1=jdi+68569L
t2=(4*t1)/146097L
t1=t1-(146097L*t2+3L)/4L
t3=(4000L*(t1+1L))/1461001L
t1=t1-(1461L*t3)/4L + 31L
t4=(80L*t1)/2447L

dd=t1-(2447L*t4)/80L
t1=t4/11L
mm=t4+2L-12L*t1
yyyy=100L*(t2-49L)+t3+t1

dt=TRANSPOSE([[yyyy],[mm],[dd],[hh],[mn],[ss]])
IF (KEYWORD_SET(iso)) OR (KEYWORD_SET(isoms)) THEN BEGIN
  dts=STRARR(6)
  IF (KEYWORD_SET(iso)) AND (ss-FIX(ss) GE 0.5) THEN BEGIN
    ;recalculate to get correct seconds value
    jdhold=jdf+0.000008D ;add ~0.7 secs
    IF KEYWORD_SET(mjd2000) THEN jdhold=jdhold+j0
    CALDAT,jdhold,mm,dd,yyyy,hh,mn,ss
    dt=TRANSPOSE([[yyyy],[mm],[dd],[hh],[mn],[ss]])
  ENDIF
  dt=LONG(dt)
  FOR i=1,5 DO $
    IF dt[i] LT 10L THEN dts[i]='0'+STRTRIM(dt[i],2) ELSE dts[i]=STRTRIM(dt[i],2)
  IF KEYWORD_SET(isoms) THEN BEGIN
    ssd=STRING(format='(f6.3)',ss)
    IF FLOAT(ssd) LT 10.0 THEN dts[5]='0'+STRTRIM(ssd,2) ELSE dts[5]=STRTRIM(ssd,2)
  ENDIF
  RETURN,STRTRIM(dt[0],2)+dts[1]+dts[2]+'T'+dts[3]+dts[4]+dts[5]+'Z'
ENDIF ELSE RETURN, dt

END ;Function jdf_2_datetime



FUNCTION julian_date, date_ut, ISO8601=iso, MJD2000=mjd2000
;Computes the Julian date (jd) or date in MJD2000 format in double precision.
; ----------
;   Bojan R. Bojkov
;   bojan.r.bojkov@nasa.gov
;   06/03/2001
;   09/24/2001: Code cleanup                        - Function Version 1.0
;   10/16/2001: Mode corrections                    - Function Version 1.01
;   03/07/2002: Comment cleanup and variable change - Function Version 2.0
;   10/08/2002: Added jdf option                    - Function Version 2.1
;   07/21/2003: Converted to a function             - Function Version 3.0
;   05/25/2005: Added ISO8601 and MJD2000 keywords.
;               Input can either be a string or a numeric array
;               (see below for input details). Returns -99999.d
;               if the Input is invalid - Ian Boyd
;
; References ----------
;   Hughes, D.W., Yallop, B.D. and Hohenkerk, C.Y., "The Equation of Time",
;     Mon. Not. R. astr. Soc., 238, pp 1529-1535. 1989.
;
;   Results verified using:
;     Meeus, J, "Astronomical Algorithms", William-Bell, Inc., Richmond VA,
;     p62, 1991.
;
; Caveats ----------
;   NONE
;
; Input ----------
;   date_ut: Numeric array (UT: YYYY, MM, DD, [hh, mm, ss]), or if ISO8601
;            keyword set, a string containing datetime as YYYYMMDDThhmmssZ
;
; Output ---------
;   jd:      julian day, or if MJD2000 keyword set,
;            a double precision day in MJD2000 format
;
; External subroutines ---------
;   NONE

ON_IOERROR,TypeConversionError
valid=0 ;Type conversion check
inputerror=0 ;Check that input is as expected
;Check input format
IF KEYWORD_SET(iso) THEN BEGIN ;input is YYYYMMDDThhmmssZ
  IF STRLEN(date_ut) EQ 16 THEN BEGIN
    ;test for numeric values (will return type conversion error value if not a number)
    FOR i=0,7 DO check=FIX(STRMID(date_ut,i,1))
    FOR i=9,14 DO check=FIX(STRMID(date_ut,i,1))
    dt=INTARR(6)
    dt[0]=FIX(STRMID(date_ut,0,4)) & dt[1]=FIX(STRMID(date_ut,4,2))
    dt[2]=FIX(STRMID(date_ut,6,2)) & dt[3]=FIX(STRMID(date_ut,9,2))
    dt[4]=FIX(STRMID(date_ut,11,2)) & dt[5]=FIX(STRMID(date_ut,13,2))
    date_ut=dt & achk=[0,6] ;to add hhmmss component
  ENDIF ELSE inputerror=1
ENDIF ELSE BEGIN
  ;check that input contains at least YMD (and at most YMDhms) information and is either
  ;an integer, long, float or double array
  achk=SIZE(date_ut)
  IF (achk[1] LT 3) OR (achk[1] GT 6) OR (achk[2] LT 2) OR (achk[2] GT 5) THEN inputerror=1
ENDELSE

IF inputerror EQ 0 THEN BEGIN ;can perform JD calculation
  IF date_ut[1] GT 2 THEN BEGIN
    y=DOUBLE(date_ut[0])
    m=DOUBLE(date_ut[1]-3)
    d=DOUBLE(date_ut[2])
  ENDIF ELSE BEGIN
    y=DOUBLE(date_ut[0]-1)
    m=DOUBLE(date_ut[1]+9)
    d=DOUBLE(date_ut[2])
  ENDELSE

  ;Compute Julian date:
  j=LONG(365.25D0*(y+4712.D0))+LONG(30.6D0*m+0.5D0)+59.D0+d-0.5D0

  ;Check for Julian or Gregorian calendar:
  IF j LT 2299159.5D0 THEN jd=j $ ;If Julian calendar.
  ELSE BEGIN ;If Gregorian calendar.
    gn=38.D0-LONG(3.D0*LONG(49.D0+y/100.D0)/4.D0)
    jd=j+gn
  ENDELSE

  ;add hhmmss values if present
  CASE 1 OF
    achk[1] EQ 4: jd=jd+DOUBLE(date_ut[3])/24.D ;hh only
    achk[1] EQ 5: jd=jd+(DOUBLE(date_ut[3])*3600.D + DOUBLE(date_ut[4])*60.D)/86400.D ;hhmm only
    achk[1] EQ 6: jd=jd+(DOUBLE(date_ut[3])*3600.D + DOUBLE(date_ut[4])*60.D + $
                         DOUBLE(date_ut[5]))/86400.D ;hhmmss
    ELSE:
  ENDCASE

  valid=1 ;Type conversion OK

  ;Set to MJD2000 if required
  IF KEYWORD_SET(mjd2000) THEN jd=jd-2451544.5D
ENDIF

TypeConversionError:
IF valid EQ 0 THEN RETURN,-99999.D ELSE RETURN, jd

END ;Function Julian_Date



Ian Boyd's avatar
Ian Boyd committed
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
PRO nc_dimension_chk, ftype, sd_id, di, n_sds, dgi, n_sdsg
;Procedure to check for netCDF dimension datasets. These will be quietly ignored during the
;read process 
; ----------
;Written by Ian Boyd for the EVDC - iboyd@astro.umass.edu
;
;  History:
;    20190806: Introduced - Version 4.0b22
;
;  Inputs: ftype - format type, HDF5 or netCDF
;          sd_id - GEOMS file identifier number
;          di    - index values for datasets
;          n_sds - number of datasets determined by H5G_GET_NMEMBERS call
;
;  Outputs: dgi    - index values for GEOMS datasets (not generated by netCDF libraries)
;           n_sdsg - number of GEOMS datasets
;
;  Called by: Read_HDF_SDS
;
dimen_names=['constant','independent_','_strlen']
n_dn=N_ELEMENTS(dimen_names)
ds_names=STRARR(n_sds) & gdi=INTARR(n_sds)-1
FOR i=0,n_sds-1 DO BEGIN
  IF ftype EQ 'H5' THEN sds_name=H5G_GET_MEMBER_NAME(sd_id,'/',di[i]) $
  ELSE BEGIN ;netCDF file  
    varstruct=NCDF_VARINQ(sd_id,i)
    sds_name=varstruct.name
  ENDELSE  
  ds_names[i]=STRTRIM(STRLOWCASE(sds_name),2)
ENDFOR

FOR i=0,n_sds-1 DO BEGIN
  dnicnt=0 & j=0
  WHILE (dnicnt EQ 0) AND (j LE n_dn-1) DO $
    IF STRPOS(ds_names[i],dimen_names[j]) NE -1 THEN dnicnt++ ELSE j++
  IF dnicnt NE 0 THEN BEGIN
    ;Check that names are actually netCDF dimension names o/w will treat as a dataset name
    CASE j OF
      0: IF ds_names[i] NE dimen_names[j] THEN dnicnt=0 ;i.e. not a netCDF dimension name
      1: BEGIN
           ;check that first part of the name is INDEPENDENT_ and the second part is a number
           IF STRMID(ds_names[i],0,STRLEN(dimen_names[j])) NE dimen_names[j] $
             THEN dnicnt=0 $
           ELSE BEGIN
             res=STRSPLIT(ds_names[i],'_',/EXTRACT,COUNT=cres)
             res=[res,''] ;to ensure at least 2 extracted values in ds_names[i] 
             if ~IS_A_NUMBER_ASCII(res[1]) THEN dnicnt=0 ; i.e. not a netCDF dimension name
           ENDELSE        
         END
      2: BEGIN
          ;check that first part of the name is a named dataset
          res=STRSPLIT(ds_names[i],'_',/EXTRACT)
          ni=WHERE(res[0] EQ ds_names,ncnt)
          IF ncnt EQ 0 THEN dnicnt=0 ;i.e. not a netCDF dimension name
         END
    ENDCASE
  ENDIF
  IF dnicnt EQ 0 THEN gdi[i]=i ;i.e. this is a valid dataset name and not a netCDF dimension name
ENDFOR

dgi=WHERE(gdi NE -1,n_sdsg) ;dgi and n_sdsg are returned by the procedure

END ;Procedure nc_dimension_chk



FUNCTION file_format_a, infile
;Function to identify the format of the input file - tests for HDF4, HDF5/netCDF4 and netCDF3.
;netCDF4 files will be identified as HDF5 and read in using the HDF5 library
; ----------
;Written by Ian Boyd for the EVDC - iboyd@astro.umass.edu
;
;  History:
;    20190806: Introduced - Version 4.0b22
;
;  Inputs: infile - Input GEOMS compliant file
;
;  Outputs: string identifying the type of file (H4, H5, NC) or XX for unidentified file
;
;  Called by: IDLCR8ASCII
;
;Markers
;HDF4: 14   3  19   (1   0  16   0   1  95  98   0  30)
;HDF5: 137  72  68  70 (13  10  26  10   0   0   0   0)
;nc3:  67  68  70   (1   0   0   0   0   0   0   0  10)
;nc4:  137  72  68  70 (13  10  26  10  (2   8   8   0))

h4id=[14B, 3B, 19B]
h5id=[137B, 72B, 68B, 70B]
nc3id=[67B, 68B, 70B]

;First test using start of file byte markers
fid=BYTARR(4)
openr,fu,infile,/GET_LUN
readu,fu,fid
FREE_LUN,fu

idfound='XX'
CASE 1 OF
  ARRAY_EQUAL(fid[0:2],h4id): idfound='H4'
  ARRAY_EQUAL(fid,h5id): idfound='H5'
  ARRAY_EQUAL(fid[0:2],nc3id): idfound='NC'
  ELSE:
ENDCASE

;Second test, if required, using IDL commands
IF idfound EQ 'XX' THEN BEGIN
  IF HDF_ISHDF(infile) EQ 1 THEN idfound='H4' $
  ELSE IF FLOAT(!Version.Release) GE 5.6 THEN BEGIN
    IF H5F_IS_HDF5(infile) EQ 1 THEN idfound='H5' ;netCDF4 or HDF5
  ENDIF
  IF idfound EQ 'XX' THEN BEGIN
    ;final test for netCDF3
    validname=1
    CATCH, ncdferror ;To catch the error when attempting to open the file as a netCDF
    ;Error Handler
    IF ncdferror NE 0 THEN BEGIN ;not a netCDF file
      validname=0
      CATCH, /CANCEL
    ENDIF
    IF validname EQ 1 THEN BEGIN
      fileid=NCDF_OPEN(infile)
      ;If get to here then file is netCDF
      CATCH, /CANCEL
      NCDF_CLOSE,fileid
      idfound='NC'
    ENDIF
  ENDIF
ENDIF

RETURN, idfound

END ;File_Format_A



Ian Boyd's avatar
Ian Boyd committed
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
PRO test_dim_order, va_name, va_value, va_type, sds_dim, rev_vd_vs
;Procedure to test the dimension ordering of any multi-dimensional datasets in the input DF file, and
;return correct ordering code (dimension ordering for idlcr8ascii uses IDL/Fortran convention)
; ----------
;Written by Ian Boyd for the AVDC - iboyd@astro.umass.edu
;
;  History:
;    20111208: Introduced - Version 4.0b4
;    20140325: Fix bug that meant VAR_SIZE and VAR_DEPEND values were not being automatically
;              reversed if the array sizes were the same and VAR_DEPEND did not include a DATETIME
;              value. Now defaults to the 'reverse' option if no other conditions are satisfied;
;              Fix bug that caused crash when VAR_SIZE values were not of type string - Version 4.0b8
;    20141110: Fix bug that caused multi-dimensional array ordering to not be correctly identified
;              if the first dataset to be checked in the file had the same number of elements in
;              the array (e.g. was a set of Averaging Kernels) - Version 4.0b9
;
;  Inputs: va_name - An abbreviated version of the Variable Name; either 'VD' (VAR_DEPEND) or 'VS'
;                    (VAR_SIZE)
;          va_value - The corresponding Variable Value
;          sds_dim - Dimension information for the DF Dataset being tested
;
;  Outputs: rev_vd_vs - Scalar to indicate whether VAR_DEPEND and VAR_SIZE values (and data, in the case
;                       of netCDF measurements) need to be reversed
;
;  Called by: Read_HDF_SDS
;

IF va_type NE 'STRING' THEN BEGIN ;VAR_SIZE contains numeric values instead of in the form of a string
  vavhold='' & n_vav=N_ELEMENTS(va_value)
  FOR k=0,n_vav-1 DO BEGIN
    IF k EQ n_vav-1 THEN vtxt='' ELSE vtxt=';'
    vavhold=vavhold+STRTRIM(va_value[k],2)+vtxt
  ENDFOR
ENDIF ELSE vavhold=va_value
vs_v=STRCOMPRESS(STRSPLIT(vavhold,';, ',/EXTRACT,COUNT=rcnt),/REMOVE_ALL)

IF va_name EQ 'VD' THEN BEGIN ;VAR_DEPEND tests
  dti=WHERE(STRUPCASE(vs_v) EQ 'DATETIME',dticnt)
  ;Note: if DATETIME has VAR_SIZE=1 then the ordering is dependent on the order of the VAR_SIZE values only
  IF (rcnt GT 1) AND (dticnt NE 0) THEN BEGIN ;multi-dimensions including DATETIME
    test1=(dti[0] EQ rcnt-1) AND (sds_dim[0] GT 1)
    ;DATETIME is the last value and present only once and has VAR_SIZE GT 1
    test2=(dti[0] EQ 0) AND (dticnt EQ 1) AND (sds_dim[0] GT 1)
    ;DATETIME is the first value and present only once and has VAR_SIZE GT 1
    IF (test1) AND (rev_vd_vs EQ 2) THEN rev_vd_vs=10 $
    ELSE IF (test2) AND (rev_vd_vs EQ 2) THEN rev_vd_vs=11 $
    ELSE IF ((test1) AND (rev_vd_vs MOD 10 EQ 1)) OR ((test2) AND (rev_vd_vs EQ 10)) THEN $
      rev_vd_vs=3 $ ;inconsistent rules regarding dimension ordering in file
    ELSE IF (test2) AND (rev_vd_vs EQ 0) THEN rev_vd_vs=20
    ;Dimension ordering is slowest changing first so change dataset ordering as well as
    ;VAR_DEPEND and VAR_SIZE ordering
  ENDIF
ENDIF ELSE BEGIN ;VAR_SIZE tests
  IF rcnt GT 1 THEN BEGIN ;multi-dimensions
    ;check all values are numeric
    numchk=STRJOIN(vs_v,/SINGLE) & valok=1
    FOR k=0,STRLEN(numchk)-1 DO BEGIN
      bchar=BYTE(STRMID(numchk,k,1))
      IF (bchar LT 48) OR (bchar GT 57) THEN valok=0 ;i.e. non-numeric character
    ENDFOR
    IF valok EQ 1 THEN BEGIN ;all values are numeric so do checks
      vs_vl=LONG(vs_v)
      test1=ARRAY_EQUAL(vs_vl,sds_dim)
      test2=ARRAY_EQUAL(REVERSE(vs_vl),sds_dim) ;test in case values are the same e.g. 40;40
      ;If arrays are the same size then the default will be to reverse the VS and VD values
      ;unless a different dimension ordering has already been identified
      IF (test1 EQ 1) AND (test2 EQ 0) AND (rev_vd_vs EQ 2) THEN rev_vd_vs=0 $
      ELSE IF (test1 EQ 0) AND (test2 EQ 1) AND (rev_vd_vs EQ 2) THEN rev_vd_vs=1 $
      ELSE IF ((test1 EQ 1) AND (test2 EQ 0) AND (rev_vd_vs MOD 10 EQ 1)) OR $
              ((test1 EQ 0) AND (test2 EQ 1) AND (rev_vd_vs MOD 10 EQ 0)) THEN $
        rev_vd_vs=3 $ ;inconsistent VAR_SIZE and sds_dim agreement from dataset to dataset
      ELSE IF (test1 EQ 1) AND (test2 EQ 1) AND (rev_vd_vs EQ 2) THEN $
        rev_vd_vs=2 $ ;first set of measurements tested have same dimensions so do not do anything
      ELSE IF rev_vd_vs EQ 2 THEN rev_vd_vs=1 ;default if no other criteria are found as this
      ;will then automatically reverse the VD and VS values
    ENDIF
  ENDIF
ENDELSE

END ;Procedure Test_Dim_Order



PRO read_hdf_sds, ifile, ga, sds, catinfo
;Procedure to read the contents of a GEOMS standard HDF4 or HDF5 compatible file
;into session memory
; ----------
;Written by Ian Boyd for the AVDC - iboyd@astro.umass.edu
;
;  History:
;    20050729: Original IDLCR8ASCII Routine - Version 1.0
;    20050912: Removed Common variable definition CATALOGINFO and made the variable a parameter passed
;              to the procedure; Checks that the attribute variable names read by HDF_SD_GETINFO, match
;              those recorded in DATA_VARIABLES in the Global Attributes. If not use the variable name
;              listed in DATA_VARIABLES - Version 1.1
;    20061004: Common variable definition WIDGET_WIN_A added for Error calls; Add option to read HDF5
;              files (needs IDL5.6 or newer); Add check that the HDF4 file has valid information on the
;              number of global and variable attributes; Checks added in the event that the HDF4 or
;              HDF5 file has been created by the NCSA utility programs H5toH4 or H4toH5 from an original
;              AVDC/Envisat/NDACC HDF4 or HDF5 file; Ensures variable attributes and data are listed in
;              the same order as that given under DATA_VARIABLES - Version 2.0
;    20081020 - If the HDF4 file is created using the HDF4.2r3 library, then extra information
;               regarding the dimensions of the VAR_DEPEND values may be included in the file (which
;               show up as extra datasets in the HDF_SD_FILEINFO call). A check for this is carried out
;               and, if found, the information is excluded from the output - Version 3.01
;    20090311: Improve HDF4.2r3 library checks by comparing the number of datasets recorded under
;              DATA_VARIABLES, with the number returned by the HDF_SD_FILEINFO call; Change 'WARNING' to
;              'INFORMATION and make calls to procedure INFOTXT_OUTPUT_A to avoid duplication. Add
;              'INFORMATION' text if anything irregular found with the datasets; Change 'Data dimensions
;               exceed 8' and 'Data type not allowable' ERROR messages to 'INFORMATION' messages
;               - Version 3.02
;    20091208: Incorporate HDF_SD_ISCOORDVAR to differentiate between datasets and dimension variable
;              names. Add check for allowable VAR_DATA_TYPE=STRING. Add all INFORMATION text messages to
;              INFOTXT_OUTPUT_A procedure to avoid duplication of code that creates the messages;
;              Add INFORMATION messages when reading HDF5 files; Improve checks for non-standard (i.e.
;              non-groundbased) HDF input file - Version 3.03
;    20101122: Numeric metadata values remain in their original data type when transferring to session
;              memory, meaning: variable attribute labels and values now written to separate structure
;              variables sds.va_l and sds.va_v (previously sds.va), and structure has 2 dimensions
;              (n_sds, n_atts) instead of 1 (previously n_sds); The dataset is written to the initial
;              n_atts index (i.e. sds[n_sds,0].data); Rename saved SDS datatype according to GEOMS rules
;              (affects INT, LONG, and FLOAT); Compare HDF dimensions with VAR_SIZE values and transpose
;              VAR_SIZE and VAR_DEPEND values if required - Version 4.0b1
;    20111014: Add netCDF read capability - writes variable attributes to standard GEOMS variable
;              attribute labels where possible, otherwise appends information to VAR_NOTES - Version 4.0b3
;    20120426: Bug fix when a netCDF file only contains datasets with single dimensions, then VAR_SIZE
;              information was not being extracted during the read process - Version 4.0b6
;    20140325: Fix bug that caused crash when VAR_SIZE values were not of type string - Version 4.0b8
;    20150127: Allow for VAR_FILL_VALUE values for string datasets to be written and saved as an
;              (empty) string of the correct length. Ensure string datasets are all written and
;              saved to the correct length (that of the longest string in the dataset) - Version 4.0b10
;    20150217: Do not remove whitespace of any variable attribute values that are written as strings
;              - Version 4.0b11
;    20160614: Some HDF4 string datasets may not have 2 dimensions, so do not need to remove the first
;              dimension (= maximum number of characters) - Version 4.0b14
;    20160725: Fix bug causing program to crash when the number of DATA_VARIABLES values is greater
;              than the number of datasets in the file - Version 4.0b15
;    20161130: Assign SDS_name to catinfo for attributes when the VAR_NAME is not present; rewrite
;              netCDF section to align it with HDF5 section and to account for dataset ordering that
;              does not match that in DATA_VARIABLES - Version 4.0b16
;    20161213: Fix bug when reading in netCDF files to stop all dataset variable attribute values to
;              be converted to string format (now matches HDF4 and HDF5 checks). Fix bug that caused
;              the program to crash when VAR_DEPEND=CONSTANT in netCDF files (sds_ndim=0) - Version 4.0b17
;    20161218: Fix bug when reading netCDF attribute values that are of type byte rather than string -
;              Version 4.0b18
;    20180218  Fix bug that caused multi-dimensional array ordering to not be correctly identified
;              if the datasets have the same VAR_SIZE. Now rev_vd_vs changes to 1 (VAR_SIZE and
;              VAR_DEPEND ordering is reversed) if rev_vd_vs=2 after all the checks - Version 4.0b19
Ian Boyd's avatar
Ian Boyd committed
1079
1080
1081
1082
;    20181116  Fix bug that caused multi-dimensional array ordering to not be correctly written if
;              not consistent through the file i.e. rev_vd_vs eq 3. Now rev_vd_vs changes to 1
;              (VAR_SIZE, VAR_DEPEND and dataset ordering is reversed), but information message
;              advising of the issue is still reported - Version 4.0b20
Ian Boyd's avatar
Ian Boyd committed
1083
1084
1085
;    20190514  Fix bug that occurred when GEOMS variable attribute information is missing from an
;              HDF4 file but written to the heap structure based on the contents of the file. Previously
;              the variable attribute label was not written correctly - Version 4.0b21
Ian Boyd's avatar
Ian Boyd committed
1086
1087
1088
1089
1090
;    20190806  Provide support for reading netCDF4 files. Currently uses the HDF5 routines to do this
;              so need to account for features unique to netCDF4 such as; the inclusion of dimension
;              names; standard netCDF attribute names; empty attribute values that do not include a
;              null termination character (which causes an error in the H5A_Read routine)
;              - Version 4.0b22
Ian Boyd's avatar
Ian Boyd committed
1091
;    20190821  Fixed bugs associated with H5 TAG_NAMES checks introduced in v4.0b22 - Version 4.0b23
Ian Boyd's avatar
Ian Boyd committed
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
;
;  Inputs: ifile - a string containing the name of the input file to be read in.
;          catinfo - a string array identifying the type of input file ('H4','H5','NC')
;
;  Outputs: ga - a string array containing the global attribute labels and values extracted from the HDF
;                file.
;           sds - a structure using pointers, of size [n_sds,n_atts], containing the variable attribute
;                 labels and values (sds[n,m].va_l and sds[n,m].va_v) and the data (sds[n,0].data)
;                 extracted from the HDF file.
;           catinfo - a string array of size [n_sds,4] (where n_sds is the number of datasets in the HDF
;                     file), containing information on the variable name, data type, data dimension, and
;                     number of attributes.
;
;  Called by: IDLCR8ASCII
;
;  Subroutines Called: STOP_WITH_ERROR_A (if error state detected)
;                      INFOTXT_OUTPUT_A (if program can make a change)
;    Possible Conditions for STOP_WITH_ERROR call (plus [line number] where called):
;      1. No global attributes or datasets present in the root group of an HDF5 file.
;      2. The number of global attributes or datasets is zero or not able to be determined
;         in an HDF4 file.
;
;    Information Conditions (when the program is able to make changes):
;      1. DATA_VARIABLES global attribute not found or has no values.
;      2. Number of datasets given under DATA_VARIABLES is not equal to the number saved to the
;         file.
;      3. The number of dimensions in a dataset exceeds 8.
;      4. The data type of a dataset is not BYTE, SHORT, INTEGER, LONG, REAL, DOUBLE, or STRING.
;      5. Dataset name is truncated or does not match VAR_NAME value.
;      6. Metadata label VAR_NAME not found for the dataset.
;      7. VAR_NAME does not match any DATA_VARIABLES values.
;      8. No Variable Attributes returned after HDF call.
;      9. Metadata Dataset listing order may not match the order of the Global Attribute
;         DATA_VARIABLES Values.
;     10. Variable is written to the HDF file as a coordinate variable.

COMMON WIDGET_WIN_A

;Note any calibrated (scaled) HDF4 dataset is corrected using the formula specified in
;UG_print42r3.pdf pg. 3-107: orig = cal * (cal_val - offset)
ncsa_cal=['scale_factor','scale_factor_err','add_offset','add_offset_err','calibrated_nt']

;Initialize scalar to indicate whether VAR_DEPEND and VAR_SIZE values need to be reversed
rev_vd_vs=2 ;1 = reverse attribute values; 0 = no reverse (and will generate message);
;2 = no action required; 3 = rev_vd_vs changed between 0 and 1 during checking, therefore error
Ian Boyd's avatar
Ian Boyd committed
1137
;Note if rev_vd_vs is 2 or 3 after all the checks then it will be changed to 1 so that dimensions
Ian Boyd's avatar
Ian Boyd committed
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
;are swapped by default

;List of Standard GEOMS Variable Attributes
attr_arr_data=['VAR_NAME','VAR_DESCRIPTION','VAR_NOTES','VAR_SIZE','VAR_DEPEND',$
               'VAR_DATA_TYPE','VAR_UNITS','VAR_SI_CONVERSION','VAR_VALID_MIN',$
               'VAR_VALID_MAX','VAR_FILL_VALUE']
n_aad=N_ELEMENTS(attr_arr_data)

;Possible error messages for this procedure
proname='Read_HDF_SDS procedure: '
errtxt=STRARR(2)
errtxt[0]=' found in the Root Group of the HDF5 File.'
errtxt[1]=' is zero or not able to be determined (check for corrupted file).'
lu=-1

;Allowable VAR_DATA_TYPEs
avdt=['BYTE','SHORT','INTEGER','LONG','REAL','DOUBLE','STRING']
slabel=['VAR_UNITS','VAR_SI_CONVERSION','VAR_VALID_MIN','VAR_VALID_MAX','VAR_FILL_VALUE'] ;save values as is if dataset is of type STRING

;If necessary, free up memory by destroying the heap variables pointed at by its pointer arguments
;from previous calls to this program
IF N_ELEMENTS(sds) NE 0 THEN PTR_FREE,sds.va_l,sds.va_v,sds.data

;Define the HDF SDS storage structure
sds_set={va_l: PTR_NEW(), $ ;Variable Attribute Labels
         va_v: PTR_NEW(), $ ;Variable Attribute Values
         data: PTR_NEW()}   ;SD data array

IF catinfo[0,0] EQ 'H4' THEN BEGIN

  ;The HDF_SD_START function opens an HDF file and initializes the SD interface.
  sd_id=HDF_SD_START(ifile,/READ)

  ;Determine the number of SDS (n_sds) and global attributes (n_ga) found
  ;in the current file.
  HDF_SD_FILEINFO,sd_id,n_sds,n_ga
  IF n_sds EQ 0L THEN ntxt='Number of SDS datasets' $
  ELSE IF n_ga EQ 0L THEN ntxt='Number of Global Attributes' $
  ELSE ntxt=''
  IF ntxt NE '' THEN BEGIN
    STOP_WITH_ERROR_A,o3[3]+proname,ntxt+errtxt[1],lu & RETURN
  ENDIF
  ga=STRARR(n_ga) ;set the Global Attribute dimensions
  vn=[''] ;initialize vn array (to hold Dataset names)
  n_sds_hold=n_sds & n_vn=0L & do_coordvar=0

  ;Read the file's Global Attributes and determine number of DATA_VARIABLES
  FOR i=0L,n_ga-1L DO BEGIN
    ;The HDF_SD_ATTRINFO procedure reads or retrieves info about an SD attribute.
    HDF_SD_ATTRINFO,sd_id,i,NAME=ga_name,DATA=ga_data
    ga_name=STRTRIM(ga_name,2) & ga_data=STRTRIM(ga_data,2)
    ;Assign global attributes to ga
    ga[i]=ga_name+'='+ga_data
    ;read list of DATA_VARIABLES into array
    IF STRUPCASE(ga_name) EQ 'DATA_VARIABLES' THEN vn=STRSPLIT(ga_data,' ;',/Extract,COUNT=n_vn)
  ENDFOR

  IF n_sds_hold NE n_vn THEN BEGIN
    ;Do check for sds being a Dimension attribute
    do_coordvar=1
    FOR i=0L,n_sds-1L DO BEGIN
      ;The HDF_SD_SELECT function returns an SD dataset ID given the current
      ;SD interface ID, and the zero-based SD dataset index.
      sds_id=HDF_SD_SELECT(sd_id,i)
      IF HDF_SD_ISCOORDVAR(sds_id) THEN n_sds_hold=n_sds_hold-1L
      ;Closes the SDS interface.
      HDF_SD_ENDACCESS,sds_id
    ENDFOR
    IF n_sds_hold EQ 0L THEN BEGIN
      STOP_WITH_ERROR_A,o3[3]+proname,'Number of SDS datasets'+errtxt[1],lu
      RETURN
    ENDIF

    IF n_sds_hold NE n_vn THEN $
      INFOTXT_OUTPUT_A,[0],[STRTRIM(n_vn,2),STRTRIM(n_sds_hold,2),'HDF_SD_FILEINFO']
      ;DATA_VARIABLES values not read successfully, or number of datasets given under DATA_VARIABLES
      ;is not equal to the number saved to the file
  ENDIF

  ;Determine maximum number of attributes, and match the dataset order to DATA_VARIABLES list if possible
  max_atts=0L & oi=LONARR(n_sds_hold)
  dv_order=1 & c_sds=0L
  catinfo=STRARR(n_sds_hold,4) ;output info for catalog output
  FOR i=0L,n_sds-1L DO BEGIN
    ;The HDF_SD_SELECT function returns an SD dataset ID given the current
    ;SD interface ID, and the zero-based SD dataset index.
    sds_id=HDF_SD_SELECT(sd_id,i)

    IF (do_coordvar EQ 0) OR (NOT HDF_SD_ISCOORDVAR(sds_id)) THEN BEGIN
      ;The HDF_SD_GETINFO procedure retrieves information about an SD dataset.
      HDF_SD_GETINFO,sds_id,NATTS=sds_natts, $ ;no. attributes
                            TYPE=sds_type, $   ;data type
                            DIMS=sds_dim, $    ;dimension information (automatically reversed by IDL)
                            NAME=sds_name      ;dataset name


      ;Check for string attribute and, if so, remove the first dimension
      IF (sds_type EQ 'STRING') AND (N_ELEMENTS(sds_dim) NE 1) THEN sds_dim=sds_dim[1:N_ELEMENTS(sds_dim)-1]

      ;Check for multi-dimensional dataset
      n_sds_dim=N_ELEMENTS(sds_dim)
      IF n_sds_dim GT 1 THEN multi_dim=1

      ;Check number of dimensions
      IF n_sds_dim gt 8 THEN INFOTXT_OUTPUT_A,[1],[sds_name,STRTRIM(n_sds_dim,2)]

      ;Check for coordinate variable
      IF HDF_SD_ISCOORDVAR(sds_id) THEN INFOTXT_OUTPUT_A,[8],[sds_name]

      ;Check that a dataset name has been successfully read
      sds_name=STRTRIM(sds_name,2)
      IF sds_name EQ '' THEN sds_name='N/A'

      ;Rename format to be compatible with the Metadata guidelines
      sds_type=STRTRIM(STRUPCASE(sds_type),2)
      IF sds_type EQ 'INT' THEN sds_type='SHORT' $
      ELSE IF sds_type EQ 'LONG' THEN sds_type='INTEGER' $
      ELSE IF sds_type EQ 'FLOAT' THEN sds_type='REAL'
      vi=WHERE(sds_type EQ avdt,vcnt)
      IF vcnt EQ 0 THEN INFOTXT_OUTPUT_A,[2],[sds_name,sds_type]

      IF sds_natts NE 0L THEN BEGIN
        IF sds_natts GT max_atts THEN max_atts=sds_natts
        vnf=0 & lcnt=0 & vnv='' & vcnt=sds_natts
        ;Extract the variable attributes
        FOR j=0L,sds_natts-1L DO BEGIN
          ;The HDF_SD_ATTRINFO procedure reads or retrieves information about an SD attribute.
          HDF_SD_ATTRINFO,sds_id,j,NAME=va_name,DATA=va_value,TYPE=va_type
          va_name=STRTRIM(va_name,2)
          schk=WHERE(STRUPCASE(va_name) EQ slabel,schkcnt)
          IF ((sds_type EQ 'STRING') AND (schkcnt EQ 0)) OR ((sds_type NE 'STRING') AND (va_type EQ 'STRING')) THEN $
            va_value=STRTRIM(va_value,2)
          ;Check returned dataset attributes (sds_) with saved attributes (va_)
          IF (STRUPCASE(va_name) EQ 'VAR_NAME') AND (vnf EQ 0) THEN BEGIN
            ;check that the dataset name matches the VAR_NAME
            vnf=1
            IF STRUPCASE(va_value) NE STRUPCASE(sds_name) THEN BEGIN
              ;Try and determine which is wrong - Dataset name or VAR_NAME - default is Dataset name
              li=WHERE((STRUPCASE(sds_name) EQ STRUPCASE(vn)) AND (vn[0] NE ''),lcnt)
              ;If lcnt NE 0 then Dataset name matches DATA_VARIABLES value so assume that VAR_NAME is wrong
              IF lcnt NE 0 THEN usesds=1 ELSE usesds=0
              INFOTXT_OUTPUT_A,[3,4,usesds],[sds_name,va_value]
              IF usesds EQ 0 THEN sds_name=va_value ELSE va_value=sds_name
            ENDIF
            vnv=va_value
            ;match the order of the dataset to the DATA_VARIABLES list
            li=WHERE(STRUPCASE(va_value) EQ STRUPCASE(vn),lcnt)
          ENDIF
          IF STRUPCASE(va_name) EQ 'VAR_DATA_TYPE' THEN BEGIN
            va_value=STRUPCASE(va_value)
            IF va_value NE sds_type THEN BEGIN
              ;Dataset is not saved in the same data type as that described by VAR_DATA_TYPE
              IF (sds_type EQ 'INTEGER') AND (va_value EQ 'LONG') THEN $
                itxt=['32-bit INTEGER','64-bit LONG'] $
              ELSE IF (sds_type EQ 'SHORT') AND (va_value EQ 'INTEGER') THEN $
                itxt=['16-bit SHORT','32-bit INTEGER'] $
              ELSE itxt=[sds_type,va_value]
            ENDIF
          ENDIF ELSE IF STRUPCASE(va_name) EQ 'VAR_DEPEND' THEN $ ;do dimension ordering checks
            TEST_DIM_ORDER,'VD',va_value,va_type,sds_dim,rev_vd_vs $
          ELSE IF STRUPCASE(va_name) EQ 'VAR_SIZE' THEN $ ;do dimension ordering checks
            TEST_DIM_ORDER,'VS',va_value,va_type,sds_dim,rev_vd_vs
        ENDFOR
      ENDIF ELSE BEGIN ;No Variable Attributes found
        vcnt=3
        ;Can the SDS_NAME be matched with a DATA_VARIABLES value?
        li=WHERE(STRUPCASE(sds_name) EQ STRUPCASE(vn),lcnt)
        IF lcnt NE 0 THEN vnf=1 ELSE vnf=0
        INFOTXT_OUTPUT_A,[5],[sds_name,'HDF_SD_GETINFO']
      ENDELSE

      ;Check for possible problems with determining array index
      IF lcnt NE 0 THEN BEGIN
        IF li[0] GE n_sds_hold THEN dv_order=0 $ ;can occur when number of DATA_VARIABLE values is greater than number of datasets
        ELSE IF catinfo[li[0],0] NE '' THEN dv_order=0 ;This array has already been written to
      ENDIF ELSE IF (lcnt EQ 0) OR (vnf EQ 0) THEN dv_order=0
      ;Write out information text if sds_name cannot be matched and determine array index
      IF dv_order EQ 0 THEN BEGIN
        li=WHERE(catinfo[*,0] EQ '') ;determine lowest available array index to write info to
        in0=[4,vnf,lcnt,n_vn]
        IF (vnf EQ 0) OR ((lcnt EQ 0) AND (n_vn NE 0L)) THEN INFOTXT_OUTPUT_A,in0,[sds_name,vnv]
      ENDIF

      oi[c_sds]=li[0] ;Attribute order index
      c_sds=c_sds+1L

      catinfo[li[0],0]=sds_name & catinfo[li[0],1]=sds_type
      catinfo[li[0],2]=STRTRIM(sds_dim[0],2) & catinfo[li[0],3]=vcnt
      IF N_ELEMENTS(sds_dim) GT 1 THEN $
        FOR j=1,N_ELEMENTS(sds_dim)-1 DO catinfo[li[0],2]=catinfo[li[0],2]+';'+STRTRIM(sds_dim[j],2)
    ENDIF

    ;Closes the SDS interface.
    HDF_SD_ENDACCESS,sds_id
  ENDFOR

  IF max_atts EQ 0L THEN max_atts=3 ;Use information from HDF_SD_GETINFO call only
Ian Boyd's avatar
Ian Boyd committed
1335
1336
1337
1338
1339
  
  rev_vd_vsh=rev_vd_vs ;keep original value in hold variable (required if rev_vd_vs eq 3)
  IF (rev_vd_vs EQ 2) OR (rev_vd_vs EQ 3) THEN rev_vd_vs=1 ;change default so that VAR_DEPEND and VAR_SIZE 
  ;are reversed if not o/w changed in TEST_DIM_ORDER
  
Ian Boyd's avatar
Ian Boyd committed
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
  ;Dimension the structure to the number of datasets x number of attributes
  sds=REPLICATE(sds_set, n_sds_hold, max_atts)

  ;Write attributes to structure
  c_sds=0L
  !QUIET=1 ;suppress system error and information messages (used for CALDATA call in HDF_SD_GETINFO)
  FOR i=0L,n_sds-1L DO BEGIN
    notranspose=1 ;0/1 will change to 0 if the dataset needs to be transposed
    ;The HDF_SD_SELECT function returns an SD dataset ID given the current
    ;SD interface ID, and the zero-based SD dataset index.
    sds_id=HDF_SD_SELECT(sd_id,i)
    nocal=1 ;Boolean to identify dataset which has been calibrated
    IF (do_coordvar EQ 0) OR (NOT HDF_SD_ISCOORDVAR(sds_id)) THEN BEGIN
      ;The HDF_SD_GETINFO procedure retrieves information about an SD dataset.
      ;Note - any saved pre-defined attributes will be called with HDF_SD_ATTRINFO call
      HDF_SD_GETINFO,sds_id,NATTS=sds_natts, $ ;no. attributes
                            TYPE=sds_type, $   ;data type
                            DIMS=sds_dim, $    ;dimension information
                            NAME=sds_name, $   ;dataset name
                            CALDATA=sds_cal    ;pre-defined calibration info

      ;Check to see whether data has had scale factor and offset applied
      IF (sds_cal.Cal NE 0.D) OR (sds_cal.Offset NE 0.D) THEN BEGIN
        nocal=0 ;Dataset has been calibrated
        ;identify datatype of the original dataset
        CASE 1 OF
          sds_cal.Num_Type EQ 21L: vdt_val='BYTE'
          sds_cal.Num_Type EQ 22L: vdt_val='SHORT'
          sds_cal.Num_Type EQ 24L: vdt_val='INTEGER'
          sds_cal.Num_Type EQ 5L: vdt_val='REAL'
          sds_cal.Num_Type EQ 6L: vdt_val='DOUBLE'
          ELSE: vdt_val='' ;invalid or cannot determine original datatype
        ENDCASE
      ENDIF

      IF sds_natts NE 0L THEN BEGIN
        ;Extract the variable attributes
        jh=0 ;will only loop if attribute is not a pre-defined calibration attribute
        FOR j=0L,sds_natts-1L DO BEGIN
          ;The HDF_SD_ATTRINFO procedure reads or retrieves information about an SD attribute.
          HDF_SD_ATTRINFO,sds_id,j,NAME=va_name,DATA=va_value,TYPE=va_type
          va_name=STRTRIM(va_name,2)

          schk=WHERE(STRUPCASE(va_name) EQ slabel,schkcnt)
          IF ((sds_type EQ 'STRING') AND (schkcnt EQ 0)) OR ((sds_type NE 'STRING') AND (va_type EQ 'STRING')) THEN $
            va_value=STRTRIM(va_value,2)
          IF (nocal EQ 0) AND (STRUPCASE(va_name) EQ 'VAR_DATA_TYPE') THEN BEGIN
            ;If required prepare INFORMATION text (written after DATA has been corrected)
            IF (vdt_val NE STRUPCASE(va_value)) AND (vdt_val NE '') THEN BEGIN
              io_arr=[STRTRIM(sds_name,2),va_value,vdt_val]
              va_value=vdt_val
            ENDIF ELSE BEGIN
              io_arr=[''] & vdt_val=va_value
            ENDELSE
          ENDIF

          IF ((STRUPCASE(va_name) EQ 'VAR_DEPEND') OR (STRUPCASE(va_name) EQ 'VAR_SIZE')) AND $
            ((rev_vd_vs MOD 10 EQ 1) OR (rev_vd_vs EQ 20)) THEN BEGIN ;need to reverse the va_values

            IF va_type NE 'STRING' THEN BEGIN ;VAR_SIZE contains numeric values instead of in the form of a string
              vavhold='' & n_vav=N_ELEMENTS(va_value)
              FOR k=0,n_vav-1 DO BEGIN
                IF k EQ n_vav-1 THEN vtxt='' ELSE vtxt=';'
                vavhold=vavhold+STRTRIM(va_value[k],2)+vtxt
              ENDFOR
            ENDIF ELSE vavhold=va_value

            vs_v=STRCOMPRESS(STRSPLIT(vavhold,';, ',/EXTRACT,COUNT=rcnt),/REMOVE_ALL)
            IF rcnt GT 1 THEN BEGIN ;multi-dimensions
              IF (rev_vd_vs EQ 20) THEN notranspose=0 ;need to transpose dataset as well
              vs_v=REVERSE(vs_v)
              FOR k=0,rcnt-1 DO $
                IF k EQ 0 THEN va_value=vs_v[k] ELSE va_value=va_value+';'+vs_v[k]
            ENDIF
          ENDIF

          ;Test for pre-defined calibration attribute and do not save as the data have
          ;been coverted back to original values
          pi=WHERE(STRLOWCASE(va_name) EQ ncsa_cal,pcnt)
          IF pcnt EQ 0 THEN BEGIN
            sds[oi[c_sds],jh].va_l=PTR_NEW(va_name)
            sds[oi[c_sds],jh].va_v=PTR_NEW(va_value)
            jh=jh+1
          ENDIF
        ENDFOR
      ENDIF ELSE BEGIN ;No Variable Attributes found
        va_lab=['VAR_NAME','VAR_SIZE','VAR_DATA_TYPE']
Ian Boyd's avatar
Ian Boyd committed
1427
        FOR j=0,2 DO sds[oi[c_sds],j].va_l=PTR_NEW(va_lab[j])
Ian Boyd's avatar
Ian Boyd committed
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
        sds[oi[c_sds],0].va_v=PTR_NEW(sds_name)
        sds[oi[c_sds],1].va_v=PTR_NEW(sds_dim)
        ;Rename format to be compatible with the Metadata guidelines
        IF sds_type EQ 'INT' THEN sds_type='SHORT' $
        ELSE IF sds_type EQ 'LONG' THEN sds_type='INTEGER' $
        ELSE IF sds_type EQ 'FLOAT' THEN sds_type='REAL'
        sds[oi[c_sds],2].va_v=PTR_NEW(sds_type)
      ENDELSE
      ;Extract the data
      HDF_SD_GETDATA,sds_id,datasize
      ;Test to see if dataset dimension ordering needs to be changed
      IF notranspose EQ 0 THEN datasize=TRANSPOSE(datasize)

      IF nocal EQ 0 THEN BEGIN
        ;apply scale factor and offset corrections to data and write text to log file
        CASE 1 OF
          vdt_val EQ 'BYTE': datasize=BYTE(sds_cal.Cal*(datasize-sds_cal.Offset))
          vdt_val EQ 'SHORT': datasize=FIX(sds_cal.Cal*(datasize-sds_cal.Offset))
          vdt_val EQ 'INTEGER': datasize=LONG(sds_cal.Cal*(datasize-sds_cal.Offset))
          vdt_val EQ 'REAL': datasize=FLOAT(sds_cal.Cal*(datasize-sds_cal.Offset))
          ELSE: datasize=DOUBLE(sds_cal.Cal*(datasize-sds_cal.Offset))
        ENDCASE
        INFOTXT_OUTPUT_A,[9],[STRTRIM(sds_name,2)]
        IF io_arr[0] NE '' THEN INFOTXT_OUTPUT_A,[10],io_arr
      ENDIF
      sds[oi[c_sds],0].data=PTR_NEW(datasize)
      c_sds=c_sds+1L
    ENDIF

    ;Closes the SDS interface.
    HDF_SD_ENDACCESS,sds_id
  ENDFOR
  !QUIET=0 ;Allow system error and information messages
  ;The HDF_SD_END function closes the SD interface to an HDF file.
  HDF_SD_END,sd_id
Ian Boyd's avatar
Ian Boyd committed
1463
ENDIF ELSE IF catinfo[0,0] EQ 'H5' THEN BEGIN ;HDF5 or netCDF4 format file
Ian Boyd's avatar
Ian Boyd committed
1464
1465
1466
  stdfm=1 ;Boolean indicating standard AVDC/EVDC/NDACC H5 format or not
  dv_order=1 ;Boolean indicating whether Metadata Variable listing order matches DATA_VARIABLES values
  n_vn=0L ;Initialize the number of Variable Names in the standard format h5 file
Ian Boyd's avatar
Ian Boyd committed
1467
1468
1469
1470
1471
1472
1473
1474
  
  ;labels to quietly ignore if the file is netCDF4
  nc4_ignore=['reference_list','dimension_list','class','name','_netcdf4dimid']
  
  ;Do bulk read so errors generated by netCDF4 inconsistencies can be identified without
  ;crashing the program e.g. null string attribute values 
  h5p=H5_PARSE(ifile) 
  
Ian Boyd's avatar
Ian Boyd committed
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
  hdf_file_id=H5F_OPEN(ifile)
  sd_id=H5G_OPEN(hdf_file_id,'/')

  ;Get number of attributes within the root group (these should be the Global Attributes)
  n_ga=H5A_GET_NUM_ATTRS(sd_id)
  ;Get the number of objects within the root group (including datasets, groups etc)
  n_obj=H5G_GET_NMEMBERS(sd_id,'/')

  IF n_obj NE 0L THEN BEGIN
    d_obj=INTARR(n_obj) & g_obj=INTARR(n_obj)
    ;Determine number of datasets.  Note: if H5 file created using H4toH5 then an extra
    ;group will have been created which will not be used.  Also datasets may be listed
    ;alphabetically. This section also determines whether the H5 file is a standard
    ;AVDC/EVDC/NDACC groundbased file or otherwise.
    FOR i=0L,n_obj-1L DO BEGIN
      sds_name=H5G_GET_MEMBER_NAME(sd_id,'/',i)
      h5fstat=H5G_GET_OBJINFO(sd_id,sds_name)
      IF h5fstat.type EQ 'DATASET' THEN d_obj[i]=1 $
      ELSE IF h5fstat.type EQ 'GROUP' THEN g_obj[i]=1
    ENDFOR
    di=WHERE(d_obj EQ 1,n_sds)
    gi=WHERE(g_obj EQ 1,n_grp)
    IF (n_ga EQ 0) AND (n_sds EQ 0) AND (n_grp NE 0) THEN stdfm=0
  ENDIF ELSE n_sds=0L

  IF stdfm EQ 0 THEN catinfo[0,0]='HE5' ;non-standard dataset so need to call READ_NONSTD_H5

  IF n_ga NE 0 THEN BEGIN
    ga=STRARR(n_ga) ;set the Global Attribute dimensions
    vn=[''] ;initialize vn array (to hold Dataset names)
    ;Read in the Global Attribute labels and values
    FOR i=0L,n_ga-1L DO BEGIN
      ga_id=H5A_OPEN_IDX(sd_id,i)
      ga_name=H5A_GET_NAME(ga_id) & ga_name=STRTRIM(ga_name,2)
Ian Boyd's avatar
Ian Boyd committed
1509
1510
            
      ;test for valid attribute value - can be invalid if it is netCDF4 and equal to ''
Ian Boyd's avatar
Ian Boyd committed
1511
      gni=WHERE(TAG_NAMES(h5p) EQ STRUPCASE(ga_name))
Ian Boyd's avatar
Ian Boyd committed
1512
      attest=h5p.(gni[0])
Ian Boyd's avatar
Ian Boyd committed
1513
      IF STRTRIM(attest._data[0],2) EQ '<read error>' THEN ga_data='' $
Ian Boyd's avatar
Ian Boyd committed
1514
1515
1516
      ELSE BEGIN
        ga_data=H5A_READ(ga_id) & ga_data=STRTRIM(ga_data,2)
      ENDELSE
Ian Boyd's avatar
Ian Boyd committed
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
      ;Check for _GLOSDS suffix and strip (added for compatibility with H4toH5 utility)
      IF STRMID(STRUPCASE(ga_name),STRLEN(ga_name)-7) EQ '_GLOSDS' THEN $
        ga_name=STRMID(ga_name,0,STRPOS(ga_name,'_',/Reverse_Search))
      WHILE STRMID(ga_name,STRLEN(ga_name)-1) EQ '_' DO ga_name=STRMID(ga_name,0,STRLEN(ga_name)-1)
      ga[i]=ga_name+'='+ga_data
      H5A_CLOSE,ga_id
      ;Extract the list of the DATA_VARIABLES
      IF STRUPCASE(ga_name) EQ 'DATA_VARIABLES' THEN vn=STRSPLIT(ga_data,' ;',/Extract,COUNT=n_vn)
    ENDFOR
  ENDIF ELSE IF stdfm EQ 1 THEN BEGIN
    STOP_WITH_ERROR_A,o3[3]+proname,'No Global Attributes'+errtxt[0],lu
    RETURN
  ENDIF

  IF n_sds NE 0L THEN BEGIN
Ian Boyd's avatar
Ian Boyd committed
1532
1533
    NC_DIMENSION_CHK,'H5',sd_id,di,n_sds,dgi,n_sdsg ;check for netCDF dimension names (will quietly ignore)
    di=dgi & n_sds=n_sdsg
Ian Boyd's avatar
Ian Boyd committed
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
    IF n_vn NE n_sds THEN $
      INFOTXT_OUTPUT_A,[0],[STRTRIM(n_vn,2),STRTRIM(n_sds,2),'H5G_GET_NMEMBERS']
      ;DATA_VARIABLES values not read successfully, or number of datasets given under DATA_VARIABLES
      ;is not equal to the number saved to the file

    ;Determine maximum number of Attributes, correct dataset order, and dimension order
    max_atts=0L & oi=LONARR(n_sds)
    dv_order=1
    catinfo=STRARR(n_sds,4) ;output info for catalog output
    FOR i=0,n_sds-1 DO BEGIN
      sds_name=H5G_GET_MEMBER_NAME(sd_id,'/',di[i])
      ds_id=H5D_OPEN(sd_id,sds_name)
Ian Boyd's avatar
Ian Boyd committed
1546
1547
1548
      ;need to swap non-alphanumeric characters to '_' to match names from h5_parse for attribute values test
      h5p_name=ALPHA_NUMERIC_UNDERSCORE(STRUPCASE(sds_name))
 
Ian Boyd's avatar
Ian Boyd committed
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
      IF sds_name EQ '' THEN sds_name='N/A'
      datasize=H5D_READ(ds_id)
      sds_dim=SIZE(datasize,/DIMENSIONS) & n_sds_dim=N_ELEMENTS(sds_dim)
      sds_type=SIZE(datasize,/TYPE)

      ;Determine the number of attributes associated with the dataset
      sds_natts=H5A_GET_NUM_ATTRS(ds_id)
      IF sds_natts GT max_atts THEN max_atts=sds_natts

      IF sds_natts NE 0L THEN BEGIN
        vnf=0 & lcnt=0 & vnv=''
        ci=STRARR(4) ;array containing catalog attributes
        ;Read in Dataset Attributes
Ian Boyd's avatar
Ian Boyd committed
1562
        jj=0 ;use this index in the event of invalid attribute names (e.g. dimension_list from netCDF4 file)
Ian Boyd's avatar
Ian Boyd committed
1563
1564
1565
        FOR j=0L,sds_natts-1L DO BEGIN
          da_id=H5A_OPEN_IDX(ds_id,j)
          da_name=H5A_GET_NAME(da_id) & da_name=STRTRIM(da_name,2)
Ian Boyd's avatar
Ian Boyd committed
1566
1567
1568
1569
          
          nc4i=WHERE(STRLOWCASE(da_name) EQ nc4_ignore,nc4icnt)
          IF nc4icnt EQ 0 THEN BEGIN
            ;test for valid attribute value - can be invalid if it is netCDF4 and equal to ''
Ian Boyd's avatar
Ian Boyd committed
1570
            gni=WHERE(TAG_NAMES(h5p) EQ STRUPCASE(h5p_name)) ;Tag index from h5_parse read
Ian Boyd's avatar
Ian Boyd committed
1571
            attest=h5p.(gni[0])
Ian Boyd's avatar
Ian Boyd committed
1572
            gni=WHERE(TAG_NAMES(attest) EQ STRUPCASE(da_name))
Ian Boyd's avatar
Ian Boyd committed
1573
            attestx=attest.(gni[0])
Ian Boyd's avatar
Ian Boyd committed
1574
            IF STRTRIM(attestx._data[0],2) EQ '<read error>' THEN va_value='' $
Ian Boyd's avatar
Ian Boyd committed
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
            ELSE va_value=H5A_READ(da_id)
            IF SIZE(va_value,/TYPE) EQ 7 THEN va_type='STRING' ELSE va_type='NON_STRING'
            schk=WHERE(STRUPCASE(da_name) EQ slabel,schkcnt)
            IF ((sds_type EQ 7) AND (schkcnt EQ 0)) OR ((sds_type NE 7) AND (va_type EQ 'STRING')) THEN $
              va_value=STRTRIM(va_value,2)
            CASE 1 OF
              (STRUPCASE(da_name) EQ 'VAR_NAME') AND (vnf EQ 0):BEGIN
                  vnf=1
                  ;check that the dataset name matches the VAR_NAME
                  IF STRUPCASE(va_value) NE STRUPCASE(sds_name) THEN BEGIN
                    ;Try and determine which is wrong - Dataset name or VAR_NAME - default is Dataset name
                    li=WHERE((STRUPCASE(sds_name) EQ STRUPCASE(vn)) AND (vn[0] NE ''),lcnt)
                    ;If lcnt NE 0 then Dataset name matches DATA_VARIABLES value so assume that VAR_NAME is wrong
                    IF lcnt NE 0 THEN usesds=1 ELSE usesds=0
                    INFOTXT_OUTPUT_A,[3,5,usesds],[sds_name,va_value]
                    IF usesds EQ 0 THEN sds_name=va_value ELSE va_value=sds_name
                  ENDIF
                  ;Determine actual list order from DATA_VARIABLE listing (H5 lists datasets alphabetically)
                  li=WHERE(STRUPCASE(va_value[0]) EQ STRUPCASE(vn),lcnt)
                  ci[0]=va_value & vnv=va_value
                END
              STRUPCASE(da_name) EQ 'VAR_DATA_TYPE': BEGIN
                  ci[1]=va_value
                  vi=WHERE(STRUPCASE(va_value[0]) EQ avdt,vcnt)
                  IF vcnt EQ 0 THEN INFOTXT_OUTPUT_A,[2],[sds_name,va_value]
                END
              STRUPCASE(da_name) EQ 'VAR_DEPEND': BEGIN
                  TEST_DIM_ORDER,'VD',va_value,va_type,sds_dim,rev_vd_vs
                END
              STRUPCASE(da_name) EQ 'VAR_SIZE': BEGIN
                  TEST_DIM_ORDER,'VS',va_value,va_type,sds_dim,rev_vd_vs
                END
              ELSE:
            ENDCASE
            jj++
          ENDIF ;ELSE IF nc4i[0] EQ 4 THEN catinfo[0,0]='H5N' ;netCDF4 file - not implemented yet
Ian Boyd's avatar
Ian Boyd committed
1611
1612
          H5A_CLOSE,da_id
        ENDFOR
Ian Boyd's avatar
Ian Boyd committed
1613
        ci[3]=STRTRIM(jj,2)
Ian Boyd's avatar
Ian Boyd committed
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
        IF ci[0] EQ '' THEN ci[0]=sds_name
      ENDIF ELSE BEGIN ;No Attributes, so can only write dataset name to Metadata
        ci=[STRUPCASE(sds_name),'','','1'] & vnv=sds_name
        ;Can the SDS_NAME be matched with a DATA_VARIABLES value?
        li=WHERE(STRUPCASE(sds_name) EQ STRUPCASE(vn),lcnt)
        IF lcnt NE 0 THEN vnf=1 ELSE vnf=0
        INFOTXT_OUTPUT_A,[5],[sds_name,'H5A_GET_NUM_ATTRS']
      ENDELSE
      FOR k=0,N_ELEMENTS(sds_dim)-1 DO BEGIN
        IF k EQ 0 THEN ci[2]=STRTRIM(sds_dim[k],2) ELSE ci[2]=ci[2]+';'+STRTRIM(sds_dim[k],2)
      ENDFOR

      ;Check for possible problems with determining array index
      IF lcnt NE 0 THEN BEGIN
        IF li[0] GE n_sds THEN dv_order=0 $ ;can occur when number of DATA_VARIABLE values is greater than number of datasets
        ELSE IF catinfo[li[0],0] NE '' THEN dv_order=0 ;This array has already been written to
      ENDIF ELSE IF (lcnt EQ 0) OR (vnf EQ 0) THEN dv_order=0
      ;Write out information text if sds_name cannot be matched and determine array index
      IF dv_order EQ 0 THEN BEGIN
        li=WHERE(catinfo[*,0] EQ '') ;determine lowest available array index to write info to
        in0=[4,vnf,lcnt,n_vn]
        IF (vnf EQ 0) OR ((lcnt EQ 0) AND (n_vn NE 0L)) THEN INFOTXT_OUTPUT_A,in0,[sds_name,vnv]
      ENDIF

      H5D_CLOSE,ds_id
      oi[i]=li[0] ;to put datasets in the correct order
      catinfo[li[0],*]=ci[*]
    ENDFOR
  ENDIF ELSE IF stdfm EQ 1 THEN BEGIN
    STOP_WITH_ERROR_A,o3[3]+proname,'No Datasets'+errtxt[0],lu & RETURN
  ENDIF

Ian Boyd's avatar
Ian Boyd committed
1646
1647
1648
  rev_vd_vsh=rev_vd_vs ;keep original value in hold variable (required if rev_vd_vs eq 3)
  IF (rev_vd_vs EQ 2) OR (rev_vd_vs EQ 3) THEN rev_vd_vs=1 ;change default so that VAR_DEPEND and VAR_SIZE 
  ;are reversed if not o/w changed in TEST_DIM_ORDER
Ian Boyd's avatar
Ian Boyd committed
1649
1650
1651
1652
1653
1654
1655
1656

  ;Dimension the structure to the size of the SDS datasets (with dimension n_sds)
  sds=REPLICATE(sds_set, n_sds, max_atts)
  ;Put datasets and attributes into sds structure
  FOR i=0,n_sds-1 DO BEGIN
    notranspose=1 ;0/1 will change to 0 if the dataset needs to be transposed
    sds_name=H5G_GET_MEMBER_NAME(sd_id,'/',di[i])
    ds_id=H5D_OPEN(sd_id,sds_name)
Ian Boyd's avatar
Ian Boyd committed
1657
1658
1659
    ;need to swap non-alphanumeric characters to '_' to match names from h5_parse for attribute values test
    h5p_name=ALPHA_NUMERIC_UNDERSCORE(STRUPCASE(sds_name))

Ian Boyd's avatar
Ian Boyd committed
1660
1661
1662
1663
1664
1665
1666
1667
    IF sds_name EQ '' THEN sds_name='N/A'
    datasize=H5D_READ(ds_id)
    sds_type=SIZE(datasize,/TYPE)
    ;Determine the number of attributes associated with the dataset
    sds_natts=H5A_GET_NUM_ATTRS(ds_id)

    IF sds_natts NE 0L THEN BEGIN
      ;Read in Dataset Attributes
Ian Boyd's avatar
Ian Boyd committed
1668
      jj=0 ;use this index in the event of invalid attribute names (e.g. dimension_list fro netCDF4 file)
Ian Boyd's avatar
Ian Boyd committed
1669
1670
1671
      FOR j=0L,sds_natts-1L DO BEGIN
        da_id=H5A_OPEN_IDX(ds_id,j)
        da_name=H5A_GET_NAME(da_id) & da_name=STRTRIM(da_name,2)
Ian Boyd's avatar
Ian Boyd committed
1672
1673
1674
        nc4i=WHERE(STRLOWCASE(da_name) EQ nc4_ignore,nc4icnt)
        IF nc4icnt EQ 0 THEN BEGIN
          ;test for valid attribute value - can be invalid if it is netCDF4 and equal to ''
Ian Boyd's avatar
Ian Boyd committed
1675
          gni=WHERE(TAG_NAMES(h5p) EQ STRUPCASE(h5p_name)) ;Tag index from h5_parse read
Ian Boyd's avatar
Ian Boyd committed
1676
          attest=h5p.(gni[0])
Ian Boyd's avatar
Ian Boyd committed
1677
          gni=WHERE(TAG_NAMES(attest) EQ STRUPCASE(da_name))
Ian Boyd's avatar
Ian Boyd committed
1678
          attestx=attest.(gni[0])
Ian Boyd's avatar
Ian Boyd committed
1679
          IF STRTRIM(attestx._data[0],2) EQ '<read error>' THEN va_value='' $
Ian Boyd's avatar
Ian Boyd committed
1680
          ELSE va_value=H5A_READ(da_id)
Ian Boyd's avatar
Ian Boyd committed
1681

Ian Boyd's avatar
Ian Boyd committed
1682
1683
1684
1685
1686
          IF SIZE(va_value,/TYPE) EQ 7 THEN va_type='STRING' ELSE va_type='NON_STRING'
          schk=WHERE(STRUPCASE(da_name) EQ slabel,schkcnt)
          IF ((sds_type EQ 7) AND (schkcnt EQ 0)) OR ((sds_type NE 7) AND (va_type EQ 'STRING')) THEN $
            va_value=STRTRIM(va_value,2)
          H5A_CLOSE,da_id
Ian Boyd's avatar
Ian Boyd committed
1687

Ian Boyd's avatar
Ian Boyd committed
1688
1689
          IF ((STRUPCASE(da_name) EQ 'VAR_DEPEND') OR (STRUPCASE(da_name) EQ 'VAR_SIZE')) AND $
             ((rev_vd_vs MOD 10 EQ 1) OR (rev_vd_vs EQ 20)) THEN BEGIN ;need to reverse the va_values
Ian Boyd's avatar
Ian Boyd committed
1690

Ian Boyd's avatar
Ian Boyd committed
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
            IF va_type NE 'STRING' THEN BEGIN ;VAR_SIZE contains numeric values instead of in the form of a string
              vavhold='' & n_vav=N_ELEMENTS(va_value)
              FOR k=0,n_vav-1 DO BEGIN
                IF k EQ n_vav-1 THEN vtxt='' ELSE vtxt=';'
                vavhold=vavhold+STRTRIM(va_value[k],2)+vtxt
              ENDFOR
            ENDIF ELSE vavhold=va_value

            vs_v=STRCOMPRESS(STRSPLIT(vavhold,';',/EXTRACT,COUNT=rcnt),/REMOVE_ALL)
            IF rcnt GT 1 THEN BEGIN ;multi-dimensions
              IF (rev_vd_vs EQ 20) THEN $ ;AND (STRPOS(STRUPCASE(va_value),'DATETIME') NE -1) THEN $
                notranspose=0 ;need to transpose dataset as well
              vs_v=REVERSE(vs_v)
              FOR k=0,rcnt-1 DO $
                IF k EQ 0 THEN va_value=vs_v[k] ELSE va_value=va_value+';'+vs_v[k]
            ENDIF
Ian Boyd's avatar
Ian Boyd committed
1707
          ENDIF
Ian Boyd's avatar
Ian Boyd committed
1708
1709
1710
          sds[oi[i],jj].va_l=PTR_NEW(da_name)
          sds[oi[i],jj].va_v=PTR_NEW(va_value)
          jj++
Ian Boyd's avatar
Ian Boyd committed
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
        ENDIF
      ENDFOR
    ENDIF ELSE BEGIN ;No Attributes, so can only write dataset name to structure
      val='VAR_NAME'
      sds[oi[i],0].va_l=PTR_NEW(val)
      sds[oi[i],0].va_v=PTR_NEW(sds_name)
    ENDELSE

    H5D_CLOSE,ds_id
    ;Test to see if dataset dimension ordering needs to be changed
    IF notranspose EQ 0 THEN datasize=TRANSPOSE(datasize)
    sds[oi[i],0].data=PTR_NEW(datasize)
  ENDFOR

  H5G_CLOSE,sd_id
  H5F_CLOSE,hdf_file_id
Ian Boyd's avatar
Ian Boyd committed
1727
ENDIF ELSE BEGIN ;netCDF3 file
Ian Boyd's avatar
Ian Boyd committed
1728
1729
  ncdf_desc=['long_name']
  ncdf_std=['units','valid_range','_fillvalue']
Ian Boyd's avatar
Ian Boyd committed
1730
  ncdf_dimen=['constant','independent']
Ian Boyd's avatar
Ian Boyd committed
1731
1732

  dv_order=1 ;Boolean indicating whether Metadata Variable listing order matches DATA_VARIABLES values
Ian Boyd's avatar
Ian Boyd committed
1733
  n_vn=0L ;Initialize the number of Variable Names in the standard format netCDF file
Ian Boyd's avatar
Ian Boyd committed
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769

  fileid=NCDF_OPEN(ifile)
  ;Get number of global atts and variables
  filestruct=NCDF_INQUIRE(fileid)
  n_dim=filestruct.ndims
  n_ga=filestruct.ngatts
  n_sds=filestruct.nvars

  IF n_sds EQ 0L THEN ntxt='Number of SDS datasets' $
  ELSE IF n_ga EQ 0L THEN ntxt='Number of Global Attributes' $
  ELSE ntxt=''
  IF ntxt NE '' THEN BEGIN
    STOP_WITH_ERROR_A,o3[3]+proname,ntxt+errtxt[1],lu & RETURN
  ENDIF

  ;read global attributes
  ga=STRARR(n_ga) & vn=['']
  FOR j=0,n_ga-1 DO BEGIN
    ga_name=NCDF_ATTNAME(fileid,j,/GLOBAL)
    NCDF_ATTGET,fileid,ga_name,ga_data,/GLOBAL
    ga_data=STRTRIM(ga_data,2)
    ga[j]=STRTRIM(STRUPCASE(ga_name),2)+'='+ga_data
    ;read list of DATA_VARIABLES into array
    IF STRUPCASE(ga_name) EQ 'DATA_VARIABLES' THEN vn=STRSPLIT(ga_data,' ;',/Extract,COUNT=n_vn)
  ENDFOR

  ;read dimension information
  IF n_dim NE 0 THEN BEGIN
    mv_dim=STRARR(n_dim,2)
    FOR j=0,n_dim-1 DO BEGIN
      NCDF_DIMINQ,fileid,j,dimname,dimsize
      mv_dim[j,0]=STRTRIM(dimname,2)
      mv_dim[j,1]=STRTRIM(dimsize,2)
    ENDFOR
  ENDIF ELSE mv_dim=['']

Ian Boyd's avatar
Ian Boyd committed
1770
1771
1772
1773
  di=INDGEN(n_sds)
  NC_DIMENSION_CHK,'NC',fileid,di,n_sds,dgi,n_sdsg
  di=dgi & n_sds=n_sdsg ;di contains index values of valid sds datasets

Ian Boyd's avatar
Ian Boyd committed
1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
  IF n_vn NE n_sds THEN $
    INFOTXT_OUTPUT_A,[0],[STRTRIM(n_vn,2),STRTRIM(n_sds,2),'NCDF_INQUIRE']
    ;DATA_VARIABLES values not read successfully, or number of datasets given under DATA_VARIABLES
    ;is not equal to the number saved to the file

  ;Determine maximum number of Attributes, correct dataset order, and dimension order
  max_atts=0L & oi=LONARR(n_sds)
  catinfo=STRARR(n_sds,4) ;output info for catalog output

  FOR j=0,n_sds-1 DO BEGIN
Ian Boyd's avatar
Ian Boyd committed
1784
    varstruct=NCDF_VARINQ(fileid,di[j])
Ian Boyd's avatar
Ian Boyd committed
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
    sds_natts=varstruct.natts
    IF sds_natts GT max_atts THEN max_atts=sds_natts
    sds_name=varstruct.name
    sds_datatype=varstruct.datatype
    sds_dimid=varstruct.dim
    sds_ndim=varstruct.ndims

    ;determine the actual dimension values from the dim IDs given in varstruct.dim
    IF sds_ndim NE 0 THEN BEGIN
      sds_dim=LONARR(sds_ndim)
      FOR k=0,sds_ndim-1 DO sds_dim[k]=mv_dim[sds_dimid[k],1]
    ENDIF ELSE sds_dim=[1L]

    ;Read Variable Attributes
    IF sds_natts NE 0L THEN BEGIN
      vnf=0 & lcnt=0 & vnv=''
      ci=STRARR(4) ;array containing catalog attributes
      std_fnd=INTARR(n_aad)
      ;Read in Dataset Attributes
      FOR k=0,sds_natts-1 DO BEGIN
Ian Boyd's avatar
Ian Boyd committed
1805
1806
        varattsname=NCDF_ATTNAME(fileid,di[j],k)
        NCDF_ATTGET,fileid,di[j],varattsname,va_value
Ian Boyd's avatar
Ian Boyd committed
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
        va_name=STRUPCASE(varattsname)
        vavtype=SIZE(va_value,/TYPE)

        IF (vavtype EQ 7) OR (vavtype EQ 1) THEN va_type='STRING' ELSE va_type='NON-STRING'

        schk=WHERE(va_name EQ slabel,schkcnt)
        test1=(sds_datatype EQ 'CHAR') AND (schkcnt EQ 0) ;string dataset but not one of the slabel values
        test2=(sds_datatype NE 'CHAR') AND (va_type EQ 'STRING') ;not a string dataset but string attribute value
        test3=(sds_datatype EQ 'CHAR') AND (schkcnt NE 0) AND (vavtype EQ 1) ;string dataset, slabel and attribute value is of type byte
        IF (test1) OR (test2) THEN va_value=STRTRIM(va_value,2) $
        ELSE IF test3 THEN va_value=STRING(va_value)

        CASE 1 OF
          (STRUPCASE(va_name) EQ 'VAR_NAME') AND (vnf EQ 0):BEGIN
              vnf=1
              ;check that the dataset name matches the VAR_NAME
              IF STRUPCASE(va_value) NE STRUPCASE(sds_name) THEN BEGIN
                ;Try and determine which is wrong - Dataset name or VAR_NAME - default is Dataset name
                li=WHERE((STRUPCASE(sds_name) EQ STRUPCASE(vn)) AND (vn[0] NE ''),lcnt)
                ;If lcnt NE 0 then Dataset name matches DATA_VARIABLES value so assume that VAR_NAME is wrong
                IF lcnt NE 0 THEN usesds=1 ELSE usesds=0
                INFOTXT_OUTPUT_A,[3,5,usesds],[sds_name,va_value]
                IF usesds EQ 0 THEN sds_name=va_value ELSE va_value=sds_name
              ENDIF
              ;Determine actual list order from DATA_VARIABLE listing
              li=WHERE(STRUPCASE(va_value[0]) EQ STRUPCASE(vn),lcnt)
              ci[0]=va_value & vnv=va_value
            END
          STRUPCASE(va_name) EQ 'VAR_DATA_TYPE': BEGIN
              ci[1]=va_value
              vi=WHERE(STRUPCASE(va_value[0]) EQ avdt,vcnt)
              IF vcnt EQ 0 THEN INFOTXT_OUTPUT_A,[2],[sds_name,va_value]
            END
          STRUPCASE(va_name) EQ 'VAR_DEPEND': BEGIN
              TEST_DIM_ORDER,'VD',va_value,va_type,sds_dim,rev_vd_vs
            END
          STRUPCASE(va_name) EQ 'VAR_SIZE': BEGIN
              TEST_DIM_ORDER,'VS',va_value,va_type,sds_dim,rev_vd_vs
            END
          ELSE:
        ENDCASE
      ENDFOR
      ci[3]=STRTRIM(sds_natts,2)
      IF ci[0] EQ '' THEN ci[0]=sds_name
    ENDIF ELSE BEGIN ;No Attributes, so can only write dataset name to Metadata
      ci=[STRUPCASE(sds_name),'','','1'] & vnv=sds_name
      ;Can the SDS_NAME be matched with a DATA_VARIABLES value?
      li=WHERE(STRUPCASE(sds_name) EQ STRUPCASE(vn),lcnt)
      IF lcnt NE 0 THEN vnf=1 ELSE vnf=0
      INFOTXT_OUTPUT_A,[5],[sds_name,'NCDF_VARINQ']
    ENDELSE
    FOR k=0,N_ELEMENTS(sds_dim)-1 DO BEGIN
      IF k EQ 0 THEN ci[2]=STRTRIM(sds_dim[k],2) ELSE ci[2]=ci[2]+';'+STRTRIM(sds_dim[k],2)
    ENDFOR

    ;Check for possible problems with determining array index
    IF lcnt NE 0 THEN BEGIN
      IF li[0] GE n_sds THEN dv_order=0 $ ;can occur when number of DATA_VARIABLE values is greater than number of datasets
      ELSE IF catinfo[li[0],0] NE '' THEN dv_order=0 ;This array has already been written to
    ENDIF ELSE IF (lcnt EQ 0) OR (vnf EQ 0) THEN dv_order=0
    ;Write out information text if sds_name cannot be matched and determine array index
    IF dv_order EQ 0 THEN BEGIN
      li=WHERE(catinfo[*,0] EQ '') ;determine lowest available array index to write info to
      in0=[4,vnf,lcnt,n_vn]
      IF (vnf EQ 0) OR ((lcnt EQ 0) AND (n_vn NE 0L)) THEN INFOTXT_OUTPUT_A,in0,[sds_name,vnv]
    ENDIF

    oi[j]=li[0] ;to put datasets in the correct order
    catinfo[li[0],*]=ci[*]
  ENDFOR

Ian Boyd's avatar
Ian Boyd committed
1878
1879
1880
1881
  rev_vd_vsh=rev_vd_vs ;keep original value in hold variable (required if rev_vd_vs eq 3)
  IF (rev_vd_vs EQ 2) OR (rev_vd_vs EQ 3) THEN rev_vd_vs=1 ;change default so that VAR_DEPEND and VAR_SIZE
  ;are reversed if not o/w changed in TEST_DIM_ORDER
 
Ian Boyd's avatar
Ian Boyd committed
1882
1883
1884
1885
1886
  ;Dimension the structure to the size of the SDS datasets (with dimension n_sds)
  sds=REPLICATE(sds_set, n_sds, max_atts)
  ;Put datasets and attributes into sds structure
  FOR j=0,n_sds-1 DO BEGIN
    notranspose=1 ;0/1 will change to 0 if the dataset needs to be transposed
Ian Boyd's avatar
Ian Boyd committed
1887
    varstruct=NCDF_VARINQ(fileid,di[j])
Ian Boyd's avatar
Ian Boyd committed
1888
1889
1890
1891
1892
1893