Documentation for /home/davin/idl/socware/

Generated by IDLdoc

single page | use frames     summary     class     fields     routine details     file attributes

external/CDAWlib/

read_myCDF.pro


This package of IDL functions facilitates reading data and metadata from Common Data Format (CDF) files. While CDF provides all the benefits of a portable, self-documenting scientific data format, reading them is not always a simple matter. To make it simple, I have created this IDL package so that all of the data and metadata from multiple variables can be read from multiple CDF files ... in one single, simple command. The function is called 'READ_MYCDF' and it returns an anonymous structure of the form: structure_name.variable_name.attribute_name.attribute_value From this structure, all data and metadata for the requested variables is easily accessed. AUTHOR: Richard Burley, NASA/GSFC/Code 632.0, Feb 13, 1996 burley@nssdca.gsfc.nasa.gov (301)286-2864 NOTES: Three additional 'attributes' will be included in the sub-structure for each variable. The first is the 'VARNAME' field. Because IDL structure tags are always uppercase, and because CDF variable names are case sen- sitive, a case sensitive copy of the variable name is created. The second 'attribute' to be added is the 'CDFTYPE' field. This field will hold a string value holding the cdf data type. The last 'attribute' to be artificially added will be either the 'DAT' field or, if the keyword NODATASTRUCT is set, the 'HANDLE' field. The 'DAT' field will contain the actual data values read from the CDF's for the variable. The 'HANDLE' field will hold a handle_id where the data will reside. This package will look for and utilize certain special attributes required by the International Solar Terrestrial Physics Key Parameters Generation Software Standards and Guidelines. The existance of these attributes is not required for the operation of this software, but will enhance its usefullness, primarily by reading variables that will be needed for proper utilization of the data, even though you may not have asked for them explicitly. This package was tested under IDL version 4.0.1b. This package was tested on CDF's up to version 2.5 and on both r-variables and z-variables. CDF variables defined as unsigned integers are, unfortunately, currently returned by the IDL CDF_VARGET procedure as signed integers. This can cause sign flips. This software detects and corrects for this defect for data values. However, it cannot detect and correct for this defect for attribute values because the IDL procedure CDF_ATTINQ does not return the CDF data type of the attribute. These problems have been reported to RSI. Modifications: As of October 2, 2000, this software can run on all of the following IDL versions, 5.1, 5.2 and 5.3 (testing for 5.4 will commence soon). Some fairly major changes were necessary in order for read_myCDF to work under 5.3. IDL 5.3 enforces the variable naming rules for structure tag names. This change affects this s/w because we basically had never checked our tag names, e.g. we used the CDF variable names and label attribute values directly. So in read_myCDF the general concept to fixing this problem was to set up a table (which is shared in a common block - not my favorite way to go, but definitely the easiest), where there are two tags, equiv and varname. varname contains the real CDF variable name, equiv contains the "cleaned up, IDL acceptable" variable name that can be used as a structure tag name... TJK 04/02/2000 1996, NASA/Goddard Space Flight Center This software may be used, copied, or redistributed as long as it is not sold and this copyright notice is reproduced on each copy made. This routine is provided as is without any express or implied warranties whatsoever.

RCJ 03/30/2012 Commented out this function. It was called once and that call is commented out too. ;check the variables_comp array for existence of the variable name, for ;the current cdf. function check_varcompare, variables_comp, cdf_index, variable_name ;print,'**** ' & help,variables_comp ;print, variable_name, cdf_index, variables_comp ;stop; x = where(variable_name eq variables_comp(cdf_index,*), xcnt) if (xcnt gt 0)then print, variable_name, ' found 1' else print, variable_name, ' not found 0' if (xcnt gt 0)then return, 1 else return, 0 end +------------------------------------------------------------------------ NAME: READ_MYCDF PURPOSE: Read all data and metadata for given variables, from given CDF files, and return all information in a single anonymous structure of the form: structure_name.variable_name.attribute_name.attribute_value CALLING SEQUENCE: out = read_mycdf(vnames,cnames) INPUTS: vnames = string, array of variable names or a single string of names separated by a comma. (ex. 'Epoch,Magfld,Bmax') cnames = string, array of CDF filenames or a single string of names separated by a comma. KEYWORD PARAMETERS: ALL = 0: get data and metadata for requested variable(s) only. 1: get data and metadata for ALL variables in the CDFs. 2: get data and metadata for all var_type='data' variables. NODATASTRUCT = If set, instead of returning the data for each variable in the 'DAT' attribute field, create a 'HANDLE' field and set it to the handle id of a data handle which holds the data for each variable. NOQUIET = If set, do NOT set the !QUIET system variable before reading the cdf file(s). DEBUG = If set, print out some progress information during reading. TSTART = epoch starting value - YYYYMMDD etc. string. TSTOP = epoch ending value - YYYYMMDD etc. string. OUTPUTS: out = anonymous structure holding all data and metadata for the requested variables. If an error occurs, that we know how to deal w/, an alternate structure is returned, its structure is as follows: ('DATASET',d_set,'ERROR',v_err,'STATUS',v_stat) AUTHOR: Richard Burley, NASA/GSFC/Code 632.0, Feb 13, 1996 burley@nssdca.gsfc.nasa.gov (301)286-2864 MODIFICATION HISTORY: Tami Kovalick, HSTX, 12/16/96 modified to verify whether variables requested in vnames array are actually in the "data" cdfs prior to requesting the data from these variables. If variables aren't valid then they are removed from the vnames array and the code continues on to create a valid structure. Tami Kovalick, HSTX, 12/20/96 modified to allow the use of TSTART and TSTOP keywords (see above). Use of these keywords will force the code to only read the necessary records in the CDF, otherwise the code will read the entire CDF. Could enhance the code to deal w/ one or the other keyword - right now they are only used if both are set. Tami Kovalick, RSTX, 02/13/98, Carrie Gallap started modifications to read_myCDF to accommodate "virtual variables" (VV) . Tami finished up the code and made corrections to several sections. One new routine was written add_myCOMPONENTS, this routine is called when a valid virtual variable is found in order to add any additional variables needed for actually generating the data for the VV. The routine looks for variable attributes w/ the naming convention COMPONENT_n where n is a digit. The basic methodology to the changes is to determine whether any of the variables selected are virtual variables, if so then the variable name and the source (where the VV was defined - master or data cdfs) are stored in a structure called vir_vars, then add the component variables to the vnames array. Do the usual checking to see if the variables requested in vnames actually exist. Then continue on w/ getting the metadata for all variables (including VV), and continue on w/ the getting the data from the CDFs for all variables except the VV. Population of the VV's data field in the "burley" structure are handled at the very end in a case statement which looks for each VV's variable attribute FUNCTION to determine which actual "IDL function" to call, ie. conv_pos.

Routines

Routines from read_myCDF.pro

result = break_mySTRING(s, DELIMITER=DELIMITER)
result = amI_ISTPptr(aname)

Search the tnames array for the instring, returning the index in tnames if it is present, or -1 if it is not.

result = amI_VAR(aname)

NAME: AMI_VAR PURPOSE: Return true(1) or false(0) depending on whether or not the given attribute name's value is assigned to a real CDF variable name.

result = parse_DISPLAY_TYPE(instring)

NAME: PARSE_DISPLAY_TYPE PURPOSE: Parse and examine the input string.

result = correct_vnames(vnames)
follow_myDEPENDS, metadata, vnames, vvarys, ctypes, dhids, mhids

NAME: follow_myDEPENDS PURPOSE: Search the metadata anonymous structure for ISTP 'DEPEND' attributes.

result = CDFtype_to_myIDLtype(cdftype)
result = append_myDATA(new, old, dict_key=dict_key, vector=vector)

NAME: APPEND_MYDATA PURPOSE: Append the 'new' data to the 'old' data using array concatenation.

add_myDEPENDS, metadata, vnames

NAME: add_myDEPENDS PURPOSE: Search the metadata anonymous structure for ISTP 'DEPEND' attributes and add the variable name that it points to to the vnames array if it is not already present.

add_myCOMPONENTS, metadata, vnames

NAME: add_myCOMPONENTS PURPOSE: Search the metadata anonymous structure for ISTP 'COMPONENT' attributes and add the variable name that it points to to the vnames array if it is not already present.

add_myDELTAS, metadata, vnames
result = read_myVARIABLE(vname, CDFid, vary, dtype, recs, START_REC=START_REC, REC_COUNT=REC_COUNT, DEBUG=DEBUG)

NAME: READ_MYVARIABLE PURPOSE: Return the data for the requested variable.

result = read_myATTRIBUTE(vname, anum, CDFid, isglobal=isglobal)

NAME: READ_MYATTRIBUTE PURPOSE: Return the value of the requested attribute for the requested variable.

result = read_myMETADATA(vname, CDFid)

NAME: READ_MYMETADATA PURPOSE: To read all of the attribute values for the requested variable, and to return this information as an anonymous structure.

result = getvar_attribute_names(CDFid, ALL=ALL)

NAME: Getvar_attribute_names CDFid PURPOSE: To return all of the attribute names for the requested variable, as an array.

result = get_numallvars(CNAME=CNAME, CDFid=CDFid)

NAME: GET_NUMALLVARS PURPOSE: To return the total number of variables in the cdf.

result = get_allvarnames(CNAME=CNAME, CDFid=CDFid, VAR_TYPE=VAR_TYPE)

NAME: GET_ALLVARNAMES PURPOSE: To return a string array containing the names of all of the variables in the given CDF file.

result = write_fill(vn_sdat, burley, tmp_str)
result = correct_varname(struct, varnames, index)
result = find_var(CDFid, variable)
result = find_epochvar(CDFid)
result = merge_metadata(cnames, base_struct, all=all)
result = unique_array(first, second)
result = read_myCDF(vnames, cnames, ALL=ALL, NODATASTRUCT=NODATASTRUCT, NOQUIET=NOQUIET, DEBUG=DEBUG, TSTART=TSTART, TSTOP=TSTOP, START_MSEC=START_MSEC, STOP_MSEC=STOP_MSEC, START_USEC=START_USEC, STOP_USEC=STOP_USEC, START_NSEC=START_NSEC, STOP_NSEC=STOP_NSEC, START_PSEC=START_PSEC, STOP_PSEC=STOP_PSEC, NOVIRTUAL=NOVIRTUAL)

Routine details

top source break_mySTRING

result = break_mySTRING(s, DELIMITER=DELIMITER)

Parameters

s

Keywords

DELIMITER

top source amI_ISTPptr

result = amI_ISTPptr(aname)

Search the tnames array for the instring, returning the index in tnames if it is present, or -1 if it is not. TJK this function is in a separate file called TAGindex.pro since its called from many different routines in this system. FUNCTION TAGindex, instring, tnames instring = STRUPCASE(instring) ; tagnames are always uppercase a = where(tnames eq instring,count) if count eq 0 then return, -1 $ else return, a(0) end +------------------------------------------------------------------------ NAME: AMI_ISTPPTR PURPOSE: Return true(1) or false(0) depending on whether or not the given attribute name qualifies as an ISTP pointer-class attribute. CALLING SEQUENCE: out = amI_ISTPptr(attribute_name) INPUTS: attribute_name = name of a CDF attribute as a string KEYWORD PARAMETERS: OUTPUTS: True(1) or False(0) AUTHOR: Richard Burley, NASA/GSFC/Code 632.0, Feb 13, 1996 burley@nssdca.gsfc.nasa.gov (301)286-2864 MODIFICATION HISTORY:

Parameters

aname

top source amI_VAR

result = amI_VAR(aname)

NAME: AMI_VAR PURPOSE: Return true(1) or false(0) depending on whether or not the given attribute name's value is assigned to a real CDF variable name. CALLING SEQUENCE: out = amI_VAR(attribute_name) INPUTS: attribute_name = name of a CDF attribute as a string KEYWORD PARAMETERS: OUTPUTS: True(1) or False(0) AUTHOR: Tami Kovalick March 6, 2000 MODIFICATION HISTORY:

Parameters

aname

top source parse_DISPLAY_TYPE

result = parse_DISPLAY_TYPE(instring)

NAME: PARSE_DISPLAY_TYPE PURPOSE: Parse and examine the input string. It should be the value of the CDF attribute 'DISPLAY_TYPE'. Return an array of variable names that it 'points' to. CALLING SEQUENCE: out = parse_display_type(instring) INPUTS: instring = string, value of a CDF attribute called 'DISPLAY_TYPE' KEYWORD PARAMETERS: OUTPUTS: out = string array, names of other variables required for display NOTES: This routine expects to find 'DISPLAY_TYPE' values looking like: PLOT_TYPE>x=vname,y=vname ... PLOT_TYPE>y=vname,z=vname(*,1),z=vname(*,2) ... AUTHOR: Richard Burley, NASA/GSFC/Code 632.0, Feb 13, 1996 burley@nssdca.gsfc.nasa.gov (301)286-2864 MODIFICATION HISTORY: TJK modified 01/27/98 to not parse orbit display type here - the items specified for the orbit plot type aren't additional variables. TJK modified 09/25/2001 to not parse the "symsize" keyword because its value isn't a variable.

Parameters

instring

top source correct_vnames

result = correct_vnames(vnames)

Parameters

vnames

top source follow_myDEPENDS

follow_myDEPENDS, metadata, vnames, vvarys, ctypes, dhids, mhids

NAME: follow_myDEPENDS PURPOSE: Search the metadata anonymous structure for ISTP 'DEPEND' attributes. If and when found, add the variable name that it points to to the vnames array if it is not already present, and increase the size of the dhids and mhids arrays. CALLING SEQUENCE: follow_myDEPENDS, metadata, vnames, dhids, mhids INPUTS: metadata = anonymous structure holding attribute values vnames = string array of the names of variables already processed vvarys = string array of the record variance for each variable dhids = array of data handle id's mhids = array of metadata handle id's KEYWORD PARAMETERS: OUTPUTS: dhids = array of data handle id's mhids = array of metadata handle id's AUTHOR: Richard Burley, NASA/GSFC/Code 632.0, Feb 13, 1996 burley@nssdca.gsfc.nasa.gov (301)286-2864 MODIFICATION HISTORY:

Parameters

metadata
vnames
vvarys
ctypes
dhids
mhids

top source CDFtype_to_myIDLtype

result = CDFtype_to_myIDLtype(cdftype)

Parameters

cdftype

top source append_myDATA

result = append_myDATA(new, old, dict_key=dict_key, vector=vector)

NAME: APPEND_MYDATA PURPOSE: Append the 'new' data to the 'old' data using array concatenation. CALLING SEQUENCE: out = append_mydata(new,old) INPUTS: new = data to be appended to the old data old = older data that new data is to be appended to KEYWORD PARAMETERS: OUTPUTS: out = product of concatenating the old and new data arrays NOTES: Special case check: if old data was from either a skeleton CDF or from a CDF with only a single record, then the last dimension was dropped during the process of saving/retrieving the data from a handle. Must compare the dimensionality of the new and old data to determine if this drop has occured, and if so, reform the old data to include the extra dimension so that the data can be appended. AUTHOR: Richard Burley, NASA/GSFC/Code 632.0, Feb 13, 1996 burley@nssdca.gsfc.nasa.gov (301)286-2864 MODIFICATION HISTORY:

Parameters

new
old

Keywords

dict_key
vector

top source add_myDEPENDS

add_myDEPENDS, metadata, vnames

NAME: add_myDEPENDS PURPOSE: Search the metadata anonymous structure for ISTP 'DEPEND' attributes and add the variable name that it points to to the vnames array if it is not already present. If the DEPEND variable is not present in the list, change the data_type so it won't be plotted. CALLING SEQUENCE: add_myDEPENDS, metadata, vnames INPUTS: metadata = anonymous structure holding attribute values vnames = string array of virtual variables found OUTPUTS: vnames = modified variable name that includes component variable names NOTES - this is similar to follow_myDEPENDS, except it does less. AUTHOR: Tami Kovalick, QSS, 11/29/2006

Parameters

metadata
vnames

top source add_myCOMPONENTS

add_myCOMPONENTS, metadata, vnames

NAME: add_myCOMPONENTS PURPOSE: Search the metadata anonymous structure for ISTP 'COMPONENT' attributes and add the variable name that it points to to the vnames array if it is not already present. If the component variable is not present in the list, change the data_type so it won't be plotted. CALLING SEQUENCE: add_myCOMPONENTS, metadata, vnames INPUTS: metadata = anonymous structure holding attribute values vnames = string array of virtual variables found OUTPUTS: vnames = modified variable name that includes component variable names AUTHOR: Carrie Gallap, Raytheon STX, 1/5/98

Parameters

metadata
vnames

top source add_myDELTAS

add_myDELTAS, metadata, vnames

Parameters

metadata
vnames

top source read_myVARIABLE

result = read_myVARIABLE(vname, CDFid, vary, dtype, recs, START_REC=START_REC, REC_COUNT=REC_COUNT, DEBUG=DEBUG)

NAME: READ_MYVARIABLE PURPOSE: Return the data for the requested variable. CALLING SEQUENCE: out = read_myvariable(vname, CDFid, vary, dtype, recs) INPUTS: vname = string, name of variable to be read from the CDF CDFid = integer, id or already opened CDF file. KEYWORD PARAMETERS: START_REC = first record to read. REC_COUNT = number of records to read. OUTPUTS: out = all data from the CDF for the variable being read vary = True(1) or False(0) is variable record-varying dtype= string, CDF data type recs = integer, number of data records AUTHOR: Richard Burley, NASA/GSFC/Code 632.0, Feb 13, 1996 burley@nssdca.gsfc.nasa.gov (301)286-2864 MODIFICATION HISTORY: 96/04/11 : R.Burley :zVar handling when MAXRECS = -1 changed to read REC_COUNT of MAXRECS + 2 & return,DAT 96/12/20 ; T. Kovalick modified to take START_REC and REC_COUNT keywords (see above). If they aren't set you will get all of the records in a cdf.

Parameters

vname
CDFid
vary
dtype
recs

Keywords

START_REC
REC_COUNT
DEBUG

top source read_myATTRIBUTE

result = read_myATTRIBUTE(vname, anum, CDFid, isglobal=isglobal)

NAME: READ_MYATTRIBUTE PURPOSE: Return the value of the requested attribute for the requested variable. CALLING SEQUENCE: out = read_myattribute(vname,anum,CDFid) INPUTS: vname = string, name of variable whose attribute is being read anum = integer, number of attribute being read CDFid = integer, id of already opened CDF file. KEYWORD PARAMETERS: OUTPUTS: out = anonymous structure holding both the name of the attribute and the value of the attribute AUTHOR: Richard Burley, NASA/GSFC/Code 632.0, Feb 13, 1996 burley@nssdca.gsfc.nasa.gov (301)286-2864 MODIFICATION HISTORY: RCJ 11/2003 Added keyword isglobal

Parameters

vname
anum
CDFid

Keywords

isglobal

top source read_myMETADATA

result = read_myMETADATA(vname, CDFid)

NAME: READ_MYMETADATA PURPOSE: To read all of the attribute values for the requested variable, and to return this information as an anonymous structure. CALLING SEQUENCE: metadata = read_mymetadata(vname,CDFid) INPUTS: vname = string, name of variable whose metadata is being read CDFid = integer, id of already opened CDF file KEYWORD PARAMETERS: OUTPUTS: metadata = anonymous structure whose tags are the attribute names and whose fields are the corresponding attribute values. AUTHOR: Richard Burley, NASA/GSFC/Code 632.0, Feb 13, 1996 burley@nssdca.gsfc.nasa.gov (301)286-2864 MODIFICATION HISTORY:

Parameters

vname
CDFid

top source getvar_attribute_names

result = getvar_attribute_names(CDFid, ALL=ALL)

NAME: Getvar_attribute_names CDFid PURPOSE: To return all of the attribute names for the requested variable, as an array. CALLING SEQUENCE: att_array = getvar_attribute_names(vname,CDFid, ALL=ALL) INPUTS: CDFid = integer, id of already opened CDF file KEYWORD PARAMETERS: ALL - all attributes are returned default is that just variable scoped attributes are returned OUTPUTS: att_array = string array of attribute names AUTHOR: Tami Kovalick tami.kovalick@gsfc.nasa.gov (301)286-9422 MODIFICATION HISTORY:

Parameters

CDFid

Keywords

ALL

top source get_numallvars

result = get_numallvars(CNAME=CNAME, CDFid=CDFid)

NAME: GET_NUMALLVARS PURPOSE: To return the total number of variables in the cdf. CALLING SEQUENCE: num_vars = get_numallvars(CNAME=CNAME) INPUTS: KEYWORD PARAMETERS: CNAME = string, name of a CDF file to be opened and read CDFid = integer, id of an already opened CDF file OUTPUTS: num_vars = number of variables in the CDF AUTHOR: Tami Kovalick, RITSS, October 27, 2000 MODIFICATION HISTORY:

Keywords

CNAME
CDFid

top source get_allvarnames

result = get_allvarnames(CNAME=CNAME, CDFid=CDFid, VAR_TYPE=VAR_TYPE)

NAME: GET_ALLVARNAMES PURPOSE: To return a string array containing the names of all of the variables in the given CDF file. CALLING SEQUENCE: vnames = get_allvarnames() INPUTS: KEYWORD PARAMETERS: CNAME = string, name of a CDF file to be opened and read CDFid = integer, id of an already opened CDF file VAR_TYPE = string, only return the names for variables who have an attribute called 'VAR_TYPE' and whose value matches the value given by this keyword. (ex. VAR_TYPE='data') OUTPUTS: vnames = string array of variable names AUTHOR: Richard Burley, NASA/GSFC/Code 632.0, Feb 13, 1996 burley@nssdca.gsfc.nasa.gov (301)286-2864 MODIFICATION HISTORY: 4/9/1998 - TJK modified to include all variable when the "var_type" keyword isn't used. The original code only included variables that vary by record so some important "support_data" variables were being thrown out.

Keywords

CNAME
CDFid
VAR_TYPE

top source write_fill

result = write_fill(vn_sdat, burley, tmp_str)

Parameters

vn_sdat
burley
tmp_str

top source correct_varname

result = correct_varname(struct, varnames, index)

Parameters

struct
varnames
index

top source find_var

result = find_var(CDFid, variable)

Parameters

CDFid
variable

top source find_epochvar

result = find_epochvar(CDFid)

Parameters

CDFid

top source merge_metadata

result = merge_metadata(cnames, base_struct, all=all)

Parameters

cnames
base_struct

Keywords

all

top source unique_array

result = unique_array(first, second)

Parameters

first
second

top source read_myCDF

result = read_myCDF(vnames, cnames, ALL=ALL, NODATASTRUCT=NODATASTRUCT, NOQUIET=NOQUIET, DEBUG=DEBUG, TSTART=TSTART, TSTOP=TSTOP, START_MSEC=START_MSEC, STOP_MSEC=STOP_MSEC, START_USEC=START_USEC, STOP_USEC=STOP_USEC, START_NSEC=START_NSEC, STOP_NSEC=STOP_NSEC, START_PSEC=START_PSEC, STOP_PSEC=STOP_PSEC, NOVIRTUAL=NOVIRTUAL)

Parameters

vnames
cnames

Keywords

ALL
NODATASTRUCT
NOQUIET
DEBUG
TSTART
TSTOP
START_MSEC
STOP_MSEC
START_USEC
STOP_USEC
START_NSEC
STOP_NSEC
START_PSEC
STOP_PSEC
NOVIRTUAL

File attributes

Modification date: Tue Oct 21 13:53:47 2014
Lines: 1,719