Cloudy SolarSoftware

From RHESSI Wiki

(Difference between revisions)
Jump to: navigation, search
(Step 2: Pre-processing)
m (fixed next nugget link)
 
(48 intermediate revisions not shown)
Line 2: Line 2:
|name = Nugget
|name = Nugget
|title = Cloudy SolarSoftware
|title = Cloudy SolarSoftware
-
|number = XXX
+
|number = 131
|first_author = Laszlo I. Etesi
|first_author = Laszlo I. Etesi
|second_author = Brian R. Dennis
|second_author = Brian R. Dennis
|publish_date = 2010 July 5
|publish_date = 2010 July 5
-
|next_nugget = TBD
+
|next_nugget = [[Harmonic Oscillations]]
-
|previous_nugget = TBD
+
|previous_nugget = [[Albedo and the modification of RHESSI results]]
}}
}}
-
 
-
== NOTE ==
 
-
 
-
This nugget is unfinished and still undergoing changes.
 
== Introduction ==
== Introduction ==
-
In our project "Extending the Virtual Solar Observatory ([http://umbra.nascom.nasa.gov/vso/ VSO])," we have combined features available in Solar Software ([http://hesperia.gsfc.nasa.gov/rhessidatacenter/software.html SSW]) to produce an environment that supports not only the data location capability of the VSO but also data retrieval, preparation, display, and analysis. Our goal is an integrated analysis experience in IDL that is both easy-to-use for the casual observer but also of value for detailed scientific analysis by more knowledgeable users. To that end, we have made the transition from a locally oriented setting, where all the analysis is done on the user’s computer, to an extended analysis environment, where IDL has access to services available on other computers through the Internet. We have implemented a form of [http://en.wikipedia.org/wiki/Cloud_computing Cloud Computing] that uses the VSO to locate data files of interest through our IDL procedure, SHOW_SYNOP. The data files can be downloaded either directly to the user's computer or through a pre-processing server ([https://team.i4ds.ch/projects/JIDL PrepServer]) that provides remote execution of instrument-specific data preparation procedures. The raw and/or pre-processed data can then be displayed with our GUI plotting suite, [http://hesperia.gsfc.nasa.gov/ssw/gen/idl/plotman/doc/plotman_help.htm PLOTMAN], which can handle different data types (light curves, images, and spectra) and perform basic data analysis. The workflow within this overall integrated environment is shown in Figures 1, 2, and 3, where the numbers correspond to steps 1-2-3 explained below in the text. Note that all of the operations can be controlled using either graphical user interfaces (GUIs) or through the IDL command line, or, and this is often the most efficient approach, through a combination of the two.
+
The analysis of solar observations, for example from RHESSI data, almost invariably requires the use of data from other sources.
 +
A single database seldom has enough breadth to tackle the complex problems that arise.
 +
Flares happen in the [http://www.scholarpedia.org/article/Solar_activity solar atmosphere], over vast ranges of physical parameters such as plasma density and temperature.
 +
Accordingly we often turn to many sets of data from [http://www.scholarpedia.org/article/Solar_Satellites other satellites] and from [http://www.scholarpedia.org/article/Solar_telescopes ground-based solar telescopes] to try to complete the picture.
 +
But how do we find out what data may exist for a given event, and then conveniently access these data and incorporate them into a comprehensive analysis?
-
Our environment supports data from a growing number of solar instruments that currently includes RHESSI, SOHO/EIT, TRACE, SECCHI/EUVI, HINODE/XRT, and HINODE/EIS.
+
In this Nugget we describe a new tool that we have developed for the solar physics community in collaboration with the Virtual Solar Observatory ([http://umbra.nascom.nasa.gov/vso/ VSO]). Our goal is to provide an integrated analysis experience in the [http://en.wikipedia.org/wiki/IDL_(programming_language IDL] programming language that is both easy-to-use for the casual observers but can also be used for detailed scientific analysis by more knowledgeable users.
-
<gallery caption="Overview of the Cloudy SolarSoftware concept" perrow="3">
+
The current mode of operation for solar data analysis begins with the retrieval of raw data from somewhere on the internet (i.e. the  
-
Image:CloudySSW.Step1.png|'''Figure 1.''' SHOW_SYNOP allows searching the VSO from within SSW IDL for solar data (Step 1).
+
[http://en.wikipedia.org/wiki/Cloud_computing Cloud]).  
-
Image:CloudySSW.Step2.png|'''Figure 2.''' Data files found with SHOW_SYNOP can be sent to the PrepServer for pre-processing (Step 2).
+
New tools such as the VSO have been developed to make this task easier as solar data are hosted on a multitude of websites. After the appropriate data files have been retrieved, the bulk of the work (the preparation, display, and analysis of the data) is performed locally on a user's desktop or laptop computer. This mode of operation places a large burden on the user as it is then their responsibility to keep up to date with all the changes in software and instrument-specific calibrations, etc. necessary for each of these steps. In order to alleviate this problem, we have developed a new extended analysis environment ([http://en.wikipedia.org/wiki/Cloud_computing Cloud Computing]), where IDL has access to services available on other computers through the Internet.
-
Image:CloudySSW.Step3.png|'''Figure 3.''' This figure shows examples of images displayed with PLOTMAN. The images in the first two columns show pre-processed images. The third column shows the images in column one overlaid as contours on the images in column two (Step 3).
+
This environment can easily be accessed either by using the graphical user interface (GUI) called SHOW_SYNOP (to show synoptic solar data) or by typing commands directly into the IDL command-line.
-
</gallery>
+
-
== Minimum Requirements ==
+
== SHOW_SYNOP IDL Widget ==
-
* IDL 6.4
+
Before we begin, it is important to note the following minimum requirements to run our software suite.  You will need
 +
* IDL 6.4 or later
* Sun Java 1.5
* Sun Java 1.5
* SolarSoftware (SSW) with GEN package (standard)
* SolarSoftware (SSW) with GEN package (standard)
 +
This should not be a problem for a standard computer running up-to-date IDL.
-
== SHOW_SYNOP IDL Widget ==
+
SHOW_SYNOP provides the starting point for retrieving, "prepping," displaying, and analyzing solar data of interest to the user.  You can most easily begin your session either by typing <code>SHOW_SYNOP</code> in the IDL command-line or by using the "Synoptic..." link in the RHESSI GUI (started by typing <code>HESSI</code> in the command line). SHOW_SYNOP provides the simple GUI interface shown in Figure 5.
-
=== Step 1: Finding the Data ===
+
The first function of SHOW_SYNOP is that it provides an easy-to-use interface to the VSO directly in IDL, and can download raw data directly to your computer. These data can then be displayed with our GUI plotting suite, [http://hesperia.gsfc.nasa.gov/ssw/gen/idl/plotman/doc/plotman_help.htm PLOTMAN], which can handle different data types (light curves, images, and spectra), and perform basic data analysis.
-
[http://hesperia.gsfc.nasa.gov/~zarro/synop/show_synop.html SHOW_SYNOP] shown in Figure 4 is an IDL graphical user interface (GUI) to search for and retrieve instrument data within a specified time interval using the VSO or other data finding facilities.  
+
-
Search results can be directly downloaded into the active SSW IDL environment or first pre-processed through our prep-server. To start searching with SHOW_SYNOP, first open the GUI by typing <code>SHOW_SYNOP</code> in your SSW IDL environment command-line. The red box highlights the VSO search form with "Start Time" and "End Time" specifying the search interval and "remote sites ->" defining the instrument (TRACE in this example). Click on the "Search" button to query the VSO for data files that will be displayed in the list below the search form shown in the green box in Figure 5. All of this capability can also be obtained by executing [[#Step_1:_Finding_the_Data_2|IDL scripts ]] from the command-line.
+
The workflow within this overall integrated environment is shown in Figure 1, where the numbers correspond to the three steps explained below. Note that all of the operations can be controlled using either  GUIs or through the IDL command-line, or, and this is often the most efficient approach, through a combination of the two.
-
<gallery caption="SHOW_SYNOP for finding data" perrow="2">
+
Another important feature of SHOW_SYNOP is that it can access our new pre-processing server ([https://team.i4ds.ch/projects/JIDL PrepServer]) that provides remote execution of instrument-specific data preparation procedures. This means that SHOW_SYNOP can provide you with up-to-date fully-calibrated data!  The way this works is that when you ask SHOW_SYNOP (running on your local computer) for a particular data file, it asks the PrepServer (running on a remote computer) to retrieve the file, process it as necessary, and send the calibrated file directly to you.  You can then display and analyze the data as usual in SHOW_SYNOP.
-
Image:CloudySSW.Show_synop.Search.png|'''Figure 4.''' Screen shot of the SHOW_SYNOP GUI used to find and retrieve specific data sets. The user-specified start and end times are entered inside the red box, and the instrument of interest is selected from the extensive list in the pull-down menu. Data files of interest are then located by pressing the "Search" button.
+
 
-
Image:CloudySSW.Show_Synop.Searched.png|'''Figure 5.''' The retrieved data files are shown in the green box after pressing the "Search" button.  
+
Our environment supports data from a growing number of solar instruments listed below, with additional instruments to be added as the data become available.
 +
 
 +
<gallery caption="Overview of the Cloudy SolarSoftware concept" perrow="2" widths="320px" heights="320px">
 +
Image:CloudySSW.Step-1-2-3.png|'''Figure 1.''' SHOW_SYNOP allows searching the VSO from within SSW IDL for solar data (Step 1). Data files found with SHOW_SYNOP can be sent to the PrepServer for pre-processing (Step 2). PLOTMAN is used for image display (Step 3, also see Figures 2, 3, and 4).
 +
Image:CloudySSW.EUVI.RHESSI.png|'''Figure 2.''' In this figure the EUVI image from Figure 4 is overlaid with the RHESSI contours from Figure 3.
 +
Image:CloudySSW.RHESSI.Overlay.png|'''Figure 3.''' This figure shows PLOTMAN displaying a RHESSI clean 6-12keV image that was reconstructed using the PrepServer. The RHESSI image is overlaid with the RHESSI 20, 30, 50, 70, and 90% contours (in white).
 +
Image:CloudySSW.EUVI.png|'''Figure 4.''' This is a PLOTMAN screenshot showing a STEREO-B EUVI image of the same flare as shown in Figure 3. This image was pre-processed using the PrepServer.
</gallery>
</gallery>
-
=== Step 2: Pre-processing ===
+
As an introduction to this we will guide you through a simple example on how to use SHOW_SYNOP.
-
Typically, instrument data found with the VSO are unprocessed level-0 data. The PrepServer offers remote pre-processing of those data from within SHOW_SYNOP or from the IDL command-line before or after downloading to the user's computer. The big advantage is that the user does not have to maintain instrument-specific data analysis procedures or calibration files on his computer. Once the files containing the data of interest has been retrieved and prepped if necessary, the file names are displayed in the SHOW_SYNOP window (green box in Figure 5).  
+
-
'''Move all bolded text to the script examples page. There are two different pre-processing behaviors configurable:'''
+
=== Step 1: Finding the Data ===
-
* '''By default a click on "Download" retrieves the level-0 data file, stores it on the local hard drive and adds it to a list (Figure 6, blue box). The level-0 data file is pre-processed by selecting it in the blue box and clicking on "Display". This uploads the level-0 data file to the PrepServer where it is pre-processed, and downloads the pre-processed product to the local hard drive.'''
+
[http://hesperia.gsfc.nasa.gov/~zarro/synop/show_synop.html SHOW_SYNOP] shown in Figures 5 and 6 is an IDL GUI to search for and retrieve instrument data within a specified time interval using the VSO or other data finding facilities.  
-
* '''In auto-preprocessing mode, which can be activated in the configuration menu (Figures 5 and 7, yellow box), SHOW_SYNOP sends the URL of the level-0 data file to the PrepServer and only downloads the pre-processed data.
+
-
(Note that in case of zipped data files (e.g. TRACE), SHOW_SYNOP opens a selection widget to help in choosing the desired image(s) (see Figure 8).
+
-
'''
+
-
<span style="color:#FF0000">'''Please note: SHOW_SYNOP and the PrepServer are dependent on data providers for downloading speed. Be aware that downloading data may take a few minutes, during which SHOW_SYNOP or your SSW IDL session may be unresponsive.</span>'''
+
-
For more details on the prep-server including accessing it from the IDL command-line, please click [[#Step_2:_Pre-processing_2|here]]
+
Search results can be directly downloaded into the active SSW IDL environment or first pre-processed through our prep-server. To start searching with SHOW_SYNOP, first open the GUI by typing <code>SHOW_SYNOP</code> in your SSW IDL environment command-line. The red box in Figure 5 highlights the VSO search form with "Start Time" and "End Time" specifying the search interval, and "remote sites ->" defining the instrument (TRACE in this example). Click on the "Search" button to query the VSO for data files that will be displayed in the list below the search form shown in the green box in Figure 6. All of this capability can also be obtained by executing [[#Step_1:_Finding_the_Data_2|IDL scripts ]] from the command-line.
-
<gallery caption="SHOW_SYNOP for downloading and pre-processing data" perrow="3">
+
Currently, SHOW_SYNOP can search for data from the following instruments:
-
Image:CloudySSW.Show_synop.Downloaded.png|'''Figure 6.''' In this screen shot of SHOW_SYNOP the window highlighted in blue indicates the local file repository with level-0 and pre-processed data files.
+
* Big Bear Solar Observatory ([http://www.bbso.njit.edu/ BBSO])
 +
* [http://www.exp-astro.phys.ethz.ch/astro1/Users/cmonstei/instrument/callisto/index.php Callisto Radio Observations]
 +
* [http://www.mpe.mpg.de/gamma/instruments/glast/GBM/www/ FERMI GBM]
 +
* [http://msslxr.mssl.ucl.ac.uk:8080/SolarB/ HINODE/EIS]
 +
* [http://xrt.cfa.harvard.edu/ HINODE/XRT]
 +
* [http://www.kso.ac.at/index_en.php Kanzelhohe Solar Observatory]
 +
* [http://www.grandpublic.obspm.fr/?lang=en Meudon Observatory]
 +
* [http://nrt.obspm.fr/ Nancay Radio Observatory]
 +
* [http://solar.nro.nao.ac.jp/norh/ Nobeyama Radioheliograph]
 +
* [http://www.astro1obs.phys.ethz.ch/rapp/catalog/catalog.html#phoenix Phoenix ETH Zurich]
 +
* [http://umbra.nascom.nasa.gov/eit/ SOHO/EIT]
 +
* [http://soi.stanford.edu/ SOHO/MDI]
 +
* [http://www.prl.res.in/~soxs-data/index.html Solar X-ray Spectrometer (SOXS)]
 +
* [http://secchi.nrl.navy.mil/ STEREO/SECCHI-COR1/2]
 +
* [http://secchi.lmsal.com/EUVI/ STEREO/SECCHI-EUVI]
 +
* [http://trace.lmsal.com/ TRACE]
-
'''Delete Figures 7 and 8.'''
+
<gallery caption="SHOW_SYNOP" perrow="2" widths="320px" heights="320px">
-
Image:CloudySSW.Show synop.Options.png|'''Figure 7.''' This options menu allows configuring the pre-processing behavior: download level-0 data or request pre-processed data.
+
Image:CloudySSW.Show_synop.Search.png|'''Figure 5.''' Screenshot of the SHOW_SYNOP GUI used to find and retrieve specific data sets. The user-specified start and end times are entered inside the red box, and the instrument of interest is selected from the extensive list in the pull-down menu. Data files satisfying these requirements are then located by pressing the "Search" button.
-
Image:CloudySSW.Selection.Widget.Figure4.png|'''Figure 8.''' When pre-processing a TRACE data file that contains multiple TRACE images, a selection widget will open allowing picking a set of images.
+
Image:CloudySSW.Show_synop.Searched_Downloaded.png|'''Figure 6.''' The file names are shown in the green box.  The window highlighted in blue indicates the local file repository with level-0 and pre-processed data files.
</gallery>
</gallery>
-
=== Step 3: Visualizing ===
+
=== Step 2: Pre-processing ===
 +
Typically, instrument data found with the VSO are unprocessed level-0 data (i.e., no calibrations have been performed).
 +
The PrepServer offers remote pre-processing of those data from within SHOW_SYNOP or from the IDL command-line before or after downloading to the user's computer. The big advantage is that the user does not have to maintain instrument-specific data analysis procedures or calibration files on his computer. Once the files containing the data of interest have been retrieved and prepped (calibrated) if necessary, the file names are displayed in the SHOW_SYNOP window (green box in Figure 6).
-
The data are visualized with PLOTMAN, which handles different data types such as light curves, images, spectra, and spectrograms. It provides basic display operations such as zooming, image overlays, solar rotation, etc. To display data with PLOTMAN from within SHOW_SYNOP, simply select a data file (Figure 6, blue box) and click on "Display". If the data file has already been pre-processed, it will be displayed immediately in a PLOTMAN window. Otherwise, it is first sent to the PrepServer for processing. If multiple files have been selected, all are displayed in the same PLOTMAN window (see Figures 9, 10, and 11).
+
For more details on the prep-server including accessing it from the IDL command-line, click [[#Step_2:_Pre-processing_2|here]]
-
Like all other GUIs discussed in this nugget, PLOTMAN can also be called from the command-line. Examples can be found [[#Step_3:_Visualizing_2|here]].
+
=== Step 3: Visualizing ===
-
<gallery caption="Example images displayed with PLOTMAN" perrow="3">
+
The data are visualized with PLOTMAN, which handles different data types such as light curves, images, spectra, and spectrograms. PLOTMAN provides basic display operations such as zooming, image overlays, solar rotation, etc. To display data with PLOTMAN from within SHOW_SYNOP, simply select a data file (Figure 6, blue box) and click "Display". If the data file has already been pre-processed, it will be displayed immediately in a PLOTMAN window. Otherwise, it is first sent to the PrepServer for processing. If multiple files have been selected, all are displayed in the same PLOTMAN window for easy comparison. See Figures 2, 3, and 4 for images that were displayed using PLOTMAN.
-
Image:CloudySSW.XRT.Figure3.png|'''Figure 9.''' This figure shows PLOTMAN displaying an XRT image (January 25 2007 06:56:56) that has been pre-processed with default settings by the PrepServer
+
 
-
Image:CloudySSW.TRACE.Figure5.png|'''Figure 10.''' This figure shows PLOTMAN displaying a TRACE image (171 A, January 25 2007, 06:15:09) that has been pre-processed with default settings by the PrepServer. If multiple images were selected in the selection window (see Figure 8), they can be accessed by clicking on "Window_Control" in PLOTMAN's menu panel at the top.
+
Like all other GUIs discussed in this nugget, all the capabilities of PLOTMAN are also available from the command-line. Examples can be found [[#Step_3:_Visualizing_2|here]].
-
Image:CloudySSW.RHESSI.EIT.Overlay.Figure.6.png|'''Figure 11.''' This figure shows PLOTMAN displaying a contour overlay of a reconstructed RHESSI clean 6-12keV image on an EIT image (171 A, January 25 2007, 07:00:13). The reconstruction of the RHESSI image as well as the pre-processing of the EIT images was done with the PrepServer.
+
-
</gallery>
+
== IDL Command-Line Tools ==
== IDL Command-Line Tools ==
=== Step 1: Finding the Data ===
=== Step 1: Finding the Data ===
-
The VSO search can be initiated directly from the IDL command-line using the procedure [https://team.i4ds.ch/projects/JIDL/wiki/vso_files VSO_FILES]. Two different search strategies are supported:
+
The VSO search can be initiated directly from the IDL command-line using the procedure [https://team.i4ds.ch/projects/JIDL/wiki/vso_files VSO_FILES]. Data for all the instruments available through the VSO can be searched with this procedure (please see the VSO web page for a list of supported [http://sdac.virtualsolar.org/cgi/show_details?keyword=INSTRUMENT instruments] and [http://sdac.virtualsolar.org/cgi/show_details?keyword=DETECTOR detectors]). VSO_FILES returns just the URLs of the data files that it finds. The files themselves can then be downloaded using [http://hesperia.gsfc.nasa.gov/~zarro/idl/sockets/sockets.html SOCK_COPY] or [https://team.i4ds.ch/projects/JIDL/wiki/vso_prep VSO_PREP] (see [[#Step_2:_Pre-processing_2|next section]]).
 +
 
 +
Two different search strategies are supported in VSO_FILES:
* An interval search that returns URLs to files containing data for the specified interval.
* An interval search that returns URLs to files containing data for the specified interval.
* A proximity search that returns a URL to the data file that is closest to the specified time.
* A proximity search that returns a URL to the data file that is closest to the specified time.
-
 
-
Data for the following instruments can be searched:
 
-
* euvi
 
-
* eit
 
-
* xrt
 
-
* eis
 
-
* trace
 
-
* aia
 
-
 
-
VSO_FILES does not download any data files. Instead, [http://hesperia.gsfc.nasa.gov/~zarro/idl/sockets/sockets.html SOCK_COPY] or [https://team.i4ds.ch/projects/JIDL/wiki/vso_prep VSO_PREP] (see [[#Step_2:_Pre-processing_2|next section]]) are used.
 
=== Step 2: Pre-processing ===
=== Step 2: Pre-processing ===
Line 108: Line 116:
=== Examples ===
=== Examples ===
-
* All examples demonstrate steps 1-2-3 on the command-line with the tools described above
+
* Each example performs these three steps - finding, pre-processing and visualizing the data.
-
* All examples can be copy-pasted into IDL and run from the command-line
+
* To run an example, copy the lines of code and paste them into your IDL session.
 +
* The images from all of the examples will be added to the same PLOTMAN instance for further manipulation. Use the Window_Control button to select an image, and the Plot_Control button to select overlays or change the display options.
 +
* Note that some of these commands take time, so please be patient!  Also network glitches can cause failures.
==== Example 1 ====
==== Example 1 ====
-
<source lang="ittvis_idl">
 
-
; Step 1: Search the VSO and return a URL to the Hinode/XRT image closest to January 25 2007 06:57 UT.
 
-
; Step 2: Pre-process that data file and create an XRT object (xrt_obj)
 
-
; Step 3: Display the data file with PLOTMAN using a standard color table (see Figure 9).
 
-
xrt_file = [f]vso_files[/f]('25-Jan-2007 06:57', instr='xrt')
+
* Search the VSO and return a URL to the Hinode/XRT image closest to January 25 2007 06:57 UT.
-
[p]vso_prep[/p], xrt_file, oprep=xrt_obj
+
* Pre-process that data file and create an XRT object (xrt_obj)
-
xrt_obj->[p]plotman[/p], /colors
+
* Display the data with PLOTMAN using the XRT standard color table.
 +
 
 +
 
 +
<source lang="ittvis_idl">
 +
  xrt_file = [f]vso_files[/f]('25-Jan-2007 06:57', instr='xrt')
 +
  [p]vso_prep[/p], xrt_file, oprep=xrt_obj
 +
  xrt_obj->[p]plotman[/p], /colors, plotman=p
</source>
</source>
 +
==== Example 2 ====
==== Example 2 ====
-
<source lang="ittvis_idl">
 
-
; Step 1: Search the VSO and return URLs to EIT images that have been observed between January 25 2007 06:45 and January 25 20007 07:15
 
-
; Step 2: Pre-process one EIT image at the time and...
 
-
; Step 3: ...display it with PLOTMAN using standard coloring. All images are displayed in the same PLOTMAN window.
 
-
eit_files = [f]vso_files[/f]('25-Jan-2007 06:45', '25-Jan-2007 07:15', instr='eit')
+
* Search the VSO and return URLs to EIT images that have been observed between January 25 2007 06:45 and January 25 2007 07:05 (in this case 2 URLs)
-
FOR i=0, N_ELEMENTS(eit_files)-1 DO BEGIN $
+
* Pre-process one EIT image at a time and...
-
[p]vso_prep[/p], eit_files[i], oprep=eit_obj & $
+
* ...display it with PLOTMAN using the EIT standard color table.
-
eit_obj->[p]plotman[/p], /colors, plotman=p & $
+
 
-
ENDFOR
+
 
 +
<source lang="ittvis_idl">
 +
  eit_files = [f]vso_files[/f]('25-Jan-2007 06:45', '25-Jan-2007 07:05', instr='eit')
 +
  FOR i=0, N_ELEMENTS(eit_files)-1 DO BEGIN $
 +
  [p]vso_prep[/p], eit_files[i], oprep=eit_obj & $
 +
  eit_obj->[p]plotman[/p], /colors, plotman=p & $
 +
  ENDFOR
</source>
</source>
 +
==== Example 3 ====
==== Example 3 ====
-
<source lang="ittvis_idl">
 
-
; Step 1: Search the VSO and return a URL to the TRACE data file closest to January 25 2007 07:00
 
-
; Step 2: The TRACE data file contains multiple images. Pre-process images one to 5 and create TRACE objects for them (trace_obj).
 
-
;        Not specifying an index would result in VSO_PREP opening the selection widget as shown in Figure 8.
 
-
; Step 3: Display all TRACE images in PLOTMAN using standard coloring '''(see Figure XXX)'''
 
-
trace_file = [f]vso_files[/f]('25-Jan-2007 07:00', instr='trace')
 
-
[p]vso_prep[/p], trace_file, image_no=[1,2,3,4,5], oprep=trace_obj
 
-
trace_obj->[p]plotman[/p], /colors
 
-
</source>
 
-
==== Example 4 ====
+
* Use VSO_PREP to reconstruct a RHESSI 6-12keV image using the CLEAN algorithm on the PrepServer
-
<source lang="ittvis_idl">
+
* Display RHESSI image with PLOTMAN using your PLOTMAN default color (see Figure 3)
-
; Step 1: Not required
+
-
; Step 2: Use VSO_PREP to reconstruct a RHESSI clean 6-12keV image on the PrepServer
+
-
; Step 3: Display RHESSI image with PLOTMAN using standard coloring '''(see Figure XXX)'''
+
-
[p]vso_prep[/p], instrument='rhessi', im_time_interval=['25-Jan-2007 06:53:44', '25-Jan-2007 06:57:40'], image_alg='clean', im_energy_binning=[6,12], oprep=rhessi_obj
 
-
rhessi_obj->[p]plotman[/p], /colors
 
-
</source >
 
-
==== Example 5 ====
 
<source lang="ittvis_idl">
<source lang="ittvis_idl">
-
; Step 1: Search the VSO and return a URL to the EUVI data file closest to January 25 2007 06:57
+
  [p]vso_prep[/p], instrument='rhessi', im_time_interval=['25-Jan-2007 06:53:44', '25-Jan-2007 06:57:40'], $
-
; Step 2: The EUVI image is pre-processed (incl. rolling) and an EUVI object is created (euvi_obj)
+
  image_alg='clean', im_energy_binning=[6,12], oprep=rhessi_obj
-
; Step 3: Display the EUVI image in PLOTMAN using standard coloring '''(see Figure XXX)'''
+
  rhessi_obj->[p]plotman[/p], plotman=p
-
 
+
-
euvi_file = [f]vso_files[/f]('25-Jan-2007 06:57', instr='euvi')
+
-
[p]vso_prep[/p], euvi_file, /roll, oprep=euvi_obj
+
-
euvi_obj->[p]plotman[/p], /colors
+
</source>
</source>
== Conclusion ==
== Conclusion ==
 +
In this Nugget, we have described an integrated way of searching and retrieving, pre-processing, and visualizing solar data from an IDL session. We introduced three new computer tools - SHOW_SYNOP, the PrepServer, and PLOTMAN. They are designed to simplify the display and inter-comparison of data from a wide range of solar instruments. Further details are available in the documentation listed below. We encourage you to try the examples given here. You can use the SHOW_SYNOP GUI or enter the scripts directly in the IDL command line, as you wish. Then try out these tools for your own specific needs. We will be very interested in your feedback and any suggestions for improvements. Please use the contact information below to let us know what you think, both positive and negative.
 +
 +
For the future, we plan to extend the list of instruments that SHOW_SYNOP and the PrepServer can handle, and further improve our support for multi-instrument analysis. We will continue working closely with the solar physics community and the service providers (e.g. the VSO) to refine and enhance these tools.
== Documentation ==
== Documentation ==
Line 176: Line 175:
* [https://team.i4ds.ch/projects/JIDL PrepServer]
* [https://team.i4ds.ch/projects/JIDL PrepServer]
* [http://hesperia.gsfc.nasa.gov/ssw/gen/idl/plotman/doc/plotman_help.htm PLOTMAN]
* [http://hesperia.gsfc.nasa.gov/ssw/gen/idl/plotman/doc/plotman_help.htm PLOTMAN]
 +
* [http://hesperia.gsfc.nasa.gov/~zarro/synop/show_synop.html SHOW_SYNOP]  (Note: This documentation is out of date and will be updated soon.)
* [https://team.i4ds.ch/projects/JIDL/wiki/vso_files VSO_FILES]
* [https://team.i4ds.ch/projects/JIDL/wiki/vso_files VSO_FILES]
* [https://team.i4ds.ch/projects/JIDL/wiki/vso_prep VSO_PREP]
* [https://team.i4ds.ch/projects/JIDL/wiki/vso_prep VSO_PREP]
Line 185: Line 185:
* PrepServer: Laszlo I. Etesi (laszlo dot etesi at nasa dot gov)
* PrepServer: Laszlo I. Etesi (laszlo dot etesi at nasa dot gov)
* PLOTMAN: Kim Tolbert (kim dot tolbert at nasa dot gov)
* PLOTMAN: Kim Tolbert (kim dot tolbert at nasa dot gov)
 +
 +
[[Category: Nugget]]

Latest revision as of 21:49, 18 November 2010


Nugget
Number: 131
1st Author: Laszlo I. Etesi
2nd Author: Brian R. Dennis
Published: 2010 July 5
Next Nugget: Harmonic Oscillations
Previous Nugget: Albedo and the modification of RHESSI results
List all



Contents

Introduction

The analysis of solar observations, for example from RHESSI data, almost invariably requires the use of data from other sources. A single database seldom has enough breadth to tackle the complex problems that arise. Flares happen in the solar atmosphere, over vast ranges of physical parameters such as plasma density and temperature. Accordingly we often turn to many sets of data from other satellites and from ground-based solar telescopes to try to complete the picture. But how do we find out what data may exist for a given event, and then conveniently access these data and incorporate them into a comprehensive analysis?

In this Nugget we describe a new tool that we have developed for the solar physics community in collaboration with the Virtual Solar Observatory (VSO). Our goal is to provide an integrated analysis experience in the IDL programming language that is both easy-to-use for the casual observers but can also be used for detailed scientific analysis by more knowledgeable users.

The current mode of operation for solar data analysis begins with the retrieval of raw data from somewhere on the internet (i.e. the Cloud). New tools such as the VSO have been developed to make this task easier as solar data are hosted on a multitude of websites. After the appropriate data files have been retrieved, the bulk of the work (the preparation, display, and analysis of the data) is performed locally on a user's desktop or laptop computer. This mode of operation places a large burden on the user as it is then their responsibility to keep up to date with all the changes in software and instrument-specific calibrations, etc. necessary for each of these steps. In order to alleviate this problem, we have developed a new extended analysis environment (Cloud Computing), where IDL has access to services available on other computers through the Internet. This environment can easily be accessed either by using the graphical user interface (GUI) called SHOW_SYNOP (to show synoptic solar data) or by typing commands directly into the IDL command-line.

SHOW_SYNOP IDL Widget

Before we begin, it is important to note the following minimum requirements to run our software suite. You will need

This should not be a problem for a standard computer running up-to-date IDL.

SHOW_SYNOP provides the starting point for retrieving, "prepping," displaying, and analyzing solar data of interest to the user. You can most easily begin your session either by typing SHOW_SYNOP in the IDL command-line or by using the "Synoptic..." link in the RHESSI GUI (started by typing HESSI in the command line). SHOW_SYNOP provides the simple GUI interface shown in Figure 5.

The first function of SHOW_SYNOP is that it provides an easy-to-use interface to the VSO directly in IDL, and can download raw data directly to your computer. These data can then be displayed with our GUI plotting suite, PLOTMAN, which can handle different data types (light curves, images, and spectra), and perform basic data analysis.

The workflow within this overall integrated environment is shown in Figure 1, where the numbers correspond to the three steps explained below. Note that all of the operations can be controlled using either GUIs or through the IDL command-line, or, and this is often the most efficient approach, through a combination of the two.

Another important feature of SHOW_SYNOP is that it can access our new pre-processing server (PrepServer) that provides remote execution of instrument-specific data preparation procedures. This means that SHOW_SYNOP can provide you with up-to-date fully-calibrated data! The way this works is that when you ask SHOW_SYNOP (running on your local computer) for a particular data file, it asks the PrepServer (running on a remote computer) to retrieve the file, process it as necessary, and send the calibrated file directly to you. You can then display and analyze the data as usual in SHOW_SYNOP.

Our environment supports data from a growing number of solar instruments listed below, with additional instruments to be added as the data become available.

As an introduction to this we will guide you through a simple example on how to use SHOW_SYNOP.

Step 1: Finding the Data

SHOW_SYNOP shown in Figures 5 and 6 is an IDL GUI to search for and retrieve instrument data within a specified time interval using the VSO or other data finding facilities.

Search results can be directly downloaded into the active SSW IDL environment or first pre-processed through our prep-server. To start searching with SHOW_SYNOP, first open the GUI by typing SHOW_SYNOP in your SSW IDL environment command-line. The red box in Figure 5 highlights the VSO search form with "Start Time" and "End Time" specifying the search interval, and "remote sites ->" defining the instrument (TRACE in this example). Click on the "Search" button to query the VSO for data files that will be displayed in the list below the search form shown in the green box in Figure 6. All of this capability can also be obtained by executing IDL scripts from the command-line.

Currently, SHOW_SYNOP can search for data from the following instruments:

Step 2: Pre-processing

Typically, instrument data found with the VSO are unprocessed level-0 data (i.e., no calibrations have been performed). The PrepServer offers remote pre-processing of those data from within SHOW_SYNOP or from the IDL command-line before or after downloading to the user's computer. The big advantage is that the user does not have to maintain instrument-specific data analysis procedures or calibration files on his computer. Once the files containing the data of interest have been retrieved and prepped (calibrated) if necessary, the file names are displayed in the SHOW_SYNOP window (green box in Figure 6).

For more details on the prep-server including accessing it from the IDL command-line, click here

Step 3: Visualizing

The data are visualized with PLOTMAN, which handles different data types such as light curves, images, spectra, and spectrograms. PLOTMAN provides basic display operations such as zooming, image overlays, solar rotation, etc. To display data with PLOTMAN from within SHOW_SYNOP, simply select a data file (Figure 6, blue box) and click "Display". If the data file has already been pre-processed, it will be displayed immediately in a PLOTMAN window. Otherwise, it is first sent to the PrepServer for processing. If multiple files have been selected, all are displayed in the same PLOTMAN window for easy comparison. See Figures 2, 3, and 4 for images that were displayed using PLOTMAN.

Like all other GUIs discussed in this nugget, all the capabilities of PLOTMAN are also available from the command-line. Examples can be found here.

IDL Command-Line Tools

Step 1: Finding the Data

The VSO search can be initiated directly from the IDL command-line using the procedure VSO_FILES. Data for all the instruments available through the VSO can be searched with this procedure (please see the VSO web page for a list of supported instruments and detectors). VSO_FILES returns just the URLs of the data files that it finds. The files themselves can then be downloaded using SOCK_COPY or VSO_PREP (see next section).

Two different search strategies are supported in VSO_FILES:

Step 2: Pre-processing

VSO_PREP allows for data pre-processing from an IDL command-line without the requirement of a local installation of instrument software or calibration data. VSO_PREP takes as a minimum a local file or a URL to a remote file for a parameter. If a URL is provided then the PrepServer will download the data and send them back pre-processed; otherwise they are uploaded to the PrepServer, pre-processed, and downloaded.

The following instrument data can be pre-processed:

Step 3: Visualizing

PLOTMAN displays level-0 and pre-processed data. It is integrated with VSO_PREP and allows visualizing data returned by VSO_PREP with one command.

Examples

Example 1


  xrt_file = vso_files('25-Jan-2007 06:57', instr='xrt')
  vso_prep, xrt_file, oprep=xrt_obj
  xrt_obj->plotman, /colors, plotman=p


Example 2


  eit_files = vso_files('25-Jan-2007 06:45', '25-Jan-2007 07:05', instr='eit')
  FOR i=0, N_ELEMENTS(eit_files)-1 DO BEGIN $
   vso_prep, eit_files[i], oprep=eit_obj & $
   eit_obj->plotman, /colors, plotman=p & $
  ENDFOR


Example 3


  vso_prep, instrument='rhessi', im_time_interval=['25-Jan-2007 06:53:44', '25-Jan-2007 06:57:40'], $
   image_alg='clean', im_energy_binning=[6,12], oprep=rhessi_obj
  rhessi_obj->plotman, plotman=p

Conclusion

In this Nugget, we have described an integrated way of searching and retrieving, pre-processing, and visualizing solar data from an IDL session. We introduced three new computer tools - SHOW_SYNOP, the PrepServer, and PLOTMAN. They are designed to simplify the display and inter-comparison of data from a wide range of solar instruments. Further details are available in the documentation listed below. We encourage you to try the examples given here. You can use the SHOW_SYNOP GUI or enter the scripts directly in the IDL command line, as you wish. Then try out these tools for your own specific needs. We will be very interested in your feedback and any suggestions for improvements. Please use the contact information below to let us know what you think, both positive and negative.

For the future, we plan to extend the list of instruments that SHOW_SYNOP and the PrepServer can handle, and further improve our support for multi-instrument analysis. We will continue working closely with the solar physics community and the service providers (e.g. the VSO) to refine and enhance these tools.

Documentation

Contacts

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox