Hsi flare list fill: How the RHESSI Flare List is Generated

'''HSI_FLARE_LIST_FILL:'''

[http://hessi.ssl.berkeley.edu/hessidata/dbase/hessi_flare_list.txt The RHESSI flare list] is created during quicklook processing (and is often re-processed). It is distributed with the software and the flare list files can be found in the directory $ssw/hessi/dbase. Both FITS (hessi_flare_list_yyyymm.fits) and text files are available. The full list includes many events which are not necessarily of solar origin. 

The IDL routine used to create the flare list for a given time interval is called HSI_FLARE_LIST_FILL. It is distributed with the [http://hesperia.gsfc.nasa.gov/rhessidatacenter/software/documentation.html RHESSI software] (as is all of the software used to create the RHESSI database), and is found in the directory $ssw/hessi/idl/qlook_archive. 

Steps in hsi_flare_list_fill:

The time range is 1 orbit, starting and ending in SC night. Note that most parameters are changeable via keyword (mostly true; it seems that min, max of 8.0, 300.0 seconds are hardcoded for imaging interval tests).

1) Get flags, saa, eclipse, gap, atten_state, max_det_vs_tot from obs_summ_flag object.

2) Get count_rate, corrected count rates from obs_summ_rate object. Ch0=1, the 6 to 12 keV channel is the primary channel. This was changed from 12-25 keV in 2007.

3) Calculate correction factor, uncorrected/corrected rates, used in calculation of uncertainties

4) Use HSI_ALT_THRESHOLD to get threshold values for "possibly solar". This currently defaults to use front_ratio, the front to total counts ratio, the alt threshold is the lowest flat nonzero background level (using hsi_obs_bck0) plus 3.0*sigma. Post 2016 anneal, the MOD_VARIANCE (variance in count rates for detector 8 is used instead.

5) Find and flag nonzero count rates, no gap or SAA flags are allowed. If there is no good data, return.

6) Save corrected unsmoothed rates for output.

7) Get background rate. The background level at a given point is then the average value for the previous minute, unless it is less than the min daylight background level, which is possible if there are gaps, or at the day start; then the min daylight value is used. The min daylight background value is found using the program HSI_OBS_BCK0, which finds the minimum value in the daytime rate, subject to a flatness test. Also we set an upper limit to daytime_bck, to avoid missing flares that start at daytime and last the whole orbit. This is a function of energy; for ch0=1, the value is 20.0; that is, the daylight background level is never allowed to go above 20 cps/detector.
 
So to recap, the procedure is as follows: find the min value of count rate during daytime; if it's larger than 20, set this value to 20; then for each time interval, get the count rate for the previous minute. If this is more than the min daytime rate, keep it. Otherwise set it to the min daytime rate.

8) The excess flare threshold is the background level plus 3*sigma. Sigma is the standard deviation of the count rate for the previous minute used for the calculation of the background level.

9) Rates are smoothed after threshold calculation. Smoothing is done over 15 points (1 minute).

10) Count rates during eclipse are now set to zero.

[[Image:Hsi_flare_list_simple_flare.png|600px|thumb|center|'''Figure 1''':Here is a sample of the results of the EXCESS_INTERVALS calculation for a familiar orbit from 2002-02-20; the white line is the count rate from 6 to 12 keV, the red line is the threshold rate. The blue lines show the flare time intervals.]]


11) Call EXCESS_INTERVALS to get flare candidates. EXCESS_INTERVALS compares the count rate at each point to it's threshold value. If the value goes above the threshold, then an event start is recorded. The event ends when the count rate drops below the threshold value*1.2.


[[Image:Hsi_flare_list_mult_flare.png|600px|thumb|center|'''Figure 2''':Here is a sample of the results for an orbit from 2006-12-05; the white line is the count rate from 6 to 12 keV, and the red line again shows the threshold level. The blue lines show the flare time intervals. The first, second, fifth, and sixth intervals have multiple peaks, and may possibly be split up.]]


12) Back in HSI_FLARE_LIST_FILL, if events were found, then loop over each one. For each event: 

a) events with data gaps longer than 5 minutes are split.

b) events which have multiple peaks are split using TEMP_SPLIT_FLARE. Here splitting occurs unless:

b1) the minimum value between peaks does not drop to less than 1/2 the value of the 1st peak excess count rate. AND  

b2) the minimum is not "flat";i.e. there is not a 5 minute period around the minimum where the count rate varies by less than 10%. AND 

b3) the minimum value for excess counts does not drop to a value below twice the threshold value. (This helps for small flares.)

So, if the value of the excess count rate drops below 1/2 the value of the first peak, OR if there is a flat period of more than 5 minutes between peaks, OR a 5 minute data gap between peaks, OR if the rate drops to less than twice the threshold value, the next peak is considered to be a separate flare. 

Note that here excess counts refers to counts - threshold value (background plus 3*sigma). Also the count rate gets extra smoothing (31 points, 124 seconds) before peak-finding. This is shown in the next Figure,the extrasmoothing has removed some smaller peaks, so only one of the intervals is split.


[[Image:Hsi_flare_list_split_flare_sm.png|600px|thumb|center|'''Figure 2b''':Here is a sample of the results of the TEMP_SPLIT_FLARE calculation for a flare on 2006-12-05; the white line is the count rate from 6 to 12 keV. The blue lines show the flare time intervals. The fifth interval has now been split into three separate contiguous intervals. ]]



13) Call HSI_FLARE_ATTEN_CHECK, which checks for possiblilty of a new "flare" being caused by an att_state change. If flares are separated only by an att_state change, then they are combined.

14) Next for each event, 

a) check for duration, flares must be at least 12 seconds.  

b) check max_det_vs_tot flag, if the rate in the front detector with the most counts is more than 50% of the total, then this one is thrown out. 

c) Uncorrected peak count rate must be greater than 5.0 cps/detector. 

15) If there are still events, then call EXCESS_INTERVALS again, this time for each energy channel, to get possible intervals for higher energies.

16) For each flare, check to decide if its solar: 

a) If the front_ratio is greater than the alt_threshold (see step 4) OR the variance in the count rate due to modulation in channel ch0 (6-12 keV) is 1.5 times that expected from poisson, then we continue. If not then this one is tagged "non-solar".

b) Use HSI_OK_INTV to find "good" intervals for imaging. Intervals may be as long as 300.0 sec, or as short as 8.0. (As noted before, the 300 and 8 are hard coded, probably sometime in the future these will be changeable parameters.)

c) Use HSI_XY_TEST_NEW to check for valid position. This finds flare position, using HSI_LOCATE_FLARE, which compares the peak brightness position for different detectors. Each possible "good" interval, starting with the one closest to the flare peak, is tested until it gets a result. Or not -- if it fails on all intervals, this one is flagged "possible solar". If a valid position is found, the flare is finally flagged as "solar". 

d) If we do get a position, then we test position for each energy band. This uses HSI_XY_TEST_NEW, /VERIFY, which returns a valid posiiton if the position value is close (within an arcminute) to the 6-12 position. The highest energy channel that succeeds is put in the energy_hi field of the flare list data structure.


17) The final step is to fill the bck_rate, bck_rate_time fields in the output structure. Tags are filled with the background rate (from the corrected rates) and time interval (1 minute before each flare).


'''Post HSI_FLARE_LIST_FILL:''' 

Results are passed back to the calling routine, HSI_FLARE_LIST::FILL in an anonymous structure, here the rest of the flarelistdata structure is filled.

1) Peak and total count rates are calculated for the 6 to 12 keV channel. 

2) A correction factor, (corrected_rate/uncorrected_rate) is calculated for the peak and total.

3) Data.sflag1 is set: sflag1=0 means not solar, sflag1=1 means solar (i.e., valid position), sflag1=2 means "possibly solar" (in the front detectors, but no valid position).

3) Flare flags are filled in hsi_flare_flag_fill.pro

4) Filename tag is filled.

5) Flare Id's are assigned using HSI_FLARE_ID_CHECK. HSI_FLARE_ID_CHECK checks the previous flare list for flares in this time range and assigns id's so that old id's are kept for flares that have overlapping time ranges with previous ones.

6) alt_id (just time2file for the flare start) is assigned. (This will change soon to use a new IAU naming convention).

7) HSI_FLARELIST2GOES assigns [http://www.swpc.noaa.gov/today.html#xray GOES] events, if there is one for the given time range. It also finds the value of the GOES flux at the flare peak, to give an idea of flare size for small flares without corresponding GOES flares.

8) HSI_FLARELIST2AR assigns the closest active region. (Position is compared to position found at the start of the day from NAR files).

9) The [http://sprg.ssl.berkeley.edu/~jimm/hessi/hsi_obs_summ_soc.html OBS_SUMMARY] object for the object is updated with flare flags.


'''HSI_DO_QLOOK_IMAGE for images and spectra:'''

The flare list is also changed in the program HSI_DO_QLOOK_IMAGE, which is run in a separate cronjob.

HSI_DO_QLOOK_IMAGE handles both images and spectra. (This is done to cut down on the number of cronjobs). For each flare, images are done for each energy band for which a valid position was found, up to the 300 to 800 keV band. The image pixel size is 3 arcsecond, with 64x64 pixels, and the time range is the one used to find the flare position. This time range is typically a 5 minute range at the flare peak in the given energy channel.

Spectra are calculated at approximately 1 minute time resolution, depending on attenuator state changes. The energy bands start out as 1 keV resolution from 3 to 10 keV then increasing by 10% for each
succeeding energy channel. Background levels are calculated using the program HSI_NEW_SPEC_BCK, which uses a database of background level versus SC position calculated during spacecraft night. Next the detector response matrix is inverted (HSI_INVERT_SPECTRUM) to obtain a photon flux spectrum for the energy range > 6 keV. This is
used to get the peak, total and e-folding photon flux. These flux values are appended to the flare list data structure. Note that the background levels used for this procedure are less than reliable for times after 2006, [http://sprg.ssl.berkeley.edu/~tohban/nuggets/?page=article&article_id=27 when radiation damage began to affect detectors]. We're working on a better time-dependent background calculation.