PROCEDURE: THM_SST_REMOVE_SUNPULSE Purpose: Routine to perform a variety of calibrations on full distribution sst data. These can remove sun contamination and on-board masking. They can also scale the data to account for the loss of solid angle from the inability of the sst to measure directly along the probe geometric Z axis and the inability to measure directly along the probe geometric xy plane.(ie X=0,Y=0,Z = n or X=n,Y=m,Z=0, are SST 'blind spots') THM_REMOVE_SUNPULSE routine should not generally be called directly. Keywords to it will be passed down from higher level routines such as, thm_part_moments, thm_part_moments2, thm_part_dist,thm_part_getspec, thm_sst_psif, and thm_sst_psef Arguments: dat: the dat structure used in thm_part_dist, etc... Keywords: method_clean: Simplified sun cleaning methods: Allowed options are: 'manual': To use this option you need to specify the contaminated bins using the sun_bins keyword. Specified bins are removed and interpolated over phi. 'automatic': This method attempts to find contaminated bins using a statistical outlier test(modified z-score) then fills the bins by interpolation over phi. sun_bins: method_clean manual removal bin selection. Must be either a 64 element or a 16x64 element 0-1 array. Bins set to 1 will be used, bins set to 0 will be removed and filled. The 64 element input is the same format as the output from edit3dbins, it removes and fills all energies at a particular angle of measure. The 16x64 element input allows the user to remove some bins at some of the 16 energies but to keep others. all_angle_median: set this option to replace the angular distribution with the median of the data calculated over the all angles(thetas & phis) for each energy. This will generally eliminate contamination in some of the moments, but will make analysis of angular plots impossible. It will also eliminate the velocity moment. scale_sphere: set this option to increase the value of all counts by 16%. This accounts for the loss of phase space mentioned above. method_sunpulse_clean: set this to a string: Either 'median' or 'spin_fit' or 'z_score_mod' 'median': This will remove all points that are greater than 2.0 standard deviations from the median.By default they will be filled by a linear interpolation across the phi angle by default. 'spin_fit': This will remove all points that are greater than 2.0 standard deviations from a spin fit across phi angle. The equation used to fit is A+B*sin(phi)+C*cos(phi). By default these points will be filled by a linear interpolation across the phi angle. The fitting is done using the svdfit routine from the idl distribution. 'z_score_mod': This will remove all points that have a modified z-score(calculated across phi) greater than 3.5 The modified z-score is a normalized outlier detection test defined as follows: #1 X_Bar = median(X+1) #2 Sigma = MAD = Median Absolute Deviation = median(abs(X-X_Bar)) #3 Z_Score_Mod = .6745*(X - X_Bar)/Sigma This test can often get excellent results because it is insensitive to variation in standard deviation and skew in the distributions. limit_sunpulse_clean: set this equal to a floating point number that will override the default of 2.0 standard deviation tolerance or 3.5 z_score_tolerance, used by the sunpulse cleaning methods by default. This keyword will only have an effect if the method_sunpulse_clean keyword is set. fillin_method: Set this keyword to a string that specifies the method used to fill the points that are removed via the method_sunpulse_clean or the mask_remove keywords, and bins removed when enoise_remove_method='fill'. 'interpolation': this routine will interpolate across the phi angle. This is the default behavior. Interpolation is done using the interp_gap routine. 'spin_fit': this routine will perform a spin fit to the data after the points have been removed using the equation A+B*sin(phi)+C*cos(phi). It will then generate expected values for each removed phi using the equation of fit. The fitting is done using the svdfit routine from the idl distribution. Note that if 'spin_fit' is selected for the clean method and the fill method, this routine will perform two spin fits. 'blank' : Just removes the data but doesn't fill with anything mask_remove: Set this keyword to the proportion of values that must be 0 at all energies to determine that a mask is present. Generally .99 or 1.0 is a good value. The mask is a set of points that are set to 0 on-board the spacecraft. By default they will be filled by linear interpolation across phi. NOTE: This argument is not actually accepted by this routine, it is only documented here. If you provide this argument to thm_part_moments, thm_part_moments2, or thm_part_getspec, those routines will appropriately set the value for the mask_tot keyword to this routine. enoise_bins: A 0-1 array that indicates which bins should be used to calculate electronic noise. A 0 indicates that the bin should be used for electronic noise calculations. This is basically the output from the bins argument of edit3dbins. It should have dimensions 16x4. NOTE: This argument is not actually accepted by this routine, it is only documented here. If you provide this argument to thm_part_moments, thm_part_moments2, or thm_part_getspec, those routines will appropriately set the value for the enoise_tot keyword to this routine. enoise_bgnd_time: This should be either a 2 element array or a 2xN element array(where n is the number of elements in enoise_bins). The arguments represents the start and end times over which the electronic background will be calculated for each bin. If you pass a 2 element array the same start and end times can be used for each bin. If you pass a 2xN element array, then the Ith bin in enoise_bins will use the time enoise_bgnd_time[0,I] as the start time and enoise_bgnd_time[1,I] as the end time for the calculation of the background for that bin. If this keyword is not set then electronic noise will not be subtracted.NOTE: This argument is not actually accepted by this routine, it is only documented here. If you provide this argument to thm_part_moments, thm_part_moments2, or thm_part_getspec, those routines will appropriately set the value for the enoise_tot keyword to this routine. enoise_remove_method(default: 'fit_median') set the keyword to a string specifying the method you want to use to calculate the electronic noise that will be subtracted This function combines values across time. The allowable options are: 'min': Use the minimum value in the time interval for each bin/energy combination. 'average': Use the average value in the time interval for each bin/energy combination. 'median': Use the median value in the time interval for each bin/energy combination. 'fit_average': Fill in selected bins with a value that is interpolated across phi, then use the average of these values across the time interval for each bin/energy combination. 'fit_median' : Fill in selected bins with a value that is interpolated across phi, then use the mean of these values across the time interval for each bin/energy combination. 'fill': Fill in selected bins across phi, do not perform any subtraction. This will provide the cleanest looking plot, but the signal in that bin will be entirely removed. NOTE: This argument is not actually accepted by this routine, it is only documented here. If you provide this argument to thm_part_moments, thm_part_moments2, or thm_part_getspec, those routines will appropriately set the value for the enoise_tot keyword to this routine. enoise_remove_fit_method(default:'interpolation'): Set this keyword to control the method used in 'fit_average' & 'fit_median' to fit across phi. Options are: 'interpolation' 'spin_fit' By default, missing bins are interpolated across phi. Setting enoise_remove_fit_method='spin_fit' will instead try to fill by fitting to a curve of the form A+B*sin(phi)+C*cos(phi). mask_tot: The user should never manually set this keyword. thm_part_moments,thm_part_moments2, & thm_part_getspec will properly set this keyword, if the mask_remove keyword is set when they are called. enoise_tot: The user should never manually set this keyword. thm_part_moments,thm_part_moments2, & thm_part_getspec will properly set this keyword, if enoise keywords are set when called. Examples: thm_part_moments,probe='a',inst='ps?f',method_clean='automatic' sun_bins=dblarr(64)+1 sun_bins[[30,39,40,46,55,56,61,62]] = 0 ;these are the bin numbers that will be removed thm_part_moments,probe='a',inst='ps?f',sun_method='manual',sun_bins=sun_bins thm_part_moments,probe='a',instrum=['ps?f'],mag_suffix='_peif_magf',scpot_suffix='_peif_sc_pot',moments='*', $ ,fillin_method='spin_fit',method_sunpulse_clean='spin_fit',limit_sunpulse_clean=1.8, $ trange=['2008-05-19','2008-05-20'],tplotsuffix='_fit_mask_fit' thm_part_getspec, probe='a', trange=['2007-03-23','2007-03-23'],theta=[0,45], data_type='ps?f', angle='phi', $ erange=[50000,100000], /mask_remove,method_sunpulse_clean='median',limit_sunpulse_clean=1.5, $ suffix='_fit_mask_med_t2' edit3dbins,thm_sst_psef(probe='a', time_double('2008-03-01'),method_sunpulse_clean='spin_fit', $ limit_sunpulse_clean=1.2),ebins=4,sum_ebins=1 SEE ALSO: thm_part_moments.pro, thm_part_moments2.pro, thm_part_getspec.pro thm_part_dist.pro, thm_sst_psif.pro, thm_sst_psef.pro,thm_crib_sst_contamination.pro thm_sst_find_masking.pro, thm_sst_erange_bin_val.pro $LastChangedBy: pcruce $ $LastChangedDate: 2012-08-10 09:42:49 -0700 (Fri, 10 Aug 2012) $ $LastChangedRevision: 10800 $ $URL: svn+ssh://thmsvn@ambrosia.ssl.berkeley.edu/repos/spdsoft/trunk/projects/themis/spacecraft/particles/SST/thm_sst_remove_sunpulse.pro $