Tohban Report 2015-08-26

From RHESSI Wiki

Revision as of 23:55, 25 August 2015 by Lglesener (Talk | contribs)
Jump to: navigation, search


Tohban Reports
Start Date: 19 Aug 2015
End Date: 26 Aug 2015
Tohban: Lindsay Glesener
Tohban email: glesener@ssl.berkeley.edu
Next Tohban: TBD
List all reports



Contents

Solar Activity

We had an active week with many M flares.


Memory Management

On Aug 20/21, the SSR filled up due to skyrocketing ULD events in D9 rear. See description below.


Spacecraft Status

CP1 is regularly >135K on the spacecraft status page, showing up as a yellow warning.


Data Gaps

Some long gaps (~half a day) due to need to skip the pointer after the SSR filled up.

Detector issues

Some thresholds were changed on Aug. 20 around ~22:20:

Detectors 3 and 4: The front fast thresholds were raised in order to lower fast rates and free up livetime. Both changes were successful in achieving this. For D3, livetime rose from ~50% to >90%. For D4, livetime rose from ~0% to ~50%. The D4 threshold should be raised again to further raise the livetime. Over the next few days this action was not repeated because the D9 business ate up my own livetime.

Detector 9: More on the D9 rear ULD escapade below; here is only a note on the front threshold change, which occurred on Aug. 20. This was an attempt to free up livetime, but it was unsuccessful; there was no measured effect. D9 front livetime is very low, about 15%. On Aug. 21 we backed off the previous front fast adjustment as a test then reimplemented it later. (Commanding not shown here for brevity.)

Detector 6: D6 is not in good shape, with high slow and fast rates, high resets, and somewhat reduced livetime in the front. Livetime is also somewhat low in the rear segment, and fast rates are high, but the rear resets are not terrible. This detector needs attention (first try fast threshold changes in front and rear) but there wasn't time for it this week.

15-232-22:16:08 start idib_chg_thrshld (3, FRONT, FAST, 0x50)
15-232-22:16:30 /IDPUDUMPTABL TABLE=DIBTBL3
15-232-22:16:30 /IDPUTABLE3 OFFSET=FRONTFASTDAC
15-232-22:16:41 /IDPLOAD VALUE=0x50
15-232-22:17:53 start idib_chg_thrshld (4, FRONT, FAST, 0xA0)
15-232-22:18:12 /IDPUDUMPTABL TABLE=DIBTBL4
15-232-22:18:12 /IDPUTABLE4 OFFSET=FRONTFASTDAC
15-232-22:18:25 /IDPLOAD VALUE=0xA0
15-232-22:19:13 start idib_chg_thrshld (4, FRONT, FAST, 0xB0)
15-232-22:19:23 /IDPUDUMPTABL TABLE=DIBTBL4
15-232-22:19:23 /IDPUTABLE4 OFFSET=FRONTFASTDAC
15-232-22:19:32 /IDPLOAD VALUE=0xB0
15-232-22:21:09 start idib_chg_thrshld (9, FRONT, FAST, 0x90)
15-232-22:21:38 /IDPUDUMPTABL TABLE=DIBTBL9 
15-232-22:21:38 /IDPUTABLE9 OFFSET=FRONTFASTDAC
15-232-22:21:46 /IDPLOAD VALUE=0x90

More on D9 ULD anomaly:

Late on the 20th (early on the 21st UTC), detector 9 started spouting out ULD events and quickly filled up the SSR. Once this problem was identified on Friday, detector 9 events (both front and rear) were disabled, and the write pointer motion slowed to a reasonable value. The read pointer was skipped in order to empty the SSR of these bad events (though of course some good data was thrown out with the bath water; see summary below). Since the D9 resets were high (~30k), the HV was turned far down to 900V, and then to 800V, in case there was arcing, in order to prevent permanent damage. At this voltage, resets were ~23k. It was seen later that the ULDs stopped going crazy at the time of this change. (The hyperactive ULD rate never showed in the monitor rates that we see during the passes, which let to delays in its diagnosis.) See attached for a plot of D9 events in the event list, and see this link for the monitor rate plots, at the time D9 started its ULD tantrum.

http://sprg.ssl.berkeley.edu/~tohban/browser/?show=monru&date=20150820&time=201739&bar=1

D9 events were left off and the detector was left at 800V over the weekend, so that there would be no danger of a repeat occurrence during a time when tohban/ops reaction time is likely to be slower.

On Monday, Aug. 24 we turned on D9 events for a partial orbit to assess the state. While the write pointer speed did increase during the pass, we believe this was due to other changes we made around the same time (fast threshold raises, with corresponding increases in slow rates due to higher livetime). Examination of the VC1 plots shows that the ULDs remain at the “normal” value.

In other news, we also raised some detector thresholds on Thursday and Monday. These changes were not related to the D9 ULD activity. There were several changes today since several detectors had been in need of attention, but D9 was eating up all the tohban’s livetime! We also changed to active/vigorous decimation since there’s a Major Flare Watch in effect. Note that the thin attenuator probably needs some tweaking, since it’s staying in for long periods (though eventually coming out).


Longer version:

Here’s the full list of everything we tried in order to diagnose the problem on Friday. Apologies that these times are in PDT. (Plus 7 for UTC.)

1:15 pm Reversed Thursday’s threshold change on D9 only, in case that this had triggered a problem. (The SSR fill-up started not far from the time of Thursday’s threshold changes.) This produced no discernible change. The problem does not manifest itself in any of the monitor rates as seen in the MOC, so instead we looked at the SSR write pointer speed as an indicator.

2:50 pm Disabled D9 events (front and rear). This caused a sharp decrease in the write pointer velocity, confirming that D9 was the guilty party. At the end of the pass, we skipped the pointer, thus shedding (the ability to read) several hours of data. We waiting until the end of the pass in order to try to get the data from the first of the M flares that had happened that night.

4:30 pm Tried an HV drop of 400V in two steps (to 1838V). There was no change in reset rates, which were ~30k. Mark did an RTS load to keep D9 events from coming back on after eclipse. We then re-disabled D9 events, and due to the RTS load they should now stay off. We also cranked up the D9 thresholds on the rear segment, which no discernible reaction.

6 pm Dropped HV to ~900V. Rear resets dropped from ~30k to 25.7k. Then dropped another 100V to ~800V. Resets now 23.5k. Also raised the D9 FRONT fast threshold (the same raise we did on Thursday) in order to leave the front thresholds in the same condition they had started in that day. Later examination of the VC1 plots indicates that this was the time when the D9 ULDs quieted.


And here’s a repeat of the data quality assessment while the SSR was filling up or full:

There were three M class flares on the 21st; all of them happened after the D9 ULD events started going crazy. Here's a rundown:

Data gaps: We had to skip the pointer, so we lost some data.

Missing Aug. 21 1:33:40 to 18:43:10 Missing 19:06:40 to 20:15:18, though part of this is eclipse. Missing 21:19 to 21:51:40 Missing 23:29 onward, but just might not have this down yet.

Aug. 21 M flares:

M flare round ~0115: We were able to salvage most of the preimpulsive phase by careful timing of the pointer skip (i.e. downloading the flare data during a pass and skipping the pointer at the end). We couldn't wait longer than that, so we missed ~15 minutes of the preimpulsive part, then also miss the decay phase. (Flare peak happened during RHESSI eclipse.)

M flare around ~1000: No data at all. Most of the flare occurred during eclipse, so we wouldn’t have seen much anyway.

M flare around ~2034: We have some data from this flare, though there are intermittent gaps. Data probably not of high quality since SSR was very full at this point (i.e. high decimation).

Attenuators

Other notes

Spacecraft Management

Decimation Normal/Vigorous
Night time data (fronts) +- 4 minutes
Night time data (rears) +- 4 minutes
Require extra passes? No
Requirement for moving pointer? YES
Attenuator operation Normal
Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox