Tohban Report 2015-08-26

From RHESSI Wiki

(Difference between revisions)
Jump to: navigation, search
(Detector issues)
(Attenuators)
Line 98: Line 98:
== Attenuators ==
== Attenuators ==
 +
The attenuators came in and out during the set of M flares on Thursday, so that's nice.  Over the weekend the thin attenuator came in several times and stayed in for an inordinate amount of time before coming out, indicating that it was on the verge of getting stuck in.  On Tuesday (Aug 25) it did get stuck in, so we changed the livetime threshold.  Both the thin and thick attenuator out thresholds were changed by 1 count each.  (The thick attenuator threshold had also just been changed on Monday to stay one tick ahead of the thin attenuator.  With this latest change to the thin we also kept the thick 1 tick ahead.)
-
 
+
The thin out threshold is now "8" or 3.14% and the thick out is "9" or 3.53% dead time.  These changes were made August 25, 18:42:24 UTC.  Once they were made the thin attenuator came out.
-
 
+
-
 
+
== Other notes ==
== Other notes ==

Revision as of 00:19, 26 August 2015


Tohban Reports
Start Date: 19 Aug 2015
End Date: 26 Aug 2015
Tohban: Lindsay Glesener
Tohban email: glesener@ssl.berkeley.edu
Next Tohban: TBD
List all reports



Contents

Solar Activity

We had an active week with many M flares.


Memory Management

On Aug 20/21, the SSR filled up due to skyrocketing ULD events in D9 rear. See description below.


Spacecraft Status

CP1 is regularly >135K on the spacecraft status page, showing up as a yellow warning.


Data Gaps

Some long gaps (~half a day) due to need to skip the pointer after the SSR filled up.

Detector issues

(For convenience, I'm leaving out the commanding logs, but these can be found on the individual detector update pages.)

Some thresholds were changed on Aug. 20 around ~22:20:

Detectors 3 and 4: The front fast thresholds were raised in order to lower fast rates and free up livetime. Both changes were successful in achieving this. For D3, livetime rose from ~50% to >90%. For D4, livetime rose from ~0% to ~50%. The D4 threshold should be raised again to further raise the livetime. Over the next few days this action was not repeated because the D9 business ate up my own livetime.

Detector 9: More on the D9 rear ULD escapade below; here is only a note on the front threshold change, which occurred on Aug. 20. This was an attempt to free up livetime, but it was unsuccessful; there was no measured effect. D9 front livetime is very low, about 15%. On Aug. 21 we backed off the previous front fast adjustment as a test then reimplemented it later. (Commanding not shown here for brevity.)

Detector 6: D6 is not in good shape, with high slow and fast rates, high resets, and somewhat reduced livetime in the front. Livetime is also somewhat low in the rear segment, and fast rates are high, but the rear resets are not terrible. This detector needs attention (first try fast threshold changes in front and rear) but there wasn't time for it this week.

More thresholds were changed on Monday, Aug. 24. These are not related to the D9 antics:

Detector 3: The rear fast threshold was raised. This caused the confusing result that the fast rates increased and the livetime decreased. Upon realizing that D3 had actually gotten so bad that it had switched into the negative universe where up is down and down is up, we raised the threshold again. This was successful in bringing down the rates and restored the livetime to 64%.

Detector 4: The front fast threshold was raised again, bringing the livetime up to 55%. Even after this, fast and slow rates were both high. The fast threshold was raised again, which brought the fast rates down by an order of magnitude. Once the data is down, the next tohban should examine whether this detector is putting in a bunch of noise into the event list; if so, the slow threshold should be raised.

Detector 5: The rear fast threshold was raised, successfully bringing the livetime up to >75%.

Detector 6: Both front and rear fast thresholds were raised again. This successfully brought the livetime up to 75% in the front, but didn't really change anything in the rear. Another attempt was made with the rear fast threshold, but again no change. The slow threshold was also raised, which brought the slow rates down from 200k to 5k.

Those last sets of changes were made in two batches: one at Aug. 24 20:42--20:47 and the other at Aug. 25 00:02--00:04.

More on D9 ULD anomaly:

Late on the 20th (early on the 21st UTC), detector 9 started spouting out ULD events and quickly filled up the SSR. Once this problem was identified on Friday, detector 9 events (both front and rear) were disabled, and the write pointer motion slowed to a reasonable value. The read pointer was skipped in order to empty the SSR of these bad events (though of course some good data was thrown out with the bath water; see summary below). Since the D9 resets were high (~30k), the HV was turned far down to 900V, and then to 800V, in case there was arcing, in order to prevent permanent damage. At this voltage, resets were ~23k. It was seen later that the ULDs stopped going crazy at the time of this change. (The hyperactive ULD rate never showed in the monitor rates that we see during the passes, which let to delays in its diagnosis.) See attached for a plot of D9 events in the event list, and see this link for the monitor rate plots, at the time D9 started its ULD tantrum.

http://sprg.ssl.berkeley.edu/~tohban/browser/?show=monru&date=20150820&time=201739&bar=1

D9 events were left off and the detector was left at 800V over the weekend, so that there would be no danger of a repeat occurrence during a time when tohban/ops reaction time is likely to be slower.

On Monday, Aug. 24 we turned on D9 events for a partial orbit to assess the state. (2015-236-20:42:45 /itmon value=AFE8.) While the write pointer speed did increase during the pass, we believe this was due to other changes we made around the same time (fast threshold raises, with corresponding increases in slow rates due to higher livetime). Examination of the VC1 plots shows that the ULDs remain at the “normal” value.

In other news, we also raised some detector thresholds on Thursday and Monday. These changes were not related to the D9 ULD activity. There were several changes today since several detectors had been in need of attention, but D9 was eating up all the tohban’s livetime! We also changed to active/vigorous decimation since there’s a Major Flare Watch in effect. Note that the thin attenuator probably needs some tweaking, since it’s staying in for long periods (though eventually coming out).


Longer version:

Here’s the full list of everything we tried in order to diagnose the problem on Friday. Apologies that these times are in PDT. (Plus 7 for UTC.)

1:15 pm Reversed Thursday’s threshold change on D9 only, in case that this had triggered a problem. (The SSR fill-up started not far from the time of Thursday’s threshold changes.) This produced no discernible change. The problem does not manifest itself in any of the monitor rates as seen in the MOC, so instead we looked at the SSR write pointer speed as an indicator.

2:50 pm Disabled D9 events (front and rear). This caused a sharp decrease in the write pointer velocity, confirming that D9 was the guilty party. At the end of the pass, we skipped the pointer, thus shedding (the ability to read) several hours of data. We waiting until the end of the pass in order to try to get the data from the first of the M flares that had happened that night.

4:30 pm Tried an HV drop of 400V in two steps (to 1838V). There was no change in reset rates, which were ~30k. Mark did an RTS load to keep D9 events from coming back on after eclipse. We then re-disabled D9 events, and due to the RTS load they should now stay off. We also cranked up the D9 thresholds on the rear segment, which no discernible reaction.

6 pm Dropped HV to ~900V. Rear resets dropped from ~30k to 25.7k. Then dropped another 100V to ~800V. Resets now 23.5k. Also raised the D9 FRONT fast threshold (the same raise we did on Thursday) in order to leave the front thresholds in the same condition they had started in that day. Later examination of the VC1 plots indicates that this was the time when the D9 ULDs quieted.


And here’s a repeat of the data quality assessment while the SSR was filling up or full:

There were three M class flares on the 21st; all of them happened after the D9 ULD events started going crazy. Here's a rundown:

Data gaps: We had to skip the pointer, so we lost some data.

Missing Aug. 21 1:33:40 to 18:43:10 Missing 19:06:40 to 20:15:18, though part of this is eclipse. Missing 21:19 to 21:51:40 Missing 23:29 onward, but just might not have this down yet.

Aug. 21 M flares:

M flare round ~0115: We were able to salvage most of the preimpulsive phase by careful timing of the pointer skip (i.e. downloading the flare data during a pass and skipping the pointer at the end). We couldn't wait longer than that, so we missed ~15 minutes of the preimpulsive part, then also miss the decay phase. (Flare peak happened during RHESSI eclipse.)

M flare around ~1000: No data at all. Most of the flare occurred during eclipse, so we wouldn’t have seen much anyway.

M flare around ~2034: We have some data from this flare, though there are intermittent gaps. Data probably not of high quality since SSR was very full at this point (i.e. high decimation).

Attenuators

The attenuators came in and out during the set of M flares on Thursday, so that's nice. Over the weekend the thin attenuator came in several times and stayed in for an inordinate amount of time before coming out, indicating that it was on the verge of getting stuck in. On Tuesday (Aug 25) it did get stuck in, so we changed the livetime threshold. Both the thin and thick attenuator out thresholds were changed by 1 count each. (The thick attenuator threshold had also just been changed on Monday to stay one tick ahead of the thin attenuator. With this latest change to the thin we also kept the thick 1 tick ahead.)

The thin out threshold is now "8" or 3.14% and the thick out is "9" or 3.53% dead time. These changes were made August 25, 18:42:24 UTC. Once they were made the thin attenuator came out.

Other notes

Spacecraft Management

Decimation Normal/Vigorous
Night time data (fronts) +- 4 minutes
Night time data (rears) +- 4 minutes
Require extra passes? No
Requirement for moving pointer? YES
Attenuator operation Normal
Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox