Tohban Report 2015-08-26

From RHESSI Wiki

(Difference between revisions)
Jump to: navigation, search
(Data Gaps)
(Detector issues)
 
(6 intermediate revisions not shown)
Line 24: Line 24:
== Memory Management ==
== Memory Management ==
-
On Aug 20/21, the SSR filled up due to skyrocketing ULD events in D9 rear.  See description below.
+
On Aug 20/21, the SSR filled up due to skyrocketing ULD events in D9 rear.  See description below.  Since the pointer skip, the fill is still large, probably not *just* due to solar activity (max 50% yesterday, fell to 30% at end of pass set).  Detectors may be putting in noise counts.
== Spacecraft Status ==
== Spacecraft Status ==
-
CP1 is regularly >135K on the spacecraft status page, showing up as a yellow warning.
+
CP1 is regularly >135K on the spacecraft status page, showing up as a yellow warning.  Hard limit is at 145K.
Line 37: Line 37:
  GAP START TIME              GAP END TIME                  GAP (SEC)
  GAP START TIME              GAP END TIME                  GAP (SEC)
  2015-08-21T01:30:00.000 -- 2015-08-21T18:45:00.000      62100.000
  2015-08-21T01:30:00.000 -- 2015-08-21T18:45:00.000      62100.000
 +
 +
There are also a couple ~half orbits missing after that, probably due to SSR being essentially full.
== Detector issues ==
== Detector issues ==
Line 48: Line 50:
Detector 9:  More on the D9 rear ULD escapade below; here is only a note on the front threshold change, which occurred on Aug. 20.  This was an attempt to free up livetime, but it was unsuccessful; there was no measured effect.  D9 front livetime is very low, about 15%.  On Aug. 21 we backed off the previous front fast adjustment as a test then reimplemented it later.  (Commanding not shown here for brevity.)
Detector 9:  More on the D9 rear ULD escapade below; here is only a note on the front threshold change, which occurred on Aug. 20.  This was an attempt to free up livetime, but it was unsuccessful; there was no measured effect.  D9 front livetime is very low, about 15%.  On Aug. 21 we backed off the previous front fast adjustment as a test then reimplemented it later.  (Commanding not shown here for brevity.)
-
Detector 6:  D6 is not in good shape, with high slow and fast rates, high resets, and somewhat reduced livetime in the front.  Livetime is also somewhat low in the rear segment, and fast rates are high, but the rear resets are not terrible.  This detector needs attention (first try fast threshold changes in front and rear) but there wasn't time for it this week.
+
Detector 6:  D6 is not in good shape, with high slow and fast rates, high resets, and somewhat reduced livetime in the front.  Livetime is also somewhat low in the rear segment, and fast rates are high, but the rear resets are not terrible.  This detector needs attention (first try fast threshold changes in front and rear).
'''More thresholds were changed on Monday, Aug. 24.  These are not related to the D9 antics:'''
'''More thresholds were changed on Monday, Aug. 24.  These are not related to the D9 antics:'''
Line 54: Line 56:
Detector 3:  The rear fast threshold was raised.  This caused the confusing result that the fast rates increased and the livetime decreased.  Upon realizing that D3 had actually gotten so bad that it had switched into the negative universe where up is down and down is up, we raised the threshold again.  This was successful in bringing down the rates and restored the livetime to 64%.
Detector 3:  The rear fast threshold was raised.  This caused the confusing result that the fast rates increased and the livetime decreased.  Upon realizing that D3 had actually gotten so bad that it had switched into the negative universe where up is down and down is up, we raised the threshold again.  This was successful in bringing down the rates and restored the livetime to 64%.
-
Detector 4:  The front fast threshold was raised again, bringing the livetime up to 55%.  Even after this, fast and slow rates were both high.  The fast threshold was raised again, which brought the fast rates down by an order of magnitude.  Once the data is down, the next tohban should examine whether this detector is putting in a bunch of noise into the event list; if so, the slow threshold should be raised.
+
Detector 4:  The front fast threshold was raised again, bringing the livetime up to 55%.  Even after this, fast and slow rates were both high.  The fast threshold was raised again, which brought the fast rates down by an order of magnitude.  Once the data is down, the next tohban should examine whether this detector is putting in a bunch of noise into the event list; if so, the slow threshold should be raised. The fast threshold changes are working, but we're almost out of head room!  (At 0xE0 now.)
Detector 5:  The rear fast threshold was raised, successfully bringing the livetime up to >75%.
Detector 5:  The rear fast threshold was raised, successfully bringing the livetime up to >75%.
-
Detector 6:  Both front and rear fast thresholds were raised again.  This successfully brought the livetime up to 75% in the front, but didn't really change anything in the rear.  Another attempt was made with the rear fast threshold, but again no change.  The slow threshold was also raised, which brought the slow rates down from 200k to 5k.
+
Detector 6:  Both front and rear fast thresholds were raised again.  This successfully brought the livetime up to 75% in the front, but didn't really change anything in the rear.  Another attempt was made with the rear fast threshold, but again no change.  The front slow threshold was also raised, which brought the slow rates down from 200k to 5k.
Those last sets of changes were made in two batches:  one at Aug. 24 20:42--20:47 and the other at Aug. 25 00:02--00:04.
Those last sets of changes were made in two batches:  one at Aug. 24 20:42--20:47 and the other at Aug. 25 00:02--00:04.
Line 67: Line 69:
http://sprg.ssl.berkeley.edu/~tohban/browser/?show=monru&date=20150820&time=201739&bar=1
http://sprg.ssl.berkeley.edu/~tohban/browser/?show=monru&date=20150820&time=201739&bar=1
 +
 +
[[File:rhessi-2.png]]
D9 events were left off and the detector was left at 800V over the weekend, so that there would be no danger of a repeat occurrence during a time when tohban/ops reaction time is likely to be slower.  
D9 events were left off and the detector was left at 800V over the weekend, so that there would be no danger of a repeat occurrence during a time when tohban/ops reaction time is likely to be slower.  
Line 115: Line 119:
== Other notes ==
== Other notes ==
 +
There have been no quicklook images/spectra recently, and it looks like flares aren't pulling a flare trigger!  Jim thinks this could be due to D9 being out, plus some other detectors being in very poor state over the last week, so that the routines didn't think there was a source on the Sun.  He'll look into ways to deal with this.

Latest revision as of 00:11, 27 August 2015


Tohban Reports
Start Date: 19 Aug 2015
End Date: 26 Aug 2015
Tohban: Lindsay Glesener
Tohban email: glesener@ssl.berkeley.edu
Next Tohban: TBD
List all reports



Contents

Solar Activity

We had an active week with many M flares! There remains in effect a major flare watch, which has been in place the last few days. The AR (12403) that produced these flares is in the SW quadrant and will be on disk for the next few days.

How many GOES flares occurred?

 Flares above B, C, M, X class were     25    47     7     0

And how many of these are listed in the RHESSI flare list?

 Flares above B, C, M, X class were     10     5     1     0

And how many had EXCELLENT coverage?

 Flares above B, C, M, X class were      0     0     0     0

There were RHESSI flares/GOES flares 88 / 79 over the time range 19-Aug-15 26-Aug-15


Memory Management

On Aug 20/21, the SSR filled up due to skyrocketing ULD events in D9 rear. See description below. Since the pointer skip, the fill is still large, probably not *just* due to solar activity (max 50% yesterday, fell to 30% at end of pass set). Detectors may be putting in noise counts.


Spacecraft Status

CP1 is regularly >135K on the spacecraft status page, showing up as a yellow warning. Hard limit is at 145K.


Data Gaps

A long gap (~half a day) due to need to skip the pointer after the SSR filled up:

GAP START TIME              GAP END TIME                   GAP (SEC)
2015-08-21T01:30:00.000 -- 2015-08-21T18:45:00.000       62100.000

There are also a couple ~half orbits missing after that, probably due to SSR being essentially full.

Detector issues

(For convenience, I'm leaving out the commanding logs, but these can be found on the individual detector update pages.)

Some thresholds were changed on Aug. 20 around ~22:20:

Detectors 3 and 4: The front fast thresholds were raised in order to lower fast rates and free up livetime. Both changes were successful in achieving this. For D3, livetime rose from ~50% to >90%. For D4, livetime rose from ~0% to ~50%. The D4 threshold should be raised again to further raise the livetime. Over the next few days this action was not repeated because the D9 business ate up my own livetime.

Detector 9: More on the D9 rear ULD escapade below; here is only a note on the front threshold change, which occurred on Aug. 20. This was an attempt to free up livetime, but it was unsuccessful; there was no measured effect. D9 front livetime is very low, about 15%. On Aug. 21 we backed off the previous front fast adjustment as a test then reimplemented it later. (Commanding not shown here for brevity.)

Detector 6: D6 is not in good shape, with high slow and fast rates, high resets, and somewhat reduced livetime in the front. Livetime is also somewhat low in the rear segment, and fast rates are high, but the rear resets are not terrible. This detector needs attention (first try fast threshold changes in front and rear).

More thresholds were changed on Monday, Aug. 24. These are not related to the D9 antics:

Detector 3: The rear fast threshold was raised. This caused the confusing result that the fast rates increased and the livetime decreased. Upon realizing that D3 had actually gotten so bad that it had switched into the negative universe where up is down and down is up, we raised the threshold again. This was successful in bringing down the rates and restored the livetime to 64%.

Detector 4: The front fast threshold was raised again, bringing the livetime up to 55%. Even after this, fast and slow rates were both high. The fast threshold was raised again, which brought the fast rates down by an order of magnitude. Once the data is down, the next tohban should examine whether this detector is putting in a bunch of noise into the event list; if so, the slow threshold should be raised. The fast threshold changes are working, but we're almost out of head room! (At 0xE0 now.)

Detector 5: The rear fast threshold was raised, successfully bringing the livetime up to >75%.

Detector 6: Both front and rear fast thresholds were raised again. This successfully brought the livetime up to 75% in the front, but didn't really change anything in the rear. Another attempt was made with the rear fast threshold, but again no change. The front slow threshold was also raised, which brought the slow rates down from 200k to 5k.

Those last sets of changes were made in two batches: one at Aug. 24 20:42--20:47 and the other at Aug. 25 00:02--00:04.

More on D9 ULD anomaly:

Late on the 20th (early on the 21st UTC), detector 9 started spouting out ULD events and quickly filled up the SSR. Once this problem was identified on Friday, detector 9 events (both front and rear) were disabled, and the write pointer motion slowed to a reasonable value. The read pointer was skipped in order to empty the SSR of these bad events (though of course some good data was thrown out with the bath water; see summary below). Since the D9 resets were high (~30k), the HV was turned far down to 900V, and then to 800V, in case there was arcing, in order to prevent permanent damage. At this voltage, resets were ~23k. It was seen later that the ULDs stopped going crazy at the time of this change. (The hyperactive ULD rate never showed in the monitor rates that we see during the passes, which let to delays in its diagnosis.) See attached for a plot of D9 events in the event list, and see this link for the monitor rate plots, at the time D9 started its ULD tantrum.

http://sprg.ssl.berkeley.edu/~tohban/browser/?show=monru&date=20150820&time=201739&bar=1

Rhessi-2.png

D9 events were left off and the detector was left at 800V over the weekend, so that there would be no danger of a repeat occurrence during a time when tohban/ops reaction time is likely to be slower.

On Monday, Aug. 24 we turned on D9 events for a partial orbit to assess the state. (2015-236-20:42:45 /itmon value=AFE8.) While the write pointer speed did increase during the pass, we believe this was due to other changes we made around the same time (fast threshold raises, with corresponding increases in slow rates due to higher livetime). Examination of the VC1 plots shows that the ULDs remain at the “normal” value.

In other news, we also raised some detector thresholds on Thursday and Monday. These changes were not related to the D9 ULD activity. There were several changes today since several detectors had been in need of attention, but D9 was eating up all the tohban’s livetime! We also changed to active/vigorous decimation since there’s a Major Flare Watch in effect. Note that the thin attenuator probably needs some tweaking, since it’s staying in for long periods (though eventually coming out).


Longer version:

Here’s the full list of everything we tried in order to diagnose the problem on Friday. Apologies that these times are in PDT. (Plus 7 for UTC.)

1:15 pm Reversed Thursday’s threshold change on D9 only, in case that this had triggered a problem. (The SSR fill-up started not far from the time of Thursday’s threshold changes.) This produced no discernible change. The problem does not manifest itself in any of the monitor rates as seen in the MOC, so instead we looked at the SSR write pointer speed as an indicator.

2:50 pm Disabled D9 events (front and rear). This caused a sharp decrease in the write pointer velocity, confirming that D9 was the guilty party. At the end of the pass, we skipped the pointer, thus shedding (the ability to read) several hours of data. We waiting until the end of the pass in order to try to get the data from the first of the M flares that had happened that night.

4:30 pm Tried an HV drop of 400V in two steps (to 1838V). There was no change in reset rates, which were ~30k. Mark did an RTS load to keep D9 events from coming back on after eclipse. We then re-disabled D9 events, and due to the RTS load they should now stay off. We also cranked up the D9 thresholds on the rear segment, which no discernible reaction.

6 pm Dropped HV to ~900V. Rear resets dropped from ~30k to 25.7k. Then dropped another 100V to ~800V. Resets now 23.5k. Also raised the D9 FRONT fast threshold (the same raise we did on Thursday) in order to leave the front thresholds in the same condition they had started in that day. Later examination of the VC1 plots indicates that this was the time when the D9 ULDs quieted.


And here’s a repeat of the data quality assessment while the SSR was filling up or full:

There were three M class flares on the 21st; all of them happened after the D9 ULD events started going crazy. Here's a rundown:

Data gaps: We had to skip the pointer, so we lost some data.

Missing Aug. 21 1:33:40 to 18:43:10 Missing 19:06:40 to 20:15:18, though part of this is eclipse. Missing 21:19 to 21:51:40 Missing 23:29 onward, but just might not have this down yet.

Aug. 21 M flares:

M flare round ~0115: We were able to salvage most of the preimpulsive phase by careful timing of the pointer skip (i.e. downloading the flare data during a pass and skipping the pointer at the end). We couldn't wait longer than that, so we missed ~15 minutes of the preimpulsive part, then also miss the decay phase. (Flare peak happened during RHESSI eclipse.)

M flare around ~1000: No data at all. Most of the flare occurred during eclipse, so we wouldn’t have seen much anyway.

M flare around ~2034: We have some data from this flare, though there are intermittent gaps. Data probably not of high quality since SSR was very full at this point (i.e. high decimation).

Attenuators

The attenuators came in and out during the set of M flares on Thursday, so that's nice. Over the weekend the thin attenuator came in several times and stayed in for an inordinate amount of time before coming out, indicating that it was on the verge of getting stuck in. On Tuesday (Aug 25) it did get stuck in, so we changed the livetime threshold. Both the thin and thick attenuator out thresholds were changed by 1 count each. (The thick attenuator threshold had also just been changed on Monday to stay one tick ahead of the thin attenuator. With this latest change to the thin we also kept the thick 1 tick ahead.)

The thin out threshold is now "8" or 3.14% and the thick out is "9" or 3.53% dead time. These changes were made August 25, 18:42:24 UTC. Once they were made the thin attenuator came out.

Other notes

There have been no quicklook images/spectra recently, and it looks like flares aren't pulling a flare trigger! Jim thinks this could be due to D9 being out, plus some other detectors being in very poor state over the last week, so that the routines didn't think there was a source on the Sun. He'll look into ways to deal with this.


Spacecraft Management

Decimation Active/Vigorous
Night time data (fronts) +- 4 minutes
Night time data (rears) +- 4 minutes
Require extra passes? No
Requirement for moving pointer? YES
Attenuator operation Thin got stuck in; both thin and thick thresholds changed.
Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox