corgidrp.darks#

Exceptions#

CalDarksLSQException

Exception class for calibrate_darks_lsq.

Functions#

`mean_combine`(dataset_or_image_list, bpmap_list[, err])	Get mean frame and corresponding bad-pixel map from L2b data frames. The
`build_trad_dark`(dataset, detector_params[, ...])	This function produces a traditional master dark from a stack of darks
`calibrate_darks_lsq`(dataset, detector_params[, ...])	The input dataset represents a collection of frame stacks of the
`build_synthesized_dark`(dataset, noisemaps[, ...])	Assemble a master dark SCI frame (full or image)

Module Contents#

corgidrp.darks.mean_combine(dataset_or_image_list, bpmap_list, err=False)[source]#

Get mean frame and corresponding bad-pixel map from L2b data frames. The input image_list should consist of frames with no bad pixels marked or removed. This function takes the bad-pixels maps into account when taking the mean.

The two lists must be the same length, and each 2D array in each list must be the same size, both within a list and across lists.

If the inputs are instead np.ndarray (a single frame or a stack), the function will accommodate and convert them to lists of arrays.

Also Includes outputs for processing darks used for calibrating the master dark.

Parameters:: dataset_or_image_list (data.Dataset, list, or array_like) – Dataset or list (or stack) of L2b data frames

(with no bad pixels applied to them).: bpmap_list (list or array_like): List (or stack) of bad-pixel maps associated with L2b data frames. Each must be 0 (good) or 1 (bad) at every pixel. If first input is a Dataset, this input is ignored. err (bool): If True, calculates the standard error over all the frames. Intended for the corgidrp.Data.Dataset.all_err arrays. Defaults to False.

Returns:

Mean-combined frame from input list data.

comb_bpmap (array_like): Mean-combined bad-pixel map.

map_im (array-like): Array showing how many frames per pixel were unmasked. Used for getting read noise in the calibration of the master dark.

enough_for_rn (bool): Useful only for the calibration of the master dark. False: Fewer than half the frames available for at least one pixel in the averaging due to masking, so noise maps cannot be effectively determined for all pixels. True: Half or more of the frames available for all pixels, so noise mpas can be effectively determined for all pixels.

Return type:

comb_image (array_like)

corgidrp.darks.build_trad_dark(dataset, detector_params, detector_regions=None, full_frame=False)[source]#

This function produces a traditional master dark from a stack of darks taken at a specific EM gain and exposure time to match a corresponding observation. The input dataset represents a stack of dark frames (in e-). The frames should be SCI full frames that:

have had their bias subtracted (assuming full frame)
have had masks made for cosmic rays
have been corrected for nonlinearity
have been converted from DN to e-
have been divided by EM gain
have NOT been desmeared. Darks should not be desmeared. The only component

of dark frames that would be subject to a smearing effect is dark current since it linearly increases with time, so the extra row read time affects the dark current per pixel. However, illuminated images would also contain this smeared dark current, so dark subtraction should remove this smeared dark current (and then desmearing may be applied to the processed image if appropriate).

Also, add_shot_noise_to_err() should NOT have been applied to the frames in dataset. And note that creation of the fixed bad pixel mask containing warm/hot pixels and pixels with sub-optimal functionality requires a master dark, which requires this function first.

There are rows of each frame that are used for telemetry and are irrelevant for making a master dark. So this function disregards telemetry rows and does not do any fitting for master dark for those rows. They are set to NaN.

Args: dataset (corgidrp.data.Dataset):

This is an instance of corgidrp.data.Dataset. Each frame should accord with the SCI full frame geometry. If Dataset has metadata only (as in RAM-heavy case), each frame is read in from its filepath one at a time. If Dataset has its data, then all the frames are processed at once.

detector_params (corgidrp.data.DetectorParams):: a calibration file storing detector calibration values
detector_regions (dict):: a dictionary of detector geometry properties. Keys should be as found in detector_areas in detector.py. Defaults to None, in which case detector_areas from detector.py is used.
full_frame (bool):: If True, a full-frame master dark is generated (which may be useful for the module that statistically fits a frame to find the empirically applied EM gain, for example). If False, an image-area master dark is generated. Defaults to False.

Returns: master_dark : corgidrp.data.DetectorNoiseMaps instance

The mean-combined master dark, in detected electrons. master_dark.err includes the statistical error across all the frames as well as any individual err from each frame (and accounts for masked pixels in the calculations). master_dark.dq: pixels that are masked for all frames have non-zero values.

exception corgidrp.darks.CalDarksLSQException[source]#

Bases: Exception

Exception class for calibrate_darks_lsq.

corgidrp.darks.calibrate_darks_lsq(dataset, detector_params, weighting=True, detector_regions=None)[source]#

The input dataset represents a collection of frame stacks of the (in e- units), where the stacks are for various EM gain values and exposure times. Stacks with fewer frames than other stacks are accordingly weighed less in the fit. The frames in each stack should be SCI full frames that:

have had their bias subtracted (assuming full frame)
have had masks made for cosmic rays
have been corrected for nonlinearity
have been converted from DN to e-
have NOT been desmeared. Darks should not be desmeared. The only component

The steps shown above are a subset of the total number of steps involved in going from L1 to L2b. This function averages each stack (which minimizes read noise since it has a mean of 0) while accounting for masks. It then computes a per-pixel map of fixed-pattern noise (due to electromagnetic pick-up before going through the amplifier), dark current, and the clock-induced charge (CIC), and it also returns the bias offset value. The function assumes the stacks have the same noise profile (at least the same CIC, fixed-pattern noise, and dark current).

Args: dataset (corgidrp.data.Dataset):

This is an instance of corgidrp.data.Dataset. The function sorts it into stacks where each stack is a stack of dark frames, and each stack is for a unique EM gain and frame time combination. Each stack should have the same number of frames. Each frame should accord with the SCI frame geometry. We recommend >= 1176 frames for each stack if calibrating darks for analog frames, thousands for photon counting depending on the maximum number of frames that will be used for photon counting. If Dataset has metadata only (as in RAM-heavy case), each frame is read in from its filepath one at a time. If Dataset has its data, then all the frames are processed at once.

detector_params (corgidrp.data.DetectorParams):: a calibration file storing detector calibration values
weighting (bool):: If True, weighting is used for the least squares fit, and the weighting takes into account the err coming from the input frames, the statistical variation among the supposedly identical frames in each sub-stack, and the effect of any DQ masking. If False, all data is evenly weighted in the least squares fit. Defaults to True.
detector_regions (dict):: a dictionary of detector geometry properties. Keys should be as found in detector_areas in detector.py. Defaults to None, in which case detector_areas from detector.py is used.

Returns: noise_maps : corgidrp.data.DetectorNoiseMaps instance

Includes a 3-D stack of frames for the data, err, and the dq. input data: np.stack([FPN_map, CIC_map, DC_map]) input err: np.stack([FPN_std_map, C_std_map, DC_std_map]) FPN_std_map, C_std_map, and DC_std_map contain the fitting error. In all the err, masked pixels are accounted for in the calculations, and the err from the input frames, along with statistical error due to having fewer frames available per sub-stack due to any masking, is used for weighting the data in the least squares fit. input dq: np.stack([output_dq, output_dq, output_dq]) The pixels that are masked for EVERY frame in all sub-stacks but 3 (or less) are assigned a flag value from the combination of the frames. These pixels would have no reliability for dark subtraction.

The header info is taken from that of one of the frames from the input datasets and can be changed via a call to the DetectorNoiseMaps class if necessary. The bias offset info is found in the exthdr under these keys: ‘B_O’: bias offset ‘B_O_ERR’: bias offset error ‘B_O_UNIT’: DN

Info on intermediate products in this function: FPN_map : array-like (full frame)

A per-pixel map of fixed-pattern noise (in detected electrons). Any negative values from the fit are made positive in the end.

CIC_maparray-like (full frame): A per-pixel map of EXCAM clock-induced charge (in detected electrons). Any negative values from the fit are made positive in the end.
DC_maparray-like (full frame): A per-pixel map of dark current (in detected electrons/s). Any negative values from the fit are made positive in the end.
bias_offsetfloat: The median for the residual FPN+CIC in the region where bias was calculated (i.e., prescan). In DN.
bias_offset_upfloat: The upper bound of bias offset, accounting for error in input datasets and the fit.
bias_offset_lowfloat: The lower bound of bias offset, accounting for error in input datasets and the fit.
FPN_image_maparray-like (image area): A per-pixel map of fixed-pattern noise in the image area (in detected electrons). Any negative values from the fit are made positive in the end.
CIC_image_maparray-like (image area): A per-pixel map of EXCAM clock-induced charge in the image area (in deteceted electrons). Any negative values from the fit are made positive in the end.
DC_image_maparray-like (image area): A per-pixel map of dark current in the image area (in detected electrons/s). Any negative values from the fit are made positive in the end.
FPNvarfloat: Variance of fixed-pattern noise map (in detected electrons).
CICvarfloat: Variance of clock-induced charge map (in detected electrons).
DCvarfloat: Variance of dark current map (in detected electrons).
read_noisefloat: Read noise estimate from the noise profile of a mean frame (in detected electrons). It’s read off from the sub-stack with the lowest product of EM gain and frame time so that the gained variance of C and D is comparable to or lower than read noise variance, thus making reading it off doable. If read_noise is returned as NaN, the read noise estimate is not trustworthy, possibly because not enough frames were used per substack for that or because the next lowest gain setting is much larger than the gain used in the sub-stack. The official calibrated read noise comes from the k gain calibration, and this is just a rough estimate that can be used as a sanity check, for checking agreement with the official calibrated value.
R_maparray-like: A per-pixel map of the adjusted coefficient of determination (adjusted R^2) value for the fit.
FPN_image_meanfloat: F averaged over all pixels, before any negative ones are made positive. Should be roughly the same as taking the mean of F_image_map. This is just for comparison.
CIC_image_meanfloat: C averaged over all pixels, before any negative ones are made positive. Should be roughly the same as taking the mean of C_image_map. This is just for comparison.
DC_image_meanfloat: D averaged over all pixels, before any negative ones are made positive. Should be roughly the same as taking the mean of D_image_map. This is just for comparison.
unreliable_pix_maparray-like (full frame): A pixel value in this array indicates how many sub-stacks are usable for a fit for that pixel. For each sub-stack for which a pixel is masked for more than half of the frames in the sub-stack, 1 is added to that pixel’s value in unreliable_pix_map. Since the least-squares fit function has 3 parameters, at least 4 sub-stacks are needed for a given pixel in order to perform a fit for that pixel. The pixels in unreliable_pix_map that are >= len(stack_arr)-3 cannot be fit. The pixels that are masked for EVERY frame in all sub-stacks but 3 (or less) are assigned a flag value from the combination of the frames. These pixels would have no reliability for dark subtraction.
FPN_std_maparray-like (full frame): The standard deviation per pixel for the calibrated FPN.
CIC_std_maparray-like (full frame): The standard deviation per pixel for the calibrated CIC.
DC_std_maparray-like (full frame): The standard deviation per pixel for the calibrated dark current.

corgidrp.darks.build_synthesized_dark(dataset, noisemaps, detector_regions=None, full_frame=False)[source]#

Assemble a master dark SCI frame (full or image) from individual noise components.

This is done this way because the actual dark frame varies with gain and exposure time, and both of these values may vary over orders of magnitude in the course of acquisition, alignment, and HOWFSC. Better to take data sets that don’t vary.

Output is a bias-subtracted, gain-divided master dark in detected electrons. (Bias is inherently subtracted as we don’t use it as one of the building blocks to assemble the dark frame.)

Master dark = (F + g*t*D + g*C)/g = F/g + t*D + C where F = FPN (fixed-pattern noise) map g = EM gain t = exposure time D = dark current map C = CIC (clock-induced charge) map

Arguments: dataset: corgidrp.data.Dataset instance. The dataset should consist of

frames all with the same EM gain and exposure time, which are read off from the dataset headers.

noisemaps: corgidrp.data.DetectorNoiseMaps instance. The noise maps used: to build the master dark.
detector_regions: dict. A dictionary of detector geometry properties.: Keys should be as found in detector_areas in detector.py. Defaults to detector_areas in detector.py.
full_frame: bool. If True, a full-frame master dark is generated (which: may be useful for the module that statistically fits a frame to find the empirically applied EM gain, for example). If False, an image-area master dark is generated. Defaults to False.

Returns: master_dark: corgidrp.data.Dark instance.

This contains the master dark in detected electrons.