The Walker and Recipe System#

corgidrp.walker is the top-level driver for the pipeline. It reads a list of raw FITS files, selects an appropriate processing recipe, fills in calibration file paths from the calibration database (caldb), and executes the recipe step-by-step. Given a list of filepaths and a CPGS XML filepath, most users only need to call a single function:

import corgidrp.walker as walker

walker.walk_corgidrp(filelist, CPGS_XML_filepath, outputdir)

You can also pass a completely custom template:

walker.walk_corgidrp(filelist, CPGS_XML_filepath, outputdir,
    template="/path/to/my_custom_recipe.json"
)

Public Functions#

corgidrp.walker.walk_corgidrp() — the main entry point; combines recipe generation and execution (functions described below).
corgidrp.walker.autogen_recipe() — finds and returns a recipe without running it (including filling in necessary calibrations), useful for inspection or modification before execution.
corgidrp.walker.run_recipe() — executes a pre-built recipe

Example Usages#

import glob
import corgidrp.walker as walker

# Collect all raw L1 frames from the visit directory
filelist = sorted(glob.glob("/data/raw/visit_042/*.fits"))

# Point to the accompanying CPGS observation specification
cpgs_xml = "/data/raw/visit_042/CPGS_obs_042.xml"

# Run the full pipeline; template is chosen automatically
recipe = walker.walk_corgidrp(filelist, cpgs_xml, "/data/processed/visit_042/")

print("Used recipe:", recipe["name"])
print("Pipeline steps:", [s["name"] for s in recipe["steps"]])

To inspect the recipe before running it:

import json
import corgidrp.walker as walker

recipe = walker.autogen_recipe(filelist, "/data/processed/visit_042/")
print(json.dumps(recipe, indent=2))

# Optionally modify the recipe, then run it
walker.run_recipe(recipe)

Recipe Format (JSON)#

Recipes are JSON text files. The DRP ships with a library of templates in corgidrp/recipe_templates/. When a recipe is generated by autogen_recipe, template is set to false and inputs/outputdir are populated. Here, we describe the possible fields in a recipe.

Top-level fields#

Field	Type	Description
`name`	string	Human-readable name for the recipe (e.g. `"l1_to_l2a_basic"`).
`template`	bool	`true` in template files; the walker sets it to `false` on generated recipes to distinguish templates from full recipes.
`inputs`	list[str]	Absolute paths to the input FITS files. Empty in templates; populated by the walker at generation time.
`outputdir`	string	Absolute path to the output directory. Empty in templates; populated by the walker.
`steps`	list[str]	Ordered list of pipeline steps. See Pipeline Steps below.
`drpconfig`	object (optional)	Key/value pairs that temporarily override global `corgidrp` settings for the duration of this recipe (e.g. `"track_individual_errors": false`). Original values are restored after the recipe finishes.
`process_in_chunks`	bool (optional)	If `true`, the input filelist is split into chunks of `corgidrp.chunk_size` frames and processed sequentially. Reduces peak memory usage. Default: `false`.
`ram_heavy`	bool (optional)	If `true`, frame data is not loaded into RAM at the start; each step reads frames individually from disk. Supersedes `process_in_chunks`. Only certain pipeline steps support RAM heavy mode. Default: `false`.

Pipeline Steps#

Each entry in the steps list is a dict with the following fields:

Field	Type	Description
`name`	string	Required. Must match a key in `walker.all_steps` or the special step function `"save"`.
`calibs`	object (optional)	Specifies any required calibrations needed for the step. Use `"AUTOMATIC"` to let the caldb choose the best file at recipe generation time, and add `",OPTIONAL"` (e.g. `"KGain": "AUTOMATIC,OPTIONAL"`) to skip the step gracefully if no matching calibration exists. When filled in by the caldb, each calibration file has a corresponding file path (e.g., `"KGain" : "/path/to/file.fits"`)
`keywords`	object (optional)	Keyword arguments forwarded to the step function. Values must be one of the following types (int, float, str, bool, list of primitives, dict of primitives).
`skip`	bool (optional)	If `true`, the step is skipped entirely. Set automatically when `skip_missing_cal_steps = True` and a required calibration cannot be found.

The special step "save" writes the current dataset to outputdir. It accepts two optional keywords:

suffix — string appended to each output filename before .fits (e.g. "drk" turns frame.fits into frame_drk.fits).
ram_heavy_save — bool; if true, frames are read from disk one at a time during the save. Used only in ram_heavy recipe mode.

Minimal example#

{
    "name": "l1_to_l2a_basic",
    "template": false,
    "inputs": [
        "/data/raw/frame_001.fits",
        "/data/raw/frame_002.fits"
    ],
    "outputdir": "/data/processed/",
    "drpconfig": {
        "track_individual_errors": false
    },
    "steps": [
        {
            "name": "prescan_biassub",
            "calibs": {
                "DetectorNoiseMaps": "/path/to/calib1.fits"
            },
            "keywords": {
                "return_full_frame": false,
                "dataset_copy": false
            }
        },
        {
            "name": "detect_cosmic_rays",
            "calibs": {
                "DetectorParams": "/path/to/calib0.fits",
                "KGain": "/path/to/calib2.fits"
            }
        },
        {
            "name": "correct_nonlinearity",
            "calibs": {
                "NonLinearityCalibration": "/path/to/calib3.fits"
            }
        },
        {
            "name": "update_to_l2a"
        },
        {
            "name": "save"
        }
    ]
}

The full set of recognised step names is defined in walker.all_steps.

Custom Templates#

You can override any default template by placing a JSON file with the same filename in the user templates directory (corgidrp.user_templates_dir, which defaults to ~/.corgidrp/recipe_templates/). The walker checks this directory first.

By default (corgidrp.enforce_template_structure = True), user templates must contain the same step names in the same order as the corresponding default template. You can modify keywords and calibs freely, but cannot add, remove, or reorder steps. To lift this restriction, set this setting to false in the pipeline config file (~/.corgidrp/corgidrp.cfg).