API reference¶
Rendered from docstrings via
mkdocstrings for every symbol in the
public API. See Public API for the canonical list of that
surface.
Sweep entry points¶
Sweep orchestrator: owns the run iterable, backend, manifest, and output dir.
The single public class is :class:Sweep. It binds a list of
:class:gmat_sweep.spec.RunSpec to a backend :class:gmat_sweep.backends.base.Pool,
fans the specs out, drains the resulting outcomes in completion order, and
records each one as a :class:gmat_sweep.manifest.ManifestEntry with an
fsynced append so a mid-sweep Ctrl-C leaves a parseable manifest on disk.
The class does not own the pool's lifecycle — wrap the supplied
:class:Pool in a with block at the call site (or call close())
so worker processes are cleaned up. The thin :func:gmat_sweep.api.sweep
wrapper takes care of this for the common case.
Sweep(
*,
runs: Iterable[RunSpec],
backend: Pool,
manifest_path: Path,
output_dir: Path,
script_path: Path,
parameter_spec: Mapping[str, Any] | None = None,
sweep_seed: int | None = None,
progress: bool = True,
allow_unisolated_pool: bool = False,
fsync_each: bool = True,
fsync_batch: int = 50,
expected_run_count: int | None = None,
postprocess: str | None = None,
)
Bind run specs, a pool, and a manifest path into a runnable orchestrator.
Parameters¶
runs:
The :class:RunSpec instances to dispatch. run_id values must be
unique. Order is preserved on the submission side; outcomes return in
completion order. Pass an :class:Iterable (e.g. the streaming
generator from :func:gmat_sweep.grids.iter_grid_run_specs) to keep a
large factorial out of driver memory; in that case
expected_run_count is required for the manifest header.
expected_run_count:
Total run count for the manifest header and progress bar. Required
when runs is a non-:class:Sequence iterable (no __len__);
derived as len(runs) otherwise. The :meth:resume and
:meth:extend paths always pass a finite :class:Sequence so this
stays None for them.
backend:
A constructed :class:Pool. The caller owns its lifecycle — typically
a with LocalJoblibPool(...) as pool: block.
manifest_path:
Where the JSON Lines manifest will be written. Parent directories are
created on :meth:run.
output_dir:
Sweep root the per-run output directories live under. Used as the
anchor for any relative paths the manifest records.
script_path:
The .script every run loads. Hashed via
:func:canonical_script_sha256 for the manifest header.
parameter_spec:
The original sweep parameterisation (e.g. the materialised grid) —
recorded verbatim in the manifest header for reproducibility.
None (the default) auto-derives the explicit-row shape from
the overrides of runs via :func:parameter_spec_from_runs,
so a sweep built straight from a :class:RunSpec list still
records what varied without the caller hand-building the dict.
Auto-derivation needs a materialised runs :class:Sequence;
pass parameter_spec explicitly when runs is a streaming
iterator.
sweep_seed:
Optional integer seed recorded on the manifest. Sweep does not
consume it directly; the Monte Carlo and Latin hypercube wrappers
in :mod:gmat_sweep.api use it to derive their per-run draws.
progress:
True (default) wraps the drain loop in a :mod:tqdm bar.
Set to False for non-interactive use (tests, CI logs).
allow_unisolated_pool:
Acknowledgement flag for backends whose subprocess_isolated is
not :data:True (today: only
:class:gmat_sweep.backends.debug.DebugPool with the "debug"
sentinel). Defaults to :data:False, in which case constructing a
:class:Sweep over an unisolated backend raises
:class:gmat_sweep.errors.BackendError. Pass :data:True together
with the matching flag on the pool to opt in to in-process,
single-run debug dispatch.
fsync_each:
Forwarded to :class:Manifest. True (default) preserves the
v0.3 strict-per-entry fsync cadence — every appended entry is
durable before the next run is dispatched. False defers
fsyncs to fsync_batch-entry boundaries plus the
end-of-sweep :meth:Manifest.close; on a host crash between
boundaries up to fsync_batch - 1 recently-completed entries
can be lost from the on-disk manifest, but the per-run Parquet
outputs and the script hash are unaffected and the resume flow
re-runs only the missing slice. A KeyboardInterrupt mid-sweep
deliberately skips close() so the same recovery window
applies to user-aborted sweeps.
fsync_batch:
Forwarded to :class:Manifest. Number of entries between
fsyncs when fsync_each is False. Ignored when
fsync_each is True.
postprocess:
Optional import-path string ("package.module:function")
naming a per-run postprocess hook. Stamped onto every dispatched
:class:RunSpec and recorded on the manifest header, so
:meth:from_manifest re-applies it to resumed and extended runs.
None (default) runs no postprocessing. See
:func:gmat_sweep.worker.run_one for the worker-side contract.
run() -> Sweep
Submit every run, drain outcomes in completion order, return self.
Builds and saves the manifest header up front (one fsync, with the
parent directory created on demand). For each completed
:class:RunOutcome an entry is appended via
:meth:Manifest.append_entry, which fsyncs each line — a Ctrl-C
between any two iterations leaves a parseable file containing exactly
the runs that finished.
Dispatch streams the run iterable through :meth:Pool.imap, which
bounds the in-flight set to roughly 4 * max_workers so a 10⁵-run
sweep does not pin 10⁵ :class:RunSpec payloads + 10⁵ futures in
driver memory. The grid expansion is itself lazy
(:func:gmat_sweep.grids.iter_grid_run_specs), so on a large
factorial neither the spec list nor the future list materialises in
full.
:exc:KeyboardInterrupt is not caught; it propagates so the caller's
with-managed pool exits and cancels still-pending futures.
This is the eager form of :meth:iter_outcomes: run() is
exactly iter_outcomes() drained to exhaustion with each pair
discarded. Reach for :meth:iter_outcomes instead when you want to
observe runs as they land — a live dashboard, an early-exit
condition, or interleaving sweep progress with other work.
iter_outcomes() -> Iterator[tuple[RunSpec, RunOutcome]]
Stream the sweep, yielding (spec, outcome) as each run completes.
The manifest-aware streaming form of :meth:run. Like run() it
builds and saves the manifest header, dispatches every run through
:meth:Pool.imap, and appends one :class:ManifestEntry per
:class:RunOutcome with the same fsynced-append crash safety — but
instead of discarding each outcome it yields the (spec, outcome)
pair to the caller in completion order. The yielded pair has the
same shape :meth:Pool.imap produces.
Being a generator, the body runs lazily: the manifest header is
written when iteration starts (the first next()), not when
iter_outcomes() is called. Abandoning the iterator partway
through leaves the manifest on disk exactly as a Ctrl-C would —
every run that finished is recorded and parseable; the trailing
:meth:Manifest.close fsync is reached only once the stream drains
in full. :exc:KeyboardInterrupt is not caught; it propagates so
the caller's with-managed pool exits and cancels still-pending
futures.
Example — record outcomes as they land, then aggregate once the stream is exhausted:
classmethod
¶
from_manifest(
manifest_path: str | Path,
script_path: str | Path,
*,
backend: Pool,
output_dir: str | Path | None = None,
allow_script_drift: bool = False,
context_provider: Callable[[int], dict[str, Any]]
| None = None,
progress: bool = True,
fsync_each: bool = True,
fsync_batch: int = 50,
) -> Sweep
Rebuild a :class:Sweep from a manifest written by a prior run.
Reads manifest_path, validates that the on-disk script still
matches the manifest's recorded script_sha256, reconstructs the
run iterable from the manifest's parameter_spec, and returns a
:class:Sweep whose manifest is pre-bound to the loaded one. The
returned sweep is suitable input to :meth:resume; calling
:meth:run on it would re-execute every run from scratch and is
not the intended flow.
Each rebuilt run's per-run :attr:RunSpec.context is restored from
its manifest entry, so a resumed context-dependent postprocess
hook behaves identically to the original run. A run that never
produced an entry — never dispatched before the original sweep was
interrupted — has no recorded context; pass context_provider
to recompute it.
Parameters¶
manifest_path:
Path to the existing manifest.jsonl. By default its parent
directory is treated as the sweep's output directory; pass
output_dir to point elsewhere.
script_path:
Path to the GMAT .script to load. The file's canonical
SHA-256 must equal the manifest's script_sha256 unless
allow_script_drift is set.
backend:
A constructed :class:Pool. The caller owns its lifecycle —
same contract as the regular constructor.
output_dir:
Base directory for per-run output subdirectories
(output_dir / run-<id>) and the root the resumed sweep
reads successful runs' Parquet files from. None (default)
derives it from manifest_path's parent, matching a sweep
whose manifest sits alongside its per-run directories. Pass an
explicit path when the original dispatch wrote per-run outputs
to a different tree than the manifest. Resolved to an absolute
path and must already exist on disk.
allow_script_drift:
False (default) raises :class:SweepConfigError on a hash
mismatch with both hashes in the message. True proceeds
anyway and emits a :class:RuntimeWarning.
context_provider:
Optional callable supplying the context for runs that have
no manifest entry — runs never dispatched before the original
sweep was interrupted. It is called once per such run with the
run's integer run_id and must return a JSON-encodable
dict. Runs that do have an entry restore the context
they ran with and never reach the provider. None (default)
leaves entry-less runs with an empty context — harmless
unless a postprocess hook needs the per-run payload, in which
case a resumed run that never completed would see {}. The
provider runs in the driver process before dispatch, so it may
be any callable.
progress:
Forwarded to the constructor — controls the :mod:tqdm bar in
:meth:resume.
fsync_each, fsync_batch:
Forwarded to the constructor; control the manifest's fsync
cadence on the appended resume / extend entries. The on-disk
manifest's existing entries are not affected — these knobs
govern only the writes the returned sweep performs.
Raises¶
SweepConfigError
If the resume output directory does not exist, the script hash
drifted and allow_script_drift is False, the manifest's
parameter_spec carries an unknown _kind, or a
context_provider call returns a non-dict.
resume() -> Sweep
Re-run only the failed and missing runs from the loaded manifest.
Submits specs for the union of manifest.find_failed() and manifest.find_missing(...)
through the bound backend, appends one new :class:ManifestEntry
per outcome (with the same run_id as the original), and reloads
the manifest so the in-memory entries reflect the last-wins
merge. Returns self for chaining.
Each retried run is dispatched with overwrite enabled: a run
that failed or was interrupted mid-flight can leave a partial GMAT
output file in its per-run directory, and gmat_run refuses to
start a run whose output path collides with an existing file.
Forcing overwrite clears the stale fragment so the retry runs
cleanly. Successful runs are never re-dispatched and are untouched.
Raises :exc:RuntimeError when called on a :class:Sweep not
produced by :meth:from_manifest.
extend(*, n: int) -> Sweep
Append n more bit-deterministic Monte Carlo runs to a loaded sweep.
Only valid on a :class:Sweep produced by :meth:from_manifest
whose manifest's parameter_spec._kind is "monte_carlo".
The new runs occupy run_id range [old_n, old_n + n) where
old_n is the cumulative high-water mark on disk
(parameter_spec["n"] + manifest.extension_run_count); their
per-parameter draws are bit-equal to the same indices of a fresh
monte_carlo(n=old_n+n, ...) call thanks to the position-
determinism of :func:numpy.random.SeedSequence.spawn.
Refuses with :class:SweepConfigError if any run_id in
[0, old_n) is missing on disk or has a failed latest
status — extending across an unfinished base would silently
produce a manifest with gaps. Call :meth:resume first to fill
them in.
Returns self for chaining (typical use is
sweep.extend(n=...).to_dataframe()).
Pack the sweep — script, manifest, per-run Parquets — into a .zip.
The bundle is suitable for archival deposit (Zenodo, JOSS supplementary
material) or internal handoff. Layout, path-rewrite rules, and the
accompanying MANIFEST.hash are documented in
:mod:gmat_sweep.archive.
Parameters¶
out:
Destination .zip path. Parent directories are created on demand.
include_logs:
When True, every per-run worker.log is bundled and the
manifest's log_path field continues to point at it (rewritten
to bundle-relative form). The default False drops the logs and
sets log_path to None in the bundled manifest, keeping the
archive small.
Returns¶
Path
The resolved path to the produced .zip.
Aggregate the sweep's ReportFile outputs into one DataFrame.
name selects which report to aggregate when the sweep produced
multiple ReportFile resources per run; None (default) picks
the sole report when exactly one was produced. engine="polars"
returns a :class:polars.DataFrame with the (run_id, time)
MultiIndex flattened into two leading columns; the default
engine="pandas" preserves the MultiIndex. See
:func:gmat_sweep.aggregate.lazy_multiindex for the full contract.
Aggregate the sweep's EphemerisFile outputs into one DataFrame.
See :func:gmat_sweep.aggregate.lazy_ephemerides for the contract,
including the engine knob.
Aggregate the sweep's ContactLocator outputs into one DataFrame.
See :func:gmat_sweep.aggregate.lazy_contacts for the contract,
including the engine knob.
Aggregate the sweep's postprocess-hook extra outputs into one DataFrame.
Only meaningful when the sweep ran with a postprocess hook.
name selects which extra-output key to aggregate and is
required — there is no sole "natural" extra output to fall back
to. See :func:gmat_sweep.aggregate.lazy_extra_outputs for the
contract, including the adaptive run_id / (run_id, time)
index and the engine knob.
Aggregate the sweep's Target / Optimize iteration history.
Returns a (run_id, solver, iteration)-MultiIndexed DataFrame of
every solver run across the sweep. Only meaningful when the mission
sequence declared a solver. See
:func:gmat_sweep.aggregate.lazy_solver_runs for the full
contract, including the engine knob.
Summarise solver convergence as a (run_id, solver)-indexed boolean view.
Read straight from the manifest's per-run converged maps — no
Parquet is touched. See
:func:gmat_sweep.aggregate.lazy_solver_convergence for the
contract, including the engine knob.
Aggregate the sweep's ReportFile outputs into a polars DataFrame.
Shortcut for :meth:to_dataframe with engine="polars": returns
a :class:polars.DataFrame whose (run_id, time) MultiIndex is
flattened into two leading sorted columns. Requires the [polars]
extra; raises :class:ImportError with the install hint otherwise.
to_fused_reports(
names: Sequence[str],
*,
tolerance: str | Timedelta,
spool: bool = True,
) -> DataFrame
Fuse N ReportFile outputs per run into one wide MultiIndex-column DataFrame.
See :func:gmat_sweep.aggregate.lazy_fused_reports for the
contract — this is a thin convenience that binds the sweep's own
manifest and output directory.
parameter_spec_from_runs(
runs: Iterable[RunSpec],
) -> dict[str, Any]
Derive an explicit-row parameter_spec from a sequence of run specs.
Builds the _kind="explicit" manifest parameter_spec shape — the
same one :func:gmat_sweep.sweep records for a samples= sweep —
directly from each :class:~gmat_sweep.spec.RunSpec's overrides:
columnsis the union of every spec'soverrideskeys in first-seen order — the order keys are encountered scanningrunsfront to back.rowsholds one list per spec, in the orderrunsyields them, carrying that spec's value for each column. A spec that does not set a given column contributesNonein that slot.
This is the explicit counterpart to constructing a :class:Sweep
without a parameter_spec: the constructor calls this function for
you when the argument is omitted. Call it directly when you want to
inspect or amend the derived spec before handing it back to
:class:Sweep, or to record the run set somewhere other than a sweep
manifest.
The derived spec round-trips — it is valid input to the
:meth:Sweep.from_manifest reconstruction path. One caveat: a column
that every spec leaves unset except as an explicit
overrides={"col": None} derives as an all-None column, which
the explicit-row expander rejects on resume. Specs that all carry the
same override keys — the common case — never reach that edge.
runs is iterated once and may be any iterable; pass a list if you
need to reuse it afterwards.
monte_carlo(
mission: str | Path,
*,
n: int,
perturb: Mapping[str, DistSpec],
seed: int | None = ...,
backend: Pool | None = ...,
out: str | Path | None = ...,
postprocess: str | None = ...,
progress: bool = ...,
fsync_each: bool = ...,
fsync_batch: int = ...,
engine: Literal["pandas"] = ...,
) -> DataFrame
monte_carlo(
mission: str | Path,
*,
n: int,
perturb: Mapping[str, DistSpec],
seed: int | None = ...,
backend: Pool | None = ...,
out: str | Path | None = ...,
postprocess: str | None = ...,
progress: bool = ...,
fsync_each: bool = ...,
fsync_batch: int = ...,
engine: Literal["polars"],
) -> DataFrame
monte_carlo(
mission: str | Path,
*,
n: int,
perturb: Mapping[str, DistSpec],
seed: int | None = None,
backend: Pool | None = None,
out: str | Path | None = None,
postprocess: str | None = None,
progress: bool = True,
fsync_each: bool = True,
fsync_batch: int = 50,
engine: str = "pandas",
) -> DataFrame
Run a Monte Carlo dispersion sweep over a GMAT mission.
Builds an explicit-row run set of n runs by independently sampling
each perturb parameter from its own distribution. Per-parameter
sub-seeds are derived from the parameter name via
:func:derive_param_seed <gmat_sweep.distributions.derive_param_seed>,
so adding a perturbed parameter to an existing sweep does not change
the draws of any other parameter at any run_id.
Parameters¶
mission:
Path to the GMAT .script file every run loads.
n:
Number of stochastic runs. Must be >= 1.
perturb:
Mapping from dotted-path field name to a distribution spec. Each
value is one of the three shorthand tuples (("normal", mu,
sigma), ("uniform", lo, hi), ("lognormal", mu, sigma)) or
any pre-frozen :class:scipy.stats._distn_infrastructure.rv_frozen.
See :data:gmat_sweep.distributions.DistSpec for the full surface.
seed:
Optional integer parent seed. None falls back to OS entropy and
is not reproducible. With an integer seed two calls at the same
(mission, n, perturb, seed) produce bit-equal DataFrames.
backend:
Execution backend; same semantics as :func:sweep.
out:
Sweep output directory; same semantics as :func:sweep.
postprocess:
Per-run postprocess hook import path; same semantics as
:func:sweep.
progress:
Whether to draw the :mod:tqdm progress bar; same semantics as
:func:sweep.
engine:
Output engine; same semantics as :func:sweep.
Returns¶
pandas.DataFrame or polars.DataFrame
(run_id, time)-MultiIndexed frame (or polars flat-key
equivalent under engine="polars"), one row per (run,
time-step) pair, with run_id cardinality n. A failed run
lands as one NaN row with __status="failed" — same contract
as :func:sweep.
Raises¶
SweepConfigError
If perturb is empty, n < 1, any parameter spec is
ill-formed, or engine is neither "pandas" nor
"polars".
monte_carlo_extend(
manifest: str | Path,
script: str | Path,
*,
n: int,
backend: Pool | None = ...,
allow_script_drift: bool = ...,
progress: bool = ...,
fsync_each: bool = ...,
fsync_batch: int = ...,
engine: Literal["pandas"] = ...,
) -> DataFrame
monte_carlo_extend(
manifest: str | Path,
script: str | Path,
*,
n: int,
backend: Pool | None = ...,
allow_script_drift: bool = ...,
progress: bool = ...,
fsync_each: bool = ...,
fsync_batch: int = ...,
engine: Literal["polars"],
) -> DataFrame
monte_carlo_extend(
manifest: str | Path,
script: str | Path,
*,
n: int,
backend: Pool | None = ...,
allow_script_drift: bool = ...,
progress: bool = ...,
fsync_each: bool = ...,
fsync_batch: int = ...,
engine: str,
) -> DataFrame
monte_carlo_extend(
manifest: str | Path,
script: str | Path,
*,
n: int,
backend: Pool | None = None,
allow_script_drift: bool = False,
progress: bool = True,
fsync_each: bool = True,
fsync_batch: int = 50,
engine: str = "pandas",
) -> DataFrame
Append n more bit-deterministic Monte Carlo runs to an existing sweep.
Loads the manifest written by a prior :func:monte_carlo call,
dispatches n new runs at run_id range
[old_n, old_n + n) (where old_n is the cumulative high-water
mark including any prior extensions), and returns the aggregated
DataFrame over all runs (original + every extension applied so
far). Per-parameter draws at the new run_id\ s are bit-equal to
the same indices of a fresh monte_carlo(n=old_n+n, seed=...)
call thanks to the position-determinism of
:func:numpy.random.SeedSequence.spawn.
The original perturb mapping and seed are read from the
manifest's parameter_spec — the caller does not (and cannot)
change them. Adding new perturbed parameters mid-sweep is not
supported and would break determinism.
Parameters¶
manifest:
Path to the existing manifest.jsonl. Its parent is the
sweep's output_dir and must still exist on disk —
successful runs' Parquet files are read from there as-is when
the aggregated DataFrame is built.
script:
Path to the same GMAT .script the original sweep loaded.
Its canonical SHA-256 must equal the manifest's
script_sha256 unless allow_script_drift is set —
otherwise the original runs and the new ones would have loaded
different scripts and the aggregated DataFrame would mix them.
n:
Number of additional runs to dispatch. Must be >= 1.
backend:
Execution backend; same semantics as :func:monte_carlo.
allow_script_drift:
False (default) raises :class:SweepConfigError on a hash
mismatch with both hashes in the message. True proceeds
anyway and emits a :class:RuntimeWarning. Same surface as
:meth:gmat_sweep.Sweep.from_manifest.
progress:
Whether to draw the :mod:tqdm progress bar over the new runs.
engine:
Output engine; same semantics as :func:sweep.
Returns¶
pandas.DataFrame or polars.DataFrame
(run_id, time)-MultiIndexed frame (or polars flat-key
equivalent under engine="polars") whose run_id
cardinality is old_n + n.
Raises¶
SweepConfigError
If the manifest's parameter_spec._kind is not
"monte_carlo", n < 1, the script hash drifted with
allow_script_drift=False, the base sweep is incomplete
(any run_id in [0, old_n) is failed or missing), or
engine is neither "pandas" nor "polars".
latin_hypercube(
mission: str | Path,
*,
n: int,
perturb: Mapping[str, DistSpec],
seed: int | None = ...,
backend: Pool | None = ...,
out: str | Path | None = ...,
postprocess: str | None = ...,
progress: bool = ...,
fsync_each: bool = ...,
fsync_batch: int = ...,
engine: Literal["pandas"] = ...,
) -> DataFrame
latin_hypercube(
mission: str | Path,
*,
n: int,
perturb: Mapping[str, DistSpec],
seed: int | None = ...,
backend: Pool | None = ...,
out: str | Path | None = ...,
postprocess: str | None = ...,
progress: bool = ...,
fsync_each: bool = ...,
fsync_batch: int = ...,
engine: Literal["polars"],
) -> DataFrame
latin_hypercube(
mission: str | Path,
*,
n: int,
perturb: Mapping[str, DistSpec],
seed: int | None = None,
backend: Pool | None = None,
out: str | Path | None = None,
postprocess: str | None = None,
progress: bool = True,
fsync_each: bool = True,
fsync_batch: int = 50,
engine: str = "pandas",
) -> DataFrame
Run a Latin hypercube sweep over a GMAT mission.
Backed by :class:scipy.stats.qmc.LatinHypercube: draws n unit-cube
points stratified across each of len(perturb) axes and maps each
column through the user's distribution via rv.ppf(...). Latin
hypercube sampling typically beats plain Monte Carlo when n is
small relative to the problem's dimensionality, because the coverage
of each axis is enforced by construction.
Parameters¶
mission:
Path to the GMAT .script file every run loads.
n:
Number of Latin hypercube points. Must be >= 1.
perturb:
Mapping from dotted-path field name to a distribution spec — same
accepted shapes as :func:monte_carlo.
seed:
Optional integer seed forwarded to
:class:scipy.stats.qmc.LatinHypercube. None falls back to OS
entropy and is not reproducible. With an integer seed two calls
at the same (mission, n, perturb, seed) produce bit-equal
DataFrames.
backend:
Same semantics as :func:sweep.
out:
Same semantics as :func:sweep.
postprocess:
Per-run postprocess hook import path; same semantics as
:func:sweep.
progress:
Same semantics as :func:sweep.
engine:
Same semantics as :func:sweep.
Returns¶
pandas.DataFrame or polars.DataFrame
(run_id, time)-MultiIndexed frame (or polars flat-key
equivalent under engine="polars") with run_id cardinality
n. Same failure-as-row contract as :func:sweep.
Raises¶
SweepConfigError
If perturb is empty, n < 1, any parameter spec is
ill-formed, or engine is neither "pandas" nor
"polars".
Sweep(
*,
runs: Iterable[RunSpec],
backend: Pool,
manifest_path: Path,
output_dir: Path,
script_path: Path,
parameter_spec: Mapping[str, Any] | None = None,
sweep_seed: int | None = None,
progress: bool = True,
allow_unisolated_pool: bool = False,
fsync_each: bool = True,
fsync_batch: int = 50,
expected_run_count: int | None = None,
postprocess: str | None = None,
)
Bind run specs, a pool, and a manifest path into a runnable orchestrator.
Parameters¶
runs:
The :class:RunSpec instances to dispatch. run_id values must be
unique. Order is preserved on the submission side; outcomes return in
completion order. Pass an :class:Iterable (e.g. the streaming
generator from :func:gmat_sweep.grids.iter_grid_run_specs) to keep a
large factorial out of driver memory; in that case
expected_run_count is required for the manifest header.
expected_run_count:
Total run count for the manifest header and progress bar. Required
when runs is a non-:class:Sequence iterable (no __len__);
derived as len(runs) otherwise. The :meth:resume and
:meth:extend paths always pass a finite :class:Sequence so this
stays None for them.
backend:
A constructed :class:Pool. The caller owns its lifecycle — typically
a with LocalJoblibPool(...) as pool: block.
manifest_path:
Where the JSON Lines manifest will be written. Parent directories are
created on :meth:run.
output_dir:
Sweep root the per-run output directories live under. Used as the
anchor for any relative paths the manifest records.
script_path:
The .script every run loads. Hashed via
:func:canonical_script_sha256 for the manifest header.
parameter_spec:
The original sweep parameterisation (e.g. the materialised grid) —
recorded verbatim in the manifest header for reproducibility.
None (the default) auto-derives the explicit-row shape from
the overrides of runs via :func:parameter_spec_from_runs,
so a sweep built straight from a :class:RunSpec list still
records what varied without the caller hand-building the dict.
Auto-derivation needs a materialised runs :class:Sequence;
pass parameter_spec explicitly when runs is a streaming
iterator.
sweep_seed:
Optional integer seed recorded on the manifest. Sweep does not
consume it directly; the Monte Carlo and Latin hypercube wrappers
in :mod:gmat_sweep.api use it to derive their per-run draws.
progress:
True (default) wraps the drain loop in a :mod:tqdm bar.
Set to False for non-interactive use (tests, CI logs).
allow_unisolated_pool:
Acknowledgement flag for backends whose subprocess_isolated is
not :data:True (today: only
:class:gmat_sweep.backends.debug.DebugPool with the "debug"
sentinel). Defaults to :data:False, in which case constructing a
:class:Sweep over an unisolated backend raises
:class:gmat_sweep.errors.BackendError. Pass :data:True together
with the matching flag on the pool to opt in to in-process,
single-run debug dispatch.
fsync_each:
Forwarded to :class:Manifest. True (default) preserves the
v0.3 strict-per-entry fsync cadence — every appended entry is
durable before the next run is dispatched. False defers
fsyncs to fsync_batch-entry boundaries plus the
end-of-sweep :meth:Manifest.close; on a host crash between
boundaries up to fsync_batch - 1 recently-completed entries
can be lost from the on-disk manifest, but the per-run Parquet
outputs and the script hash are unaffected and the resume flow
re-runs only the missing slice. A KeyboardInterrupt mid-sweep
deliberately skips close() so the same recovery window
applies to user-aborted sweeps.
fsync_batch:
Forwarded to :class:Manifest. Number of entries between
fsyncs when fsync_each is False. Ignored when
fsync_each is True.
postprocess:
Optional import-path string ("package.module:function")
naming a per-run postprocess hook. Stamped onto every dispatched
:class:RunSpec and recorded on the manifest header, so
:meth:from_manifest re-applies it to resumed and extended runs.
None (default) runs no postprocessing. See
:func:gmat_sweep.worker.run_one for the worker-side contract.
run() -> Sweep
Submit every run, drain outcomes in completion order, return self.
Builds and saves the manifest header up front (one fsync, with the
parent directory created on demand). For each completed
:class:RunOutcome an entry is appended via
:meth:Manifest.append_entry, which fsyncs each line — a Ctrl-C
between any two iterations leaves a parseable file containing exactly
the runs that finished.
Dispatch streams the run iterable through :meth:Pool.imap, which
bounds the in-flight set to roughly 4 * max_workers so a 10⁵-run
sweep does not pin 10⁵ :class:RunSpec payloads + 10⁵ futures in
driver memory. The grid expansion is itself lazy
(:func:gmat_sweep.grids.iter_grid_run_specs), so on a large
factorial neither the spec list nor the future list materialises in
full.
:exc:KeyboardInterrupt is not caught; it propagates so the caller's
with-managed pool exits and cancels still-pending futures.
This is the eager form of :meth:iter_outcomes: run() is
exactly iter_outcomes() drained to exhaustion with each pair
discarded. Reach for :meth:iter_outcomes instead when you want to
observe runs as they land — a live dashboard, an early-exit
condition, or interleaving sweep progress with other work.
iter_outcomes() -> Iterator[tuple[RunSpec, RunOutcome]]
Stream the sweep, yielding (spec, outcome) as each run completes.
The manifest-aware streaming form of :meth:run. Like run() it
builds and saves the manifest header, dispatches every run through
:meth:Pool.imap, and appends one :class:ManifestEntry per
:class:RunOutcome with the same fsynced-append crash safety — but
instead of discarding each outcome it yields the (spec, outcome)
pair to the caller in completion order. The yielded pair has the
same shape :meth:Pool.imap produces.
Being a generator, the body runs lazily: the manifest header is
written when iteration starts (the first next()), not when
iter_outcomes() is called. Abandoning the iterator partway
through leaves the manifest on disk exactly as a Ctrl-C would —
every run that finished is recorded and parseable; the trailing
:meth:Manifest.close fsync is reached only once the stream drains
in full. :exc:KeyboardInterrupt is not caught; it propagates so
the caller's with-managed pool exits and cancels still-pending
futures.
Example — record outcomes as they land, then aggregate once the stream is exhausted:
classmethod
¶
from_manifest(
manifest_path: str | Path,
script_path: str | Path,
*,
backend: Pool,
output_dir: str | Path | None = None,
allow_script_drift: bool = False,
context_provider: Callable[[int], dict[str, Any]]
| None = None,
progress: bool = True,
fsync_each: bool = True,
fsync_batch: int = 50,
) -> Sweep
Rebuild a :class:Sweep from a manifest written by a prior run.
Reads manifest_path, validates that the on-disk script still
matches the manifest's recorded script_sha256, reconstructs the
run iterable from the manifest's parameter_spec, and returns a
:class:Sweep whose manifest is pre-bound to the loaded one. The
returned sweep is suitable input to :meth:resume; calling
:meth:run on it would re-execute every run from scratch and is
not the intended flow.
Each rebuilt run's per-run :attr:RunSpec.context is restored from
its manifest entry, so a resumed context-dependent postprocess
hook behaves identically to the original run. A run that never
produced an entry — never dispatched before the original sweep was
interrupted — has no recorded context; pass context_provider
to recompute it.
Parameters¶
manifest_path:
Path to the existing manifest.jsonl. By default its parent
directory is treated as the sweep's output directory; pass
output_dir to point elsewhere.
script_path:
Path to the GMAT .script to load. The file's canonical
SHA-256 must equal the manifest's script_sha256 unless
allow_script_drift is set.
backend:
A constructed :class:Pool. The caller owns its lifecycle —
same contract as the regular constructor.
output_dir:
Base directory for per-run output subdirectories
(output_dir / run-<id>) and the root the resumed sweep
reads successful runs' Parquet files from. None (default)
derives it from manifest_path's parent, matching a sweep
whose manifest sits alongside its per-run directories. Pass an
explicit path when the original dispatch wrote per-run outputs
to a different tree than the manifest. Resolved to an absolute
path and must already exist on disk.
allow_script_drift:
False (default) raises :class:SweepConfigError on a hash
mismatch with both hashes in the message. True proceeds
anyway and emits a :class:RuntimeWarning.
context_provider:
Optional callable supplying the context for runs that have
no manifest entry — runs never dispatched before the original
sweep was interrupted. It is called once per such run with the
run's integer run_id and must return a JSON-encodable
dict. Runs that do have an entry restore the context
they ran with and never reach the provider. None (default)
leaves entry-less runs with an empty context — harmless
unless a postprocess hook needs the per-run payload, in which
case a resumed run that never completed would see {}. The
provider runs in the driver process before dispatch, so it may
be any callable.
progress:
Forwarded to the constructor — controls the :mod:tqdm bar in
:meth:resume.
fsync_each, fsync_batch:
Forwarded to the constructor; control the manifest's fsync
cadence on the appended resume / extend entries. The on-disk
manifest's existing entries are not affected — these knobs
govern only the writes the returned sweep performs.
Raises¶
SweepConfigError
If the resume output directory does not exist, the script hash
drifted and allow_script_drift is False, the manifest's
parameter_spec carries an unknown _kind, or a
context_provider call returns a non-dict.
resume() -> Sweep
Re-run only the failed and missing runs from the loaded manifest.
Submits specs for the union of manifest.find_failed() and manifest.find_missing(...)
through the bound backend, appends one new :class:ManifestEntry
per outcome (with the same run_id as the original), and reloads
the manifest so the in-memory entries reflect the last-wins
merge. Returns self for chaining.
Each retried run is dispatched with overwrite enabled: a run
that failed or was interrupted mid-flight can leave a partial GMAT
output file in its per-run directory, and gmat_run refuses to
start a run whose output path collides with an existing file.
Forcing overwrite clears the stale fragment so the retry runs
cleanly. Successful runs are never re-dispatched and are untouched.
Raises :exc:RuntimeError when called on a :class:Sweep not
produced by :meth:from_manifest.
extend(*, n: int) -> Sweep
Append n more bit-deterministic Monte Carlo runs to a loaded sweep.
Only valid on a :class:Sweep produced by :meth:from_manifest
whose manifest's parameter_spec._kind is "monte_carlo".
The new runs occupy run_id range [old_n, old_n + n) where
old_n is the cumulative high-water mark on disk
(parameter_spec["n"] + manifest.extension_run_count); their
per-parameter draws are bit-equal to the same indices of a fresh
monte_carlo(n=old_n+n, ...) call thanks to the position-
determinism of :func:numpy.random.SeedSequence.spawn.
Refuses with :class:SweepConfigError if any run_id in
[0, old_n) is missing on disk or has a failed latest
status — extending across an unfinished base would silently
produce a manifest with gaps. Call :meth:resume first to fill
them in.
Returns self for chaining (typical use is
sweep.extend(n=...).to_dataframe()).
Pack the sweep — script, manifest, per-run Parquets — into a .zip.
The bundle is suitable for archival deposit (Zenodo, JOSS supplementary
material) or internal handoff. Layout, path-rewrite rules, and the
accompanying MANIFEST.hash are documented in
:mod:gmat_sweep.archive.
Parameters¶
out:
Destination .zip path. Parent directories are created on demand.
include_logs:
When True, every per-run worker.log is bundled and the
manifest's log_path field continues to point at it (rewritten
to bundle-relative form). The default False drops the logs and
sets log_path to None in the bundled manifest, keeping the
archive small.
Returns¶
Path
The resolved path to the produced .zip.
Aggregate the sweep's ReportFile outputs into one DataFrame.
name selects which report to aggregate when the sweep produced
multiple ReportFile resources per run; None (default) picks
the sole report when exactly one was produced. engine="polars"
returns a :class:polars.DataFrame with the (run_id, time)
MultiIndex flattened into two leading columns; the default
engine="pandas" preserves the MultiIndex. See
:func:gmat_sweep.aggregate.lazy_multiindex for the full contract.
Aggregate the sweep's EphemerisFile outputs into one DataFrame.
See :func:gmat_sweep.aggregate.lazy_ephemerides for the contract,
including the engine knob.
Aggregate the sweep's ContactLocator outputs into one DataFrame.
See :func:gmat_sweep.aggregate.lazy_contacts for the contract,
including the engine knob.
Aggregate the sweep's postprocess-hook extra outputs into one DataFrame.
Only meaningful when the sweep ran with a postprocess hook.
name selects which extra-output key to aggregate and is
required — there is no sole "natural" extra output to fall back
to. See :func:gmat_sweep.aggregate.lazy_extra_outputs for the
contract, including the adaptive run_id / (run_id, time)
index and the engine knob.
Aggregate the sweep's Target / Optimize iteration history.
Returns a (run_id, solver, iteration)-MultiIndexed DataFrame of
every solver run across the sweep. Only meaningful when the mission
sequence declared a solver. See
:func:gmat_sweep.aggregate.lazy_solver_runs for the full
contract, including the engine knob.
Summarise solver convergence as a (run_id, solver)-indexed boolean view.
Read straight from the manifest's per-run converged maps — no
Parquet is touched. See
:func:gmat_sweep.aggregate.lazy_solver_convergence for the
contract, including the engine knob.
Aggregate the sweep's ReportFile outputs into a polars DataFrame.
Shortcut for :meth:to_dataframe with engine="polars": returns
a :class:polars.DataFrame whose (run_id, time) MultiIndex is
flattened into two leading sorted columns. Requires the [polars]
extra; raises :class:ImportError with the install hint otherwise.
to_fused_reports(
names: Sequence[str],
*,
tolerance: str | Timedelta,
spool: bool = True,
) -> DataFrame
Fuse N ReportFile outputs per run into one wide MultiIndex-column DataFrame.
See :func:gmat_sweep.aggregate.lazy_fused_reports for the
contract — this is a thin convenience that binds the sweep's own
manifest and output directory.
parameter_spec_from_runs(
runs: Iterable[RunSpec],
) -> dict[str, Any]
Derive an explicit-row parameter_spec from a sequence of run specs.
Builds the _kind="explicit" manifest parameter_spec shape — the
same one :func:gmat_sweep.sweep records for a samples= sweep —
directly from each :class:~gmat_sweep.spec.RunSpec's overrides:
columnsis the union of every spec'soverrideskeys in first-seen order — the order keys are encountered scanningrunsfront to back.rowsholds one list per spec, in the orderrunsyields them, carrying that spec's value for each column. A spec that does not set a given column contributesNonein that slot.
This is the explicit counterpart to constructing a :class:Sweep
without a parameter_spec: the constructor calls this function for
you when the argument is omitted. Call it directly when you want to
inspect or amend the derived spec before handing it back to
:class:Sweep, or to record the run set somewhere other than a sweep
manifest.
The derived spec round-trips — it is valid input to the
:meth:Sweep.from_manifest reconstruction path. One caveat: a column
that every spec leaves unset except as an explicit
overrides={"col": None} derives as an all-None column, which
the explicit-row expander rejects on resume. Specs that all carry the
same override keys — the common case — never reach that edge.
runs is iterated once and may be any iterable; pass a list if you
need to reuse it afterwards.
Execution backend¶
Bases: ABC
Abstract execution backend.
Subclasses MUST keep :attr:subprocess_isolated set to :data:True,
signalling that they implement both the per-worker-reuse and per-task-
fresh-bootstrap modes correctly. Setting the attribute to anything else
(False, None, a truthy non-True value) raises
:class:gmat_sweep.errors.BackendError from :meth:__init_subclass__ so
the error fires when the bad backend's module is imported, not at sweep
time.
The single recognised opt-out is the literal string "debug", used by
:class:gmat_sweep.backends.debug.DebugPool to declare in-process,
single-run dispatch for breakpoint()-driven debugging. Sweeps refuse
to dispatch through any pool whose subprocess_isolated is not
:data:True unless the caller acknowledges the violation via
Sweep(..., allow_unisolated_pool=True).
Subclasses accept reuse_gmat_context: bool = True as a keyword-only
parameter on __init__ and store it; concrete dispatch in
:meth:as_completed reads self._reuse_gmat_context to choose between
calling :func:gmat_sweep.worker.run_one directly (fast path) and
delegating to
:func:gmat_sweep.backends._subprocess.run_spec_in_subprocess (isolation
path).
property
¶
Best-effort worker count for sizing the in-flight cap.
Subclasses should override to expose their actual worker count
(loky's n_jobs resolved against cpu_count, Ray's
cluster size, …). Returning 1 is the safe fallback when
the backend cannot answer — the caller's bounded-submit loop
will dispatch sequentially, which is correct (if slow) for
every backend.
abstractmethod
¶
submit(spec: RunSpec) -> Future[RunOutcome]
Enqueue spec and return a future that resolves once it runs.
abstractmethod
¶
as_completed(
futures: Iterable[Future[RunOutcome]],
) -> Iterator[RunOutcome]
Drain futures, yielding outcomes in completion order.
imap(
specs: Iterable[RunSpec],
*,
in_flight: int | None = None,
) -> Iterator[tuple[RunSpec, RunOutcome]]
Stream specs through the pool, yielding (spec, outcome) pairs.
Outcomes are yielded in completion order, paired with the
:class:RunSpec that produced them. Bounds the in-flight set
to in_flight (default 4 * self.max_workers) so a 10⁵-spec
iterator does not pin 10⁵ payloads + 10⁵ futures in driver memory.
Specs are pulled from the iterator lazily — only in_flight
specs are materialised at any one time.
Each yielded pair carries the :class:RunSpec that produced
the :class:RunOutcome: the caller does not need to maintain a
side-table from outcome.run_id back to the spec, and the
spec object becomes garbage-collectable the moment the caller
finishes processing it.
Default implementation: chunked submit / drain. Specs are
consumed in_flight at a time, each chunk drained through
:meth:as_completed before the next chunk is submitted. This
bounds RSS but loses pipelining within a chunk — backends with
true future-by-future progress (e.g. those backed by
:class:concurrent.futures.Executor) should override with a
sliding-window submit/wait loop for better throughput.
imap is the lowest-level public dispatch surface. Reach for it
directly when driving a pool without a :class:~gmat_sweep.Sweep
wrapper — a custom resume path, a partial sweep, or interleaving
completions with other work. :meth:gmat_sweep.Sweep.iter_outcomes
sits one level up: the same (spec, outcome) stream, plus the
manifest bookkeeping.
Example — drive a pool directly, handling each outcome as it lands:
LocalJoblibPool(
max_workers: int | None = None,
*,
workers: int | None = None,
reuse_gmat_context: bool = True,
)
Bases: Pool
Local subprocess pool backed by joblib.Parallel(backend="loky").
Parameters¶
max_workers:
Number of loky worker processes. -1 (the default) uses every
available core. Any other negative value or 0 is rejected with
:class:gmat_sweep.errors.BackendError. workers= is accepted as
a deprecated alias.
workers:
Deprecated alias for max_workers=. Emits a
:class:DeprecationWarning; will be removed in a future release.
reuse_gmat_context:
True (default) dispatches each task as
:func:gmat_sweep.worker.run_one, which imports gmat_run once
per loky worker and reuses the import across tasks — fast, but only
safe when every spec dispatched through this pool loads the same
script. False dispatches each task through
:func:gmat_sweep.backends._subprocess.run_spec_in_subprocess,
spawning a fresh Python interpreter inside the loky worker so each
task bootstraps gmatpy from scratch — slower, but supports
cross-script sweeps. See
:class:gmat_sweep.backends.base.Pool for the contract.
DaskPool(
*,
client: Client | None = None,
n_workers: int | None = None,
threads_per_worker: int = 1,
reuse_gmat_context: bool = True,
)
Bases: Pool
Distributed pool backed by dask.distributed.
Parameters¶
client:
An existing :class:distributed.Client to dispatch through. When
supplied, the pool does not create or own a cluster, and
:meth:close does not shut the client down.
n_workers:
Number of workers in the auto-spawned :class:distributed.LocalCluster.
Ignored when client is supplied. None (default) uses
:func:os.cpu_count.
threads_per_worker:
Threads per worker for the auto-spawned :class:distributed.LocalCluster.
Ignored when client is supplied. Defaults to 1 so each worker
is a single-threaded subprocess shell.
reuse_gmat_context:
True (default) lets Dask workers reuse a single gmatpy import
across tasks — fast, but only safe when every spec dispatched
through this pool loads the same script. False runs each task
through :func:_dask_run_one, which spawns a fresh Python
interpreter per task and bootstraps gmatpy from scratch — slower,
but supports cross-script sweeps. See
:class:gmat_sweep.backends.base.Pool for the contract.
RayPool(
*,
address: str | None = None,
num_cpus: int | None = None,
reuse_gmat_context: bool = True,
**ray_init_kwargs: Any,
)
Bases: Pool
Distributed pool backed by Ray.
Parameters¶
address:
Forwarded to :func:ray.init to connect to an existing cluster
("auto" for a local cluster, "ray://host:port" for a remote
Ray Client server, or a raw GCS address). None (default) starts
a local Ray runtime.
num_cpus:
Forwarded to :func:ray.init for the local-runtime case. Ignored
when connecting to an existing cluster via address.
reuse_gmat_context:
True (default) lets Ray workers reuse a single gmatpy
import across tasks — fast, but only safe when every spec
dispatched through this pool loads the same script. False
binds the Ray remote to :func:_ray_run_one_impl, which spawns a
fresh Python interpreter per task and bootstraps gmatpy from
scratch — slower, but supports cross-script sweeps. See
:class:gmat_sweep.backends.base.Pool for the contract.
**ray_init_kwargs:
Extra keyword arguments forwarded verbatim to :func:ray.init.
KubernetesJobPool(
*,
image: str,
pvc_name: str,
pvc_mount_path: str = "/sweep",
driver_mount_path: str | Path | None = None,
namespace: str = "default",
parallelism: int | None = _DEFAULT_PARALLELISM,
backoff_limit: int = _DEFAULT_BACKOFF_LIMIT,
ttl_seconds_after_finished: int = _DEFAULT_TTL_SECONDS,
job_deadline_seconds: int = _DEFAULT_JOB_DEADLINE_SECONDS,
resources: _ResourcesArg = None,
default_resources: Mapping[str, Any] | None = None,
kubeconfig: str | Path | None = None,
in_cluster: bool | None = None,
reuse_gmat_context: bool = True,
)
Bases: Pool
Distributed pool that submits each :class:RunSpec as a Kubernetes Job.
Parameters¶
image:
Fully-qualified container image with gmat-sweep[k8s] plus a
working GMAT install. Pods run python -m
gmat_sweep._run_subprocess inside this image. Required —
no published default image is shipped today.
pvc_name:
Name of an existing :class:PersistentVolumeClaim in
namespace that the Pods will mount. Must be visible to the
driver under driver_mount_path and to the Pods under
pvc_mount_path; ReadWriteMany is the typical access mode,
but any topology that gives the driver and the Pods a shared
view of the same files works.
pvc_mount_path:
Path inside each Pod where the PVC is mounted. The pool uses it
to compute the in-Pod spec / outcome paths it passes to
_run_subprocess via --spec / --outcome. Defaults to
/sweep.
driver_mount_path:
Path on the driver side that resolves to the same PVC contents.
Defaults to pvc_mount_path, which is correct when the driver
runs as a Pod mounting the PVC at the same path as the workers.
For an out-of-cluster driver, set this to the local path where
the PVC's backing storage is mounted (NFS, EFS, GCS Fuse, …).
namespace:
Kubernetes namespace for the Jobs and the watch loop. Defaults
to "default".
parallelism:
Maximum number of in-flight Jobs at any moment. None means
no cap (one Job created per spec, all up-front). Defaults to
64 so a 10000-run sweep does not stampede the API server.
backoff_limit:
Forwarded to V1JobSpec.backoff_limit. Defaults to 0 so
Pod failures map 1:1 to outcome failures; silent retries break
the failure-as-row contract.
ttl_seconds_after_finished:
Forwarded to V1JobSpec.ttl_seconds_after_finished. Defaults
to 300 (5 min) — enough for a kubectl-window for inspection
before the cluster GCs.
job_deadline_seconds:
Driver-side wall-clock deadline per Job. A Job that has not
reached a terminal status (succeeded or failed) within
this many seconds is deleted (propagationPolicy=Background)
and folded into a synthetic :meth:RunOutcome.failed, instead
of letting the driver hang forever on Pods stuck in
Pending / ImagePullBackOff / Unschedulable.
Granularity is bounded by the watch reconnect cadence
(~60 s); the deadline is a hang preventer, not a hard SLA.
Defaults to 3600 (1 h). Must be a positive integer.
resources:
Per-run resource overrides keyed by RunSpec.run_id, or a
callable taking the spec and returning a resources dict. The
resolved value populates V1ResourceRequirements.requests
(and .limits if the user includes a "limits" key, see
below). When unresolved for a given run, the pool falls back to
default_resources.
default_resources:
Resources applied to every Pod that resources does not
provide a value for. The dict shape is the standard k8s shape:
{"cpu": "1", "memory": "4Gi"} for requests only, or
{"requests": {...}, "limits": {...}} for both. None
leaves resources unset on the Job spec entirely.
kubeconfig:
Path to an explicit kubeconfig file. Forwarded to
kubernetes.config.load_kube_config. Mutually exclusive with
in_cluster=True.
in_cluster:
True forces in-cluster auth (load_incluster_config).
False forces out-of-cluster auth (load_kube_config).
None (default) auto-detects: in-cluster if the
ServiceAccount token file exists, out-of-cluster otherwise.
reuse_gmat_context:
Accepted for :class:Pool API parity. Pods are always fresh
interpreters on this backend, so the flag has no effect.
MPIPool(
*,
max_workers: int | None = None,
reuse_gmat_context: bool = True,
**mpi_executor_kwargs: Any,
)
Bases: Pool
Distributed pool backed by mpi4py.futures.
Parameters¶
max_workers:
Number of MPI worker ranks. Forwarded verbatim to
:class:mpi4py.futures.MPIPoolExecutor. None (default) lets
the executor pick — under
mpirun -n K python -m mpi4py.futures … that means K-1
pre-allocated workers; under plain python … the executor
falls back to MPI_Comm_spawn with an implementation-defined
default count, so an explicit max_workers is recommended for
the dynamic-spawn path.
reuse_gmat_context:
True (default) lets MPI worker ranks reuse a single gmatpy
import across tasks — fast, but only safe when every spec
dispatched through this pool loads the same script. False
binds the executor task to :func:_mpi_run_one_impl, which spawns
a fresh Python interpreter per task and bootstraps gmatpy from
scratch — slower, but supports cross-script sweeps. See
:class:gmat_sweep.backends.base.Pool for the contract.
**mpi_executor_kwargs:
Extra keyword arguments forwarded verbatim to
:class:mpi4py.futures.MPIPoolExecutor.
Bases: Pool
Local subprocess pool backed by :class:concurrent.futures.ProcessPoolExecutor.
Always constructs the executor with max_tasks_per_child=1 so every
task runs in a fresh Python interpreter. The subprocess-isolation
contract from :class:gmat_sweep.backends.base.Pool therefore holds
by construction, and the reuse_gmat_context flag chooses only the
in-worker dispatch path.
Parameters¶
max_workers:
Number of worker processes. Forwarded verbatim to
:class:concurrent.futures.ProcessPoolExecutor. None (the
default) lets the executor pick — :func:os.process_cpu_count
on Python 3.13+, :func:os.cpu_count on 3.11-3.12.
reuse_gmat_context:
Accepted for :class:Pool API parity. Both values dispatch each
task as :func:gmat_sweep.worker.run_one directly inside the
worker process — the max_tasks_per_child=1 baked into the
executor already provides the per-task fresh-interpreter
guarantee the reuse_gmat_context=False contract requires.
Nested run_spec_in_subprocess hops on this backend would
double-pay the subprocess cost without changing the contract.
Bases: Pool
In-process, single-run pool for breakpoint()-driven debugging.
Dispatches each spec by calling :func:gmat_sweep.worker.run_one
synchronously on the driver process. No worker pool is constructed, no
subprocess is spawned, and nothing is parallelised — the point is for
the driver's debugger to be the worker's debugger. The trade-off is
that the GMAT singletons in the driver process get dirtied by the
run; reusing the same Python interpreter for a second run is not
supported, and :class:gmat_sweep.sweep.Sweep enforces the limit.
Parameters¶
allow_unisolated_pool:
Required opt-in. Defaults to :data:False, which raises
:class:gmat_sweep.errors.BackendError from __init__ so the
violation cannot happen accidentally. Pass :data:True to
construct the pool — and remember to pass the matching flag to
:class:gmat_sweep.sweep.Sweep as well.
Examples¶
from gmat_sweep import Sweep from gmat_sweep.backends.debug import DebugPool pool = DebugPool(allow_unisolated_pool=True) # doctest: +SKIP sweep = Sweep( # doctest: +SKIP ... runs=[only_spec], ... backend=pool, ... manifest_path=out / "manifest.jsonl", ... output_dir=out, ... script_path=mission, ... parameter_spec={"_kind": "explicit", ...}, ... allow_unisolated_pool=True, ... ) with pool: # doctest: +SKIP ... sweep.run()
Specs and outcomes¶
dataclass
¶
RunSpec(
script_path: Path,
overrides: dict[str, Any],
output_dir: Path,
run_id: int,
seed: int | None,
run_options: dict[str, Any],
postprocess: str | None = None,
context: dict[str, Any] = dict(),
)
A single run's worth of work — script + overrides + run_id + seed.
A worker reconstructs the full run from this record alone:
instantiate :class:gmat_run.Mission from script_path, apply
overrides via the dotted-path setter, run with run_options,
write outputs under output_dir.
postprocess is an optional import-path string
("package.module:function") naming a per-run hook the worker
invokes after a successful GMAT run. It is a string, not a callable,
because the spec is JSON-serialised across the worker boundary —
every backend round-trips it through :func:json.dumps, and a bare
function cannot survive that. The hook must therefore be a
module-level function (closures and lambdas have no importable
name). None means no postprocessing.
context is an optional free-form mapping of per-run data the
caller wants the postprocess hook to see but GMAT must not. Unlike
overrides it is never applied to the :class:gmat_run.Mission
and never folded into the manifest's derived parameter_spec — it
rides to the worker untouched and reaches the hook as
run_spec.context. Use it to carry per-run data computed before
dispatch (e.g. a reference state the hook compares the GMAT result
against). Its values must be JSON-encodable, since the spec crosses
the worker boundary as JSON. Empty by default. context is
recorded in the run's manifest entry, so a run rebuilt by
:meth:gmat_sweep.Sweep.from_manifest (resume) restores the
context it ran with. A run that never produced an entry — never
dispatched before the sweep was interrupted — has none to restore;
pass context_provider to recompute it.
dataclass
¶
SweepSpec(
mission_script_path: Path,
runs: tuple[RunSpec, ...],
backend: str,
backend_kwargs: dict[str, Any],
output_dir: Path,
manifest_path: Path,
sweep_seed: int | None = None,
)
A whole sweep's metadata — script, runs, backend, outputs.
runs is a materialised :class:tuple of :class:RunSpec so the
spec round-trips through JSON cleanly. run_id ordering is the
contract the manifest and resume flow depend on: runs[i].run_id ==
i for every well-formed sweep.
dataclass
¶
RunOutcome(
run_id: int,
status: RunStatus,
output_paths: dict[str, Path],
extra_outputs: dict[str, Path],
solver_paths: dict[str, Path],
converged: dict[str, bool],
postprocess_status: PostprocessStatus,
duration_s: float,
stderr: str | None,
started_at: datetime,
ended_at: datetime,
)
The result of one run after the worker returns.
output_paths maps a worker-chosen key (e.g. the parsed
ReportFile resource name) to the on-disk Parquet artefact written
under :attr:RunSpec.output_dir. Empty for failed and skipped runs.
stderr is None for successful runs and the captured worker
stderr / traceback string for failed runs.
extra_outputs maps a hook-chosen key to each artefact a
:attr:RunSpec.postprocess hook produced. It is populated only when
the hook ran and returned (postprocess_status="ok"); empty
otherwise. postprocess_status records the hook's own outcome
independently of status — a hook that raises makes the run a
plain status="failed" (so resume retries it) while
postprocess_status="failed" keeps the failure distinguishable
from a GMAT-engine failure.
solver_paths maps each Solver resource name to the per-run
Parquet of its Target / Optimize iteration history (gmat-run's
Results.solver_runs); converged records the matching
{solver: bool} map (Results.converged). Both are empty for
failed and skipped runs, and for ok runs whose mission sequence
declared no solver. Convergence is orthogonal to status: a run
where GMAT completed but the targeter hit MaximumIterations is
status="ok" with converged carrying False for that solver.
duration_s is measured from a :func:time.monotonic bookend
pair around the run body, not from
(ended_at - started_at).total_seconds(): a wall-clock
correction (NTP step) mid-run would otherwise drive duration_s
negative or zero. started_at / ended_at remain wall-clock
audit timestamps. Callers pass the monotonic delta in via the
:meth:ok / :meth:failed helpers.
classmethod
¶
failed(
*,
run_id: int,
stderr: str,
started_at: datetime,
ended_at: datetime,
duration_s: float,
postprocess_status: PostprocessStatus = "none",
) -> RunOutcome
Build a failed outcome.
postprocess_status defaults to "none" — the common case is
a GMAT-engine failure, where the hook never ran. The worker passes
"failed" when the GMAT step succeeded but the postprocess hook
raised: the run is still status="failed" (and resumable), but
the field marks the failure as a postprocess failure.
module-attribute
¶
Terminal status of a single run, carried on :attr:RunOutcome.status and
:attr:gmat_sweep.manifest.ManifestEntry.status.
"ok" — the GMAT step completed and any postprocess hook returned.
"failed" — the run raised (GMAT bootstrap, override rejection, a hook
exception, …); stderr carries the traceback and resume retries it.
"skipped" — the run was not executed, e.g. it was already present and
ok in a manifest being resumed.
module-attribute
¶
Outcome of a run's postprocess hook, recorded independently of status.
"none" — no hook ran (the sweep registered none, or the GMAT step
failed before the hook was reached). "ok" — the hook ran and returned.
"failed" — the hook raised; the run's status is "failed" and
stderr carries the hook traceback. Keeping this separate from
status means a hook failure is a plain failed run — retried by
resume for free — while still being distinguishable from a GMAT
failure.
Grid expansion¶
Yield override dicts for the cartesian product of grid.
Keys are emitted in lexicographic order; the cartesian product enumerates
in :func:itertools.product order over the input iterables, so the outer
loop varies the lexicographically-first key slowest and the last key
fastest. For {"a": [1, 2], "b": [10, 20, 30]} the six dicts come out as
(a=1, b=10), (a=1, b=20), (a=1, b=30), (a=2, b=10), (a=2, b=20),
(a=2, b=30).
Each input iterable is materialised once at entry so callers may pass generators without surprising exhaustion and so empty-iterable validation can run before the cartesian product begins.
An empty mapping is valid and yields a single empty override dict — the cartesian product of nothing has one element.
Raises :class:SweepConfigError if any key is not a :class:str or any
value materialises to an empty sequence.
Return the number of runs a :func:full_factorial expansion of grid produces.
prod(len(list(v)) for v in grid.values()) — but tolerant of
generators (materialises each axis once) and aware that the empty
mapping is the identity (one empty-override run, matching
:func:full_factorial).
Used by :func:gmat_sweep.sweep to size the manifest header's
run_count and the progress bar without materialising the
cartesian product itself.
expand_grid_to_run_specs(
grid: Mapping[str, Iterable[Any]],
script_path: str | Path,
output_dir: str | Path,
) -> list[RunSpec]
Build a list of :class:RunSpec from a full-factorial expansion of grid.
Each spec gets a sequential run_id starting at 0, script_path
propagated through, output_dir set to <output_dir>/run-<run_id>,
seed=None, and run_options={}. The ordering contract from
:func:full_factorial carries through unchanged: specs[i].run_id == i
and the override dicts appear in cartesian-product order.
Materialises the full cartesian product up front — fine for small
grids but spends O(N) memory on the spec list before the first
worker starts. Prefer :func:iter_grid_run_specs when the
cartesian product is large (10⁴+ runs): the streaming variant
yields the same specs one at a time.
Raises :class:SweepConfigError for the same reasons as
:func:full_factorial.
iter_grid_run_specs(
grid: Mapping[str, Iterable[Any]],
script_path: str | Path,
output_dir: str | Path,
) -> Iterator[RunSpec]
Stream :class:RunSpec instances from a full-factorial expansion of grid.
Same per-spec shape as :func:expand_grid_to_run_specs (sequential
run_id, per-run output_dir, seed=None, run_options={})
but yields lazily — for a 10⁵-row factorial the driver never holds
more than one :class:RunSpec plus :func:full_factorial's
iterator state in memory.
Validation (string keys, non-empty values) still runs eagerly at
the start of iteration via :func:full_factorial, so malformed
grids fail loudly before any spec is yielded.
expand_samples_to_run_specs(
samples: DataFrame,
script_path: str | Path,
output_dir: str | Path,
) -> list[RunSpec]
Build a list of :class:RunSpec from an explicit-row sample DataFrame.
Each row becomes one :class:RunSpec with overrides = row.to_dict(),
run_id equal to the row's positional index, output_dir set to
<output_dir>/run-<run_id>, seed=None, and run_options={}. The
DataFrame's column names are dotted-path field names — the same shape
:func:expand_grid_to_run_specs already produces — so analysts may
pre-build any sampling design (Latin hypercube, Halton/Sobol, custom)
themselves and hand the result in directly.
Per-cell NaN is forwarded as-is. gmat-run is the line that decides
whether NaN is a valid value for a given dotted path; this expander does
not second-guess it.
Validation is strict and runs before any spec is built:
samplesmust be a :class:pandas.DataFrame.- All column names must be :class:
strinstances. - Column names must be unique — duplicates would silently lose data when
:meth:
pandas.Series.to_dictcollapses them into a single key. - The DataFrame index must equal :class:
pandas.RangeIndex(start=0, stop=len(samples))sorun_idand the row's positional index agree. - No column may be entirely NaN (an all-NaN axis carries no signal).
Any violation raises :class:SweepConfigError with a message naming the
offending column or index.
expand_monte_carlo_to_run_specs(
perturb: Mapping[str, DistSpec],
n: int,
seed: int | None,
script_path: str | Path,
output_dir: str | Path,
) -> list[RunSpec]
Build :class:RunSpec instances for a Monte Carlo dispersion sweep.
For each run_id i in range(n):
- Derive the run-level seed via :func:
derive_run_seeds(seed, n)[i] <gmat_sweep.distributions.derive_run_seeds>— recorded on :attr:RunSpec.seed. - For each parameter
kin lexicographically-sortedperturb: derive a per-parameter sub-seed via :func:derive_param_seed(run_seed, k) <gmat_sweep.distributions.derive_param_seed>and sample one float through :func:sample(perturb[k], sub_seed) <gmat_sweep.distributions.sample>.
Per-parameter sub-seeds are derived from the parameter name, not its
position in the mapping, so adding a perturbed parameter to an existing
sweep does not change the draws of any other parameter at any
run_id. Two calls at the same (perturb, n, seed, script_path)
return identical specs.
Raises :class:SweepConfigError if perturb is empty, n < 1, or
any parameter spec fails its own validation in
:func:to_rv_frozen <gmat_sweep.distributions.to_rv_frozen>.
expand_monte_carlo_extension_to_run_specs(
perturb: Mapping[str, DistSpec],
old_n: int,
n: int,
seed: int | None,
script_path: str | Path,
output_dir: str | Path,
) -> list[RunSpec]
Build :class:RunSpec instances for the extension slice of an MC sweep.
Mirrors :func:expand_monte_carlo_to_run_specs but emits only the
[old_n, old_n + n) tail. The two key facts that make this slice
bit-equal to the same indices of a fresh monte_carlo(n=old_n+n, ...)
call:
- :func:
numpy.random.SeedSequence.spawnis position-deterministic — thei-th child depends only on(parent, i), not onn. So :func:derive_run_seeds(seed, total) <gmat_sweep.distributions.derive_run_seeds>at indices[old_n, total)matches the same indices of a freshderive_run_seeds(seed, total)call regardless of how the original sweep was sized. - Per-parameter sub-seeds are derived from
:func:
derive_param_seed(run_seed, name) <gmat_sweep.distributions.derive_param_seed>, which keys on the parameter name. The extension reuses the sameperturbmapping by construction (extension does not let the caller change distributions), so each parameter's sub-seed at every extendedrun_idis bit-equal to a fresh sweep at the same totaln.
Together these two facts give the bit-equivalence the
:func:gmat_sweep.monte_carlo_extend contract rests on.
Raises :class:SweepConfigError if perturb is empty, old_n < 0,
n < 1, or any parameter spec fails its own validation in
:func:to_rv_frozen <gmat_sweep.distributions.to_rv_frozen>.
expand_latin_hypercube_to_run_specs(
perturb: Mapping[str, DistSpec],
n: int,
seed: int | None,
script_path: str | Path,
output_dir: str | Path,
) -> list[RunSpec]
Build :class:RunSpec instances for a Latin hypercube sweep.
Convenience wrapper that builds the samples DataFrame via
:func:latin_hypercube_samples and forwards to
:func:expand_samples_to_run_specs. Per-run seeds are not populated:
the draw set is fully determined by (perturb, n, seed) so
individual runs do not need their own RNG state.
latin_hypercube_samples(
perturb: Mapping[str, DistSpec],
n: int,
seed: int | None,
) -> DataFrame
Build the Latin hypercube samples DataFrame for a stochastic sweep.
Builds a :class:scipy.stats.qmc.LatinHypercube sampler with
d = len(perturb) and seed = seed, draws n unit-cube points,
then maps each column through to_rv_frozen(perturb[k]).ppf(...) to
leave the unit cube.
Columns are emitted in lexicographic order so the run set is stable
under perturb-dict reordering. The returned DataFrame has a default
:class:pandas.RangeIndex and is suitable input to
:func:expand_samples_to_run_specs.
Determinism: two calls with the same (perturb, n, seed) produce
bit-equal DataFrames.
Raises :class:SweepConfigError if perturb is empty, n < 1, or
any parameter spec fails its own validation in
:func:to_rv_frozen <gmat_sweep.distributions.to_rv_frozen>.
Manifest¶
module-attribute
¶
On-disk manifest schema version this gmat-sweep writes and reads.
:meth:Manifest.load accepts any header whose schema_version is
<= MANIFEST_SCHEMA_VERSION (a missing field is treated as 1 for
backwards compatibility with manifests written before the field was
introduced) and rejects anything greater. See
docs/manifest-schema.md for the field-by-field contract and the
compatibility policy that governs future bumps.
dataclass
¶
Manifest(
script_sha256: str,
gmat_sweep_version: str,
gmat_run_version: str,
gmat_install_version: str,
python_version: str,
os_platform: str,
sweep_seed: int | None,
parameter_spec: dict[str, Any],
run_count: int,
backend: str = "unknown",
postprocess: str | None = None,
schema_version: int = MANIFEST_SCHEMA_VERSION,
entries: list[ManifestEntry] = list(),
fsync_each: bool = True,
fsync_batch: int = 50,
)
Sweep manifest — header fingerprint plus a growing list of per-run entries.
class-attribute
instance-attribute
¶
Sweep-wide postprocess hook import path, or None.
Mirrors :attr:gmat_sweep.spec.RunSpec.postprocess. Recorded on the
header so :meth:gmat_sweep.Sweep.from_manifest can re-stamp the hook
onto rebuilt run specs — a resumed run re-runs the same postprocessing
instead of silently dropping it. Additive: manifests written before
the field existed load with postprocess=None.
class-attribute
instance-attribute
¶
When True (default), every :meth:append_entry fsyncs the file.
Set to False and tune :attr:fsync_batch to amortise the fsync
cost across a batch of entries — useful for sub-second runs at large
counts where the per-entry fsync dominates the driver's time. The
durability tradeoff is documented in docs/manifest-schema.md and
on :meth:append_entry. Not serialised — this is a per-process
knob, not part of the on-disk format.
class-attribute
instance-attribute
¶
Fsync interval (in entries) when :attr:fsync_each is False.
With fsync_each=False, :meth:append_entry fsyncs after every
fsync_batch entries (and :meth:close fsyncs the tail). Has no
effect when fsync_each=True.
property
¶
Cumulative number of extension runs appended beyond the original sweep.
Derived as max(0, max(run_id for entries) + 1 - parameter_spec["n"])
for Monte Carlo manifests; 0 for any other parameter_spec
kind or for an MC manifest that has not been extended. The on-disk
header is not rewritten on extension (manifest headers are
append-only), so this is the canonical way to ask "how many
:func:gmat_sweep.monte_carlo_extend calls have landed on top of
this sweep's original n."
property
¶
The live run count — the on-disk header's run_count plus any extensions.
The header's run_count field is frozen at first
:meth:save and is not rewritten on :meth:Sweep.extend
(append-only manifest header — see docs/manifest-schema.md).
Reading run_count alone therefore lags the actual run set
after every extend; this property is what callers want when
they ask "how many runs are now in this sweep, including
extensions?".
Implementation: max(header.run_count, max(e.run_id) + 1) so
a mid-sweep Ctrl-C (entries < run_count) still reports the
expected total, and an extended manifest (max_run_id ≥
run_count) reports the post-extend total.
Write header + all current entries as JSON Lines, fsyncing file and parent dir.
append_entry(entry: ManifestEntry) -> None
Append one entry to the bound file, and to the in-memory list.
With :attr:fsync_each True (default), every appended entry
is fsynced before this method returns — strict per-entry
durability, matching the v0.3 behaviour. With fsync_each
False, the file is fsynced only on the boundary set by
:attr:fsync_batch (every Nth entry); the tail is not durable
until :meth:close is called or the next batch boundary is
crossed. A host crash between fsync boundaries can therefore
lose up to fsync_batch - 1 recently-appended entries — the
Parquet outputs and the runs themselves are unaffected, and the
resume flow re-runs only the missing slice.
Fsync the manifest file and parent directory.
Idempotent and a no-op when the manifest has no bound path
(i.e. neither :meth:save nor :meth:load has been called).
Sweeps that opt into :attr:fsync_each False should call
this on successful completion so the trailing batch of entries
becomes durable; a KeyboardInterrupt deliberately skips
close() so the resume flow exercises the
fsync_batch - 1-entry recovery window.
classmethod
¶
load(path: Path) -> Manifest
Load a manifest from disk, tolerating a single torn final line.
Materialises every entry into the returned :class:Manifest's
:attr:entries list, deduplicated last-wins per run_id.
For tail-only operations on large manifests prefer
:meth:iter_entries, :meth:find_failed, or
:meth:find_missing — they stream the file without holding
every entry in memory.
classmethod
¶
iter_entries(path: Path) -> Iterator[ManifestEntry]
Stream parsed entries from disk, lazily, without folding duplicates.
Yields one :class:ManifestEntry per non-header line in file
order; tolerates a single torn final line the same way
:meth:load does (a partial trailing line is silently dropped).
Validates the header (schema version, required fields) before
the first yield, raising :class:ManifestCorruptError with
line_number=1 on header failures and line_number=i on
per-line failures.
Use this for tail-only scans (per-run_id last-status folds,
membership checks) where holding every entry in memory would be
wasteful — :meth:find_failed and :meth:find_missing are
built on top. Use :meth:load when you need the
deduplicated, fully-materialised entry list.
classmethod
¶
Return run_ids whose latest entry has status == "failed", in first-seen order.
Streams path via :meth:iter_entries and folds per-run_id
last-wins state into a small dict — does not materialise the
full entry list. Result matches [e.run_id for e in
Manifest.load(path).entries if e.status == "failed"] while
running in O(entries) time and O(unique run_ids) memory.
classmethod
¶
Return run_ids in expected_run_ids with no entry on disk, in input order.
dataclass
¶
ManifestEntry(
run_id: int,
overrides: dict[str, Any],
status: RunStatus,
output_paths: dict[str, Path],
started_at: datetime,
ended_at: datetime,
duration_s: float,
stderr: str | None,
log_path: Path | None,
extra_outputs: dict[str, Path] = dict(),
postprocess_status: PostprocessStatus = "none",
solver_paths: dict[str, Path] = dict(),
converged: dict[str, bool] = dict(),
context: dict[str, Any] = dict(),
)
One run's record in the manifest — overrides, status, outputs, timing.
extra_outputs mirrors :attr:RunOutcome.extra_outputs: the
artefacts a per-run :attr:RunSpec.postprocess hook produced, keyed
by the hook's own keys. Non-empty only when postprocess_status is
"ok". postprocess_status records the hook outcome
("none" / "ok" / "failed") independently of status —
a hook failure is a plain status="failed" run.
solver_paths mirrors :attr:RunOutcome.solver_paths: each
Solver resource's per-run iteration-history Parquet, keyed by the
solver name. converged mirrors :attr:RunOutcome.converged — the
{solver: bool} map — so convergence is queryable straight from the
manifest without reading any Parquet. Both are empty for non-ok
runs and for ok runs whose mission sequence declared no solver.
context mirrors :attr:RunSpec.context: the free-form per-run
payload the caller attached for the postprocess hook. It is recorded
so a run rebuilt by :meth:gmat_sweep.Sweep.from_manifest carries the
same context it ran with — a resumed context-dependent hook
then behaves identically to the original run. Empty {} for runs
that carried no context and for manifests written before the field
existed.
classmethod
¶
from_outcome(
outcome: RunOutcome,
*,
overrides: dict[str, Any],
context: dict[str, Any] | None = None,
log_path: Path | None = None,
) -> ManifestEntry
Build a manifest entry from a worker :class:RunOutcome.
overrides and context come from the dispatched
:class:RunSpec — neither rides on the :class:RunOutcome, which
carries only the run's results. context defaults to an empty
mapping for runs that carried no per-run payload.
SHA-256 of the script file after BOM, line-ending, and trailing-newline normalisation.
Reads the file as bytes, decodes as UTF-8, strips a leading UTF-8
byte-order mark (\ufeff), replaces \r\n and lone \r
with \n, and ensures exactly one trailing \n. The hash is
computed over the resulting UTF-8 bytes — a script saved from a
BOM-emitting Windows editor and the same script saved without a
BOM produce identical hashes.
Aggregation¶
lazy_multiindex(
manifest: Manifest,
output_dir: Path,
*,
name: str | None = ...,
spool: bool = ...,
engine: Literal["pandas"] = ...,
) -> DataFrame
lazy_multiindex(
manifest: Manifest,
output_dir: Path,
*,
name: str | None = ...,
spool: bool = ...,
engine: Literal["polars"],
) -> DataFrame
lazy_multiindex(
manifest: Manifest,
output_dir: Path,
*,
name: str | None = ...,
spool: bool = ...,
engine: str,
) -> DataFrame
lazy_multiindex(
manifest: Manifest,
output_dir: Path,
*,
name: str | None = None,
spool: bool = True,
engine: str = "pandas",
) -> DataFrame
Assemble the (run_id, time)-indexed report DataFrame from a sweep's outputs.
Iterates manifest.entries in order. For each ok entry the
Parquet listed in :attr:ManifestEntry.output_paths under the
report__<name> key is read via :mod:pyarrow.dataset and tagged
with its run_id. For each failed or skipped entry — and
for any ok entry that did not produce the requested report — one
NaN-filled row is materialised with time = NaT and __status
set to the run-level status ("failed" / "skipped" for non-ok
runs, "ok" for ok runs missing this report).
Relative paths in output_paths are resolved against output_dir;
absolute paths are used as-is.
Parameters¶
manifest
The sweep manifest. Drives both the set of runs and their status.
output_dir
Sweep output root. Used to anchor any relative paths recorded in
the manifest.
name
Report resource name to aggregate. None (default) picks the
sole report if exactly one report is present across the sweep.
Sweeps that produced multiple reports per run must pass name=
explicitly; the call raises
:class:gmat_sweep.errors.SweepConfigError listing the available
names otherwise.
spool
True (default) streams each run's record batches into pandas
one batch at a time. False reads each run's Parquet eagerly
in one shot — simpler control flow, higher peak memory.
engine
"pandas" (default) returns a (run_id, time)-MultiIndexed
:class:pandas.DataFrame. "polars" returns a
:class:polars.DataFrame whose (run_id, time) MultiIndex is
flattened into two leading sorted columns; row count and the
non-index column set match the pandas-engine equivalent. Requires
the [polars] extra; an :class:ImportError with the install
hint is raised when polars is not importable. Experimental:
the polars output shape (column names, dtypes) is not yet
contractual and may change in a future minor version.
Raises¶
SweepConfigError
name=None was passed but the sweep produced more than one
report (the exception message lists the available names), the
explicitly-named report does not appear in any ok run's outputs,
or engine is neither "pandas" nor "polars".
ValueError
An ok entry has no output_paths at all (a run that ran
successfully must have produced something), or a per-run Parquet
is missing the time column required for the index.
lazy_ephemerides(
manifest: Manifest,
output_dir: Path,
*,
name: str | None = ...,
spool: bool = ...,
engine: Literal["pandas"] = ...,
) -> DataFrame
lazy_ephemerides(
manifest: Manifest,
output_dir: Path,
*,
name: str | None = ...,
spool: bool = ...,
engine: Literal["polars"],
) -> DataFrame
lazy_ephemerides(
manifest: Manifest,
output_dir: Path,
*,
name: str | None = ...,
spool: bool = ...,
engine: str,
) -> DataFrame
lazy_ephemerides(
manifest: Manifest,
output_dir: Path,
*,
name: str | None = None,
spool: bool = True,
engine: str = "pandas",
) -> DataFrame
Assemble the (run_id, time)-indexed ephemeris DataFrame from a sweep's outputs.
Mirrors :func:lazy_multiindex but dispatches on ephemeris__<name>
keys instead of report__<name>. The worker copies the first
datetime column of each ephemeris frame (Epoch for OEM, STK, and
SPK formats) to a column named time before writing Parquet, so
the same (run_id, time) index machinery applies.
See :func:lazy_multiindex for parameter and exception semantics,
including the engine knob.
lazy_contacts(
manifest: Manifest,
output_dir: Path,
*,
name: str | None = ...,
engine: Literal["pandas"] = ...,
) -> DataFrame
lazy_contacts(
manifest: Manifest,
output_dir: Path,
*,
name: str | None = ...,
engine: Literal["polars"],
) -> DataFrame
lazy_contacts(
manifest: Manifest,
output_dir: Path,
*,
name: str | None = ...,
engine: str,
) -> DataFrame
lazy_contacts(
manifest: Manifest,
output_dir: Path,
*,
name: str | None = None,
engine: str = "pandas",
) -> DataFrame
Assemble the (run_id, interval_id)-indexed contact DataFrame from a sweep's outputs.
Mirrors :func:lazy_multiindex but dispatches on contact__<name>
keys and uses interval_id — the per-run row position the worker
assigns at write time, 0..K-1 per run — as the secondary index
level. ContactLocator outputs are typically tiny (one row per
visibility interval), so there is no spool knob; reads are
fragment-at-a-time eager.
Failed, skipped, and report-only ok runs materialise as one row
with interval_id = pd.NA (cast as the nullable Int64 dtype so
integer interval indices and missing values share one level). Under
engine="polars" the nullable Int64 round-trips into a polars
Int64 column with null for the missing slots.
See :func:lazy_multiindex for the rest of the parameter and
exception semantics, including the engine knob.
Assemble the (run_id, solver, iteration)-indexed solver-history DataFrame.
Every run of a sweep over a Target / Optimize scenario carries
one iteration-history frame per Solver resource — gmat-run's
:attr:Results.solver_runs, staged by the worker as
solver__<name>.parquet and tracked in each manifest entry's
:attr:gmat_sweep.ManifestEntry.solver_paths. This loads the manifest
at manifest_path, streams every such Parquet, and stitches them
into one multi-indexed :class:pandas.DataFrame.
The row index is a three-level :class:~pandas.MultiIndex
(run_id, solver, iteration). solver is the GMAT resource name;
iteration is the number GMAT reports — note this is not unique
for a Yukon optimiser, whose iteration spans several function
evaluations, so a Yukon run contributes repeated iteration values
under one (run_id, solver).
Columns are the union across every solver in the sweep: one
float64 column per Vary variable (verbatim script names), the
goal/constraint residual columns (a DifferentialCorrector adds the
<goal> / <goal>_desired / <goal>_residual /
<goal>_tolerance quartet; a Yukon adds cost and
<constraint>_residual), and the per-iteration status column
("running" on every row except the terminal one, which carries
"converged", "max_iter", or "failed"). A run carrying a
solver of one type shows NaN in the columns specific to the other.
Unlike the GMAT-native aggregators, no marker rows are
materialised: a run whose mission sequence declared no Target /
Optimize — and any failed or skipped run, which produced no
solver Parquet — simply contributes nothing. A sweep with no solver
anywhere returns an empty, correctly-typed frame (a three-level
(run_id, solver, iteration) index and a status column) rather
than raising.
Run-level status and solver convergence are orthogonal. A run where
GMAT completed but the targeter exhausted MaximumIterations is a
perfectly ordinary status="ok" manifest entry; its solver history
is still aggregated here, with the terminal status row carrying
"max_iter". Use :func:lazy_solver_convergence (or the manifest
entry's :attr:~gmat_sweep.ManifestEntry.converged map) to ask which
runs actually reached their goal.
Reads honour the streaming memory contract (#130): a single
:mod:pyarrow.dataset scan over a unified schema, materialised to
pandas once, not a linear concatenation of one frame per run.
Parameters¶
manifest_path
Path to the sweep's manifest.jsonl. The manifest is loaded
via :meth:gmat_sweep.Manifest.load; relative paths recorded in
solver_paths are anchored against manifest_path's parent
directory.
engine
"pandas" (default) returns a :class:pandas.DataFrame.
"polars" flattens the (run_id, solver, iteration)
MultiIndex into three leading columns and returns a
:class:polars.DataFrame; requires the [polars] extra. Same
contract — including the experimental status of the polars output
shape — as :func:lazy_multiindex.
Raises¶
SweepConfigError
engine is neither "pandas" nor "polars".
ValueError
A manifest entry references a solver Parquet that is not on disk.
Assemble the (run_id, solver)-indexed convergence summary from the manifest.
The companion of :func:lazy_solver_runs: a compact "which runs
converged?" view built entirely from each manifest entry's
:attr:gmat_sweep.ManifestEntry.converged map — the worker records
that {solver: bool} map (gmat-run's :attr:Results.converged)
directly on the entry, so no Parquet is read.
The result is a :class:pandas.DataFrame indexed by a two-level
(run_id, solver) :class:~pandas.MultiIndex with a single
boolean converged column. Only runs that actually ran a solver
appear — solver-less, failed, and skipped runs contribute no
rows, matching :func:lazy_solver_runs. A sweep with no solver
anywhere returns an empty, correctly-typed frame.
A True here means the solver reached its goal within tolerance; a
False means it did not — most commonly because it hit
MaximumIterations. Convergence is orthogonal to the run's
status: a non-converged solver in a run that GMAT otherwise
completed cleanly is a status="ok" entry with converged=False.
Parameters¶
manifest_path
Path to the sweep's manifest.jsonl, loaded via
:meth:gmat_sweep.Manifest.load.
engine
"pandas" (default) returns a :class:pandas.DataFrame.
"polars" flattens the (run_id, solver) MultiIndex into two
leading columns and returns a :class:polars.DataFrame; requires
the [polars] extra. Same contract as :func:lazy_multiindex.
Raises¶
SweepConfigError
engine is neither "pandas" nor "polars".
Assemble the DataFrame of a postprocess hook's per-run extra outputs.
For the extra-output key name registered by a sweep's
:attr:gmat_sweep.RunSpec.postprocess hook, loads the manifest at
manifest_path, streams the per-run Parquet of every ok run
that registered name, and stitches them into one multi-indexed
:class:pandas.DataFrame.
Unlike the GMAT-native aggregators, the per-run Parquet is the hook's
own artefact — there is no <kind>__ key prefix and no fixed
schema. The result index adapts to that schema:
- per-run frames that carry a
timecolumn → a(run_id, time):class:~pandas.MultiIndex, the same shape as :func:lazy_multiindex; - per-run frames without a
timecolumn → a single-levelrun_idindex (a run that emitted K rows contributes K rows sharing onerun_id).
The presence of a time column is read from the streamed schema
and applied uniformly to the whole sweep. run_id and __status
are reserved column names — a hook that emits columns of those names
will see them dropped and replaced by the aggregator's own.
Every run in the manifest appears in the result. A failed or
skipped run — and an ok run whose hook did not register
name — materialises as one NaN-filled marker row carrying the
run's status in __status (time = NaT under the
(run_id, time) index). Reads honour the streaming memory
contract: a single :mod:pyarrow.dataset scan materialised once, not
a linear concatenation of one frame per run.
Aggregation is manifest-driven: only the Parquets recorded in each
entry's :attr:gmat_sweep.ManifestEntry.extra_outputs are read, so a
stray run_<id>.parquet left in the output directory by an earlier
batch is never picked up.
Parameters¶
manifest_path
Path to the sweep's manifest.jsonl. The manifest is loaded
via :meth:gmat_sweep.Manifest.load; relative paths recorded in
extra_outputs are anchored against manifest_path's parent
directory.
name
Extra-output key to aggregate — one of the keys a postprocess
hook returned. Required: there is no <kind>__ prefix to scan
and no notion of a sole "natural" extra output, so the key is
always explicit.
engine
"pandas" (default) returns a :class:pandas.DataFrame.
"polars" flattens the index into leading columns and returns
a :class:polars.DataFrame; requires the [polars] extra.
Same contract — including the experimental status of the polars
output shape — as :func:lazy_multiindex.
Raises¶
SweepConfigError
No run in the sweep registered an extra output named name
(the message lists the available keys), or engine is neither
"pandas" nor "polars".
ValueError
A manifest entry references an extra-output Parquet that is not
on disk.
lazy_fused_reports(
manifest: Manifest,
output_dir: Path,
names: Sequence[str],
*,
tolerance: str | Timedelta,
spool: bool = True,
) -> DataFrame
Fuse N ReportFile outputs per run into one wide (run_id, time)-indexed DataFrame.
Each report is read via :func:lazy_multiindex and the per-run slices
are stitched together into a single frame whose columns form a
two-level :class:pandas.MultiIndex keyed by (report_name, column).
The first name in names is the merge anchor: subsequent reports
are joined to it per run_id via :func:pandas.merge_asof (with
the user-supplied tolerance), or via an inner join on time
when tolerance="exact" — appropriate when every report shares a
step setting.
Parameters¶
manifest
The sweep manifest.
output_dir
Sweep output root, used to anchor relative paths in the manifest.
names
The ReportFile resource names to fuse, in merge order. The
first name is the anchor (left side of every merge); later names
are joined onto it. Must contain at least two unique names — for a
single report use :func:lazy_multiindex with name=....
tolerance
Required. Either the literal string "exact" (collapses to an
inner join on time per run, appropriate when reports share a
step setting) or any value :func:pandas.merge_asof accepts as a
tolerance argument — typically a :class:pandas.Timedelta.
spool
Forwarded to each underlying :func:lazy_multiindex call.
True (default) streams per-run Parquet a batch at a time;
False reads each fragment in one shot.
Returns¶
pandas.DataFrame
Row index: (run_id, time) :class:MultiIndex. Column index:
a two-level :class:MultiIndex ("report", "field"). For each
report, data columns appear at (report_name, column) and the
per-report status (preserving the per-report contract from
:func:lazy_multiindex) at (report_name, "__status"). A
run-level status sits at ("__status", "").
Notes¶
The anchor selection is asymmetric. A run whose anchor failed
(__status != "ok") — or whose anchor parquet was missing — lands
as a single time=NaT row with all data NaN and the per-report
__status of every report preserved. The other reports' data for
that run is not surfaced. Pick the report most likely to be present
as the first entry of names.
Raises¶
SweepConfigError
names has fewer than two entries, contains duplicates, or any
entry does not match a ReportFile resource in the sweep
(raised by the underlying :func:lazy_multiindex call).
AssertionError
Anchor frame is not sorted on time before a tolerance-based
pandas.merge_asof — a defensive guard against future
refactors that drop the explicit pre-merge sort_values;
users invoking lazy_fused_reports directly will never trip
it.
sweep_summary(
df: DataFrame,
*,
by: str = _TIME_COL,
q: Sequence[float] = (0.05, 0.5, 0.95),
include: Sequence[str] = ("mean", "std"),
dropna: bool = True,
) -> DataFrame
Summarise a sweep DataFrame across runs at each by key.
Turns a (run_id, time)-MultiIndexed DataFrame — as returned by
:func:lazy_multiindex, :func:gmat_sweep.sweep,
:func:gmat_sweep.monte_carlo, or :func:gmat_sweep.latin_hypercube
— into a per-by statistics frame: one row per unique by
value, one column per (statistic, original-column) pair under a
two-level :class:pandas.MultiIndex. The default
q=(0.05, 0.5, 0.95) matches the standard 5/50/95 dispersion bands
and feeds directly into :func:gmat_sweep.plotting.sweep_band_plot.
Parameters¶
df
Input DataFrame indexed by (run_id, time) (or any other
2-level :class:pandas.MultiIndex whose levels include the
requested by key). A __status column, if present, is
treated as a per-run status flag and excluded from the statistic
columns.
by
Index level to group on. "time" (default) collapses across
runs at each time step. "run_id" collapses across time
steps within each run. Other values raise
:class:gmat_sweep.errors.SweepConfigError — categorical or
arbitrary keys are out of scope in this release.
q
Quantiles to compute. Each entry must be a float in the open
interval (0, 1). The default returns the 5th, 50th, and
95th percentiles. Pass an empty tuple to skip quantiles.
include
Non-quantile statistics to compute, in the order they appear in
the output's column-level "statistic" index. Allowed values:
"mean", "std", "min", "max", "count_ok".
"count_ok" is the per-group count of non-NaN values in each
data column. Pass an empty tuple to skip the non-quantile stats
and emit only quantiles.
dropna
True (default) drops rows whose __status != "ok" before
aggregating, so failed and skipped runs are excluded from every
statistic. False keeps them — their NaT/NaN marker rows
contribute a NaT (or run-id) group to the output, mostly NaN.
When df has no __status column the flag has no effect.
Returns¶
pandas.DataFrame
Row index: the unique values of df.index.get_level_values(by).
Column index: a two-level :class:pandas.MultiIndex
("statistic", "field"). Statistic labels are exactly the
entries of include plus f"q{q_val}" (e.g. "q0.05",
"q0.5", "q0.95") for each requested quantile, in the
order include then q. field carries the original
data-column names.
Raises¶
SweepConfigError
by is not "time" or "run_id"; any q_val falls
outside (0, 1); include contains an unknown statistic;
q or include contains duplicate entries; or by is
not an index level of df.
Examples¶
import pandas as pd from gmat_sweep import sweep_summary
df is a (run_id, time)-MultiIndexed sweep DataFrame¶
summary = sweep_summary(df) # doctest: +SKIP summary[("q0.5", "Sat.X")] # median Sat.X across runs at each time # doctest: +SKIP
sweep_diff(
df_a: DataFrame,
df_b: DataFrame,
*,
on: str | None = ...,
how: str = ...,
tolerance: float | Callable[[str], float] | None = ...,
engine: Literal["pandas"] = ...,
) -> DataFrame
sweep_diff(
df_a: DataFrame,
df_b: DataFrame,
*,
on: str | None = None,
how: str = "both",
tolerance: float | Callable[[str], float] | None = None,
engine: str = "pandas",
) -> DataFrame
Pairwise compare two same-shape sweep DataFrames into a diff frame.
Aligns df_a and df_b on the intersection of their indexes,
picks the numeric columns shared between them, and emits per-column
<col>__diff = b - a and/or <col>__rel = (b - a) / a columns
ready for plotting against the rest of the sweep helpers.
Parameters¶
df_a, df_b
Sweep DataFrames to compare. Both must share the same index level
names (e.g. both (run_id, time)); otherwise
:class:gmat_sweep.errors.SweepConfigError is raised. Index keys
present in only one side are silently dropped from the output.
on
None (default) compares row-by-row on the existing index.
"run_id" collapses each side via
groupby(level="run_id").last() first — the per-run final-step
view, useful for "did the dispersion of the final state change?"
comparisons. Other values are not supported.
how
"absolute" emits only <col>__diff = b - a. "relative"
emits only <col>__rel = (b - a) / a (entries where a == 0
land as NaN). "both" (default) emits both, interleaved per
source column.
tolerance
None (default) emits raw diffs. A float masks every diff
whose absolute value is strictly below the cutoff to NaN — both
the __diff and the matching __rel entry are masked at the
same positions, so the output highlights only the meaningful
changes. A callable is invoked once per source column as
tolerance(col_name) -> float to produce a per-column cutoff,
which is the right shape when the data columns carry mixed units
(e.g. Sat.X in km vs. Sat.VX in km/s).
engine
"pandas" (default) returns a :class:pandas.DataFrame with
the same index shape as the inputs. "polars" returns a
:class:polars.DataFrame whose index levels are flattened into
leading columns. Requires the [polars] extra; same semantics
as :func:lazy_multiindex — including the experimental status
of the polars output shape.
Returns¶
pandas.DataFrame or polars.DataFrame
Same index as the (aligned) inputs (flattened to leading columns
under engine="polars"). One column per
(<source-column>, suffix) pair, where suffix is __diff or
__rel per how. When at least one side carries a
__status column, an extra trailing __status_diff column
encodes the per-row status pair: "ok" when both sides are
"ok", otherwise "<a_status>/<b_status>" (e.g.
"failed/ok", "ok/skipped").
Raises¶
SweepConfigError
how is not one of "absolute", "relative", "both";
on is neither None nor "run_id"; df_a and df_b
do not share the same index level names; on="run_id" was
passed against a frame whose index has no run_id level; or
engine is neither "pandas" nor "polars".
Examples¶
from gmat_sweep import sweep, sweep_diff baseline = sweep("mission.script", grid={"Sat.SMA": [7000.0]}, out=...) # doctest: +SKIP perturbed = sweep("mission.script", grid={"Sat.SMA": [7050.0]}, out=...) # doctest: +SKIP diff = sweep_diff(baseline, perturbed, on="run_id") # doctest: +SKIP diff[["Sat.SMA__diff", "Sat.SMA__rel"]] # doctest: +SKIP
mc_convergence(
df: DataFrame,
metric: str | Callable[[DataFrame], float],
*,
terminal_only: bool = ...,
engine: Literal["pandas"] = ...,
) -> DataFrame
mc_convergence(
df: DataFrame,
metric: str | Callable[[DataFrame], float],
*,
terminal_only: bool = False,
engine: str = "pandas",
) -> DataFrame
Diagnose Monte Carlo convergence: running mean / std / SE of the mean over run-id prefixes.
Reduces df to a per-run scalar (or per-run-per-time scalar) under
metric and reports cumulative statistics across the first
n = 1..N runs in run_id order. Standard error of the mean is
running_std / sqrt(n) with a sample standard deviation
(ddof=1); n=1 rows therefore carry NaN for both
running_std and se_mean. Failed and skipped runs are dropped
via __status before the prefix scan, so the n axis reflects
successful runs only.
Parameters¶
df
(run_id, time)-MultiIndexed DataFrame as returned by
:func:gmat_sweep.sweep, :func:gmat_sweep.monte_carlo, or
:func:gmat_sweep.latin_hypercube. A __status column, if
present, is used to drop non-ok runs.
metric
Either a column name in df or a callable
(per_run_subframe) -> float. The callable is invoked once per
run_id with that run's time-indexed slice (__status
dropped) and must return a single float — useful for derived
metrics like final-step miss distance.
terminal_only
True collapses the time index by taking .last() per run
for column-name metrics — the canonical "did the dispersion of
the final state converge?" view. False (default) keeps every
time step and emits one running curve per time. Ignored for
callable metrics, which already return one scalar per run.
engine
"pandas" (default) returns a :class:pandas.DataFrame with a
plain :class:~pandas.RangeIndex. "polars" returns the same
flat-column frame as a :class:polars.DataFrame. Requires the
[polars] extra; same semantics as :func:lazy_multiindex —
including the experimental status of the polars output shape.
Returns¶
pandas.DataFrame or polars.DataFrame
Long-form frame with columns n, running_mean,
running_std, se_mean. When terminal_only=False and
metric is a column name, an additional leading time
column carries the per-time grouping. The frame is sorted by
time (when present) then n ascending.
Raises¶
SweepConfigError
df.index is not a MultiIndex with a run_id level; the
column-name metric is not in df; the callable
metric does not return a numeric scalar; or engine is
neither "pandas" nor "polars".
Examples¶
from gmat_sweep import mc_convergence conv = mc_convergence(df, "MissDistance", terminal_only=True) # doctest: +SKIP conv.tail() # doctest: +SKIP n running_mean running_std se_mean 995 996 ... ... ...
Sensitivity¶
sobol_sample(
perturb: Mapping[str, DistSpec],
n: int,
*,
seed: int | None = None,
calc_second_order: bool = True,
) -> DataFrame
Build a Saltelli/Sobol sample design as an explicit-row DataFrame.
The returned DataFrame has parameter columns in lexicographic order and
a default :class:pandas.RangeIndex — suitable input to
:func:gmat_sweep.sweep via samples=. Row count is n * (2*D + 2)
when calc_second_order=True (the default) and n * (D + 2)
otherwise, where D = len(perturb).
The unit-cube design from :mod:SALib.sample.sobol is lifted into each
parameter's marginal via to_rv_frozen(perturb[k]).ppf(...), so any
distribution shape :data:gmat_sweep.distributions.DistSpec accepts is
supported — not just the uniform/normal/lognormal cases SALib's own
dists knob covers.
Two calls at the same (perturb, n, seed, calc_second_order) produce
bit-equal DataFrames. seed=None falls back to OS entropy and is
not reproducible.
Parameters¶
perturb:
Mapping from dotted-path field name to a distribution spec. Same
surface as :func:gmat_sweep.monte_carlo.
n:
Saltelli base sample size. Must be >= 1. Total runs are
n * (2*D + 2) (or n * (D + 2)); SALib's authors recommend
powers of two.
seed:
Optional integer seed forwarded to SALib's sampler.
calc_second_order:
Whether to expand the design for second-order indices. Match this
flag in :func:sobol_analyze.
Raises¶
SweepConfigError
If perturb is empty, n < 1, or any parameter spec fails
validation in :func:gmat_sweep.distributions.to_rv_frozen.
sobol_analyze(
df: DataFrame,
perturb: Mapping[str, DistSpec],
metric: str | Callable[[DataFrame], Series],
*,
calc_second_order: bool = True,
seed: int | None = None,
) -> DataFrame
Compute Sobol indices on a sweep result via :mod:SALib.analyze.sobol.
Reduces df to a per-run scalar Y vector via metric, then runs
SALib's Sobol analysis and returns a tidy long DataFrame with columns
["kind", "param_a", "param_b", "value", "conf"]:
kindis"S1"(first-order),"ST"(total-order), or"S2"(second-order, only present whencalc_second_order=True).param_ais the parameter name.param_bis the second parameter forS2rows and :data:pandas.NAforS1/STrows.valueis the Sobol index.confis SALib's 95 % bootstrap confidence half-width.
Parameters¶
df:
Sweep result DataFrame. Must come from a sweep launched with the
DataFrame :func:sobol_sample produced — the row count and ordering
are part of the Saltelli design. Failed or skipped runs cannot be
ingested; filter them out beforehand (df = df[df["__status"] == "ok"]).
perturb:
Same mapping handed to :func:sobol_sample. Used to recover the
sorted parameter list.
metric:
Reduces the per-(run_id, time) input to one scalar per run.
str form takes the value of that column at each run's final
time-step (df.groupby(level="run_id")[metric].last()) — the
common end-of-mission state shape. Callable form receives df
and must return a :class:pandas.Series of length n * (2*D + 2)
(or n * (D + 2) when calc_second_order=False).
calc_second_order:
Match the value passed to :func:sobol_sample.
seed:
Optional integer seed forwarded to SALib's bootstrap resampler.
Raises¶
SweepConfigError
If perturb is empty, the input has any non-ok status row,
the metric column is missing, the callable returns a non-Series,
or the reduction yields any NaN value.
Distributions¶
module-attribute
¶
User-facing distribution specification.
One of three shorthand tuples or a pre-frozen scipy distribution:
("normal", mu, sigma)— :func:scipy.stats.normwithloc=mu, scale=sigma.("uniform", lo, hi)— :func:scipy.stats.uniformwithloc=lo, scale=hi - lo.("lognormal", mu, sigma)— :func:scipy.stats.lognormwiths=sigma, scale=exp(mu).- a pre-frozen :class:
scipy.stats._distn_infrastructure.rv_frozen— passes through :func:to_rv_frozenunchanged for callers that need a distribution shape outside the three shorthands.
Plotting¶
sweep_band_plot(
summary: DataFrame,
column: str,
*,
ax: Axes | None = None,
**kwargs: Any,
) -> Axes
Plot one column's median and quantile band from a sweep_summary frame.
Reads the two-level ("statistic", "field") column index produced
by :func:gmat_sweep.sweep_summary, slices the band — the lowest
and highest q* statistics for column — and draws a
:meth:matplotlib.axes.Axes.fill_between shaded band plus a centre
line. The centre is the median (q0.5) when present, otherwise
falls back to mean. The x-axis is the summary's row index
(time for by="time", run_id for by="run_id").
Parameters¶
summary
DataFrame as returned by :func:gmat_sweep.sweep_summary. Must
carry the two-level ("statistic", "field") column index.
column
Original data-column name to plot — the "field" slice under
every statistic in the summary's column index.
ax
Optional pre-existing :class:matplotlib.axes.Axes. None
(default) creates a fresh figure with size (8, 4) inches.
**kwargs
Forwarded to the centre :meth:matplotlib.axes.Axes.plot call
(e.g. label=, linestyle=, linewidth=). The band's
fill_between reuses the line colour at alpha=0.25.
Returns¶
matplotlib.axes.Axes
The Axes carrying the band plot. Save the figure via
ax.figure.savefig(...).
Raises¶
SweepConfigError
summary.columns is not a 2-level MultiIndex; column does
not appear in the "field" level; or no centre statistic
(neither a quantile nor mean) is available for column.
Plot a Monte Carlo convergence curve from :func:gmat_sweep.mc_convergence output.
Renders the running mean as a line and the ±1·SE envelope as a
shaded :meth:matplotlib.axes.Axes.fill_between band, with n
(sample count) on the x-axis. Visualises whether the per-run scalar
metric has stabilised: a flattening centre line and a shrinking band
indicate convergence.
Parameters¶
conv
Long-form convergence DataFrame as returned by
:func:gmat_sweep.mc_convergence with terminal_only=True
(or with a callable metric). Must carry the columns n,
running_mean, and se_mean. The per-time shape returned
by terminal_only=False is not supported in this release —
filter to a single time first or aggregate to terminal-only.
ax
Optional pre-existing :class:matplotlib.axes.Axes. None
(default) creates a fresh figure with size (8, 4) inches.
**kwargs
Forwarded to the centre :meth:matplotlib.axes.Axes.plot call
(e.g. label=, linestyle=, linewidth=). The band's
fill_between reuses the line colour at alpha=0.25.
Returns¶
matplotlib.axes.Axes The Axes carrying the convergence plot.
Raises¶
SweepConfigError
conv is missing one of the required columns, or carries the
per-time time column from the terminal_only=False shape.
sweep_corner(
df: DataFrame,
params: Sequence[str] | None = None,
metric: str
| Callable[[DataFrame], Series[Any]]
| None = None,
*,
manifest: Manifest | None = None,
axes: NDArray[Any] | None = None,
kind: Literal["scatter", "hexbin", "auto"] = "auto",
**kwargs: Any,
) -> NDArray[Any]
Render a corner/pair plot of params coloured by metric.
The helper builds one point per run_id: each axis pair shows a
scatter of the two parameter values, and the diagonal carries a
histogram of each parameter's marginal distribution. The off-diagonal
scatters share a colormap driven by a per-run scalar derived from
metric.
Parameters¶
df:
(run_id, time)-MultiIndexed DataFrame as returned by
:func:gmat_sweep.sweep, :func:gmat_sweep.monte_carlo, or
:func:gmat_sweep.latin_hypercube. The __status column is
used to drop failed and skipped runs.
params:
Sequence of dotted-path names to plot on each axis. None
(default) auto-loads them from manifest.parameter_spec —
manifest must then be supplied.
metric:
Per-run scalar that colours the off-diagonal scatters. Pass a
column name in df to reduce the column per run_id via
.last() (the final time-step's value, matching the existing
notebook idiom), or a callable df -> Series that returns a
:class:pandas.Series indexed by run_id.
manifest:
Optional :class:gmat_sweep.manifest.Manifest. Required when
params is None or any param is not already a column of
df; supplies per-run override values via
entry.overrides[param].
axes:
Optional pre-existing 2-D ndarray of :class:matplotlib.axes.Axes
(shape (N, N) where N == len(params)). None (default)
creates a fresh figure and axes grid sized
(2.5 * N, 2.5 * N) inches.
kind:
Off-diagonal panel kind: "scatter" (one point per run, coloured
by metric), "hexbin" (2-D histogram with cells coloured by
the mean of metric), or "auto" (default — picks
"hexbin" when more than 2000 runs survive the failed/skipped
filter, else "scatter"). Hexbin avoids the silent overplot
problem on large dispersions: a 10⁴-run scatter saturates every
pixel and hides the metric variation.
**kwargs:
Forwarded to :meth:matplotlib.axes.Axes.scatter (or
:meth:~matplotlib.axes.Axes.hexbin) for the off-diagonal panels.
Returns¶
numpy.ndarray
The 2-D ndarray of :class:matplotlib.axes.Axes (shape
(N, N)). Save the figure via axes[0, 0].figure.savefig(...).
Raises¶
SweepConfigError
If params is None and manifest is None, if
metric is None, if any param cannot be resolved from
either df or manifest, if a callable metric returns
a Series not indexed by run_id, or if axes has the wrong
shape.
sweep_heatmap(
df: DataFrame,
x: str,
y: str,
z: str | Callable[[DataFrame], Series[Any]],
*,
manifest: Manifest | None = None,
ax: Axes | None = None,
**kwargs: Any,
) -> Axes
Render a 2-D heatmap of z over a full-factorial (x, y) grid.
Reduces z to one scalar per run_id, looks each run's (x, y)
cell up via the same resolution rules as :func:sweep_corner, and
pivots the result into a matrix rendered with
:meth:matplotlib.axes.Axes.pcolormesh. Failed and skipped runs land
as :data:numpy.nan cells; the colormap's set_bad paints them in
a distinct light-gray so the gap is obvious.
Parameters¶
df:
(run_id, time)-MultiIndexed DataFrame from a sweep call.
x, y:
Dotted-path names of the two grid axes.
z:
Per-run metric — same shape as :func:sweep_corner's metric:
a column in df (reduced via .last()) or a callable
returning a :class:pandas.Series indexed by run_id.
manifest:
Optional :class:gmat_sweep.manifest.Manifest. Required when
x or y is not already a column of df. When supplied,
also asserts the sweep was a two-axis grid-kind sweep — a
Monte Carlo or Latin hypercube manifest raises
:class:SweepConfigError pointing at :func:sweep_corner.
ax:
Optional pre-existing :class:matplotlib.axes.Axes. None
(default) creates a fresh figure with size (8, 5) inches.
**kwargs:
Forwarded to :meth:matplotlib.axes.Axes.pcolormesh (e.g.
cmap=..., shading="auto", norm=...).
Returns¶
matplotlib.axes.Axes
The Axes carrying the heatmap. Save the figure via
ax.figure.savefig(...).
Raises¶
SweepConfigError
If manifest was supplied and its parameter_spec is not a
2-axis grid; if x or y cannot be resolved from either
df or manifest; or if z is a callable whose Series
is not indexed by run_id.
Exceptions¶
Bases: Exception
Base class for every exception raised by gmat-sweep.
Bases: GmatSweepError
Raised when a sweep configuration is invalid before any run starts.
Covers contradictory arguments, malformed grid specs, and dotted-path overrides that fail validation at sweep-construction time. The message is the only payload.
Bases: GmatSweepError
Raised when a single run fails inside a worker (raise-on-failure mode).
Per the gmat-sweep contract a failed run normally lands as a labelled
row in the parent DataFrame rather than an exception, so this class
is a typed sentinel for tests and for callers that opt into eager
failure. The run_id attribute identifies the offending run;
stderr carries the captured worker stderr or Python traceback.
Bases: GmatSweepError
Raised when an execution backend itself fails.
Covers worker-pool initialisation failures, lost workers, and backend
implementations that violate the subprocess-isolation contract
enforced by the :class:Pool ABC.
Bases: GmatSweepError
Raised when a sweep manifest cannot be parsed.
The path attribute points at the offending file so callers can
surface it in error messages without re-deriving the path. The
optional line_number attribute is the 1-indexed line in the file
that failed to parse — set by :meth:gmat_sweep.Manifest.load when the
failure is localised to one line, None for whole-file failures
(e.g. an empty file).