luminarycloud.data¶

Data lake: Object upload/download and Case/Dataset operations.

Entities (Object, Case, Dataset, and their nested types) are returned as typed wrappers over the underlying protos. Enums (ObjectStatus, FormatKind, MeshElementAssoc) are scoped under this namespace.

Attributes¶

`MetadataFilter`
`ObjectFilter`
`ParamFilter`
`ProvisionFilter`
`RangeFilter`

Classes¶

`Blob`	The unit of stored bytes backing a file.
`Case`	A Case bundles derived provisions/formats/params (unions) with explicit tags.
`CaseObject`	A member object of a Case, with an optional role label and key/label remappings.
`CaseObjectSpec`	A member object of a Case: an object id with an optional role label and key/label remappings.
`CaseStatus`	Lifecycle status of a Case (terminal once it leaves `UNSETTLED`).
`ComponentStats`	Per-component statistics within a (possibly vector) field.
`Dataset`	A set of Cases. Frozen Datasets are immutable and the only kind eligible for training.
`DatasetStatus`	Lifecycle status of a Dataset, derived from its member cases.
`Field`	A derived field provision. Field identity is `(label, element_assoc)`.
`FieldAnnotation`	A physical-semantics annotation for one derived field, matched by (label, element_assoc).
`FieldStats`	Summary statistics for a scalar series (a single component, or a field's magnitude).
`File`	A named file backed by a blob.
`FileRef`	A reference from a VTM to a child Object (verbatim relative path -> object id).
`Format`	An Object's file format (a oneof over the per-format messages).
`FormatKind`	The file format of an Object.
`Geometry`	A geometry provision: an opaque geometry file (STL/STEP) with declared units.
`Hash`	Content hashes of a blob. Both optional; populated post-upload from the storage backend.
`MeshElementAssoc`	Whether a field is associated with mesh cells or points.
`Object`	An Object the data lake tracks: a file (or multi-file manifest) plus derived metadata.
`ObjectContent`	An Object's physical contents: its backing `File` plus (VTM only) child references.
`ObjectStatus`	Lifecycle status of a data lake Object.
`Param`	A unit-bearing parameter value: a JSON `value` (scalar/vector) plus optional `units`.
`ParameterAugmentation`	Optional augmentation applied to a parameter at `freeze_dataset()` time.
`ParameterDef`	One column of a Dataset's parameter schema.
`Provision`	A machine-readable advertisement of what an Object provides (a field or a geometry).
`Step`	STEP geometry.
`Stl`	STL geometry.
`Unknown`	A recognized-but-unprocessed format.
`Vtm`	VTK MultiBlock: a manifest aggregating independent, reusable child Objects.
`Vtp`	VTK PolyData.
`Vtu`	VTK UnstructuredGrid.

Functions¶

`add_cases_to_dataset`(...)	Append cases to an unfrozen Dataset. Frozen Datasets reject this call.
`create_case`(→ luminarycloud.data._wrappers.Case)	Create a Case from member objects (each with an optional label + key/label remappings).
`create_dataset`(→ luminarycloud.data._wrappers.Dataset)	Create a Dataset, optionally seeded with cases and tags.
`delete_case`(→ None)	Delete a Case. Underlying Objects are unaffected.
`delete_dataset`(→ None)	Delete a Dataset. Underlying Cases are unaffected.
`download_object`(→ Iterator[IO[bytes]])	Download an Object's bytes (its own blob; for a VTM this is the manifest file).
`freeze_dataset`(→ luminarycloud.data._wrappers.Dataset)	Freeze a Dataset (permanently immutable; eligible for training).
`get_case`(→ luminarycloud.data._wrappers.Case)	Fetch a Case by id.
`get_dataset`(→ luminarycloud.data._wrappers.Dataset)	Fetch a Dataset by id.
`get_object`(→ luminarycloud.data._wrappers.Object)	Fetch an Object by id.
`iterate_cases`(→ CaseIterator)	Iterate over Cases visible to the caller, optionally filtered by member-object provisions.
`iterate_dataset_cases`(→ DatasetCasesIterator)	Iterate over all Cases in a Dataset.
`iterate_datasets`(→ DatasetIterator)	Iterate over Datasets, optionally filtered by member-object provisions.
`iterate_objects`(→ ObjectIterator)	Iterate over Objects visible to the caller.
`remove_cases_from_dataset`(...)	Remove cases from an unfrozen Dataset. Frozen Datasets reject this call.
`update_case`(→ luminarycloud.data._wrappers.Case)	Update a Case's mutable metadata (name, tags) and return the updated Case.
`update_dataset`(→ luminarycloud.data._wrappers.Dataset)	Update a Dataset's name, description, and/or tags. Only provided fields change.
`update_object`(→ luminarycloud.data._wrappers.Object)	Update an Object's mutable metadata and return the updated Object.
`upload_object`(→ luminarycloud.data._wrappers.Object)	Upload a file as an Object and return the completed Object.

Package Contents¶

class Blob(proto_type: google.protobuf.message.Message | None = None)¶

The unit of stored bytes backing a file.

content_hash: Hash¶

id: str¶

size: int¶

uri: str¶

class Case(proto_type: google.protobuf.message.Message | None = None)¶

A Case bundles derived provisions/formats/params (unions) with explicit tags.

account_id: str¶

property created_at: datetime.datetime¶

created_by: str¶

failure_reason: str¶

formats: List[Format]¶

id: str¶

name: str¶

objects: List[CaseObject]¶

property params: Dict[str, Param]¶: Derived union of member objects’ params after param_mapping (each a Param).

provisions: List[Provision]¶

status: luminarycloud.data._enums.CaseStatus¶

property tags: dict¶: Explicit user-provided tags (not derived from objects).

class CaseObject(proto_type: google.protobuf.message.Message | None = None)¶

A member object of a Case, with an optional role label and key/label remappings.

property field_mapping: dict¶: Remap of the object’s Field provision labels into the case’s provision union.

label: str¶

object_id: str¶

property param_mapping: dict¶: Remap of the object’s param keys into the case’s param union.

class CaseObjectSpec¶

A member object of a Case: an object id with an optional role label and key/label remappings.

param_mapping remaps the object’s param keys into the case’s param union and field_mapping remaps its Field provision labels into the case’s provision union: an absent key passes through under its own name, a value of None (or "") drops it, and any other value renames it.

field_mapping: Mapping[str, str | None]¶

label: str | None = None¶

object_id: str¶

param_mapping: Mapping[str, str | None]¶

class CaseStatus¶

Lifecycle status of a Case (terminal once it leaves UNSETTLED).

Attributes:

UNSPECIFIED: A well-formed Case will never have this value.
UNSETTLED: At least one member object is not yet in a terminal state.
READY: All members are READY and their provisions were unioned; provisions/formats are populated.
INVALID: Unusable and unrepairable: a member object FAILED, or the members’ provisions could not be unioned. See Case.failure_reason.

INVALID¶

READY¶

UNSETTLED¶

UNSPECIFIED¶

class ComponentStats(proto_type: google.protobuf.message.Message | None = None)¶

Per-component statistics within a (possibly vector) field.

name: str¶

stats: FieldStats¶

class Dataset(proto_type: google.protobuf.message.Message | None = None)¶

A set of Cases. Frozen Datasets are immutable and the only kind eligible for training.

account_id: str¶

property created_at: datetime.datetime¶

created_by: str¶

description: str¶

failure_reason: str¶

formats: List[Format]¶

property frozen_at: datetime.datetime | None¶

id: str¶

property is_frozen: bool¶: Whether this Dataset has been frozen (and is therefore permanently immutable).

name: str¶

property params_schema: dict¶: parameter_name -> ParameterDef, the type-intersection across the cases.

provisions: List[Provision]¶

status: luminarycloud.data._enums.DatasetStatus¶

property tags: dict¶: Explicit user-provided tags (not derived from cases).

class DatasetStatus¶

Lifecycle status of a Dataset, derived from its member cases.

Attributes:

UNSPECIFIED: A well-formed Dataset will never have this value.
UNSETTLED: No member case is INVALID, but at least one is UNSETTLED.
READY: Every member case is READY (an empty Dataset is READY). Only READY Datasets can be frozen; provisions/formats are populated.
INVALID: At least one member case is INVALID; remove those cases to recover. See Dataset.failure_reason.

INVALID¶

READY¶

UNSETTLED¶

UNSPECIFIED¶

class Field(proto_type: google.protobuf.message.Message | None = None)¶

A derived field provision. Field identity is (label, element_assoc).

component_stats: List[ComponentStats]¶

element_assoc: luminarycloud.data._enums.MeshElementAssoc¶

label: str¶

magnitude_stats: FieldStats¶

num_components: int¶

quantity_type: luminarycloud.enum.QuantityType¶

units: str¶

class FieldAnnotation¶

A physical-semantics annotation for one derived field, matched by (label, element_assoc).

element_assoc: luminarycloud.data._enums.MeshElementAssoc¶

label: str¶

quantity_type: luminarycloud.enum.QuantityType¶

units: str = ''¶

class FieldStats(proto_type: google.protobuf.message.Message | None = None)¶

Summary statistics for a scalar series (a single component, or a field’s magnitude).

hist128: List[float]¶

max: float¶

mean: float¶

min: float¶

stdev: float¶

class File(proto_type: google.protobuf.message.Message | None = None)¶

A named file backed by a blob.

blob: Blob¶

name: str¶

class FileRef(proto_type: google.protobuf.message.Message | None = None)¶

A reference from a VTM to a child Object (verbatim relative path -> object id).

object_id: str¶

path: str¶

class Format(proto_type: google.protobuf.message.Message | None = None)¶

An Object’s file format (a oneof over the per-format messages).

property kind: luminarycloud.data._enums.FormatKind¶: The FormatKind of the set format (UNSPECIFIED if none is set).

step: Step¶

stl: Stl¶

unknown: Unknown¶

vtm: Vtm¶

vtp: Vtp¶

vtu: Vtu¶

class FormatKind¶

The file format of an Object.

Attributes:

UNSPECIFIED: Format not yet determined.
VTP: VTK PolyData (surface).
VTU: VTK UnstructuredGrid (volume).
VTM: VTK MultiBlock manifest referencing child Objects.
STL: STL geometry.
STEP: STEP geometry.
UNKNOWN: A recognized-but-unprocessed format.

STEP¶

STL¶

UNKNOWN¶

UNSPECIFIED¶

VTM¶

VTP¶

VTU¶

class Geometry(proto_type: google.protobuf.message.Message | None = None)¶

A geometry provision: an opaque geometry file (STL/STEP) with declared units.

units: str¶

class Hash(proto_type: google.protobuf.message.Message | None = None)¶

Content hashes of a blob. Both optional; populated post-upload from the storage backend.

etag: str¶

md5: str¶

class MeshElementAssoc¶

Whether a field is associated with mesh cells or points.

Together with label this forms a field’s identity.

Attributes:

UNSPECIFIED: Association not known.
CELL: Cell-associated (one value per mesh cell).
POINT: Point-associated (one value per mesh point).

CELL¶

POINT¶

UNSPECIFIED¶

class Object(proto_type: google.protobuf.message.Message | None = None)¶

An Object the data lake tracks: a file (or multi-file manifest) plus derived metadata.

account_id: str¶

content: ObjectContent¶

property created_at: datetime.datetime¶

created_by: str¶

description: str¶

failure_reason: str¶

property file: File | None¶: The backing File (the manifest file for a VTM); None if unset.

format: Format¶

id: str¶

name: str¶

property params: Dict[str, Param]¶: Immutable user-supplied parameters describing the object (each a Param).

property processed_at: datetime.datetime | None¶

provisions: List[Provision]¶

status: luminarycloud.data._enums.ObjectStatus¶

property tags: dict¶: Mutable free-form metadata.

property uploaded_at: datetime.datetime | None¶

class ObjectContent(proto_type: google.protobuf.message.Message | None = None)¶

An Object’s physical contents: its backing File plus (VTM only) child references.

file: File¶

references: List[FileRef]¶

class ObjectStatus¶

Lifecycle status of a data lake Object.

Attributes:

UNSPECIFIED: A well-formed Object will never have this value.
PENDING_UPLOAD: The Object has been created but its bytes have not finished uploading.
UPLOADED: Upload is complete; a VTK Object is queued for / undergoing field derivation.
PROCESSING: Field statistics are being derived from the uploaded bytes.
READY: Fully processed; usable in Cases and Datasets. Field annotations (quantity_type/units) are optional and declared up-front at object creation.
FAILED: Processing failed; see Object.failure_reason.

FAILED¶

PENDING_UPLOAD¶

PROCESSING¶

READY¶

UNSPECIFIED¶

UPLOADED¶

class Param¶

A unit-bearing parameter value: a JSON value (scalar/vector) plus optional units.

Note: numbers round-trip through google.protobuf.Value as doubles, so an integral input like 1800 reads back as 1800.0.

units: str | None = None¶

value: Any¶

class ParameterAugmentation¶

Optional augmentation applied to a parameter at freeze_dataset() time.

default_reference: List[float] = []¶

description: str = ''¶

class ParameterDef(proto_type: google.protobuf.message.Message | None = None)¶

One column of a Dataset’s parameter schema.

default_reference: List[float]¶

description: str¶

type: str¶

class Provision(proto_type: google.protobuf.message.Message | None = None)¶

A machine-readable advertisement of what an Object provides (a field or a geometry).

field: Field¶

geometry: Geometry¶

property which: str | None¶: "field" or "geometry" depending on the kind of provision (None if unset).

class Step(proto_type: google.protobuf.message.Message | None = None)¶: STEP geometry.

class Stl(proto_type: google.protobuf.message.Message | None = None)¶: STL geometry.

class Unknown(proto_type: google.protobuf.message.Message | None = None)¶: A recognized-but-unprocessed format.

class Vtm(proto_type: google.protobuf.message.Message | None = None)¶

VTK MultiBlock: a manifest aggregating independent, reusable child Objects.

property referenced_formats: List[Format]¶: The formats of the referenced child Objects, in reference order.

class Vtp(proto_type: google.protobuf.message.Message | None = None)¶: VTK PolyData.

class Vtu(proto_type: google.protobuf.message.Message | None = None)¶: VTK UnstructuredGrid.

add_cases_to_dataset(dataset_id: str, case_ids: list[str]) → luminarycloud.data._wrappers.Dataset¶: Append cases to an unfrozen Dataset. Frozen Datasets reject this call.

create_case(*, objects: Sequence[CaseObjectSpec], params: Mapping[str, luminarycloud.data._wrappers.Param] | None = None, tags: Mapping[str, Any] | None = None, name: str = '') → luminarycloud.data._wrappers.Case¶

Create a Case from member objects (each with an optional label + key/label remappings).

The Case’s parameter union is derived server-side from the members’ params after applying each member’s param_mapping (identity by default; None/"" drops a key; a non-empty value renames it), then merged with the case-level params. Conflicting values for the same key are rejected. tags are explicit user metadata (not derived from objects).

create_dataset(*, name: str, description: str = '', case_ids: Sequence[str] | None = None, tags: Mapping[str, Any] | None = None) → luminarycloud.data._wrappers.Dataset¶

Create a Dataset, optionally seeded with cases and tags.

More cases can be added later via add_cases_to_dataset() (until frozen). tags are explicit user metadata (not derived from the cases).

delete_case(case_id: str) → None¶: Delete a Case. Underlying Objects are unaffected.

delete_dataset(dataset_id: str) → None¶: Delete a Dataset. Underlying Cases are unaffected.

download_object(object_id: str) → Iterator[IO[bytes]]¶: Download an Object’s bytes (its own blob; for a VTM this is the manifest file).

freeze_dataset(dataset_id: str, *, augmentations: Mapping[str, ParameterAugmentation] | None = None) → luminarycloud.data._wrappers.Dataset¶

Freeze a Dataset (permanently immutable; eligible for training).

augmentations optionally adds default_reference/description to parameters already present in the type-intersection across the dataset’s cases.

get_case(case_id: str) → luminarycloud.data._wrappers.Case¶: Fetch a Case by id.

get_dataset(dataset_id: str) → luminarycloud.data._wrappers.Dataset¶: Fetch a Dataset by id.

get_object(object_id: str) → luminarycloud.data._wrappers.Object¶: Fetch an Object by id.

iterate_cases(page_size: int = 100, *, object_filter: luminarycloud._proto.datalake.datalake_pb2.ObjectFilter | None = None) → CaseIterator¶: Iterate over Cases visible to the caller, optionally filtered by member-object provisions.

iterate_dataset_cases(dataset_id: str, page_size: int = 100) → DatasetCasesIterator¶: Iterate over all Cases in a Dataset.

iterate_datasets(page_size: int = 100, *, object_filter: luminarycloud._proto.datalake.datalake_pb2.ObjectFilter | None = None) → DatasetIterator¶: Iterate over Datasets, optionally filtered by member-object provisions.

iterate_objects(*, object_filter: luminarycloud._proto.datalake.datalake_pb2.ObjectFilter | None = None, page_size: int = 100) → ObjectIterator¶

Iterate over Objects visible to the caller.

Pass an ObjectFilter to filter by provision (quantity_type, element_assoc, magnitude ranges), by params, by tags containment, by format, or by status.

remove_cases_from_dataset(dataset_id: str, case_ids: list[str]) → luminarycloud.data._wrappers.Dataset¶: Remove cases from an unfrozen Dataset. Frozen Datasets reject this call.

update_case(case_id: str, *, name: str | None = None, tags: Mapping[str, Any] | None = None) → luminarycloud.data._wrappers.Case¶

Update a Case’s mutable metadata (name, tags) and return the updated Case.

Only provided fields change. tags is replaced wholesale (full replacement, not merged): pass a mapping to set the tags, an empty mapping to clear them, or None to leave them untouched.

update_dataset(dataset_id: str, *, name: str | None = None, description: str | None = None, tags: Mapping[str, Any] | None = None) → luminarycloud.data._wrappers.Dataset¶

Update a Dataset’s name, description, and/or tags. Only provided fields change.

tags is replaced wholesale (full replacement, not merged): pass a mapping to set the tags, an empty mapping to clear them, or None to leave them untouched. A frozen Dataset rejects this call.

update_object(object_id: str, *, name: str | None = None, description: str | None = None, tags: Mapping[str, Any] | None = None) → luminarycloud.data._wrappers.Object¶

Update an Object’s mutable metadata and return the updated Object.

Only provided fields change. tags is replaced wholesale (full replacement, not merged): pass a mapping to set the tags, an empty mapping to clear them, or None to leave them untouched. Immutable fields (params, format, content) cannot be changed.

upload_object(file_path: os.PathLike | str, *, name: str = '', description: str = '', params: Mapping[str, luminarycloud.data._wrappers.Param] | None = None, tags: Mapping[str, Any] | None = None, geometry_units: str = '', provision_annotations: Sequence[FieldAnnotation] | None = None, chunk_size: int = _DEFAULT_CHUNK_SIZE) → luminarycloud.data._wrappers.Object¶

Upload a file as an Object and return the completed Object.

The format is inferred from the file extension. For a .vtm the referenced files are discovered by parsing the manifest, uploaded as independent child Objects, and the VTM is created referencing them.

Parameters:

file_pathPathLike | str: Path to the file to upload. The basename becomes the Object’s file name.
name, descriptionstr, optional: Display name / description.
paramsMapping[str, Param], optional: Immutable unit-bearing parameters describing the object.
tagsMapping[str, Any], optional: Mutable free-form metadata.
geometry_unitsstr, optional: Units for STL/STEP geometry provisions; sends the object straight to READY.
provision_annotationssequence of FieldAnnotation, optional: Up-front provision annotations, reconciled with derived fields after processing.
chunk_sizeint, optional: Upload chunk size. Defaults to 4 MiB.

MetadataFilter¶

ObjectFilter¶

ParamFilter¶

ProvisionFilter¶

RangeFilter¶