neuralib.persistence.persistence

Persistence Class

author:

Ta-Shun Su

Define a persistence class

Import

>>> import numpy as np
>>> from neuralib import persistence

Define class

>>> @persistence.persistence_class
... class Example:
...     # key. use filename keyword to display key in filename.
...     use_animal: str = persistence.field(validator=True, filename=True)
...     use_session: str = persistence.field(validator=True)
...     use_date: str = persistence.field(validator=True, filename=True)
...     # data
...     channels: list[int]
...     data: np.ndarray

Load/Save

>>> example = Example(use_animal='A00', use_session='', use_date='1234')
>>> save(example, 'example.pkl')
>>> example_2 = load(Example, 'example.pkl') # example and example_2 should be content identical

Cooperate with PersistenceOptions

>>> from neuralib.persistence.cli_persistence import PersistenceOptions
>>> class ExampleHandle(PersistenceOptions[Example]):
...     def empty_cache(self) -> Example:
...         return Example(use_animal='A00', use_session='', use_date='1234') # with attribute initialization
...     def compute_cache(self, result: Example) -> Example:
...         result.channels = [0, 1, 2]
...         result.riglog = np.array(result.channels)
...         return result

Dynamic generated methods for persistence class

  1. __init__({foreach persistence.field})

    >>> class Example:
...     ... # as same as above
...     def __init__(self, use_animal:str,  use_session: str, use_date: str): # auto generated
...         ...

  2. __str__ return filename

    >>> class Example:
...     ... # as same as above
...     def __str__(self): # auto generated
...         return filename(self)

  3. __repr__

    >>> class Example:
...     ... # as same as above
...     def __repr__(self): # auto generated
...         return 'Example{' + f'use_animal={self.use_animal}, use_session={self.use_session}, use_date={self.use_date}' + "}"

  4. _replace when persistence define empty _replace methods. It is NamedTuple._replace like function.

    >>> class Example:
...     ... # as same as above
...     def _replace(self, **kwargs): pass # empty method
...     def _replace(self, *,  # replaced by generated
...                  use_animal=missing,
...                  use_session=missing,
...                  use_date=missing,
...                  channels=missing,
...                  data=missing) -> Example:
...         ...

Auto increment field

For some reason you want to save a persistence result that came from same data source but different contents, which are usualy generated by random or suffle process. To save them all separately, you may need a field value that keep track that it is n-th persistence result. autoinc_field() is proposed to help this case.

>>> @persistence_class
... class Result:
...     a: str = field(validator=True, filename=T)
...     b: int = autoinc_field()
...     c: str
...     def __init__(self, a: str, b: int = None): # auto generated signature
...     def _replace(self, *, a: str, c:str): # auto generated signature

There are some rule when a persistence class has an autoinc_field.

  1. only one autoinc_field in a persistence class allowed.

  2. field type only int is allowed.

  3. raise error when load a result without autoinc field resolved.

  4. autoinc field is auto resolved when saving. Its value is max(found) + 1

Pickle format

Persistence class is transformed into a dict by as_dict, which as a root instance to be saved into a pickle file.

IMPORTANT

If a persistance class has a custom __init__ function which signature is differed from auto generated, you need to define a classmethod from_dict for creating that persistance class.

>>> @persistence.persistence_class
... class Example:
...     a: int = persistence.field(validator=True, filename=True)
...     b: int = persistence.field(validator=True, filename=True)
...     c: int
...     def __init__(self): ... # custom __init__
...     @classmethod
...     def from_dict(cls, data:dict[str, Any]) -> 'Example':
...         # data = {'a', 'b', 'c'}
neuralib.persistence.persistence.field(validator=False, filename_prefix='', filename=False)[source]

Cache class’s exported field. Used as keys to find correspond persistence.

Parameters:

  • validator (bool | Callable[[T, T], bool]) – validate this field. use __eq__ by default. Can be a callable as a customized validator.

  • filename_prefix (str) – prefix word of filename

  • filename (bool | Callable[[...], str]) – display this value on filename. Can be a callable that return the string.

Returns:

Return type:

T

neuralib.persistence.persistence.autoinc_field(filename_prefix='')[source]

make a field as auto increment, which can be used to save and distinguish between same source data but different random/shuffle persistence result.

Parameters:

filename_prefix (str)

Returns:

Return type:

int

neuralib.persistence.persistence.persistence_class(cls=None, /, *, name=None, filename_field_splitter='-')[source]

A class decorator.

Decorated class …

Parameters:

  • cls (Type | None) – Persistence class.

  • name (str | None) – class name as filename.

  • filename_field_splitter – the field splitter on filename.

Returns:

neuralib.persistence.persistence.ensure_persistence_class(data)[source]

ensure data is a persistence class.

Parameters:

data (T | Type[T]) – instance or type

Returns:

persistence info

Raises:

RuntimeError – not a persistence class

Return type:

PersistentClass[T]

neuralib.persistence.persistence.as_dict(data)[source]

transform persistence data into dictionary, which field as key.

Parameters:

data (T) – persistence instance

Returns:

dict

Return type:

dict[str, Any]

neuralib.persistence.persistence.from_dict(data_cls, d)[source]

transform dictionary

Parameters:

  • data_cls (type[T])

  • d (dict[str, Any])

Returns:

Return type:

T

neuralib.persistence.persistence.load(data_cls, path)[source]

Load data as data_cls from path.

Parameters:

  • data_cls (type[T]) – data class type

  • path (str | Path) – filepath

Returns:

persistence instance

Return type:

T

neuralib.persistence.persistence.save(data, path)[source]

save persistence data under path.

Parameters:

  • data (T) – persistence instance

  • path (str | Path) – filepath

Return type:

None

neuralib.persistence.persistence.filename(result, **kwargs)[source]

Get data persistence filename.

Parameters:

  • result (T | Type[T])

  • kwargs – overwrite fields.

Returns:

filename

Raises:

  • RuntimeError – result’s autoinc field not resolved, or filename field missing.

  • TypeError – wrong field.filename type.

  • AutoIncFieldNotResolvedError – result’s autoinc field not resolved.

Return type:

str

exception neuralib.persistence.persistence.AutoIncFieldNotResolvedError[source]

Raised when a persistence’s autoinc field is not resolved, and it is unable to do following operations.

__init__(instance, field, message=None)[source]
Parameters:

  • field (str | PersistentField)

  • message (str | None)

neuralib.persistence.persistence.auto_generated_content(**kwargs)[source]

It is used to mark the function which its function body is auto generated.

Parameters:

kwargs – blackhole

Returns:

nothing

class neuralib.persistence.persistence.PersistenceHandler[source]

The handler for loading and saving persistence instance.

abstract property persistence_class: type[T]
Returns:

type T

property persistence_info: PersistentClass[T]

information for persistence class

abstract property save_root: Path

saving directory

filename(result, **kwargs)[source]

build filename for persistence instance.

Parameters:

  • result (T | None) – persistence instance

  • kwargs – overwrite field value in result.

Returns:

file name of result, may contains ‘{}’ if result’s autoinc field not resolved

Return type:

str

filepath(result, **kwargs)[source]

build filepath for persistence instance.

Parameters:

  • result (T | None) – persistence instance

  • kwargs – overwrite field value in result.

Returns:

file path of result

Raises:

RuntimeError – result’s autoinc field not resolved, or errors from filename()

Return type:

Path

validate(ref, res)[source]
Parameters:

  • ref (T)

  • res (T)

Return type:

bool

save_persistence(result, path=None)[source]

save persistence result under path.

Parameters:

  • result (T)

  • path (str | Path) – save path.

Returns:

result. autoinc field will be resolved after saving.

Raises:

AutoIncFieldNotResolvedError –

Return type:

T

load_persistence(path)[source]

Load data as data_cls from path without validation.

Parameters:

path (Path | T | dict[str, Any]) – load from path.

Returns:

persistence instance

Raises:

IsADirectoryError –

Return type:

T

load_all(result, **kwargs)[source]

load all persistent result under save_root. missing is used to make a field becomes a wildcard field.

>>> template = Example(use_animal='A00', use_session='test', use_date='20200101')
>>> # find all animal A00's persistent result.
>>> found = PickleHandler(Example, Path('.')).load_all(template, use_date=missing)
Parameters:

result (T | None)

Return type:

Iterator[tuple[Path, T]]

class neuralib.persistence.persistence.PickleHandler[source]

Support field type: all python objects.

__init__(data_cls, save_root, ext='.pkl')[source]
Parameters:

  • data_cls (type[T])

  • save_root (Path)

  • ext (str)

property persistence_class: type[T]
Returns:

type T

property save_root: Path

saving directory

filename(result, **kwargs)[source]

build filename for persistence instance.

Parameters:

  • result (T | None) – persistence instance

  • kwargs – overwrite field value in result.

Returns:

file name of result, may contains ‘{}’ if result’s autoinc field not resolved

Return type:

str

class neuralib.persistence.persistence.GzipHandler[source]

Support field type: all python objects.

__init__(data_cls, save_root, ext='.pkl.gz', compression=9)[source]
Parameters:

  • data_cls (type[T])

  • save_root (Path)

  • ext (str)

  • compression (int)

property persistence_class: type[T]
Returns:

type T

property save_root: Path

saving directory

filename(result, **kwargs)[source]

build filename for persistence instance.

Parameters:

  • result (T | None) – persistence instance

  • kwargs – overwrite field value in result.

Returns:

file name of result, may contains ‘{}’ if result’s autoinc field not resolved

Return type:

str