neuralib.persistence.persistence

field	Cache class's exported field.
autoinc_field	make a field as auto increment, which can be used to save and distinguish between same source data but different random/shuffle persistence result.
persistence_class	A class decorator.
ensure_persistence_class	ensure data is a persistence class.
as_dict	transform persistence data into dictionary, which field as key.
from_dict	transform dictionary
load	Load data as data_cls from path.
save	save persistence data under path.
filename	Get data persistence filename.
AutoIncFieldNotResolvedError	Raised when a persistence's autoinc field is not resolved, and it is unable to do following operations.
auto_generated_content	It is used to mark the function which its function body is auto generated.
PersistenceHandler	The handler for loading and saving persistence instance.
PickleHandler	Support field type: all python objects.
GzipHandler	Support field type: all python objects.

Persistence Class

author:

Ta-Shun Su

Define a persistence class

Import

>>> import numpy as np
>>> from neuralib import persistence

Define class

>>> @persistence.persistence_class
... class Example:
...     # key. use filename keyword to display key in filename.
...     use_animal: str = persistence.field(validator=True, filename=True)
...     use_session: str = persistence.field(validator=True)
...     use_date: str = persistence.field(validator=True, filename=True)
...     # data
...     channels: list[int]
...     data: np.ndarray

Load/Save

>>> example = Example(use_animal='A00', use_session='', use_date='1234')
>>> save(example, 'example.pkl')
>>> example_2 = load(Example, 'example.pkl') # example and example_2 should be content identical

Cooperate with PersistenceOptions

>>> from neuralib.persistence.cli_persistence import PersistenceOptions
>>> class ExampleHandle(PersistenceOptions[Example]):
...     def empty_cache(self) -> Example:
...         return Example(use_animal='A00', use_session='', use_date='1234') # with attribute initialization
...     def compute_cache(self, result: Example) -> Example:
...         result.channels = [0, 1, 2]
...         result.riglog = np.array(result.channels)
...         return result

Dynamic generated methods for persistence class

1. __init__({foreach persistence.field})

   >>> class Example:
   ...     ... # as same as above
   ...     def __init__(self, use_animal:str,  use_session: str, use_date: str): # auto generated
   ...         ...
   

2. __str__ return filename

   >>> class Example:
   ...     ... # as same as above
   ...     def __str__(self): # auto generated
   ...         return filename(self)
   

3. __repr__

   >>> class Example:
   ...     ... # as same as above
   ...     def __repr__(self): # auto generated
   ...         return 'Example{' + f'use_animal={self.use_animal}, use_session={self.use_session}, use_date={self.use_date}' + "}"
   

4. _replace when persistence define empty _replace methods. It is NamedTuple._replace like function.

   >>> class Example:
   ...     ... # as same as above
   ...     def _replace(self, **kwargs): pass # empty method
   ...     def _replace(self, *,  # replaced by generated
   ...                  use_animal=missing,
   ...                  use_session=missing,
   ...                  use_date=missing,
   ...                  channels=missing,
   ...                  data=missing) -> Example:
   ...         ...
   

Auto increment field

For some reason you want to save a persistence result that came from same data source but different contents, which are usualy generated by random or suffle process. To save them all separately, you may need a field value that keep track that it is n-th persistence result. autoinc_field() is proposed to help this case.

>>> @persistence_class
... class Result:
...     a: str = field(validator=True, filename=T)
...     b: int = autoinc_field()
...     c: str
...     def __init__(self, a: str, b: int = None): # auto generated signature
...     def _replace(self, *, a: str, c:str): # auto generated signature

There are some rule when a persistence class has an autoinc_field.

1. only one autoinc_field in a persistence class allowed.

2. field type only int is allowed.

3. raise error when load a result without autoinc field resolved.

4. autoinc field is auto resolved when saving. Its value is max(found) + 1

Pickle format

Persistence class is transformed into a dict by as_dict, which as a root instance to be saved into a pickle file.

IMPORTANT

If a persistance class has a custom __init__ function which signature is differed from auto generated, you need to define a classmethod from_dict for creating that persistance class.

>>> @persistence.persistence_class
... class Example:
...     a: int = persistence.field(validator=True, filename=True)
...     b: int = persistence.field(validator=True, filename=True)
...     c: int
...     def __init__(self): ... # custom __init__
...     @classmethod
...     def from_dict(cls, data:dict[str, Any]) -> 'Example':
...         # data = {'a', 'b', 'c'}