Welcome to Deeplite Profiler documentation!

class deeplite.profiler.profiler.ComputeEvalMetric(func, key=None, default_split='test', unit_name='', comparative=Comparative.DIFF)

Bases: ExternalProfilerFunction

Compute EvalMetric from an evaluation function. Examples of those functions can be found in deeplite.torch.profiler.torch_inference.

This makes the additional assumption that an evaluation function is defined only on one split ( or one data loader)

If the evaluation function returns a dict, this class can return secondary metrics from that dict. For each key corresponding to a desired metric, call add_secondary_metrics() with that key. Optionally include the metric’s unit_name and comparative if different from the eval_metric’s

add_secondary_metric(key, unit_name=None, description=None, comparative=None)
get_bounded_status_keys()

Returns a single or a tuple of StatusKey that this ProfilerFunction computes.

pipe_kwargs_to_call(model, data_splits, kwargs)

Calls itself using model, data_splits and an arbitrary kwargs dict. It uses the bind method of inspect.Signature object.

class deeplite.profiler.profiler.ExternalProfilerFunction(func)

Bases: ProfilerFunction

ProfilerFunction in which the profiling computation comes from an external callable instead of being defined in this __call__ method.

The function func still needs to respect the same limitation that ProfilerFunction.__call__ has: It needs to accept (model, data, <any keywords BUT NOT **kwargs>). It is important to stress out the fact that it is not possible to support a **kwargs in the signature. See ProfilerFunction for details.

can_pipe()

Returns False if its __call__ signature contains *args or **kwargs else True. This is checked through python introspection module inspect.signature.

pipe_kwargs_to_call(model, data_splits, kwargs)

Calls itself using model, data_splits and an arbitrary kwargs dict. It uses the bind method of inspect.Signature object.

class deeplite.profiler.profiler.Profiler(model, data_splits, name='<UnNamedModel>', display_status_filter_func=<function make_display_filter_function.<locals>.display_filter_function>)

Bases: ABC

Abstract class for holding profiler functions. It is the entrypoint for profiling a model over some data. The user simply needs to register ProfilerFunction to the Profiler instance through register_profiler_function(). Once this is done, the user can query the Profiler for some StatusKey. Profiler will take care to call the relevant ProfilerFunction that can compute this StatusKey.

Each new framework should subclass this class. See TorchProfiler for example using Pytorch.

NOTE: Status keys are held in a dictionary like fashion. However, the user should never access and write directly on this dictionary and should go though the various status methods defined on this class. This is to keep some sort of consistency between the ProfilerFunction and the values they compute. (for ex.: the recompute flag of compute_status())

Parameters

  • model (<native object type>) – An object known by ProfilerFunction of that concrete framework (Pytorch, Tensorflow, …)

  • data_splits (dict of str to <native data object>) – Splits of the data used to for the ProfilerFunction.

  • name (str, optional) – A user-friendly name for the model, defaults to “<UnNamedModel>”

  • display_status_filter_func (callable, optional) – A method to display the computed metrics in a user-friendly readable format, defaults to deeplite.profiler.formatter.default_display_filter_function

clone(model=None, data_splits=None, retain_status=False)
compare(other, print_mode='info', recompute=False, short_print=True, **kwargs)

Compare two different :class:`~Profiler`s. The two different profilers could belong to the same model or different models. Compare is for stdout / logging comparison.

Parameters

  • other (Profiler) – Another profiler object

  • print_mode (str, optional) – Logger print mode, defaults to ‘info’

  • recompute (bool, optional) – If all the StatusKey need to be recomputed, defaults to False

  • short_print (bool, optional) – If to display a short version of the profiled output or a long detailed version, defaults to True

Raises

  • ValueError – Can only compare with another profiler instance

  • ValueError – Given other profiler does not have the same set of status keys as this profiler

compute_network_status(print_mode=None, recompute=False, short_print=True, **kwargs)

Calls all registered ProfilerFunction and populates its whole status dictionary.

Parameters

  • print_mode (str, optional) – A default printing mode of the logger, defaults to None (no logging)

  • recompute (bool, optional) – If all the StatusKey need to be recomputed, defaults to False

  • short_print (bool, optional) – Display a short version of the profiled output or a long detailed version, defaults to True

Returns

A dictionary of updated key-value pairs of the metric status key and the metric value

Return type

dictionary

compute_status(status, recompute=False, **kwargs)

Computing a StatusKey. The method will fetch its associated ProfilerFunction and call it with its model, data_splits and kwargs.

Parameters

  • status (str) – Unique string identifier of wanted StatusKey

  • recompute (bool, optional) – If the metric values need to be recomputed, defaults to False

Raises

  • ValueError – If an unrecognized status key is found

  • TypeError – If the profiling function and its status key are matched incorrectly

Returns

Returns what the ProfilerFunction returned for this :param:`status`

display_status(other=None, print_mode='debug', short_print=True)

Beautiful user-friendly display of either one or two dictionaries of profiler status

Parameters

  • other (dictionary, optional) – An optional second status message to compare, defaults to None

  • print_mode (str, optional) – Logger print mode, defaults to ‘debug’

  • short_print (bool, optional) – If to display a short version of the profiled output or a long detailed version, defaults to True

abstract classmethod dl_cls()

Returns the concrete framework DataLoader

classmethod enable_forward_pass_data_splits(data_splits, forward_pass=None)

Creating deeplite dataloader from the provided data splits dictionary.

Parameters

  • native_ds (<native data loader object> (ex.: torch.utils.data.DataLoader or tf.data)) – Splits of the train-test input data (ex: {‘train’: train_loader, ‘test’: test_loader}). Internally each dataloader can one of a PyTorch or Tensorflow dataloader

  • forward_pass (ForwardPass, optional) – A forward pass utility to send the data through the model, defaults to None

Returns

deeplite dataloader

Return type

deeplite.profiler.data_loader.DataLoader

abstract classmethod fp_cls()

Returns the concrete framework ForwardPass

load_from_dict(status_dict, to_value=True)

Update stored values with values from dict

register_profiler_function(profiler_func, override=False)

Register a ProfilerFunction with the Profiler. It will become callable that the Profiler recognizes and can fetch through a compute_status() call. Since each ProfilerFunction is associated to different StatusKey, it is possible that two ProfilerFunction compete for the same StatusKey. For this reason, the :param:`override` can be provided to override a currently registered function with the incoming :param:`profiler_func`.

Parameters

  • profiler_func (ProfilerFunction) – A callable that computes some StatusKey

  • override (bool, optional) – Overrides any StatusKey computed with already registered ProfilerFunction to new profiler_func, defaults to False

Raises

  • TypeError – Cannot pipe the provided ProfilerFunction

  • RuntimeError – Two ProfilerFunction’s with the same StatusKey and both with override=True

Returns

profiler_func itself

Return type

ProfilerFunction

reset_status()
status_contains(sk)
status_get(sk)
status_items(to_value=True)
status_keys()
status_to_dict(to_value=True)
status_values(to_value=True)
class deeplite.profiler.profiler.ProfilerFunction

Bases: ABC

Abstract callable which computes any set of StatusKey on given model and data.

Every ProfilerFunction is bound to a set of StatusKey which they compute their values. The Profiler seeks this particular function upon a query for a StatusKey bounded to it.

IMPORTANT: The common use case is to request for all profiler’s functions to be computed. Since by nature its functions are dynamically registered, Profiler does not know in advance what arguments can be passed to them. It tries to magically pipe the user’s kwargs to its functions. It creates a limitation on this class’ __call__ signature when its instance is used directly (as opposed to mid-level class only defined as abstraction and never used as instance). Because of this magic, the signature of the __call__ cannot contain *args or **kwargs.

can_pipe()

Returns False if its __call__ signature contains *args or **kwargs else True. This is checked through python introspection module inspect.signature.

abstract get_bounded_status_keys()

Returns a single or a tuple of StatusKey that this ProfilerFunction computes.

pipe_kwargs_to_call(model, data_splits, kwargs)

Calls itself using model, data_splits and an arbitrary kwargs dict. It uses the bind method of inspect.Signature object.

Indices and tables