pyyeti.dsp.despike_diff

pyyeti.dsp.despike_diff(x, n, sigma=8.0, maxiter=-1, threshold_sigma=2.0, threshold_value=None, exclude_point='first', **kwargs)[source]

Delete outlier data points from signal based on level changes

Parameters:

  • x (1d array_like) – Signal to de-spike.

  • n (odd integer) – Number of points for moving average; if even, it is reset to n+1. If greater than the dimension of x, it is reset to the dimension or 1 less.

  • sigma (real scalar; optional) – Number of standard deviations beyond which a point is considered an outlier. The default value is quite high; this is possible because the point itself is excluded from the calculations.

  • maxiter (integer; optional) – Maximum number of iterations of outlier removal allowed. If exclude_point is ‘first’, only the last spike is removed on each iteration; if it is ‘last’, only the first spike is removed on each iteration. It is done this way because removing a spike can expose other points as spikes (but didn’t appear to be because the removed spike was present). If <= 0, there is no set limit and the looping will stop when no more outliers are detected. Routine will always run at least 1 loop (setting maxiter to 0 is the same as setting it to 1).

  • threshold_sigma (scalar; optional) – Number of standard deviations below which all data is kept. This standard deviation is computed from x. Let dx = np.diff(x), the standard deviation is std(dx - moving_average(dx)). The moving average uses a window of n size. This value exists to avoid deleting small deviations such as bit toggles. Set to 0.0 to not use a threshold. threshold_value overrides threshold_sigma if it is not None.

  • threshold_value (scalar or None; optional) – Optional method for specifying a minimum threshold. If not None, this scalar is used as an absolute minimum deviation from the moving average for a value to be considered a spike. Overrides threshold_sigma. Set to 0.0 to not use a threshold.

  • exclude_point (string or int; optional) – Defines where, within each window, the point that is being considered as a potential outlier is. For this routine, exclude_point must be either ‘first’ (or 0) or ‘last’ (or n-1). For example, ‘first’ compares the first point in each window the rest in that window to test if it is an outlier.

  • **kwargs (other args are ignored) – This is here to accommodate fixtime().

Returns:

  • A SimpleNamespace with the members

  • x (1d ndarray) – Despiked version of input x. Will be shorter than input x if any spikes were deleted; otherwise, it will equal input x.

  • pv (bool 1d ndarray; same size as input x) – Has True where an outlier was detected

  • niter (integer) – Number of iterations executed

Notes

Uses exclusive_sgfilter() to exclude the point being tested from the moving average and the moving standard deviation calculations. Each point is tested. The points near the ends of the signal may not be at the requested position in the window (see exclusive_sgfilter() for more information on this).

To not use a threshold, set threshold_sigma to 0.0 (or set threshold_value to 0.0).

Note

If you plan to use both fixtime() and despike_diff(), it is recommended that you let fixtime() call despike_diff() (via the delspikes option) instead of calling it directly. This is preferable because the ideal time to run despike_diff() is in the middle of fixtime(): after drop-outs have been deleted but before gaps are filled.

Examples

Set threshold_value to catch second spike but not first. The threshold is based on differences:

>>> import numpy as np
>>> from pyyeti import dsp
>>> x = [2, 2, 2, 2, 5, 2, 2, 2, 2, 7, 2, 2, 2, 2, 2]
>>> s = dsp.despike_diff(x, n=5, threshold_value=4)
>>> s.x
array([2, 2, 2, 2, 5, 2, 2, 2, 2, 2, 2, 2, 2, 2])
>>> s.pv
array([False, False, False, False, False, False, False, False,
       False,  True, False, False, False, False, False], dtype=bool)
>>> s.niter
2