ch_util.data_quality

Data quality routines

Data quality functions

Issues

Auxiliary functions are still lacking documentation.

Functions

good_channels(data[, gain_tol, noise_tol, ...])

Test data for misbehaving channels.
ch_util.data_quality.good_channels(data, gain_tol=10.0, noise_tol=2.0, fit_tol=0.02, test_freq=0, noise_synced=None, inputs=None, res_plot=False, verbose=True)[source]

Test data for misbehaving channels.

Three tests are performed:

  1. Excessively high digital gains,

  2. Compliance of noise to the radiometer equation and

  3. Goodness of fit to a template Tsky.

See Doclib:235 file ‘data_quality.pdf’ for details on how the filters and tolerances work.

Parameters:

  • data (ch_util.andata.CorrData object) – Data to run test on. If andata object contains cross-correlations, test is performed on auto-correlations only.

  • gain_tol (float) – Tolerance for digital gains filter. Flag channels whose digital gain fractional absolute deviation is above ‘gain_tol’ (default is 10.)

  • noise_tol (float) – Tolerance for radiometer noise filter. Flag channels whose noise rms is higher then ‘noise_tol’ times the expected from the radiometer equation. (default = 2.)

  • fit_tol (float) – Tolerance for the fit-to-Tsky filter. Flag channels whose fractional rms for the ‘gain’ fit parameter is above ‘fit_tol’ (default = 0.02)

  • test_freq (integer) – Index of frequency to test. Default is 0.

  • noise_synced (boolean) – Use this to force the code to call (or not call) ni_utils.process_synced_data(). If not given, the code will determine if syncronized noise injection was on. For acquisitions newer then 20150626T200540Z_pathfinder_corr, noise injection info is written in the attributes. For older acquisitions the function _check_ni() is called to determine if noise injection is On.

  • inputs (list of CorrInputs, optional) – List of CorrInput objects describing the channels in this dataset. This is optional, if not set (default), then it will look the data up in the database. This option just allows control of the database accesses.

  • res_plot (boolean, optional) – If True, a plot with all the tested channels and the Tsky fits is generated. File naming is plot_fit_{timestamp}.pdf

  • verbose (boolean, optional) – Print out useful output as the tests are run.

Returns:

  • good_gains (list of int) –

    1. for channels that pass the gains filter, 0. otherwise.

  • good_noise (list of int) –

    1. for channels that pass the noise filter, 0. otherwise.

  • good_fit (list of int) –

    1. for channels that pass the fit-to-Tsky filter, 0. otherwise.

  • test_chans (list of int) – A list of the channels tested in the same order as they appear in all the other lists returned

Examples

Run test on frequency index 3. data is an andata object:

>>> good_gains, good_noise, good_fit, test_chans = good_channels(data,test_freq=3)

And to create a plot of the results:

>>> good_gains, good_noise, good_fit, test_chans =     ... good_channels(data,test_freq=3,res_plot=True)