ch_util.ni_utils

Tools for noise injection data

Functions

gains2utvec(g)	Converts a vector into an outer product matrix and vectorizes its upper triangle to obtain a vector in same format as the CHIME visibility matrix.
gains2utvec_tf(gains)	Converts gain array to CHIME visibility format for all frequencies and time frames.
gen_prod_sel(channels_to_select, ...)	Generates correlation product indices for selected channels.
ktrprod(A, B)	Khatri-Rao or column-wise Kronecker product of two matrices.
mat2utvec(A)	Vectorizes its upper triangle of the (hermitian) matrix A.
ni_als(R, g0, Gamma, Upsilon, maxsteps, ...)	Implementation of the Alternating Least Squares algorithm for noise injection.
ni_gains_evalues(C[, normalize_vis])	Basic algorithm to compute gains and evalues from noise injection data.
ni_gains_evalues_tf(vis_gated, Nchannels[, ...])	Computes gains and evalues from noise injection visibility data.
process_gated_data(data[, only_off])	Processes fast gating data and turns it into gated form.
process_synced_data(data[, ni_params, only_off])	Turn a synced noise source observation into gated form.
sort_evalues_mag(evalues)	Sorts eigenvalue array by magnitude for all frequencies and time frames
subtract_sky_noise(vis, Nchannels, ...)	Removes sky and system noise contributions from noise injection visibility data.
utvec2mat(n, utvec)	Recovers a hermitian matrix a from its upper triangle vectorized version.

Classes

ni_data(Reader_read_obj, Nadc_channels[, ...])

Provides analysis utilities for CHIME noise injection data.

ch_util.ni_utils.gains2utvec(g)

Converts a vector into an outer product matrix and vectorizes its upper triangle to obtain a vector in same format as the CHIME visibility matrix.

Parameters:

g (1d array) – gain vector

Return type:

1d array with vectorized form of upper triangle for the outer product of g

ch_util.ni_utils.gains2utvec_tf(gains)

Converts gain array to CHIME visibility format for all frequencies and time frames.

For every frequency and time frame, converts a gain vector into an outer product matrix and then vectorizes its upper triangle to obtain a vector in the same format as the CHIME visibility matrix.

Converting the gain arrays to CHIME visibility format makes easier to apply the gain corrections to the visibility data. See example below.

Parameters:

gains (3d array) – Input array with the gains for all frequencies, channels and time frames in the fromat of ni_gains_evalues_tf. g has dimensions [frequency, channels, time].

Returns:

G_ut – Output array with dimmensions [frequency, corr. number, time]. For every frequency and time frame, contains the vectorized form of upper triangle for the outer product of the respective gain vector.

Return type:

3d array

Example

To compute the gains from a set of noise injection pass0 data and apply the gains to the visibilities run:

>>> from ch_util import andata
>>> from ch_util import import ni_utils as ni
>>> data = andata.Reader('/scratch/k/krs/jrs65/chime_archive/20140916T173334Z_blanchard_corr/000[0-3]*.h5')
>>> readdata = data.read()
>>> nidata = ni.ni_data(readdata, 16)
>>> nidata.get_ni_gains()
>>> G_ut = ni.gains2utvec(nidata.ni_gains)
>>> corrected_vis = nidata.vis_off_dec/G_ut

See also

gains2utvec, ni_gains_evalues_tf

ch_util.ni_utils.gen_prod_sel(channels_to_select, total_N_channels)

Generates correlation product indices for selected channels.

For a correlation matrix with total_N_channels total number of channels, generates indices for correlation products corresponding to channels in the list channels_to_select.

Parameters:

• channels_to_select (list of integers) – Indices of channels to select

• total_N_channels (int) – Total number of channels

Returns:

prod_sel – indices of correlation products for channels in channels_to_select

Return type:

array

ch_util.ni_utils.ktrprod(A, B)

Khatri-Rao or column-wise Kronecker product of two matrices.

A and B have the same number of columns

Parameters:

• A (2d array)

• B (2d array)

Returns:

C – Khatri-Rao product of A and B

Return type:

2d array

ch_util.ni_utils.mat2utvec(A)

Vectorizes its upper triangle of the (hermitian) matrix A.

Parameters:

A (2d array) – Hermitian matrix

Return type:

1d array with vectorized form of upper triangle of A

Example

if A is a 3x3 matrix then the output vector is outvector = [A00, A01, A02, A11, A12, A22]

See also

utvec2mat

ch_util.ni_utils.ni_als(R, g0, Gamma, Upsilon, maxsteps, abs_tol, rel_tol, weighted_als=True)

Implementation of the Alternating Least Squares algorithm for noise injection.

Implements the Alternating Least Squares algorithm to recover the system gains, sky covariance matrix and system output noise covariance matrix from the data covariance matrix R. All the variables and definitions are as in http://bao.phas.ubc.ca/doc/library/doc_0103/rev_01/chime_calibration.pdf

Parameters:

• R (2d array) – Data covariance matrix

• g0 (1d array) – First estimate of system gains

• Gamma (2d array) – Matrix that characterizes parametrization of sky covariance matrix

• Upsilon (2d array) – Matrix characterizing parametrization of system noise covariance matrix

• maxsteps (int) – Maximum number of iterations

• abs_tol (float) – Absolute tolerance on error function

• rel_tol (float) – Relative tolerance on error function

• weighted_als (bool) – If True, perform weighted ALS

Returns:

• g (1d array) – System gains

• C (2d array) – Sky covariance matrix

• N (2d array) – System output noise covariance matrix

• err (1d array) – Error function for every step

See also

http

//bao.phas.ubc.ca/doc/library/doc_0103/rev_01/chime_calibration.pdf

class ch_util.ni_utils.ni_data(Reader_read_obj, Nadc_channels, adc_ch_ref=None, fbin_ref=None)

Bases: object

Provides analysis utilities for CHIME noise injection data.

This is just a wrapper for all the utilities created in this module.

Parameters:

• Reader_read_obj (andata.Reader.read() like object) – Contains noise injection data. Must have ‘vis’ and ‘timestamp’ property. Assumed to contain all the Nadc_channels*(Nadc_channels+1)/2 correlation products, in chime’s canonical vector, for an Nadc_channels x Nadc_channels correlation matrix

• Nadc_channels (int) – Number of channels read in Reader_read_obj

• adc_ch_ref (int in the range 0 <= adc_ch_ref <= Nadc_channels-1) – Reference channel (used to find on/off points).

• fbin_ref (int in the range) – 0 <= fbin_ref <= np.size(Reader_read_obj.vis, 0)-1 Reference frequency bin (used to find on/off points).

subtract_sky_noise : Removes sky and system noise contributions from noise

injection visibility data.

get_ni_gains : Solve for gains from decimated sky-and-noise-subtracted

visibilities

get_als_gains : Compute gains, sky and system noise covariance matrices from

a combination of noise injection gains and point source gains

Processes raw noise injection data so it is ready to compute gains.

get_als_gains()

Compute gains, sky and system noise covariance matrices from a combination of noise injection gains and point source gains

get_ni_gains(normalize_vis=False, masked_channels=None)

Computes gains and evalues from noise injection visibility data.

See also

ni_gains_evalues_tf, Additional, ---------------------

masked_channels

list of integers channels which are not considered in the calculation of the gains.

save()

Save gain solutions

subtract_sky_noise()

Removes sky and system noise contributions from noise injection visibility data.

See also

subtract_sky_noise

ch_util.ni_utils.ni_gains_evalues(C, normalize_vis=False)

Basic algorithm to compute gains and evalues from noise injection data.

C is a correlation matrix from which the gains are calculated. If normalize_vis = True, the visibility matrix is weighted by the diagonal matrix that turns it into a crosscorrelation coefficient matrix before the gain calculation. The eigenvalues are not sorted. The returned gain solution vector is normalized (LA.norm(g) = 1.)

Parameters:

• C (2d array) – Data covariance matrix from which the gains are calculated. It is assumed that both the sky and system noise contributions have already been subtracted using noise injection

• normalize_vis (bool) – If True, the visibility matrix is weighted by the diagonal matrix that turns it into a crosscorrelation coefficient matrix before the gain calculation.

Returns:

• g (1d array) – Noise injection gains

• ev (1d array) – Noise injection eigenvalues

See also

ni_gains_evalues_tf, subtract_sky_noise

ch_util.ni_utils.ni_gains_evalues_tf(vis_gated, Nchannels, normalize_vis=False, vis_on=None, vis_off=None, niter=0)

Computes gains and evalues from noise injection visibility data.

Gains and eigenvalues are calculated for all frames and frequencies in vis_gated. The returned gain solution vector is normalized (LA.norm(gains[f, :, t]) = 1.)

Parameters:

• vis_gated (3d array) – Visibility array in chime’s canonical format. vis_gated has dimensions [frequency, corr. number, time]. It is assumed that both the sky and system noise contributions have already been subtracted using noise injection.

• Nchannels (int) – Order of the visibility matrix (number of channels)

• normalize_vis (bool) – If True, then the visibility matrix is weighted by the diagonal matrix that turns it into a crosscorrelation coefficient matrix before the gain calculation.

• vis_on (3d array) – If input and normalize_vis is True, then vis_gated is weighted by the diagonal elements of the matrix vis_on. vis_on must be the same shape as vis_gated.

• vis_off (3d array) – If input and normalize_vis is True, then vis_gated is weighted by the diagonal elements of the matrix: vis_on = vis_gated + vis_off. vis_off must be the same shape as vis_gated. Keyword vis_on supersedes keyword vis_off.

• niter (0) – Number of iterations to perform. At each iteration, the diagonal elements of vis_gated are replaced with their rank 1 approximation. If niter == 0 (default), then no iterations are peformed and the autocorrelations are used instead.

Returns:

• gains (3d array) – Noise injection gains

• evals (3d array) – Noise injection eigenvalues

• Dependencies

• ————

• tools.normalise_correlations, tools.eigh_no_diagonal

See also

ni_gains_evalues, subtract_sky_noise

ch_util.ni_utils.process_gated_data(data, only_off=False)

Processes fast gating data and turns it into gated form.

Parameters:

• data (andata.CorrData) – Correlator data with noise source switched synchronously with the integration.

• only_off (boolean) – Only return the off dataset. Do not return gated datasets.

Returns:

• newdata (andata.CorrData) – Correlator data folded on the noise source.

• Comments

• ——–

• For now the correlator only supports fast gating with one gate

• (gated_vis1) and 50% duty cycle. The vis dataset contains on+off

• and the gated_vis1 contains on-off. This function returns a new

• andata object with vis containing the off data only and gated_vis1

• as in the original andata object. The attribute

• ’gpu.gpu_intergration_period’ is divided by 2 since during an

• integration half of the frames have on data.

ch_util.ni_utils.process_synced_data(data, ni_params=None, only_off=False)

Turn a synced noise source observation into gated form.

This will decimate the visibility to only the noise source off bins, and will add 1 or more gated on-off dataset according to the specification in doclib:5.

Parameters:

• data (andata.CorrData) – Correlator data with noise source switched synchronously with the integration.

• ni_params (dict) – Dictionary with the noise injection parameters. Optional for data after ctime=1435349183. ni_params has the following keys - ni_period: Noise injection period in GPU integrations. It is assummed to be the same for all the enabled noise sources - ni_on_bins: A list of lists, one per enabled noise source, with the corresponding ON gates (within a period). For each noise source, the list contains the indices of the time frames for which the source is ON. Example: For 3 GPU integration period (3 gates: 0, 1, 2), two enabled noise sources, one ON during gate 0, the other ON during gate 1, and both OFF during gate 2, then ` ni_params = {'ni_period':3, 'ni_on_bins':[[0], [1]]} `

• only_off (boolean) – Only return the off dataset. Do not return gated datasets.

Returns:

• newdata (andata.CorrData) – Correlator data folded on the noise source.

• Comments

• ——–

• - The function assumes that the fpga frame counter, which is used to

• determine the noise injection gating parameters, is unwrapped.

• - For noise injection data before ctime=1435349183 (i.e. for noise

• injection data before 20150626T200540Z_pathfinder_corr) the noise

• injection information is not in the headers so this function cannot be

• used to determine the noise injection parameters. A different method is

• required. Although it is recommended to check the data directly in this

• case, the previous version of this function assumed that

• ni_params = {‘ni_period’ (2, ‘ni_on_bins’:[[0],]})

• for noise injection data before ctime=1435349183. Although this is not

• always true, it is true for big old datasets like pass1g.

• Use the value of ni_params recommended above to reproduce the

• results of the old function with the main old datasets.

• - Data (visibility, gain and weight datasets) are averaged for all the

• off gates within the noise source period, and also for all the on

• gates of each noise source.

• - For the time index map, only one timestamp per noise period is kept

• (no averaging)

ch_util.ni_utils.sort_evalues_mag(evalues)

Sorts eigenvalue array by magnitude for all frequencies and time frames

Parameters:

evalues (3d array) – Array of evalues. Its shape is [Nfreqs, Nevalues, Ntimeframes]

Returns:

ev – Array of same shape as evalues

Return type:

3d array

ch_util.ni_utils.subtract_sky_noise(vis, Nchannels, timestamp, adc_ch_ref, fbin_ref)

Removes sky and system noise contributions from noise injection visibility data.

By looking at the autocorrelation of the reference channel adc_ch_ref for frequency bin fbin_ref, finds timestamps indices for which the signal is on and off. For every noise signal period, the subcycles with the noise signal on and off are averaged separatedly and then subtracted.

It is assumed that there are at least 5 noise signal cycles in the data. The first and last noise on subcycles are discarded since those cycles may be truncated.

Parameters:

• vis (3d array) – Noise injection visibility array in chime’s canonical format. vis has dimensions [frequency, corr. number, time].

• Nchannels (int) – Order of the visibility matrix (number of channels)

• timestamp (1d array) – Timestamps for the visibility array vis

• adc_ch_ref (int in the range 0 <= adc_ch_ref <= N_channels-1) – Reference channel (typically, but not necessaritly the channel corresponding to the directly injected noise signal) used to find timestamps indices for which the signal is on and off. on and off.

• fbin_ref (int in the range 0 <= fbin_ref <= np.size(vis, 0)-1) – frequency bin used to find timestamps indices for which the signal is on and off

Returns:

• A dictionary with keys

• time_index_on (1d array) – timestamp indices for noise signal on.

• time_index_off (1d array) – timestamp indices for noise signal off.

• timestamp_on_dec (1d array) – timestamps for noise signal on after averaging.

• timestamp_off_dec (1d array) – timestamps for noise signal off after averaging.

• timestamp_dec (1d array) – timestamps for visibility data after averaging and subtracting on and off subcycles. These timestaps represent the time for every noise cycle and thus, these are the timestaps for the gain solutions.

• vis_on_dec (3d array) – visibilities for noise signal on after averaging.

• vis_off_dec (3d array) – visibilities for noise signal off after averaging.

• vis_dec_sub (3d array) – visibilities data after averaging and subtracting on and off subcycles.

• cor_prod_ref (int) – correlation index corresponding to the autocorrelation of the reference channel

ch_util.ni_utils.utvec2mat(n, utvec)

Recovers a hermitian matrix a from its upper triangle vectorized version.

Parameters:

• n (int) – order of the output hermitian matrix

• utvec (1d array) – vectorized form of upper triangle of output matrix

Returns:

A – hermitian matrix

Return type:

2d array