ch_util.layout

Interface to the CHIME components and graphs

This module interfaces to the layout tables in the CHIME database.

The peewee module is used for the ORM to the MySQL database. Because the layouts are event-driven, you should never attempt to enter events by raw inserts to the event or timestamp tables, as you could create inconsistencies. Rather, use the methods which are described in this document to do such alterations robustly.

For most uses, you probably want to import the following:

>>> from datetime import datetime
>>> import logging
>>> logging.basicConfig(level = logging.INFO)
>>> import peewee
>>> import layout
>>> layout.connect_database()

Note

The database must now be explicitly connected. This should not be done within an import statement.

Note

The logging module can be set to the level of your preference, or not imported altogether if you don’t want log messages from the layout module. Note that the peewee module sends a lot of messages to the DEBUG stream.

If you will be altering the layouts, you will need to register as a user:

>>> layout.set_user("Ahincks")

Use your CHIME wiki username here. Make sure it starts with a capital letter. Note that different users have different permissions, stored in the user_permission table. If you are simply reading from the layout, there is no need to register as a user.

Choose Your Own Adventure

If you want to …	… then see
retrieve and examine layout graphs	`graph`
add components	`component.add`, `add_component`
remove components	`component.remove`, `remove_component`
make connexions	`make_connexion`
sever connexions	`sever_connexion`
set component properties	`component.set_property` `set_property`
get component properties	`component.get_property`
perform bulk updates	`enter_ltf()`
add component history notes	`component.add_history`
add link to component documentation	`component.add_doc`
create a global flag	`global_flag.start`

Functions

add_component
compare_connexion
connect_database
enter_ltf()
make_connexion
remove_component
set_user
sever_connexion
global_flags_between()
get_global_flag_times()

Classes

subgraph_spec
graph

Database Models

component
component_history
component_type
component_type_rev
component_doc
connexion
external_repo
event
event_type
graph_obj
global_flag
predef_subgraph_spec
predef_subgraph_spec_param
property
property_component
property_type
timestamp
user_permission
user_permission_type

Exceptions

NoSubgraph
BadSubgraph
DoesNotExist
UnknownUser
NoPermission
LayoutIntegrity
PropertyType
PropertyUnchanged
ClosestDraw
NotFound

Constants

EVENT_AT
EVENT_BEFORE
EVENT_AFTER
EVENT_ALL
ORDER_ASC
ORDER_DESC

Functions

`enter_ltf`(ltf[, time, notes, force])	Enter an LTF into the database.
`get_global_flag_times`(flag)	Convenience function to get global flag times by id or name.
`global_flags_between`(start_time, end_time[, ...])	Find global flags that overlap a time interval.

Classes

`graph`(args, *kwargs)	A graph of connexions.
`subgraph_spec`(start, terminate, oneway, hide)	Specifications for extracting a subgraph from a full graph.

ch_util.layout.enter_ltf(ltf, time=datetime.datetime(2024, 9, 27, 19, 23, 0, 852913), notes=None, force=False)[source]

Enter an LTF into the database.

This is a special mark-up language for quickly entering events. See the “help” box on the LTF page of the web interface for instructions.

Parameters:

ltf (string) – Pass either the path to a file containing the LTF, or a string containing the LTF.
time (datetime.datetime) – The time at which to apply the LTF.
notes (string) – Notes for the timestamp.
force (bool) – If True, then do nothing when events that would damage database integrity are encountered; skip over them. If False, then a bad propsed event will raise the appropriate exception.

ch_util.layout.get_global_flag_times(flag)[source]

Convenience function to get global flag times by id or name.

Parameters:

flag (integer or string) – If an integer, this is a global flag id, e.g. 64. If a string this is the global flag’s name e.g. ‘run_pass0_e’.

Returns:

start (datetime.datetime) – Global flag start time (UTC).
end (datetime.datetime or None) – Global flag end time (UTC) or None if the flag hasn’t ended.

ch_util.layout.global_flags_between(start_time, end_time, severity=None)[source]

Find global flags that overlap a time interval.

Parameters:

start_time
end_time
severity (str) – One of ‘comment’, ‘warning’, ‘severe’, or None.

Returns:

flags – List of global_flag objects matching criteria.

Return type:

list

class ch_util.layout.graph(*args: Any, **kwargs: Any)[source]

Bases: Graph

A graph of connexions.

This class inherits the networkx.Graph class and adds CHIME-specific functionality.

Use the from_db() class method to construct a graph from the database.

Parameters:: time (datetime.datetime) – The time at which the graph is valid. Default is now().

Examples

To load a graph from the database, use the from_db() class method:

>>> from ch_util import graph
>>> from datetime import datetime
>>> g = layout.graph.from_db(datetime(2014, 10, 5, 12, 0))

You can now use any of the networkx.Graph methods:

>>> print(g.order(), g.size())
2483 2660

There are some convenience methods for our implementation. For example, you can easily find components by component type:

>>> print(g.component(type = "reflector"))
[<layout.component object at 0x7fd1b2cda710>,
<layout.component object at 0x7fd1b2cda810>,
<layout.component object at 0x7fd1b2cfb7d0>]

Note that the graph nodes are component objects. You can also use the component() method to search for components by serial number:

>>> ant = g.component(comp = "ANT0044B")

Node properties are stored as per usual for networkx.Graph objects:

>>> print(g.nodes[ant])
{'_rev_id': 11L, '_type_id': 2L, '_type_name': 'antenna', '_id': 32L,
'pol1_orient': <layout.property object at 0x7f31ed323fd0>,
'pol2_orient': <layout.property object at 0x7f31ed2c8790>, '_rev_name': 'B'}

Note, however, that there are some internally-used properties (starting with an underscore). The node_property() returns a dictionary of properties without these private memebers:

>>> for p in g.node_property(ant).values():
...   print(p.type.name, "=", p.value, p.type.units if p.type.units else "")
pol1_orient = S
pol2_orient = E

To search the graph for the closest component of a given type to a single component, using closest_of_type():

>>> slt_type = layout.component_type.get(name = "cassette slot")
>>> print(g.closest_of_type(ant, slt_type).sn)
CSS004C0

Use of closest_of_type() can be subtle for components separated by long paths. See its documentation for more examples.

Subgraphs can be created using a subgraph specification, encoded in a subgraph_spec object. See the documentation for that class for details, but briefly, this allows one to create a smaller, more manageable graph containing only components and connexions you are interested in. Given a subgraph, the ltf() method can be useful.

closest_of_type(comp, type, type_exclude=None, ignore_draws=True)[source]

Searches for the closest connected component of a given type.

Sometimes the closest component is through a long, convoluted path that you do not wish to explore. You can cut out these cases by including a list of component types that will block the search along a path.

The component may be passed by object or by serial number; similarly for component types.

Parameters:

comp (component or string or list of such) – The component to search from.
type (component_type or string) – The component type to find.
type_exclude (list of component_type or strings) – Any components of this type will prematurely cut off a line of investigation.
ignore_draws (boolean) – It is possible that there be more than one component of a given type the same distance from the starting component. If this parameter is set to True, then just return the first one that is found. If set to False, then raise an exception.

Returns:

comp – The closest component of the given type to start. If no component of type is found None is returned.

Return type:

component or list of such

Raises:

ClosestDraw – Raised if there is no unique closest component and ignore_draws is set to False.

Examples

Find the cassette slot an antenna is plugged into:

>>> import layout
>>> from datetime import datetime
>>> g = layout.graph.from_db(datetime(2014, 11, 5, 12, 0))
>>> print(g.closest_of_type("ANT0044B", "cassette slot").sn)
CSS004C0

The example above is simple as the two components are adjacent:

>>> print([c.sn for c in g.shortest_path_to_type("ANT0044B", "cassette slot")])
['ANT0044B', 'CSS004C0']

In general, though, you need to take care when using this method and make judicious use of the type_exclude parameter. For example, consider the following example:

>>> print(g.closest_of_type("K7BP16-00040112", "RFT thru").sn)
RFTB15B

It seems OK on the surface, but the path it has used is probably not what you want:

>>> print([c.sn for c in g.shortest_path_to_type("K7BP16-00040112","RFT thru")])
['K7BP16-00040112', 'K7BP16-000401', 'K7BP16-00040101', 'FLA0280B', 'RFTB15B']

We need to block the searcher from going into the correlator card slot and then back out another input, which we can do like so:

>>> print(g.closest_of_type("K7BP16-00040112", "RFT thru",
... type_exclude = "correlator card slot").sn)
RFTQ15B

The reason the first search went through the correlator card slot is because there are delay cables and splitters involved.

>>> print([c.sn for c in g.shortest_path_to_type("K7BP16-00040112",
... "RFT thru", type_exclude = "correlator card slot")])
['K7BP16-00040112', 'CXS0279', 'CXA0018A', 'CXA0139B', 'SPL001AP2',
'SPL001A', 'SPL001AP3', 'CXS0281', 'RFTQ15B']

The shortest path really was through the correlator card slot, until we explicitly rejected such paths.

component(comp=None, type=None, sort_sn=False)[source]

Return a component or list of components from the graph.

The components exist as graph nodes. This method provides searchable access to them.

Parameters:

comp (string or component) – If not None, then return the component with this serial number, or None if it does not exist in the graph. If this parameter is set, then type is ignored. You can also pass a component object; the instance of that component with the same serial number will be returned if it exists in this graph.
type (string or component_type) – If not None, then only return components of this type. You may pass either the name of the component type or an object.

Returns:

If the sn parameter is passed, a single component object is returned. If the type parameter is passed, a list of component objects is returned.

Return type:

component or list of such

Raises:

NotFound – Raised if no component is found.

Examples

>>> from ch_util import graph
>>> from datetime import datetime
>>> g = layout.graph.from_db(datetime(2014, 10, 5, 12, 0))
>>> print(g.component("CXA0005A").type_rev.name)
B
>>> for r in g.component(type = "reflector"):
...   print r.sn
E_cylinder
W_cylinder
26m_dish

classmethod from_db(time=datetime.datetime(2024, 9, 27, 19, 23, 0, 852894), sg_spec=None, sg_start_sn=None)[source]

Create a new graph by reading the database.

This method is designed to be efficient. It has customised SQL calls so that only a couple of queries are required. Doing this with the standard peewee functionality requires many more calls.

This method will establish a connection to the database if it doesn’t already exist.

Parameters:

time (datetime.datetime) – The time at which the graph is valid. Default is now().
sg_spec (subgraph_spec) – The subgraph specificationto use; can be set to None.
sg_start_sn (string) – If a serial number is specified, then only the subgraph starting with that component will be returned. This parameter is ignored if sg_spec is None.

Returns:

If sg_spec is not None, and sg_start_sn is not specified, then a list of graph objects is returned instead.

Return type:

graph

:raises If no graph is found, NotFound is raised.:

classmethod from_graph(g, sg_spec=None, sg_start_sn=None)[source]

Find subgraphs within this graph.

Parameters:

g (graph) – The graph from which to get the new graph.
sg_spect (subgraph_spec) – The subgraph specification to use; can be set to None.

Returns:

A list of graph objects, one for each subgraph found. If, however,
*g* is set to None, a reference to the input graph is returned.

ltf()[source]

Get an LTF representation of the graph. The graph must be a subgraph, i.e., generated with a predef_subgraph_spec.

Returns:

ltf – The LTF representation of the graph.

Return type:

string

Raises:

NoSubgraph –
Raised if no subgraph specification is associate with this layout. –

Examples

Get the LTF for a subgraph of antenna to HK.

>>> import layout
>>> from datetime import datetime
>>> start = layout.component_type.get(name = "antenna").id
>>> terminate = [layout.component_type.get(name = "reflector").id,
                 layout.component_type.get(name = "cassette slot").id,
                 layout.component_type.get(name = "correlator input").id,
                 layout.component_type.get(name = "HK preamp").id,
                 layout.component_type.get(name = "HK hydra").id]
>>> hide = [layout.component_type.get(name = "reflector").id,
            layout.component_type.get(name = "cassette slot").id,
            layout.component_type.get(name = "HK preamp").id,
            layout.component_type.get(name = "HK hydra").id]
>>> sg_spec = layout.subgraph_spec(start, terminate, [], hide)
>>> sg = layout.graph.from_db(datetime(2014, 11, 20, 12, 0), sg_spec,"ANT0108B")
>>> print(sg.ltf())
# Antenna to correlator input.
ANT0108B pol1_orient=S pol2_orient=E
PL0108B1
LNA0249B
CXA0239C
CANBJ6B
CXS0042
RFTG00B attenuation=10
FLA0196B
CXS0058
K7BP16-00041606

# Antenna to correlator input.
ANT0108B pol1_orient=S pol2_orient=E
PL0108B2
LNA0296B
CXA0067B
CANBG6B
CXS0090
RFTG01B attenuation=10
FLA0269B
CXS0266
K7BP16-00041506

neighbour_of_type(n, type)[source]

Get a list of neighbours of a given type.

This is like the networkx.Graph.neighbors() method, but selects only the neighbours of the specified type.

Parameters:

comp (component) – A node in the graph.
type (component_type or string) – The component type to find.

Returns:

nlist

Return type:

A list of nodes of type type adjacent to n.

Raises:

networkx.NetworkXError – Raised if n is not in the graph.

node_property(n)[source]

Return the properties of a node excluding internally used properties.

If you iterate over a nodes properties, you will also get the internally-used properties (starting with an underscore). This method gets the dictionary of properties without these “private” properties.

Parameters:: node (node object) – The node for which to get the properties.
Return type:: A dictionary of properties.

Examples

>>> from ch_util import graph
>>> from datetime import datetime
>>> g = layout.graph.from_db(datetime(2014, 10, 5, 12, 0))
>>> rft = g.component(comp = "RFTK07B")
>>> for p in g.node_property(rft).values():
...   print(p.type.name, "=", p.value, p.type.units if p.type.units else "")
attenuation = 10 dB
therm_avail = ch1

property sg_spec

The subgraph_spec (subgraph specification) used to get this graph.

Return type:: The subgraph_spec used to get this graph, if any.

property sg_spec_start

The subgraph starting component.

Return type:: The component that was used to begin the subgraph, if any.

shortest_path_to_type(comp, type, type_exclude=None, ignore_draws=True)[source]

Searches for the shortest path to a component of a given type.

Sometimes the closest component is through a long, convoluted path that you do not wish to explore. You can cut out these cases by including a list of component types that will block the search along a path.

The component may be passed by object or by serial number; similarly for component types.

Parameters:

comp (component or string or list of one of these) – The component(s) to search from.
type (component_type or string) – The component type to find.
type_exclude (list of component_type or strings) – Any components of this type will prematurely cut off a line of investigation.
ignore_draws (boolean) – It is possible that there be more than one component of a given type the same distance from the starting component. If this parameter is set to True, then just return the first one that is found. If set to False, then raise an exception.

Returns:

comp – The closest component of the given type to start. If no path to a component of the specified type exists, return None.

Return type:

component or list of such

Raises:

ClosestDraw – Raised if there is no unique closest component and ignore_draws is set to False.

Examples

See the examples for closest_of_type().

property time

The time of the graph.

Returns:: time – The time at which this graph existed.
Return type:: datetime.datetime

class ch_util.layout.subgraph_spec(start, terminate, oneway, hide)[source]

Bases: object

Specifications for extracting a subgraph from a full graph.

The subgraph specification can be created from scratch by passing the appropriate parameters. They can also be pulled from the database using the class method FROM_PREDef().

The parameters can be passed as ID’s, names of compoenet types or component_type instances.

Parameters:

start (integer, component_type or string) – The component type for the start of the subgraph.
terminate (list of integers, of component_type or of strings) – Component type id’s for terminating the subgraph.
oneway (list of list of integer pairs, of component_type or of strings) – Pairs of component types for defining connexions that should only be traced one way when moving from the starting to terminating components.
hide (list of integers, of component_type or of strings) – Component types for components that should be hidden and skipped over in the subgraph.

Examples

To look at subgraphs of components between the outer bulkhead and the correlator inputs, one could create the following specification:

>>> import layout
>>> from datetime import datetime
>>> sg_spec = layout.subgraph_spec(start = "c-can thru",
                                   terminate = ["correlator input", "60m coax"],
                                   oneway = [],
                                   hide = ["60m coax", "SMA coax"])

What did we do? We specified that the subgraph starts at the C-Can bulkhead. It terminates at the correlator input; in the other direction, it must also terminate at a 60 m coaxial cable plugged into the bulkhead. We hide the 60 m coaxial cable so that it doesn’t show up in the subgraph. We also hide the SMA cables so that they will be skipped over.

We can load all such subgraphs from the database now and see how many nodes they contain:

>>> sg = layout.graph.from_db(datetime(2014, 10, 5, 12, 0), sg_spec)
print [s.order() for s in sg]
[903, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 903,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 903, 3, 3, 3, 903, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 903, 3, 1, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
903, 3, 903, 3, 3, 3, 3, 3, 3, 3, 903, 903, 5, 5]

Most of them are as short as we would expect, but there are some complications. Let’s look at that first one by printing out its LTF:

>>> print(sg[0].ltf)
# C-can thru to RFT thru.
CANAD0B
RFTA15B attenuation=10 therm_avail=ch7

# RFT thru to HK preamp.
RFTA15B attenuation=10 therm_avail=ch7
CHB036C7
HPA0002A

# HK preamp to HK readout.
HPA0002A
ATMEGA49704949575721220150
HKR00

# HK readout to HK ATMega.
HKR00
ATMEGA50874956504915100100
etc...
etc...
# RFT thru to FLA.
RFTA15B attenuation=10 therm_avail=ch7
FLA0159B

Some FLA’s are connected to HK hydra cables and we need to terminate on these as well. It turns out that some outer bulkheads are connected to 200 m coaxial cables, and some FLA’s are connected to 50 m delay cables, adding to the list of terminations. Let’s exclude these as well:

>>> sg_spec.terminate += ["200m coax", "HK hydra", "50m coax"]
>>> sg_spec.hide += ["200m coax", "HK hydra", "50m coax"]
>>> sg = layout.graph.from_db(datetime(2014, 10, 5, 12, 0), sg_spec)
>>> print([s.order() for s in sg])
[3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 1, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 10, 10, 5, 5]

The remaining subgraphs with more than three components actually turn out to be errors in the layout! Let’s investigate the last one by removing any hidden components and printing its LTF.

>>> sn = sg[-1].component(type = "C-can thru")[0].sn
CANBL1B
>>> sg_spec.hide = []
>>> bad_sg = layout.graph.from_db(datetime(2014, 10, 5, 12, 0), sg_spec, sn)
>>> print(bad_sg.ltf())
# C-can thru to c-can thru.
CANBL1B
CXS0017
RFTQ00B
CXS0016
FLA0073B
RFTQ01B attenuation=9
CXS0015
CANBL0B

It appears that CXS0016 mistakenly connects RFTQ00B to FLA0073B. This is an error that should be investigated and fixed. But by way of illustration, let’s cut this subgraph short by specifying a one-way connection, and not allowing the subgrapher to trace backwards from the inner bulkhead to an SMA cable:

>>> sg_spec.oneway = [["SMA coax", "RFT thru"]]
>>> bad_sg = layout.graph.from_db(datetime(2014, 10, 5, 12, 0), sg_spec, sn)
>>> print(bad_sg.ltf())
# C-can thru to RFT thru.
CANBL1B
CXS0017
RFTQ00B

classmethod from_predef(predef)[source]

Create a subgraph specification from a predefined version in the DB.

Parameters:: predef (predef_subgraph_spec) – A predefined subgraph specification in the DB.

property hide: The component type ID(s) that are skipped over in the subgraph.

property oneway: Pairs of component type ID(s) for one-way tracing of the subgraph.

property start: The component type ID starting the subgraph.

property terminate: The component type ID(s) terminating the subgraph.