hypertrack

API

hypertrack.auxtrain

backward(optimizer: dict, scheduler: dict, model: dict, total_loss: dict, scaler: dict, max_grad_norm: float = 1.0, dtype=torch.float32)

Backward (backpropagation and scheduler update) evolution

Parameters:

• optimizer – torch optimizers

• scheduler – torch schedulers

• model – torch models

• total_loss – losses per model

• scaler – gradient scaler (for float16)

• max_grad_norm – gradient norm clipping

• dtype – floating point type

execute(trainmode: bool, event: int, rfactor: float, model: dict, do_clustering: bool, do_pdf: bool, args: dict, global_param: dict, geometric_obj: dict)

Execute HyperTrack training (or validation)

Parameters:

• trainmode – train or evaluation mode

• event – event index

• rfactor – pile-up reduction factor

• model – model(s)

• do_clustering – clustering on/off

• do_pdf – pdf network on/off

• args – setup parameters

• global_param – global model parameters

• voronoi_obj – geometric estimator object

Returns:

total combined hybrid loss loss: individual loss terms metric: event metrics

Return type:

total_loss

load_models(epoch: int, model: dict, args: dict, transfer_tag: str | None = None)

Load models from the disk

Parameters:

• epoch – epoch number (or iteration)

• model – torch model(s)

• args – setup parameters

Returns:

the checkpoint metrics found_epoch found epoch number

Input ‘model’ is updated via reference

Return type:

prev_metrics

load_optimizers(epoch: int, optimizer: dict, scheduler: dict, args: dict, transfer_tag: str | None = None)

Load optimizers from the disk

Parameters:

• epoch – epoch number (or iteration)

• optimizer – torch optimizer(s)

• args – setup parameters

Returns:

Input ‘optimizer’ is updated via reference

plot_gradient_flow(epoch: int, model: dict, args: dict)

Plot gradient flow

Parameters:

• epoch – epoch (or iteration) number

• model – models in a dictionary

• args – main parameters

Returns:

saves figures to the disk

plot_metrics(epoch: int, metrics: dict, model: dict, args: dict, train_param: dict, alpha: float = 0.5, filter_N: int = 100)

Plot training metrics

Parameters:

• epoch – epoch (or iteration) vector

• metrics – metrics dictionary

• model – neural model (for gradient flow figures)

• args – arguments object

• train_param – training parameters

• alpha – transparency

• filter_N – running average window length

Returns:

saves figures to the disk

save_models(epoch: int, model: dict, optimizer: dict, scheduler: dict, metrics: dict, args: dict)

Save model objects to the disk

Parameters:

• epoch – epoch (or iteration)

• model – torch models

• optimizer – torch optimizers

• scheduler – torch schedulers

• metrics – metrics dictionary

• args – setup parameters

Returns:

saves objects to the disk

set_optimizer(model: dict, learning_rate: float, args: dict)

Set Optimizer parameters

Parameters:

• model – different models under training by different optimizers

• learning_rate – learning rate

• args – setup parameters

Returns:

torch optimizer

set_scheduler(optimizer: dict, args: dict)

Parameters:

• optimizer – optimizers for different models

• args – setup parameters

Returns:

torch scheduler

hypertrack.dmlp

MLP(layers: List[int], act: str = 'relu', bn: bool = False, dropout: float = 0.0, last_act: bool = False)

Return a Multi Layer Perceptron with an arbitrary number of layers.

Parameters:

• layers – input structure, such as [128, 64, 64] for a 3-layer network.

• act – activation function

• bn – batch normalization

• dropout – dropout regularization

• last_act – apply activation function after the last layer

Returns:

nn.sequential object

get_act(act: str)

Returns torch activation function

Parameters:

act – activation function ‘relu’, ‘tanh’, ‘silu’, ‘elu

hypertrack.faissknn

class faisskNN(device='cpu')

fit(X, nlist=128, nprobe=8, IVF_on=True, metric='L2')

Parameters:

• X – Data array (N x dim)

• nlist – Number of cells / clusters to partition data into

• nprobe – Number of nearest cells to include in the search (default 1)

• IVF_on – Approximate (fast) IVF search

• metric – Comparison type ‘L2’ (Euclidian distance) or ‘IP’ (inner product)

search(xq, k)

Parameters:

• xq – query vectors array (N x dim)

• k – number of nearest neigbours to search

Returns:

distances, indices

Return type:

D,I

hypertrack.flows

class ActNorm(num_inputs)

An implementation of a activation normalization layer from Glow: Generative Flow with Invertible 1x1 Convolutions (https://arxiv.org/abs/1807.03039)

forward(inputs, cond_inputs=None, mode='direct')

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class BatchNormFlow(num_inputs, momentum=0.0, eps=1e-05)

An implementation of a batch normalization layer from Density estimation using Real NVP (https://arxiv.org/abs/1605.08803)

forward(inputs, cond_inputs=None, mode='direct')

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class CouplingLayer(num_inputs, num_hidden, mask, num_cond_inputs=None, s_act='tanh', t_act='relu')

An implementation of a coupling layer from RealNVP (https://arxiv.org/abs/1605.08803)

forward(inputs, cond_inputs=None, mode='direct')

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class FlowSequential(*args: Module)

class FlowSequential(arg: OrderedDict[str, Module])

A sequential container for flows. In addition to a forward pass it implements a backward pass and computes log jacobians.

forward(inputs, cond_inputs=None, mode='direct', logdets=None)

Performs a forward or backward pass for flow modules. :param inputs: a tuple of inputs and logdets :param mode: to run direct computation or inverse

log_probs(inputs, cond_inputs=None)

sample(num_samples=None, noise=None, cond_inputs=None)

class InvertibleMM(num_inputs)

An implementation of a invertible matrix multiplication layer from Glow: Generative Flow with Invertible 1x1 Convolutions (https://arxiv.org/abs/1807.03039)

forward(inputs, cond_inputs=None, mode='direct')

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class LUInvertibleMM(num_inputs)

An implementation of a invertible matrix multiplication layer from Glow: Generative Flow with Invertible 1x1 Convolutions (https://arxiv.org/abs/1807.03039)

forward(inputs, cond_inputs=None, mode='direct')

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class Logit

forward(inputs, cond_inputs=None, mode='direct')

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class MADE(num_inputs, num_hidden, num_cond_inputs=None, act='relu', pre_exp_tanh=False)

An implementation of MADE (https://arxiv.org/abs/1502.03509)

forward(inputs, cond_inputs=None, mode='direct')

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class MADESplit(num_inputs, num_hidden, num_cond_inputs=None, s_act='tanh', t_act='relu', pre_exp_tanh=False)

An implementation of MADE (https://arxiv.org/abs/1502.03509)

forward(inputs, cond_inputs=None, mode='direct')

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class MaskedLinear(in_features, out_features, mask, cond_in_features=None, bias=True)

forward(inputs, cond_inputs=None)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class Reverse(num_inputs)

An implementation of a reversing layer from Density estimation using Real NVP (https://arxiv.org/abs/1605.08803)

forward(inputs, cond_inputs=None, mode='direct')

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class Shuffle(num_inputs)

An implementation of a shuffling layer from Density estimation using Real NVP (https://arxiv.org/abs/1605.08803)

forward(inputs, cond_inputs=None, mode='direct')

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class Sigmoid

forward(inputs, cond_inputs=None, mode='direct')

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

get_mask(in_features, out_features, in_flow_features, mask_type=None)

Parameters:

• in_features – input degree (if mask_type == ‘input’)

• out_features – output degree (if mask_type == ‘output’)

• in_flow_features – number of in-flow features

• mask_type – input | None | output

(See Figure 1 of https://arxiv.org/pdf/1502.03509.pdf)

hypertrack.hyper

clusterize(shared_obj: dict, active_node_mask: List[Tensor], adjmat_local: List[Tensor], global_ind: List[Tensor], train_param: dict, cluster_param: dict, do_pdf: bool, trainmode: bool, device: device)

Multi-Pivotal seeding and Set Transformer based clustering

Parameters:

• shared_obj – main data dictionary

• active_node_mask – node mask

• adjmat_local – local adjacency matrix

• global_ind – global index list

• train_param – training parameter dictionary

• cluster_param – clustering parameter dictionary

• do_pdf – evaluate normalizing flow

• trainmode – training or inference mode

• device – torch device

Returns:

clustering results and aux information in a dictionary

hypertrack(model: dict, X: Tensor, edge_index_hat: Tensor, x_ind: ndarray | None = None, hit_weights: Tensor | None = None, edge_label: Tensor | None = None, true_clusters: Tensor | None = None, objects: Tensor | None = None, do_pdf: bool = False, do_clustering: bool = True, trainmode: bool = False, cluster_param: dict | None = None, train_param: dict | None = None, device: device = device(type='cuda'), DEBUG: bool = False)

HyperTrack model predictor function with embedded training losses

Parameters:

• model – neural model dictionary

• X – input graph node data

• edge_index_hat – input graph adjacency

• x_ind – hit to object associations array (training only)

• hit_weights – ground truth node weights (training only)

• edge_label – edge truth array (training only)

• true_clusters – ground truth cluster associations per node (hit) (training only)

• objects – ground truth object vector data (training only)

• do_pdf – run pdf module

• do_clustering – run clustering module

• trainmode – training mode turned on == True

• cluster_param – clustering parameters dictionary

• train_param – training parameters dictionary

• device – torch computing device

Returns:

list of clusters loss: torch loss data aux: auxialary information

Return type:

cluster_labels_hat

transformer_helper(model, gg: List[List[int]], Z: Tensor, edge_prob: Tensor, sparse_ind: Tensor, edge_index_hat: Tensor, objects: Tensor, X: Tensor, hit_weights: Tensor, true_clusters: Tensor, x_ind: ndarray, cluster_param: dict, train_param: dict, trainmode: bool, do_pdf: bool, device: device)

Transformer clustering helper function

hypertrack.hyperaux

cluster_loss(model: dict, Z: Tensor, global_ind: Tensor, chosen_ind: Tensor, pivots: List[int], micrograph_ind: Tensor, cluster_logits: Tensor, cluster_prob: Tensor, x_ind: ndarray, true_clusters: Tensor, hit_weights: Tensor, objects: Tensor, train_param: dict, do_pdf: bool, device: device)

Transformer output loss computation

Args:

model: model dictionary Z: GNN latents global_ind: global index list chosen_ind: chosen indices after threshold cut pivots: pivot indices micrograph_ind: micrograph indices cluster_logits: transformer output logits cluster_prob: transformer output probabilities x_ind: cluster ground truth assignments true_clusters: cluster labels hit_weights: hit (nodes) weights objects: cluster object data train_param: training parameter dictionary do_pdf: evaluate normalizing flow (experimental) device: torch device

Returns:

losses (tuple)

edge_loss(model: dict, Z: Tensor, x_ind: ndarray, edge_label: Tensor, edge_index_hat: Tensor, objects: Tensor, edge_logits: Tensor, edge_prob: Tensor, hit_weights: Tensor, do_pdf: bool, train_param: dict, device: device)

GNN output loss computation

Parameters:

• model – model dictionary

• Z – GNN latents

• x_ind – cluster ground truth assignments

• edge_label – edge ground truth labels

• edge_index_hat – adjancency list

• objects – clustering ground truth objects

• edge_logits – GNN edge logits

• edge_prob – GNN edge probabilities

• hit_weights – hit weights

• do_pdf – evaluate normalizing flow [experimental]

• train_param – training parameters dictionary

• device – torch device

Returns:

torch losses (tuple)

loss_normalization(loss: dict, output: dict, K: int, train_param: dict)

The final loss normalization

Parameters:

• loss – loss dictionary

• output – transformer output dictionary

• K – number of clusters found

• train_param – training parameters

Returns:

loss dictionary

hypertrack.iotools

construct_event_range(event_start, event_end)

Construct event range list

Parameters:

• event_start – starting index (integer or list of integers)

• event_end – ending index

Returns:

event index list

get_gpu_memory_map()

Get the GPU VRAM use in GB.

Returns:

dictionary with keys as device ids [integers] and values the memory used by the GPU.

grab_torch_file(key: str, name: str, save_tag: str, epoch: int, path: str | None = None)

Grab a torch file from a disk

Parameters:

• key – file key identifier

• name – file name identifier

• save_tag – model tag

• epoch – epoch number (or iteration), set -1 for the latest by timestamp

• path – base disk path

Returns:

filename full filename found found_epoch: epoch number (or iteration)

load_models(model: dict, epoch: int, read_tag: str, device: device, keys: List[str] | None = None, mode='eval', path: str | None = None)

Load the latest timestamp saved models

Parameters:

• model – neural models in a dictionary

• epoch – epoch number (or iteration)

• read_tag – model tag name to be read out

• device – torch device

• keys – model dictionary keys

• mode – ‘train’ or ‘eval’

• path – path to model files

Returns:

model dictionary found_epoch: loaded epoch savetag per model

Return type:

model

process_memory_use()

Return system memory (RAM) used by the process in GB.

showmem(color: str = 'red')

Print CPU memory use

showmem_cuda(device: device, color: str = 'red')

Print CUDA memory use

Parameters:

• device – torch device

• color – print color

sysinfo()

Returns system info string

torch_cuda_total_memory(device: device)

Return CUDA device VRAM available in GB.

Parameters:

device – torch device

hypertrack.losses

class DiceLogitsLoss(weight=None, reduction='sum')

Dice set loss

forward(inputs: Tensor, targets: Tensor, smooth: float = 1)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class FocalWithLogitsLoss(weight=None, gamma=2, reduction='mean')

Focal Loss with logits as input

https://arxiv.org/abs/1708.02002

forward(predicted, target)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class HingeLoss(weight=None, reduction='sum')

https://en.wikipedia.org/wiki/Hinge_loss

forward(output: Tensor, target: Tensor)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class JaccardLogitsLoss(weight=None, reduction='sum')

Intersect over Union set loss

forward(inputs: Tensor, targets: Tensor, smooth: float = 1)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class SmoothHingeLoss(weight=None, reduction='sum', sigma=0.1)

Smooth Hinge Loss (Neurocomputing 463 (2021) 379-387)

forward(output: Tensor, target: Tensor)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

contrastive_edge_loss(edge_score: Tensor, edge_index_hat_np: ndarray, x_ind: ndarray, hit_weights: Tensor, tau: float = 0.1, sub_sample: int = 100000)

Supervised contrastive loss function based on ground truth objects and edges between hits

Parameters:

• edge_score – GNN scores per edge

• edge_index_hat_np – edge indices as a numpy array

• x_ind – cluster ground truth indices

• hit_weights – weights for each graph node

• tau – temperature hyperparameter

• sub_sample – maximum number of cluster objects to consider

Returns:

torch loss

find_posneg_connections(i: int, N: int, x_ind: ndarray, edge_index_hat: ndarray, take_self_connections: bool = False)

Find all connections to ground truth particle

Parameters:

• i – current row index in x_ind

• N – number of nodes (hits)

• x_ind – truth particle hit assigments

• edge_index_hat – GNN input graph (hard) adjacency of the event

Returns:

positive connections i.e. (i,j) both found in true_ind e_neg: negative (fake) connections i.e. i (but not j) or j (but not i) found in true_ind hit_ind: ground truth hit indices

Return type:

e_pos

fisher_loss(x: ndarray, th: float)

Fisher variance criterion loss for bimodal classifier threshold optimization

Parameters:

• x – array of data to be thresholded

• th – threshold value

Returns:

loss value (float)

fisher_threshold(prob: ndarray, threshold_array: ndarray)

Find Fisher variance criterion (Otsu’s method) optimal bimodal threshold

Parameters:

• prob – probabilities to threshold

• threshold_array – threshold to test values between [0,1]

Returns:

optimal threshold (float)

intersect_loss(prob: Tensor, true_ind: Tensor, chosen_ind: Tensor, micrograph_ind: Tensor, weight: Tensor)

Intersect set loss

Parameters:

• prob – per node probability scores [0,1]

• true_ind – ground truth indices

• chosen_ind – chosen indices after the treshold cut

• micrograph_ind – indices of the micrograph

• weight – weights per node

Returns:

torch loss

hypertrack.measures

class Metric(y_true, y_pred, weights=None, num_classes=2, hist=True, valrange='prob', N_mva_bins=30, verbose=True, num_bootstrap=0, roc_points=256)

Classifier performance evaluation metrics.

adjgraph_metrics(A: ndarray, A_hat: ndarray, print_on: bool = True)

Compute adjacency graph comparison metrics

Parameters:

• A – True adjacency matrix (N x N)

• A_hat – Estimated adjacency matrix (N x N)

• print_on – Print to to screen

Returns:

metrics dictionary

compute_adj_metrics(A: ndarray, A_hat: ndarray)

Compute metrics between adjacency matrices

Parameters:

• A – True adjacency (boolean)

• A_hat – Estimated adjacency (boolean)

Returns:

metrics as a tuple

compute_metrics(num_classes: int, y_true: ndarray, y_pred: ndarray, weights: ndarray, roc_points: int)

Helper function for Metric class

histedges_equal_N(x: ndarray, nbin: int)

Generate (histogram) bin edges such that each bin contains equal number of entries

Parameters:

• x – data points

• nbin – number of bins

Returns:

histogram edges array

majority_score(x_ind_hat: ndarray, x_ind: ndarray, hit_weights: ndarray, hits_min: int = 4)

Majority score function as defined in TrackML Kaggle challenge

Parameters:

• x_ind_hat – Estimated hit indices of tracks array

• x_ind – Ground truth hit indices of tracks array

• hit_weights – Weights of hits

• hits_min – Minimum number of hits per track threshold

Returns:

Double majority score value efficiency: Efficiency = # (estimated AND matched) / # (ground truth with >= hits_min) purity: Purity = # (estimated AND matched) / # (estimated) passed: Ground truth track indices which passed the double majority match_index match_index: Pointing match indices per estimate –> to ground truth tracks

Return type:

score

See also: https://en.wikipedia.org/wiki/Jaccard_index

roc_curve(y_true: ndarray, y_score: ndarray, sample_weight: ndarray = None, points: int = 256, sampling: str = 'log10_reverse', th_type: str = '>=', epsilon: float = 1e-07, drop_intermediate: bool = True)

Compute ROC curve

Parameters:

• y_true – True class labels (0,1) array

• y_score – Predictor scores array

• sample_weight – Sample weight array

• points – Number of points on curve

• sampling – ‘log10’, ‘linear’, ‘log10_reverse’, or ‘equalized’

• th_type – ‘>=’ or ‘>’ used in the comparison

• epsilon – Minimum value (in upper tail)

Returns:

False positive rate array tpr: True positive rate array thr: Threshold array

Return type:

fpr

true_false_positive(threshold_vector: ndarray, y_true: ndarray, sample_weight: ndarray = None)

ROC curve helper function

Parameters:

• threshold_vector – thresholded values array (boolean)

• y_true – true labels array (boolean)

• sample_weight – weight per array element

hypertrack.ratioplot

create_axes(xlabel='$x$', ylabel='Counts', ylabel_ratio='Ratio', xlim=(0, 1), ylim=None, ylim_ratio=(0.7, 1.3), height_ratios=(3.333, 1), ratio_plot=True, figsize=(5, 4), fontsize=9, units={'x': '', 'y': ''}, density=False, **kwargs)

Axes creator.

plot_horizontal_line(ax, color=(0.5, 0.5, 0.5), linewidth=0.9, ypos=1.0, linestyle='-')

For the ratio plot

set_axis_ticks(ax, ticks, dim='x')

Set ticks of the axis.

stepspace(start, stop, step)

Linear binning edges between [start, stop]

tick_calc(lim, step, N=6)

Tick spacing calculator.

tick_creator(ax, xtick_step=None, ytick_step=None, ylim_ratio=(0.7, 1.3), ratio_plot=True, minorticks_on=True, ytick_ratio_step=0.15, labelsize=9, labelsize_ratio=8, **kwargs)

Axis tick constructor.

hypertrack.tools

class Graph(V)

A simple class for graphs

DFSUtil(temp, v, visited)

addEdge(v, w)

Add an undirected edge

connectedComponents()

Retrieve connected components in an undirected graph

adj2edges(A: ndarray)

Convert an adjacency matrix to an edge list

Parameters:

A – adjacency matrix (N x N)

Returns:

adjacency list (2 x E)

Return type:

edge_index

attach_fully_connected_node(A, d=1.0)

Takes in a sparse graph (csr_matrix), possibly containing multiple disconnected subgraphs and attachs a node to the graph that is connected to all other nodes

Parameters:

• A – real valued adjacency matrix (csr_matrix)

• d – value for the connection

Returns:

extended adjacency matrix

backtrack_linear_index(ind: List[bool], k: int)

Find the k-th index which is True

(should optimize this function, i.e. without for loop)

Parameters:

• ind – boolean list

• k – index

closure_test_adj(edge_index: ndarray, x_ind: ndarray, N: int, DEBUG=False)

Unit test the constructed adjacency structures against a ground truth

Parameters:

• edge_index – Adjacency list (2 x E)

• x_ind – Hit indices per tracks array

• N – Number of hits

compute_ground_truth_A(X: ndarray, x_ind: ndarray, node2node: str = 'hyper', self_connect_noise: bool = False, edgeMAX: int = 10000000)

Compute Ground Truth adjacency structure

Parameters:

• X – input data

• x_ind – hit indices per object (track)

• node2node – ground truth construction type ‘hyper’: all hits of the track connects with a ‘lasso’ (hyperedge) ‘eom’: natural minimal EOM (eq.of.motion) trajectory (linear topology) ‘cricket’: next to minimal i.e. EOM + ‘double hops’ included

• self_connect_noise – connect noise hits with-self loops (True, False)

• edgeMAX – number of edges to construct at maximum (keep it high enough)

Returns:

adjacency list (2 x N)

compute_hit_mixing(x_ind: ndarray, x_ind_hat: ndarray, match_index: ndarray, hits_max: int = 30)

Compute (true,reco) hit multiplicity mixing matrix

Parameters:

• x_ind – ground truth hit indices per cluster array

• x_ind_hat – estimated hit indices per cluster array

• match_index – match pointing indices from estimate –> ground truth

• hits_max – the maximum number of hits

Returns:

hit count mixing 2D-square matrix array (true, reco) eff: reconstruction hit set efficiency purity: reconstruction hit set purity

Return type:

H

compute_hyper(edge_index: ndarray, x_ind: ndarray)

Helper function for compute_ground_truth()

count_hit_multiplicity(M: ndarray, bins: int = None)

Count the number of hits per cluster for diagnostics

Parameters:

• M – array of clusters (rows) and corresponding assingments (cols)

• bins – number of histogram bins

Returns:

multiplicity histogram overflow: number of overflows

Return type:

counts

count_reco_cluster(x_ind: ndarray, hits_min: int = 4)

Count the number of reconstructable clusters

Parameters:

• x_ind – hit indices per cluster array

• hits_min – minimum number of hits

Returns:

count

create_cluster_ind(labels: ndarray, hmax: int = 100)

Create cluster index arrays

Parameters:

• labels – cluster labels list of integers

• hmax – maximum number of hits per cluster

Returns:

cluster assignments [K x hmax]

create_cluster_labels(N: int, x_ind: ndarray)

Create cluster labelings

Parameters:

• N – Number of nodes (hits)

• x_ind – Hit indices of tracks list

• outlier_value – Default cluster value for unassigned hits

Returns:

List of cluster label integers

Return type:

labels

create_edge_label(edge_index_hat: ndarray, A: ndarray)

Parameters:

• edge_index_hat – estimated adjacency list (2 x E)

• A – Ground truth adjacency matrix (N x N)

Returns:

edge_label list (E)

Return type:

edge_label

edges2adj(edge_index: ndarray, N: int)

Convert an edge list to an adjacency matrix A

Parameters:

• edge_index – adjacency list (2 x E)

• N – dimension of A (N x N)

Returns:

adjacency matrix

Return type:

A

explicit_range(entry_start: int, entry_stop: int, num_entries: int)

Clean None from entry_start and entry_stop

find_connected_components(edge_index: ndarray, N: int)

Find connected components based on the adjacency list

Parameters:

• edge_index – Adjacency list (2 x E)

• N – Number of nodes in the graph

Returns:

connected components

normalize_cols(x: ndarray, order=2)

Normalize matrix columns to the L-p norm

normalize_rows(x: ndarray, order=2)

Normalize matrix rows to the L-p norm

print_clusters(x_ind)

Print cluster information

Parameters:

x_ind – cluster ground truth array

running_mean_uniform_filter1d(x: ndarray, N: int)

Running mean filter

Parameters:

• x – input array

• N – filter window size

Returns:

filtered array

select_valid(x)

span_adj_mat(x_ind: ndarray, N: int, symmetric: bool = True)

Create boolean adjacency matrix from an index array such that the first element [0] is the pivot which spans the edges to other elements in the index array.

Parameters:

• x_ind – index array (K x max) with -1 for null elements

• N – dimension of A

• symmetric – symmetric (undirected) adjacency

Returns:

adjacency matrix

split(a: int, n: int)

Generator which returns approx equally sized chunks. :param a: Total number :param n: Number of chunks

Example

list(split(10, 3))

split_size(a: int, n: int)

As split_start_end() but returns only size per chunk

split_start_end(a, n: int, end_plus: int = 1)

Returns approx equally sized chunks.

Parameters:

• a – Range, define with range()

• n – Number of chunks

• end_plus – Python/nympy index style (i.e. + 1 for the end)

Examples

split_start_end(range(100), 3) returns [[0, 34], [34, 67], [67, 100]] split_start_end(range(5,25), 3) returns [[5, 12], [12, 19], [19, 25]]

timing(f)

Timing function wrapper

track_minimal_spanning_tree(X: ndarray, x_ind_single: ndarray)

Compute minimal spanning tree according to hit to hit distances in X

Parameters:

• X – hit (node) matrix array (N x dim)

• x_ind_single – single track hit indices (array)

Returns:

minimal spanning tree adjacency with indices corresponding in x_ind_single

value_and_uncertainty_scientific(x, dx, one_digit=True, rounding='round')

A value and its uncertainty (or dispersion measure) in the scientific x(dx) format

Parameters:

• x – value

• dx – dispersion

• one_digit – force only one digit output for dx

• rounding – rounding of dx, ‘ceil’, ‘floor’ or ‘round’

Returns:

a string with x(dx) format

hypertrack.torchtools

count_parameters_torch(model: Module, only_trainable: bool = False, print_table: bool = True)

Count the number of (trainable) pytorch model parameters

Parameters:

• model – torch model

• only_trainable – if true, only which require grad are returned

• print_table – print the parameter table

Returns:

number of parameters

fastcat_rand(p: Tensor)

Categorical (multinomial) sampling function

Parameters:

p – multinomial probability vector

Returns:

random index

plot_grad_flow(named_parameters, y_max=0.1, y_min=1e-12, alpha=0.2, param_types=[''])

Plots the gradients flowing through different layers in the net during training. Use for checking for possible gradient vanishing / explosion / debugging.

Usage:

Use in torch code after loss.backwards() as “plot_grad_flow(self.model.named_parameters())” to visualize the gradient flow

Parameters:

• named_parameters – as returned by model.named_parameters()

• y_max – plot y-axis maximum (‘auto’, None, or float value)

• y_min – y-axis minimum

• alpha – bar plot transparency

• param_type – a list of names to be found from the parameters, e.g. ‘weight’, ‘bias’, set [‘’] for all

Returns:

figures as a list

seperate_weight_param(model: Module, weight_decay_w: float, weight_decay_b: float)

Construct weight decays for different parameter types of the model

Parameters:

• model – torch model

• weight_decay_w – weight decay value for layer weight terms

• weight_decay_b – weight decay value for layer bias terms

Returns:

network parameters grouped

set_diff_1d(t1: Tensor, t2: Tensor, assume_unique=False)

Set difference of two 1D tensors. Returns the unique values in t1 that are not in t2.

set_random_seed(seed)

Set random seed for torch, random and numpy

Parameters:

seed – seed value

subgraph_mask(subgraph_nodes: List[int], edge_index_hat: Tensor, sparse_ind=None)

Compute subgraph boolen mask array

Parameters:

• subgraph_nodes – list of subgraph nodes

• edge_index_hat – edge index list (2 x E)

• sparse_ind – already sparsified indices (dim E) (if none, all edges are treated)

Returns:

subgraph mask indices

to_scipy_sparse_csr_matrix(edge_index: Tensor, edge_attr: Tensor | None = None, num_nodes: int | None = None) → csr_matrix

Converts a graph given by edge indices and edge attributes to a scipy sparse matrix in CSR format

Parameters:

• edge_index – adjacency list

• edge_attr – edge features

• num_nodes – number of nodes in the graph (do set, otherwise guess)

Returns:

sparse adjacency matrix (scipy csr)

torch_intersect(t1: Tensor, t2: Tensor, use_unique=False)

Set intersect A-B between two tensors

Parameters:

• t1 – tensor A

• t2 – tensor B

Returns:

torch tensor for A-B

torch_to_scipy_scr(x, is_sparse=True)

Convert a dense or sparse torch tensor to a scipy csr matrix

Parameters:

• x – tensor to convert

• is_sparse – sparse or normal tensor

Returns:

csr array

weighted_degree(index: Tensor, weight: Tensor, num_nodes: int | None = None)

Weighted (graph) degree

Parameters:

• index – adjacency list row, i.e. edge_index[0,:]

• weight – edge weights

• num_nodes – number of nodes in the graph (do set, otherwise guess)

Returns:

weighted node degree

hypertrack.trackml

generate_graph_data(event: int, rfactor: float, noise_ratio: float, node2node: str, geom_param: dict, geometric_obj: dict)

Generate TrackML data graph input (both training and inference data)

Parameters:

• event – event file number

• rfactor – pile-up reduction factor

• node2node – ground truth adjacency construction mode

• geom_param – geometric pre-algorithm algorithm parameters

• voronoi_obj – pre-loaded voronoi estimator objects

Returns:

dictionary with objects

load_reduce_event(event: int, rfactor: float | None = None, noise_ratio: float | None = None, X_DIM: List[int] = [0, 1, 2], P_DIM: List[int] = [1, 2, 3, 4, 5, 6, 7, 8])

Load event and reduce pile-up

Parameters:

• event – event number index

• rfactor – pile-up reduction factor

• noise_ratio – noise fraction

• X_DIM – cluster (hit info) parameter indices

• P_DIM – cluster (latent info) parameter indices

Returns:

hit (node) data x_ind: cluster ground truth index array pointing to X objects: cluster latent object data hit_weights: hit (node) weights

Return type:

X

process_data(PATH: str, maxhits_per_trk: int = 30, verbose: bool = False)

Read and process TrackML kaggle challenge tracking dataset

https://www.kaggle.com/competitions/trackml-particle-identification/data

Parameters:

• path – path to TrackML files

• maxhits_per_trk – maximum number of hits per track

• verbose – verbose output

Returns:

dictionary with keys X: every single hit (x, y, z, volume_id, layer_id, module_id) x_ind: indices pointing to X for particles with at least 1 hit,

with running order matching particles[visible_particle]

particles: generated particles [particle id, vx,vy,vz, px,py,pz, q, nhits] visible_particle: boolean array with length of particles (at least 1 hit == True, else False) hit_weights: weight for every single hit

reduce_tracks(X: ndarray, x_ind: ndarray, hit_weights: array, rfactor: float, noise_ratio: float = -1)

Reduce tracks (pileup) of the event by a factor (0,1]

The reduction is simplistic ‘overall’ reduction of tracks, good enough for benchmarking purposes.

(even more realistically one should drop the number of proton-proton vertices and the associated particles & hits)

Parameters:

• X – node data (N x dim)

• x_ind – a list of list of indices in X per track (track) object

• hit_weights – weights per hit (N)

• rfactor – reduction factor (0,1]

• noise_ratio – noise hit ratio (0,1] (set -1 for the original, set None for no noise)

Returns:

as above in the input x_ind_new: as above in the input hit_weights_new: as above in the input hot_ind: chosen subset row indices of x_ind (input)

Return type:

X_new

hypertrack.transforms

transform_r_eta_phi(vec)

Transform (x,y,z) -> (r,eta,phi)

Parameters:

vec – cartesian coordinate array (N x 3)

Returns:

coordinate array

hypertrack.visualize

ROC_plot(metrics, labels, title='', plot_thresholds=True, thr_points_signal=[0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95], filename='ROC', legend_fontsize=7, xmin=1e-05, alpha=0.32)

Receiver Operating Characteristics, i.e., False Positive Rate (efficiency) vs True positive Rate (efficiency)

Parameters:

• metrics – metrics object

• labels – labels

• title – plot title

• plot_thresholds – plot annotation points (True, False)

• thr_points_signal – signal efficiency

• filename – filename (full path)

• legend_fontsize – legend fontsize

• xmin – minimum FPR value (x-axis)

• alpha – transparency

Returns:

plots saved directly to the disk

annotate_heatmap(X, ax, xlabels, ylabels, x_rot=0, y_rot=0, decimals=1, color='w', fontsize=6)

Add text annotations to a matrix heatmap plot

Parameters:

• X –

• ax –

• xlabels –

• ylabels –

• x_rot –

• y_rot –

• decimals –

• color –

• fontsize –

Returns:

axis object

Return type:

ax

compare_cluster_multiplicity(multiplicity: ndarray, multiplicity_hat: ndarray, nbins: int = 30, hits_min: int = 1)

Compare ground truth and estimated cluster count multiplicity wise

Parameters:

• multiplicity – ground truth multiplicities

• multiplicity_hat – estimated multiplicities

• nbins – number of histogram bins

Returns:

fig, ax

compare_hit_multiplicity(counts, counts_hat)

Compare ground truth and estimated hit counts multiplicity wise

Parameters:

• counts – ground truth multiplicities

• counts_hat – estimated multiplicities

Returns:

fig, ax

crf(label: str, values: ndarray)

Label string helper function

Parameters:

• label – arbitrary label

• values – data array

Returns:

text string with “label (mu = x, sigma = dx)”

density_MVA_wclass(y_pred: ndarray, y: ndarray, label: str, weights: ndarray | None = None, num_classes: int | None = None, hist_edges: int = 80, filename: str = 'output')

Evaluate MVA output (1D) density per class

Parameters:

• y_pred – MVA algorithm output

• y – Output (truth level target) data

• label – Label of the MVA model (string)

• weights – Sample weights

• hist_edges – Histogram edges list (or number of bins, as an alternative)

Returns:

Plot pdf saved directly to the disk

hist_filled_error(ax, bins, y, err, color, **kwargs)

Stephist style error

Parameters:

• ax – axis object

• bins – bins (central)

• y – y-axis values

• err – error bar values

• color – color value

Returns:

directly manipulates ax object

imperial_brown = (0.8, 0.76, 0.57)

Global marker styles zorder : approximate plotting order lw : linewidth ls : linestyle

plot_C_matrix(C: ndarray, title: str = '', savename: str = '')

Visualize connectivity C-matrix

Parameters:

• C – connectivity matrix

• title – plot title

• savename – plot savename

Returns:

plots saved directly to the disk

plot_adj_matrices(A: ndarray, A_hat: ndarray, maxrows: int = 1000)

Visualize adjacency matrices

Parameters:

• A – Ground truth adjacency matrix

• A_hat – Estimated adjacency matrix

• maxrows – Maximum number of rows to plot

Returns:

fig, ax

plot_mixing_matrix(X, normalize='cols', null_empty=True, cmap='coolwarm', decimals=0, text_fontsize=6, color='w')

Plot mixing matrix

Parameters:

• X – hit mixing matrix

• normalize – normalize either ‘cols’ or ‘rows’ to sum to one

• null_empty – set empty elements to NaN

• cmap – colormap

• decimals – number of decimal digits to include

• text_fontsize – fontsize

• color – color

Returns:

fix, ax

plot_tracks(X: ndarray, x_ind: ndarray, ind: list = [0], xbound=1250, zbound=3500, title='', text_on=True)

Visualize hits of tracks as 3D-points

Parameters:

• X – Hit position matrix [number of hits x 3]

• x_ind – Hit indices of tracks array (-1 for null)

• ind – Track indices to visualize

Returns:

fig, ax

plot_voronoi(centroids: ndarray, savename: str = '')

Visualize Voronoi division

Parameters:

• centroids – centroid vectors in an array [N x dim]

• savename – name of the file to be saved (ending)

Returns:

plots saved directly to the disk

ratioerr(A: ndarray, B: ndarray, sigma_A: ndarray, sigma_B: ndarray, sigma_AB: ndarray = 0, EPS=1e-15)

Ratio f(A,B) = A/B uncertainty, by Taylor expansion of the ratio

Parameters:

• A – values A

• B – values B

• sigma_A – 1 sigma uncertainty of A

• sigma_B – 1 sigma uncertainty of B

• sigma_AB – covariance between A and B (default 0)

Returns:

ratio uncertainty array

track_effiency_plots(passed: ndarray, purity: ndarray, eff: ndarray, objects: ndarray, hits_min: int = 4, path: str = 'figs/predict/')

Create tracking efficiency and the hit set efficiency & purity plots

Parameters:

• passed – list of 1/0 indices indicating a double major score reconstructed particle

• purity – clustering constituent purity for each ground truth reconstructed particle

• eff – clustering constituent efficiency for each ground truth reconstructed particle

• objects – particle (track) data with columns: [3-vertex {1,2,3}, 3-momentum {4,5,6}, charge {7}, nhits {8}]

• hits_min – minimum number of hits per ground truth (denominator definition)

• path – figure save path

Returns:

figures saved to the disk

hypertrack.voxdyn

combine_data(data: dict, weighted: bool)

Combine data from different events

Parameters:

• data – event-by-event dictionary data in a list

• weighted – use weights

Returns:

data vectors data_A: adjacency per particle data_weights: weights linear_list: indices of hits

Return type:

data_x

compute_A_direct(I: ndarray, C: ndarray)

“Direct” – Construct hit adjacency matrix given the hit indices I which point which cell hits belong to

Parameters:

• I – cell indices for each hit (N)

• C – cell 2-point connectivity matrix (V x V) (csr-library matrix)

Returns:

adjacency list (2 x E) A: adjacency matrix (N x N)

Return type:

edge_index

compute_A_reverse(I_per_V: ndarray, H: ndarray, C: ndarray, N: int)

“Reverse” – Construct hit adjacency matrix given the hit indices in I_per_V which point which V-cell hits belong to

Parameters:

• I_per_V – cell indices for each hit (V x maxhits (buffer) per cell)

• H – number of hits per V-space cell

• C – cell connectivity matrix (V x V) (csr-library matrix)

• N – adjacency matrix dimension

Returns:

adjacency list (2 x E) A: adjacency matrix (N x N)

Return type:

edge_index

compute_A_reverse_3pt(I_per_V3: ndarray, H3: ndarray, C3: ndarray, N: int)

3-pt version [EXPERIMENTAL]

Parameters:

• I_per_V3 – cell indices for each hit (dim |V| x maxhits (buffer) per cell)

• H3 – number of hits per V-space cell

• C3 – cell connectivity matrix (dim |V| x dim |V| x dim |V|)

• N – adjacency matrix dimension

Returns:

adjacency list (2 x E) A: adjacency matrix (N x N)

Return type:

edge_index

compute_I_per_V(I: ndarray, ncell: int, buffer: int)

From an event hit list to V-space cells

Parameters:

• I – index list of hits which point to V-space cell

• ncell – number of V-cells

• buffer – buffer length (make large enough!)

Returns:

hit index per V-cell (several hits possible, empty slots are -1) H: number of hits per V-cell

Return type:

I_per_V

faiss_to_gpu(index, device)

Faiss index object to GPU

Parameters:

• index – Faiss index object

• device – ‘cpu’ or ‘cuda’

Returns:

Index object

Return type:

index

load_particle_data(event: int, event_loader, node2node: str, weighted: bool = False)

Load particle data in a format suitable for training C-matrices (tensors)

Parameters:

• event_loader – event loader function handle

• event – event index

• node2node – target graph topology type

• weighted – pick object weights

Returns:

event data vectors A_track: event adjacency matrix weights: weights for each data vector

Return type:

X

train_2pt_connectivity(C: ndarray, linear_list: List[List[int]], I: ndarray, A_track: List[ndarray])

Train a 2-point connectivity C-matrix

Parameters:

• C – a cell 2-point connectivity matrix (V x V) (csr-library matrix)

• linear_list – a list of lists

• I – index assignments

• A_track – ground truth ‘micro adjacency’ per track, a list of adjacency matrices

• ncell – number of V-cells

Returns:

Values are filled in the input C

train_2pt_connectivity_sparse(maxbuffer: int64, linear_list: List[List[int]], I: ndarray, A_track: List[ndarray])

Train a 2-point connectivity matrix [sparse matrix version]

Parameters:

• maxbuffer – maximum number of entry fills (int64)

• linear_list – a list of lists

• I – index assignments

• A_track – ground truth ‘micro adjacency’ per track, a list of adjacency matrices

• ncell – number of V-cells

Returns:

index arrays

Return type:

row, col

train_3pt_connectivity(C: ndarray, linear_list: List[List[int]], I: ndarray, A_track: List[ndarray])

Train a 3-point connectivity tensor [EXPERIMENTAL]

Parameters:

• C – 3-point connectivity tensor (V x V x V)

• linear_list – a list of lists

• I – index assignments

• A_track – ground truth ‘micro adjacency’ per track, a list of adjacency matrices

• ncell – number of V-cells

Returns:

Values are filled in the input C

train_voxdyn(event_loader, event_range: List[int], ncell_list: List[int] = [65536, 131072], ncell3: int = 1024, niter: int = 50, min_points_per_centroid: int = 1, max_points_per_centroid: int = 1000000000, node2node: str = 'hyper', maxbuffer=5000000000, weighted=False, device: str = 'cpu', verbose: bool = True)

Train the geometric Voxel-Dynamics estimator

Parameters:

• event_loader – event loader function handle

• event_range – event index list

• ncell_list – voxel counts to construct (list)

• ncell3 – 3-point voxel count (int)

• niter – number of K-means iterations

• min_points_per_centroid – minimum number of points per centroid in K-means

• max_points_per_centroid – maximum number of points per centroid in K-means

• node2node – target graph topology type

• maxbuffer – maximum (64 bit integer) number of entry fills for the sparse C-matrix

• weighted – use weights

• device – ‘cpu’ or ‘cuda’

• verbose – verbose print

Returns:

trained models are saved to the disk

voxdyn_predictor(X: array, node2node: str = 'hyper', ncell: int = 65536, obj=None, adj_algorithm='reverse', buffer=15, device: str = 'cpu')

Predict connectivity based on Voxel-Dynamic estimator

Parameters:

• X – input data array (C-contiguous, float32)

• node2node – adjacency connectivity type

• ncell – number of cells

• obj – pre-loaded estimator object dictionary

• adj_algorithm – ‘direct’ or ‘reverse’ [identical result from both but latency may differ]

• buffer – Node count default buffer size per V-cell for the ‘reverse’ algorithm (will be recursively increased by x 10 until no saturation)

• device – computing device (‘cpu’ or ‘cuda’)

Returns:

adjacency list A_hat: adjacency matrix

Return type:

edge_index_hat