motivation-and-rationale.md

Rule format - Motivation and rationale

The following documentation explains different design dimensions considered during implementation. You can find documentation about the actual implementation here:

• Rule Format
• Rule - available data structures and functions

Rule format

The overall motivation for the rule format is an easy to write and read format for rule specification that supports the necessary rule complexity. Sufficiently complex rules require a sequence of arithmetic operations. Operation specification requires to declare input and output values in form of literal values or identifiers. In our use case, the inputs consist of measured metric values, parameters and rule specific literal values. Interpreting specifications of such complexity already requires parsing the rule specification into a format that can be evaluated, e.g. parsing into an Abstract Syntax Tree (AST) that can be evaluated. There are two ways to avoid the work and complexity of parsing:

• Let the rule specification be in a machine-readable, tokenized format that can be directly evaluated. Such a format could be result of the parsing step. Here is an example of a JSON based format, that specifies the tokens separately and can be easily evaluated. However, the expressions are hard to read and write.

    "terms":[
        {"function":"mean","input":{"name":"cpu_load","scope":"node"},"output":"load_mean"},
        {"function":"<","input":["load_mean","lowload_threshold"],"output":"lowload"}
    ],

• Make use of an existing format with parsing and evaluation support. In this example, the JSON based format contains tuples of variable and expression pairs, that represent assignments. The structure is an array of objects to make the order of terms static. In fact, all objects could be rewritten as variable = expression strings, but these would require parsing. The expressions can be evaluated as Python expressions using eval and the results saved as value for the variables.

    "terms":[
        {"load_mean":           "mean(cpu_load, 'node')"},
        {"load_threshold_old":  "cores_per_node * load_threshold_factor"},
        {"load_threshold":      "job.numHwthreads * load_threshold_factor"},
        {"lowload":             "load_mean < load_threshold"},
        {"load_max":            "job.numHwthreads"},
        {"load_perc":           "1.0 - (load_mean / load_threshold)"}
    ],

Types

The rule output is the decision if a pattern was detected or not and therefore a boolean value type is a natural choice. The rule input consists of measured sample values and are naturally represented as matrices or vectors of floating point values. Additionally, scalar floating point values are necessary for single parameter values. Python also supports strings, but these could be, if necessary, replaced with identifiers for constant values.

Strings, scalar integers and scalar floating point numbers are an integral part of Python. For the representation of matrices and vectors, one could use structures of arrays or types from additional libraries such as NumPy or Pandas. Pandas DataFrame types come with support for attached metadata, but the mechanism was not flexible enough. We decided to use a new type that wraps metadata and NumPy matrices and vectors. The downside of this approach is the necessity to add all required functions to the new data type that need to be applicable to the data (e.g. mean, sum, etc.). However, when using these functions the metadata needs to be adjusted according to the function and therefore the functions need to be wrapped anyway.

Scopes

Measurement of system metrics is performed at different levels of the hard- and software that a HPC job runs on.

• Hardware thread (In case of SMT multiple threads can run on one core)
• Core
• Memory domain (In case a socket is divided into multiple NUMA domains)
• Socket
• Node
• Accelerator (Multiple accelerators can run on one node)
• Job

These match the levels used in the Job Archive Format of ClusterCockpit for the measured samples. Additionally, the job scope is added for samples that are reduced from samples on multiple nodes. Also, the Job Archive Format defines that metrics can have samples from multiple scopes.

Measured time

Input metrics consist of time series with samples measured at time intervals or at certain moments of time. For certain reductions, the sample time needs to be taken into account and, therefore, the timestamps need to be available with the sampled values. To attach timestamps to the measured data, there are two methods:

• Save timestamps as additional column to the input.
• Save timestamps as metadata attached to the matrix object.

In our case, the timestamps are stored as column stored separately in the metadata.

Operations

Operations are applied to matrix and vector values to perform arithmetic transformations and reductions. Because of the scope hierarchy, rules need to be aware of the scope a sample belongs to. To compare the measured values to thresholds, the rules need to reduce the samples to an adequate level. Therefore, input variables need to store the scope of values. To allow for convenient selection and reduction of values, operations might need to be wrapped in functions that process the metadata, consisting of timestamps and scope information.

One example would be the operation parameter for the scope the function is applied to:

mean(X, scope='time') - Average the samples in time, number of columns stays the same
mean(X, scope='node') - Average the samples in `node` scope, columns belonging to the same node are reduced to one column

Operators

To overload operators in Python, it is necessary to define functions for the class a variable belongs to. It is not possible to assign magic methods to an object instance. However, pandas redirects all operator methods, which allows for overwriting.

Therefore, there are two methods to achieve operator overloading:

• Create a new class that wraps the old class.
• Assign functions to every new object. In this case, every relevant function needs to be wrapped, to manipulate the resulting object.

In our case, we use a new class and additionally wrap the functions to adjust metadata.

Units

The ClusterCockpit Job Archive Format provides the units of measured metrics. These units can be used to check operations performed on the values.

Some things need to be considered:

• Units can contain prefixes, e.g. "KiB", "GPackets"
• Operations imply unit conversion that need to be evaluated symbolically, e.g. "cycles * instructions per cycle = instructions"
• Unit symbols might be ambiguous

There are two possible implementation methods:

• Values are stored in types that understand units and perform unit checks implicitly.
• Units are stored for each column in the metadata and unit checks are performed separately for every operation.

We are using Pint to wrap the NumPy values with the necessary unit information.

Open ideas

Conditional term evaluation

Right now every expression is evaluated. It might be easier to define certain computations by conditionally evaluate expressions.

if c:
    a = expr
    b = expr
    x = a*b
else:
    c = expr
    d = expr
    x = c*d
y = x * 2.0

One usage example would be to test the existence of certain data and only access the data if it is available. There might be different methods to deal with conditionally evaluated expressions:

• Make all expressions always evaluatable, but only use certain results:

a = expr
b = expr
x_1 = a*b

c = expr
d = expr
x_2 = c*d

x = x_1 if c else x_2
y = x * 2.0

• Wrap every expression in an if-else expression:

x = None

a = expr if c else None
b = expr if c else None
x =  a*b if c else x

c = expr if not c else None
d = expr if not c else None
x =  c*d if not c else x

y = x * 2.0

• Define a new structure that expresses conditional evaluation of expressions:

"terms":[
    {"cond":"c",
     "if":[
        {"a":"expr"},
        {"b":"expr"},
        {"x":"a*b"}
     ],
     "else":[
        {"c":"expr"},
        {"d":"expr"},
        {"x":"c*d"}
     ]},
    {"y":"x * 2.0"}
]

Here the rule evaluation logic first evaluates the condition expression "c" and afterwards evaluates the expressions in either the "if" branch or the "else" branch.

Rule format files

As an alternative to a JSON based file, one could also create a file format mixed from plain text and JSON (or YAML).

Example:

---
{
    "name":"Low CPU load",
    "tag":"lowload",
    "parameters": ["lowcpuload_threshold_factor","job_min_duration_seconds","sampling_interval_seconds"],
    "metrics": ["cpu_load"],
    "requirements": [
        "job.exclusive == 1",
        "job.duration > job_min_duration_seconds",
        "required_metrics_min_samples > job_min_duration_seconds / sampling_interval_seconds"
    ],  
    "output":"lowload",
    "output_scalar":"load_perc",
    "template":"Job ({{ job.jobId }})\nThis job was detected as lowload because the mean cpu load {{ load_mean }} falls below the threshold {{ load_threshold }}."
}
---
# rule terms
load_mean           = cpu_load[cpu_load_pre_cutoff_samples:].mean('all')
load_threshold      = job.numHwthreads * lowcpuload_threshold_factor
lowload_nodes       = load_mean < load_threshold
lowload             = lowload_nodes.any('all')
load_perc           = 1.0 - (load_mean / load_threshold)


# next rule ...
---
[..]
---
[..]

Here, the rule attributes are defined in a JSON object in a separate section enclosed by two --- lines that are easy to parse. The rule terms are defined in the following section. The left side of the term consists of only one variable name and the rest of the line constitutes the expression part of the rule term, optionally separated by an equality sign =. The seperation of the two parts would be easy to parse. This would also ease the use of comments and blank lines. The section is followed by further section pairs for other rules.

Rule requirements

Right now a rule will be always evaluated. One could explicitly define conditional expressions that check if the assumptions of the rule are met by the job metadata. Only after the list of expressions all evaluate to True the real rule terms are evaluated.

# all expressions in `requirements` need to evaluate to True, before the rule terms are evaluated
"requirements":[
    "job.SMT == 1"
],
"terms":[
    {"foo":"expression * SMT"}
]

This way, one could also identify the expression that is not met by the job and present this as debugging information.

Warning: Rule can't be evaluated, because the following requirement was not met:  job.SMT == 1