plexus.analysis.metrics.metric module

Base class for all alignment and accuracy metrics in Plexus.

This module provides a standard interface for implementing various evaluation metrics, ensuring consistent inputs and outputs across different metric types.

class plexus.analysis.metrics.metric.Metric

Bases: ABC

Abstract base class for implementing evaluation metrics in Plexus.

Metric is the foundation for standardized evaluation metrics. Each implementation represents a specific way to measure agreement or performance, such as:

  • Agreement coefficients (Gwet’s AC1, Cohen’s Kappa)

  • Accuracy metrics (raw accuracy, F1 score)

  • Distance metrics (RMSE, MAE)

The Metric class provides: - Standard input/output interfaces using Pydantic models - Consistent calculation methods - Range information for proper visualization

Common usage patterns: 1. Creating a custom metric:

class MyMetric(Metric):
def calculate(self, input_data: Metric.Input) -> Metric.Result:

# Custom metric calculation logic return Metric.Result(

name=”My Custom Metric”, value=calculated_value, range=[0, 1]

)

  1. Using a metric:

    metric = MyMetric() result = metric.calculate(Metric.Input(

    reference=[“Yes”, “No”, “Yes”], predictions=[“Yes”, “No”, “No”]

    ))

class Input(*, reference: List[Any], predictions: List[Any])

Bases: BaseModel

Standard input structure for all metric calculations in Plexus.

The Input class standardizes how data is passed to metric calculations, typically consisting of two lists: reference (ground truth) and predictions.

Attributes:

reference: List of reference/gold standard values predictions: List of predicted values to compare against reference

Common usage:
input_data = Metric.Input(

reference=[“Yes”, “No”, “Yes”, “Yes”], predictions=[“Yes”, “No”, “No”, “Yes”]

)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config: ClassVar[ConfigDict] = {'protected_namespaces': ()}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

predictions: List[Any]
reference: List[Any]
class Result(*, name: str, value: float, range: List[float], metadata: dict = {})

Bases: BaseModel

Standard output structure for all metric calculations in Plexus.

The Result class provides a consistent way to represent metric outcomes, including the metric name, calculated value, and valid range.

Attributes:

name: The name of the metric (e.g., “Gwet’s AC1”) value: The calculated metric value range: Valid range for the metric as [min, max] metadata: Optional additional information about the calculation

Common usage:
result = Metric.Result(

name=”Accuracy”, value=0.75, range=[0, 1], metadata={“sample_size”: 100}

)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

metadata: dict
model_config: ClassVar[ConfigDict] = {'protected_namespaces': ()}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str
range: List[float]
value: float
abstractmethod calculate(input_data: Input) → Result

Calculate the metric value based on the provided input data.

This abstract method must be implemented by all concrete metric classes.

Args:

input_data: Metric.Input object containing reference and prediction data

Returns:

Metric.Result object with the calculated metric value and metadata