plexus.Scorecard module

class plexus.Scorecard.Scorecard(*, scorecard)

Bases: object

Represents a collection of scores and manages the computation of these scores for given inputs.

A Scorecard is the primary way to organize and run classifications in Plexus. Each scorecard is typically defined in a YAML file and contains multiple Score instances that work together to analyze content. Scorecards support:

• Hierarchical organization of scores with dependencies

• Parallel execution of independent scores

• Cost tracking for API-based scores

• Integration with MLFlow for experiment tracking

• Integration with the Plexus dashboard for monitoring

Common usage patterns: 1. Loading from YAML:

scorecard = Scorecard.create_from_yaml(‘scorecards/qa.yaml’)

2. Scoring content:

   results = await scorecard.score_entire_text(

   text=”content to analyze”, metadata={“source”: “phone_call”}

   )

3. Batch processing:

   for text in texts:

   results = await scorecard.score_entire_text(

   text=text, metadata={“batch_id”: “123”}

   ) costs = scorecard.get_accumulated_costs()

4. Subset scoring:

   results = await scorecard.score_entire_text(

   text=”content to analyze”, subset_of_score_names=[“IVR_Score”, “Compliance_Score”]

   )

The Scorecard class is commonly used with Evaluation for measuring accuracy and with the dashboard for monitoring production deployments.

Initializes a new instance of the Scorecard class.

Args:

scorecard (str): The name of the scorecard.

__init__(*, scorecard)

Initializes a new instance of the Scorecard class.

Args:

scorecard (str): The name of the scorecard.

build_dependency_graph(subset_of_score_names)

check_dependency_conditions(score_id: str, dependency_graph: dict, results_by_score_id: dict) → bool

Check if all conditions for a score’s dependencies are met.

Args:

score_id: The ID of the score to check dependency_graph: The dependency graph containing conditions results_by_score_id: Dictionary of score results

Returns:

bool: True if all conditions are met, False otherwise

classmethod create_from_yaml(yaml_file_path)

Creates a Scorecard class dynamically from a YAML file.

Args:

yaml_file_path (str): The file path to the YAML file containing the scorecard properties.

Returns:

type: A dynamically created Scorecard class.

get_accumulated_costs()

get_model_name(name=None, id=None, key=None)

Return the model name used for a specific score or the scorecard.

async get_score_result(*, scorecard, score, text, metadata, modality, results)

Get a result for a score by looking up a Score instance for that score name and calling its predict method with the provided input.

Args:

scorecard (str): The scorecard identifier. score (str): The score identifier. text (str): The text content to analyze. metadata (dict): Additional metadata for the score. modality (str, optional): The modality of the content (e.g., ‘Development’, ‘Production’). results (list): Previous score results that may be needed for dependent scores.

Returns:

Union[List[Score.Result], Score.Result]: Either a single Result object or a list of Result objects. The Result object contains the score value and any associated metadata.

Raises:

BatchProcessingPause: When scoring needs to be suspended for batch processing. ValueError: When required environment variables are missing.

classmethod initialize_registry()

classmethod load_and_register_scorecards(directory_path)

classmethod name()

Returns the class name of the scorecard.

Returns:

str: The name of the class.

classmethod normalize_score_name(score_name)

Normalizes a score name by removing whitespace and non-word characters.

Args:

score_name (str): The original score name.

Returns:

str: A normalized score name suitable for file names or dictionary keys.

async score_entire_text(*, text: str, metadata: dict, modality: str | None = None, subset_of_score_names: List[str] | None = None) → Dict[str, Result]

classmethod score_names()

classmethod score_names_to_process()

Filters and returns score names that need to be processed directly. Some scores are computed implicitly by other scores and don’t need to be directly processed.

Returns:

list of str: Names of scores that need direct processing.

score_registry = None