plexus.Scorecard module

class plexus.Scorecard.Scorecard(*, scorecard, api_data=None, scores_config=None)

Bases: object

Represents a collection of scores and manages the computation of these scores for given inputs.

A Scorecard is the primary way to organize and run classifications in Plexus. Each scorecard is typically defined in a YAML file and contains multiple Score instances that work together to analyze content. Scorecards support:

• Hierarchical organization of scores with dependencies

• Parallel execution of independent scores

• Cost tracking for API-based scores

• Integration with MLFlow for experiment tracking

• Integration with the Plexus dashboard for monitoring

Common usage patterns: 1. Loading from YAML:

scorecard = Scorecard.create_from_yaml(‘scorecards/qa.yaml’)

2. Scoring content:

   results = await scorecard.score_entire_text(

   text=”content to analyze”, metadata={“source”: “phone_call”}

   )

3. Batch processing:

   for text in texts:

   results = await scorecard.score_entire_text(

   text=text, metadata={“batch_id”: “123”}

   ) costs = scorecard.get_accumulated_costs()

4. Subset scoring:

   results = await scorecard.score_entire_text(

   text=”content to analyze”, subset_of_score_names=[“IVR_Score”, “Compliance_Score”]

   )

The Scorecard class is commonly used with Evaluation for measuring accuracy and with the dashboard for monitoring production deployments.

Initializes a new instance of the Scorecard class.

Args:

scorecard (str): The name or identifier of the scorecard. api_data (dict, optional): API-sourced data for the scorecard when using API-first loading. scores_config (dict, optional): Dictionary of score configurations when using API-first loading.

__init__(*, scorecard, api_data=None, scores_config=None)

Initializes a new instance of the Scorecard class.

Args:

scorecard (str): The name or identifier of the scorecard. api_data (dict, optional): API-sourced data for the scorecard when using API-first loading. scores_config (dict, optional): Dictionary of score configurations when using API-first loading.

build_dependency_graph(subset_of_score_names)

Build a dependency graph for the specified subset of scores.

Args:

subset_of_score_names: List of score names to include in the graph

Returns:

Tuple of (graph, name_to_id) where: - graph is a dictionary mapping score IDs to dependency information - name_to_id is a dictionary mapping score names to IDs

check_dependency_conditions(score_id: str, dependency_graph: dict, results_by_score_id: dict) → bool

Check if all conditions for a score’s dependencies are met.

Args:

score_id: The ID of the score to check dependency_graph: The dependency graph containing conditions results_by_score_id: Dictionary of score results

Returns:

bool: True if all conditions are met, False otherwise

classmethod create_from_yaml(yaml_file_path)

Creates a Scorecard class dynamically from a YAML file.

Args:

yaml_file_path (str): The file path to the YAML file containing the scorecard properties.

Returns:

type: A dynamically created Scorecard class.

static create_instance_from_api_data(scorecard_id: str, api_data: Dict[str, Any], scores_config: List[Dict[str, Any]]) → Scorecard

Create a Scorecard instance directly from API data.

Args:

scorecard_id: The ID of the scorecard api_data: Dictionary containing scorecard metadata from the API scores_config: List of score configurations

Returns:

A Scorecard instance populated with the API data

get_accumulated_costs()

get_model_name(name=None, id=None, key=None)

Return the model name used for a specific score or the scorecard.

get_score_names()

Instance method to get score names. Works for instances created via create_instance_from_api_data.

async get_score_result(*, scorecard, score, text, metadata, modality, results)

Get a result for a score by looking up a Score instance for that score name and calling its predict method with the provided input.

Args:

scorecard (str): The scorecard identifier. score (str): The score identifier. text (str): The text content to analyze. metadata (dict): Additional metadata for the score. modality (str, optional): The modality of the content (e.g., ‘Development’, ‘Production’). results (list): Previous score results that may be needed for dependent scores.

Returns:

Union[List[Score.Result], Score.Result]: Either a single Result object or a list of Result objects. The Result object contains the score value and any associated metadata.

Raises:

BatchProcessingPause: When scoring needs to be suspended for batch processing. ValueError: When required environment variables are missing.

initialize_from_api_data()

Initialize scores from API data and register them in the instance’s score registry.

This method handles score instantiation when data is loaded directly from the API rather than from YAML files.

classmethod initialize_registry()

classmethod load_and_register_scorecards(directory_path)

name()

Returns the name of this scorecard instance.

Returns:

str: The name of the scorecard, from properties if available, otherwise the class name.

classmethod normalize_score_name(score_name)

Normalizes a score name by removing whitespace and non-word characters.

Args:

score_name (str): The original score name.

Returns:

str: A normalized score name suitable for file names or dictionary keys.

async score_entire_text(*, text: str, metadata: dict, modality: str | None = None, subset_of_score_names: List[str] | None = None) → Dict[str, Result]

classmethod score_names()

score_names_to_process()

Instance method for filtering score names that need to be processed directly.

Returns:

list of str: Names of scores that need direct processing.

score_registry = None