plexus.cli.tuning.operations module

plexus.cli.tuning.operations.append_feedback_to_conversation(conversation_history, feedback_message)

Append feedback message to conversation history as a user message.

Args:

conversation_history (list): Existing conversation messages feedback_message (str): Feedback message to append

Returns:

list: Updated conversation history with feedback appended

plexus.cli.tuning.operations.contains_disagreement(completion_text)

Check if the completion contains a disagreement with the gold standard.

Args:

completion_text (str): The generated completion text

Returns:

bool: True if the completion indicates disagreement

plexus.cli.tuning.operations.create_hallucination_feedback(verification_result, original_transcript)

Create a feedback message for the LLM about hallucinated quotes.

Args:

verification_result (dict): Result from verify_quotes_in_completion original_transcript (str): The original transcript text

Returns:

str: Feedback message for the LLM

plexus.cli.tuning.operations.create_no_quotes_feedback()

Create a feedback message instructing the LLM to generate a completion without any quotes. This is used as a last-ditch effort to save examples that repeatedly fail quote verification.

Returns:

str: Feedback message instructing no quotes usage

plexus.cli.tuning.operations.generate_llm_completion(score_instance, row, completion_template, conversation_history=None)

Generate completion using an LLM with full context and gold standard labels.

Args:

score_instance: The score instance containing configuration row: The data row containing transcript and other information completion_template: The Jinja2 template with label interpolation for the LLM conversation_history: Optional list of previous messages for iterative improvement

Returns:

tuple: (completion_text, updated_conversation_history)

plexus.cli.tuning.operations.get_file_path(output_dir, file_type)

plexus.cli.tuning.operations.get_id_file_path(output_dir, file_type)

plexus.cli.tuning.operations.get_output_dir(scorecard_name, score_name, subsampled=False, max_tokens=None)

plexus.cli.tuning.operations.verify_quotes_in_completion(completion_text, original_transcript, fuzzy_threshold=0.8, enable_fuzzy=True, debug=False)

Verify that all quotes in the completion exist in the original transcript.

Args:

completion_text (str): The generated completion text original_transcript (str): The original transcript text fuzzy_threshold (float): Threshold for fuzzy matching (0.0-1.0) enable_fuzzy (bool): Whether to enable fuzzy matching debug (bool): Whether to enable detailed debug logging

Returns:

dict: Contains verification results with keys:

• verified_quotes: list of quotes found in transcript

• hallucinated_quotes: list of quotes NOT found in transcript

• fuzzy_matched_quotes: list of quotes matched with fuzzy logic

• is_valid: boolean indicating if all quotes are valid

• verification_details: detailed information about each quote