primus.data package

The Primus data packages are for loading and caching data for use in training and evaluating scores. The abstract base class DataCache defines the interface for the various data cache implementations. Each should override the parameters to define whatever parameters it needs, and then implement the load_data method to load a dataframe from the cache.

Submodules

• primus.data.AWSDataLakeCache module
  • AWSDataLakeCache
    • AWSDataLakeCache.Parameters
      • AWSDataLakeCache.Parameters.local_cache_directory
      • AWSDataLakeCache.Parameters.model_config
    • AWSDataLakeCache.__init__()
    • AWSDataLakeCache.download_content_item()
    • AWSDataLakeCache.execute_athena_query()
    • AWSDataLakeCache.execute_batch_athena_queries()
    • AWSDataLakeCache.get_query_results()
    • AWSDataLakeCache.load_dataframe()
    • AWSDataLakeCache.process_content_item()
    • AWSDataLakeCache.split_into_batches()
• primus.data.DataCache module
  • DataCache
    • DataCache.Parameters
      • DataCache.Parameters.class_name
      • DataCache.Parameters.model_config
    • DataCache.__init__()
    • DataCache.debug_dataframe()
    • DataCache.format_value_for_display()
    • DataCache.load_dataframe()
    • DataCache.log_validation_errors()
    • DataCache.upsert_item_for_dataset_row()
• primus.data.FeedbackItems module
  • FeedbackItems
    • FeedbackItems.LABEL_SOURCE_FINAL
    • FeedbackItems.LABEL_SOURCE_SCORE_RESULT_OR_IMPORTED
    • FeedbackItems.LABEL_SOURCE_UNRESOLVED
    • FeedbackItems.LABEL_SOURCE_VETTED
    • FeedbackItems.Parameters
      • FeedbackItems.Parameters.backfill_cells
      • FeedbackItems.Parameters.cache_file
      • FeedbackItems.Parameters.column_mappings
      • FeedbackItems.Parameters.days
      • FeedbackItems.Parameters.days_must_be_positive()
      • FeedbackItems.Parameters.feedback_end_at
      • FeedbackItems.Parameters.feedback_id
      • FeedbackItems.Parameters.feedback_item_ids
      • FeedbackItems.Parameters.feedback_item_ids_must_be_nonempty_and_unique()
      • FeedbackItems.Parameters.feedback_start_at
      • FeedbackItems.Parameters.final_value
      • FeedbackItems.Parameters.identifier_extractor
      • FeedbackItems.Parameters.initial_value
      • FeedbackItems.Parameters.item_config
      • FeedbackItems.Parameters.limit
      • FeedbackItems.Parameters.limit_must_be_positive()
      • FeedbackItems.Parameters.limit_per_cell
      • FeedbackItems.Parameters.limit_per_cell_must_be_positive()
      • FeedbackItems.Parameters.local_cache_directory
      • FeedbackItems.Parameters.max_items
      • FeedbackItems.Parameters.max_items_must_be_positive()
      • FeedbackItems.Parameters.model_config
      • FeedbackItems.Parameters.sample_seed
      • FeedbackItems.Parameters.sample_seed_only_for_random()
      • FeedbackItems.Parameters.sampling_mode
      • FeedbackItems.Parameters.sampling_mode_must_be_supported()
      • FeedbackItems.Parameters.score
      • FeedbackItems.Parameters.scorecard
    • FeedbackItems.__init__()
    • FeedbackItems.load_dataframe()
• primus.data.HuggingFaceDataCache module
  • HuggingFaceDataCache
    • HuggingFaceDataCache.Parameters
      • HuggingFaceDataCache.Parameters.model_config
      • HuggingFaceDataCache.Parameters.name
    • HuggingFaceDataCache.__init__()
    • HuggingFaceDataCache.analyze_dataset()
    • HuggingFaceDataCache.load_dataframe()
    • HuggingFaceDataCache.verify_dataset()
• primus.data.ScorecardExampleReferenceItems module
  • ScorecardExampleReferenceItems
    • ScorecardExampleReferenceItems.Parameters
      • ScorecardExampleReferenceItems.Parameters.column_mappings
      • ScorecardExampleReferenceItems.Parameters.days
      • ScorecardExampleReferenceItems.Parameters.days_must_be_positive()
      • ScorecardExampleReferenceItems.Parameters.item_config
      • ScorecardExampleReferenceItems.Parameters.limit
      • ScorecardExampleReferenceItems.Parameters.limit_must_be_positive()
      • ScorecardExampleReferenceItems.Parameters.model_config
      • ScorecardExampleReferenceItems.Parameters.score
      • ScorecardExampleReferenceItems.Parameters.scorecard
    • ScorecardExampleReferenceItems.__init__()
    • ScorecardExampleReferenceItems.load_dataframe()
• primus.data.TestDataCache module
  • TestDataCache
    • TestDataCache.Parameters
      • TestDataCache.Parameters.columns
      • TestDataCache.Parameters.model_config
      • TestDataCache.Parameters.num_rows
    • TestDataCache.__init__()
    • TestDataCache.load_dataframe()
• primus.data.session_manager module
  • SessionManager
    • SessionManager.__init__()
    • SessionManager.get_session()
    • SessionManager.renew_session()
• primus.data.test_data_cache_logging module
  • test_debug_dataframe_logs_schema_without_row_payloads()
  • test_score_data_logs_schema_without_sample_rows()
• primus.data.test_feedback_items module
  • test_cache_directory_uses_runtime_environment()
  • test_cache_identifier_with_filter_values()
  • test_cache_methods()
  • test_case_insensitive_normalization()
  • test_column_mappings_applied_to_dataset()
  • test_column_mappings_case_sensitivity()
  • test_column_mappings_parameter()
  • test_column_mappings_with_empty_dataset()
  • test_create_dataset_rows_comment_logic()
  • test_create_dataset_rows_fails_when_item_metadata_missing()
  • test_create_dataset_rows_handles_missing_edit_comment()
  • test_create_dataset_rows_skips_unresolved_labels_and_reports_ids()
  • test_create_dataset_rows_structure()
  • test_create_dataset_rows_with_data()
  • test_exact_feedback_item_ids_are_unique_and_preserve_requested_order()
  • test_exact_feedback_item_ids_fail_closed_when_membership_cannot_be_materialized()
  • test_fetch_feedback_items_for_scores_excludes_invalid_items()
  • test_initial_value_and_final_value_parameters()
  • test_parameter_defaults()
  • test_parameter_validation()
  • test_reference_label_resolution_imported_and_unresolved()
  • test_reference_label_resolution_priority()
• primus.data.test_feedback_sampling module
  • MockFeedbackItem
    • MockFeedbackItem.__init__()
  • MockFeedbackItems
  • TestFeedbackItemsSampling
    • TestFeedbackItemsSampling.setUp()
    • TestFeedbackItemsSampling.test_edge_case_empty_items()
    • TestFeedbackItemsSampling.test_edge_case_limit_zero()
    • TestFeedbackItemsSampling.test_scenario_1_fewer_items_than_limit()
    • TestFeedbackItemsSampling.test_scenario_2_prioritize_edit_comments_with_room_for_others()
    • TestFeedbackItemsSampling.test_scenario_3_more_items_with_comments_than_limit()
    • TestFeedbackItemsSampling.test_scenario_4_no_items_with_comments()
    • TestFeedbackItemsSampling.test_scenario_5_all_items_have_comments()
  • main()