Skip to main content

scorebook.trismik_services.adaptive_testing_service

Trismik adaptive testing service integration.

run_adaptive_evaluation​

async def run_adaptive_evaluation(inference: Callable,
                                  adaptive_run_spec: AdaptiveEvalRunSpec,
                                  experiment_id: str, project_id: str,
                                  metadata: Any) -> AdaptiveEvalRunResult

Run an adaptive evaluation using the Trismik API.

Arguments:

  • inference - Function to run inference
  • adaptive_run_spec - Specification for the adaptive evaluation
  • experiment_id - Experiment identifier
  • project_id - Trismik project ID
  • metadata - Additional metadata

Returns:

Results from the adaptive evaluation

make_trismik_inference​

def make_trismik_inference(inference_function: Callable,
                           return_list: bool = False) -> Callable[[Any], Any]

Wrap an inference function for flexible input handling.

Takes a function expecting list[dict] and makes it accept single dict or TrismikMultipleChoiceTextItem.