Interface EvalConfig

Hierarchy

Properties

evaluatorType agentTools? chainOptions? criteria? distanceMetric? embedding? feedbackKey? llm? prepareData?

Properties

evaluatorType

evaluatorType: keyof EvaluatorType

The name of the evaluator to use. Example: labeled_criteria, criteria, etc.

Optional agentTools

agentTools?: StructuredToolInterface<ZodObject<any, any, any, any, {}>>[]

A list of tools available to the agent, for TrajectoryEvalChain.

Optional chainOptions

chainOptions?: Partial<Omit<LLMEvalChainInput<EvalOutputType, BaseLanguageModelInterface<any, BaseLanguageModelCallOptions>>, "llm">>

Optional criteria

criteria?: CriteriaLike

The criteria to use for the evaluator.

Optional distanceMetric

distanceMetric?: EmbeddingDistanceType

The distance metric to use for comparing the embeddings.

Optional embedding

embedding?: EmbeddingsInterface

The embedding objects to vectorize the outputs.

Optional feedbackKey

feedbackKey?: string

The feedback (or metric) name to use for the logged evaluation results. If none provided, we default to the evaluationName.

Optional llm

llm?: BaseLanguageModelInterface<any, BaseLanguageModelCallOptions>

Optional prepareData

prepareData?: ((data) => {
    prediction: unknown;
    input?: unknown;
    reference?: unknown;
})

Type declaration

    • (data): {
          prediction: unknown;
          input?: unknown;
          reference?: unknown;
      }

    • Convert the evaluation data into a format that can be used by the evaluator. By default, we pass the first value of the run.inputs, run.outputs (predictions), and references (example.outputs) If this is specified, it will override the prepareData function in the RunEvalConfig for this particular evaluator.

      Parameters

      • data: {
            run: Run;
            reference_outputs?: Record<string, unknown>;
        }

        The data to prepare.

        • run: Run
        • Optional reference_outputs?: Record<string, unknown>

      Returns {
          prediction: unknown;
          input?: unknown;
          reference?: unknown;
      }

      The prepared data.

      • prediction: unknown
      • Optional input?: unknown
      • Optional reference?: unknown

Settings

Member Visibility

Theme

On This Page

Generated using TypeDoc