ragrank.metric

ragrank.metric.base

Base module for metric

Base class for metrics module

class ragrank.metric.BaseMetric(*, metric_type: MetricType, llm: BaseLLM, prompt: Prompt)

Base class for defining metrics.

metric_type

The type of the metric.

Type:: MetricType

llm

The language model associated with the metric.

Type:: BaseLLM

prompt

The prompt associated with the metric.

Type:: Prompt

load() → None: Method to load the metric. Not implemented in base class.

model_computed_fields: ClassVar[dict[str, ComputedFieldInfo]] = {}: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ConfigDict = {'arbitrary_types_allowed': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[dict[str, FieldInfo]] = {'llm': FieldInfo(annotation=BaseLLM, required=True, description='The language model associated with the metric.'), 'metric_type': FieldInfo(annotation=MetricType, required=True, description='The type of the metric.'), 'prompt': FieldInfo(annotation=Prompt, required=True, description='The prompt associated with the metric.')}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].

This replaces Model.__fields__ from Pydantic V1.

abstract property name: str

Get the name for the metric.

Returns:: The name of the metric.
Return type:: str

save() → None: Method to save the metric. Not implemented in base class.

abstract score(data: DataNode) → MetricResult

Method to compute the metric score.

Parameters:: data (DataNode) – The data node for which the score is computed.
Returns:: The computed score.
Return type:: MetricResult

class ragrank.metric.CustomInstruct(*, metric_type: MetricType = None, llm: BaseLLM = None, prompt: Prompt = None, config: InstructConfig)

A custom metric for evaluating responses based on instructions.

llm

The language model used to generate the response.

Type:: BaseLLM

prompt

The prompt provided for generating the response.

Type:: Prompt

config

Instructions and configuration for the metric.

Type:: InstructConfig

name(self) → str: Get the name for the metric.

score(self, data: DataNode) -> MetricResult: Calculate the score for the custom metric based on the provided data.

_reason(self, data: DataNode, score: float) -> Optional[str]: Determine the reason for the given score. (Not implemented yet)

model_computed_fields: ClassVar[dict[str, ComputedFieldInfo]] = {}: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ConfigDict = {'arbitrary_types_allowed': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[dict[str, FieldInfo]] = {'config': FieldInfo(annotation=InstructConfig, required=True, description='Instructions for the Custom Instruct Metric'), 'llm': FieldInfo(annotation=BaseLLM, required=False, default_factory=<lambda>, description='The language model used to generate the response.'), 'metric_type': FieldInfo(annotation=MetricType, required=False, default_factory=<lambda>, description='The type of metric, which is non-binary.'), 'prompt': FieldInfo(annotation=Prompt, required=False, default_factory=<lambda>, description='The prompt provided for generating the response')}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].

This replaces Model.__fields__ from Pydantic V1.

model_post_init(_CustomInstruct__context: Any) → None: Post-initialization hook to configure the metric.

property name: str

Get the name for the metric.

Returns:: The name of the metric.
Return type:: str

score(data: DataNode) → MetricResult

Calculate the score for the custom metric based on the provided data.

Parameters:: data (DataNode) – The input data to be used for scoring.
Returns:: The result of the metric calculation.
Return type:: MetricResult

class ragrank.metric.CustomMetric(*, metric_type: MetricType, llm: BaseLLM = None, prompt: Prompt = None)

A custom metric for evaluating responses.

llm

The language model used to generate the response.

Type:: BaseLLM

prompt

The prompt provided for generating the response.

Type:: Prompt

name(self) → str: Get the name for the metric.

score(self, data: DataNode) -> MetricResult: Calculate the score for the custom metric based on

the provided data.

metric_score(self, data: DataNode) -> float: Calculate the score for the custom metric based on

the provided data.

reason(self, data: DataNode, score: float) -> Optional[str]: Determine the reason for the given score.

abstract property metric_name: str

Name of the specific Metric

Returns:: The name of the actual metric.
Return type:: str

abstract metric_score(data: DataNode) → float

Calculate the actual score for the custom metric based on the provided data.

Parameters:: data (DataNode) – The input data to be used for scoring.
Returns:: The result score of the metric
Return type:: Float

model_computed_fields: ClassVar[dict[str, ComputedFieldInfo]] = {}: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ConfigDict = {'arbitrary_types_allowed': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[dict[str, FieldInfo]] = {'llm': FieldInfo(annotation=BaseLLM, required=False, default_factory=<lambda>, description='The language model used to generate the response.'), 'metric_type': FieldInfo(annotation=MetricType, required=True, description='The type of the metric.'), 'prompt': FieldInfo(annotation=Prompt, required=False, default_factory=<lambda>, description='The prompt provisded for generating the response')}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].

This replaces Model.__fields__ from Pydantic V1.

property name: str

Get the name for the metric.

Returns:: The name of the metric.
Return type:: str

score(data: DataNode) → MetricResult

Calculate the score for the custom metric based on the provided data.

Parameters:: data (DataNode) – The input data to be used for scoring.
Returns:: The result of the metric calculation.
Return type:: MetricResult

class ragrank.metric.InstructConfig(*, metric_type: MetricType, name: str, instructions: str, examples: List[Dict[str, Any]] = None, input_fields: List[str], output_field: str = 'output')

Configuration settings for the CustomInstruct metric.

metric_type

The type of the metric.

Type:: MetricType

name

The name of the Instruct metric.

Type:: str

instructions

Instructions for the metric.

Type:: str

examples

Examples for the Instruction metric.

Type:: List[PromptExample]

input_fields

Input fields for the Instruct Metric.

Type:: List[str]

output_field

Output field for the Instruct Metric.

Type:: str

to_prompt() → Prompt: get a Prompt object from the given configuration.

model_computed_fields: ClassVar[dict[str, ComputedFieldInfo]] = {}: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ConfigDict = {'arbitrary_types_allowed': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[dict[str, FieldInfo]] = {'examples': FieldInfo(annotation=List[Dict[str, Any]], required=False, default_factory=list, description='Examples for the Instruction metric'), 'input_fields': FieldInfo(annotation=List[str], required=True, description='Input fields for the Instruct Metric'), 'instructions': FieldInfo(annotation=str, required=True, description='Instructions for the metric'), 'metric_type': FieldInfo(annotation=MetricType, required=True, description='The type of the metric'), 'name': FieldInfo(annotation=str, required=True, description='The name of the Instruct metric'), 'output_field': FieldInfo(annotation=str, required=False, default='output', description='ouput field for the Instruct Metric')}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].

This replaces Model.__fields__ from Pydantic V1.

to_prompt() → Prompt

Convert the configuration to a Prompt object.

Returns:: The generated prompt object.
Return type:: Prompt

class ragrank.metric.MetricResult(*, datanode: DataNode, metric: BaseMetric, score: float | int, reason: str | None = None, process_time: float | None = None)

Class to hold the result of a metric computation.

datanode

The data node associated with the metric result.

Type:: DataNode

metrics

List of metrics used in the computation.

Type:: List[BaseMetric]

scores

List of scores computed for each metric.

Type:: List[Union[int, float]]

reasons

List of reasons corresponding to each metric score. Defaults to None.

Type:: List[str]

process_time

Processing time for the computation. Defaults to None.

Type:: Optional[float], optional

model_computed_fields: ClassVar[dict[str, ComputedFieldInfo]] = {}: A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ConfigDict = {'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[dict[str, FieldInfo]] = {'datanode': FieldInfo(annotation=DataNode, required=True, description='The data node associated with the metric result.'), 'metric': FieldInfo(annotation=BaseMetric, required=True, description='List of metrics used in the computation.'), 'process_time': FieldInfo(annotation=Union[float, NoneType], required=False, description='Processing time for the computation.', repr=False), 'reason': FieldInfo(annotation=Union[str, NoneType], required=False, description='List of reasons corresponding to each metric score.'), 'score': FieldInfo(annotation=Union[float, int], required=True, description='List of scores computed for each metric.')}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].

This replaces Model.__fields__ from Pydantic V1.

ragrank.metric.base