Module: hf_types

HFClassificationInferenceInput

Bases: HFInferenceBaseInput

Input data for classification models

Parameters:

Name	Type	Description	Default
text	str	Text to classify	required

Source code in src/infernet_ml/utils/hf_types.py

class HFClassificationInferenceInput(HFInferenceBaseInput):
    """Input data for classification models

    Args:
        text (str): Text to classify
    """

    task_id: HFTaskId = HFTaskId.TEXT_CLASSIFICATION

    text: str

HFDiffusionInferenceInput

Bases: HFInferenceBaseInput

Input data for diffusion models

Parameters:

Name	Type	Description	Default
prompt	str	Text prompt for image generation	required
negative_prompt	Optional[str]	Negative text prompt for the model	required
height	Optional[int]	Height in pixels of the image to generate. Default 512.	required
width	Optional[int]	Width in pixels of the image to generate. Default 512.	required
num_inference_steps	Optional[int]	Number of denoising steps. More steps --> higher quality but slower inference.	required
guidance_scale	Optional[float]	Guidance scale for the model to control the influence of the prompt on the generated image. Higher values --> more influence of the prompt on the generated image but may lead to lower image quality. Default values are model dependent but usually between 7 and 8.	required

Source code in src/infernet_ml/utils/hf_types.py

class HFDiffusionInferenceInput(HFInferenceBaseInput):
    """Input data for diffusion models

    Args:
        prompt (str): Text prompt for image generation
        negative_prompt (Optional[str]): Negative text prompt for the model
        height (Optional[int]): Height in pixels of the image to generate.
            Default 512.
        width (Optional[int]): Width in pixels of the image to generate.
            Default 512.
        num_inference_steps (Optional[int]): Number of denoising steps.
            More steps --> higher quality but slower inference.
        guidance_scale (Optional[float]): Guidance scale for the model to
            control the influence of the prompt on the generated image.
            Higher values --> more influence of the prompt on the generated
            image but may lead to lower image quality. Default values are
            model dependent but usually between 7 and 8.
    """

    task_id: HFTaskId = HFTaskId.TEXT_TO_IMAGE
    prompt: str
    negative_prompt: Optional[str] = None
    height: Optional[int] = None
    width: Optional[int] = None
    num_inference_steps: Optional[int] = None
    guidance_scale: Optional[float] = None

HFInferenceBaseInput

Bases: BaseModel

Base class for input data

Source code in src/infernet_ml/utils/hf_types.py

class HFInferenceBaseInput(BaseModel):
    """Base class for input data"""

    model_config = ConfigDict(protected_namespaces=())

    model: Optional[str] = None

    task_id: HFTaskId

HFSummarizationConfig

Bases: TypedDict

Summarization model configuration

Parameters:

Name	Type	Description	Default
model	str	Model name	required
max_length	int	Maximum length in tokens of the generated summary	required
min_length	int	Minimum length in tokens of the generated summary	required
top_k	int	Number of top tokens to sample from	required
top_p	float	Cumulative probability for top-k sampling	required
temperature	float	Temperature for sampling. Default 1.0	required
repetition_penalty	float	Repetition penalty for beam search	required
num_return_sequences	int	Number of sequences to return	required
use_cache	bool	Whether to use cache during inference	required

Source code in src/infernet_ml/utils/hf_types.py

class HFSummarizationConfig(TypedDict):
    """Summarization model configuration

    Args:
        model (str): Model name
        max_length (int): Maximum length in tokens of the generated summary
        min_length (int): Minimum length in tokens of the generated summary
        top_k (int): Number of top tokens to sample from
        top_p (float): Cumulative probability for top-k sampling
        temperature (float): Temperature for sampling. Default 1.0
        repetition_penalty (float): Repetition penalty for beam search
        num_return_sequences (int): Number of sequences to return
        use_cache (bool): Whether to use cache during inference
    """

    max_length: NotRequired[int]
    min_length: NotRequired[int]
    top_k: NotRequired[int]
    top_p: NotRequired[float]
    temperature: NotRequired[float]
    repetition_penalty: NotRequired[float]
    max_time: NotRequired[float]

HFSummarizationInferenceInput

Bases: HFInferenceBaseInput

Input data for summarization models

Parameters:

Name	Type	Description	Default
text	str	Text to summarize	required
parameters	Optional[HFSummarizationConfig]	Summarization model	required

Source code in src/infernet_ml/utils/hf_types.py

class HFSummarizationInferenceInput(HFInferenceBaseInput):
    """Input data for summarization models

    Args:
        text (str): Text to summarize
        parameters (Optional[HFSummarizationConfig]): Summarization model
    """

    task_id: HFTaskId = HFTaskId.SUMMARIZATION
    text: str
    parameters: Optional[HFSummarizationConfig] = None

HFTaskId

Bases: IntEnum

Hugging Face task types

Parameters:

Name	Type	Description	Default
UNSET	int	Unset task	required
TEXT_GENERATION	int	Text generation task	required
TEXT_CLASSIFICATION	int	Text classification task	required
TOKEN_CLASSIFICATION	int	Token classification task	required
SUMMARIZATION	int	Summarization task	required
TEXT_TO_IMAGE	int	Text to image task	required

Source code in src/infernet_ml/utils/hf_types.py

class HFTaskId(IntEnum):
    """Hugging Face task types

    Args:
        UNSET (int): Unset task
        TEXT_GENERATION (int): Text generation task
        TEXT_CLASSIFICATION (int): Text classification task
        TOKEN_CLASSIFICATION (int): Token classification task
        SUMMARIZATION (int): Summarization task
        TEXT_TO_IMAGE (int): Text to image task
    """

    UNSET = 0
    TEXT_GENERATION = 1
    TEXT_CLASSIFICATION = 2
    TOKEN_CLASSIFICATION = 3
    SUMMARIZATION = 4
    TEXT_TO_IMAGE = 5

HFTextGenerationInferenceInput

Bases: HFInferenceBaseInput

Input data for text generation models

Parameters:

Name	Type	Description	Default
prompt	str	Prompt for text generation	required
details	bool	Whether to return detailed output (tokens, probabilities, seed, finish reason, etc.)	required
stream	bool	Whether to stream output. Only available for models running with the text-generation-interface backend.	required
do_sample	bool	Whether to use logits sampling	required
max_new_tokens	int	Maximum number of tokens to generate	required
best_of	int	Number of best sequences to generate and return with highest token logprobs	required
repetition_penalty	float	Repetition penalty for greedy decoding. 1.0 is no penalty	required
return_full_text	bool	Whether to preprend the prompt to the generated text	required
seed	int	Random seed for generation sampling	required
stop_sequences	str	Sequence to stop generation if a member of stop_sequences is generated	required
temperature	float	Sampling temperature for logits sampling	required
top_k	int	Number of highest probability vocabulary tokens to keep for top-k sampling	required
top_p	float	If <1, only the most probable tokens with probabilities that add up to top_p or higher are kept for top-p sampling	required
truncate	int	Truncate input to this length if set	required
typical_p	float	Typical decoding mass.	required
watermark	bool	Whether to add a watermark to the generated text Defaults to False.	required
decoder_input_details	bool	Whether to return the decoder input token logprobs and ids. Requires details to be set to True as well. Defaults to False.	required

Source code in src/infernet_ml/utils/hf_types.py

class HFTextGenerationInferenceInput(HFInferenceBaseInput):
    """Input data for text generation models

    Args:
        prompt (str): Prompt for text generation
        details (bool): Whether to return detailed output (tokens, probabilities,
            seed, finish reason, etc.)
        stream (bool): Whether to stream output. Only available for models
            running with the `text-generation-interface` backend.
        do_sample (bool): Whether to use logits sampling
        max_new_tokens (int): Maximum number of tokens to generate
        best_of (int): Number of best sequences to generate and return
            with highest token logprobs
        repetition_penalty (float): Repetition penalty for greedy decoding.
            1.0 is no penalty
        return_full_text (bool): Whether to preprend the prompt to
            the generated text
        seed (int): Random seed for generation sampling
        stop_sequences (str): Sequence to stop generation if a member of
          `stop_sequences` is generated
        temperature (float): Sampling temperature for logits sampling
        top_k (int): Number of highest probability vocabulary tokens to keep for top-k
            sampling
        top_p (float): If <1, only the most probable tokens with probabilities that add
            up to `top_p` or higher are kept for top-p sampling
        truncate (int): Truncate input to this length if set
        typical_p (float): Typical decoding mass.
        watermark (bool): Whether to add a watermark to the generated text
            Defaults to False.
        decoder_input_details (bool): Whether to return the decoder input token
            logprobs and ids. Requires `details` to be set to True as well.
            Defaults to False.

    """

    task_id: HFTaskId = HFTaskId.TEXT_GENERATION
    prompt: str
    details: bool = Field(default=False)
    stream: bool = Field(default=False)
    do_sample: bool = Field(default=False)
    max_new_tokens: int = Field(default=20)
    best_of: Optional[int] = None
    repetition_penalty: Optional[float] = None
    return_full_text: bool = Field(default=False)
    seed: Optional[int] = None
    stop_sequences: Optional[str] = None
    temperature: Optional[float] = None
    top_k: Optional[int] = None
    top_p: Optional[float] = None
    truncate: Optional[int] = None
    typical_p: Optional[float] = None
    watermark: bool = Field(default=False)
    decoder_input_details: bool = Field(default=False)

HFTokenClassificationInferenceInput

Bases: HFInferenceBaseInput

Input data for token classification models

Parameters:

Name	Type	Description	Default
text	str	Text to classify	required

Source code in src/infernet_ml/utils/hf_types.py

class HFTokenClassificationInferenceInput(HFInferenceBaseInput):
    """Input data for token classification models

    Args:
        text (str): Text to classify
    """

    task_id: HFTaskId = HFTaskId.TOKEN_CLASSIFICATION

    text: str

parse_hf_inference_input_from_dict(r)

Parse input data from dictionary

Source code in src/infernet_ml/utils/hf_types.py

def parse_hf_inference_input_from_dict(r: Dict[str, Any]) -> HFInferenceClientInput:
    """Parse input data from dictionary"""
    if r["task_id"] == HFTaskId.TEXT_CLASSIFICATION:
        return HFClassificationInferenceInput(**r)
    if r["task_id"] == HFTaskId.TOKEN_CLASSIFICATION:
        return HFTokenClassificationInferenceInput(**r)
    if r["task_id"] == HFTaskId.TEXT_GENERATION:
        return HFTextGenerationInferenceInput(**r)
    if r["task_id"] == HFTaskId.SUMMARIZATION:
        return HFSummarizationInferenceInput(**r)
    raise ValueError(f"Unknown task_id: {r['task_id']}")