textflint.input_layer.model.flint_model.flint_model

class textflint.input_layer.model.flint_model.flint_model.FlintModel(model, tokenizer, task='SA', batch_size=1)

Bases: abc.ABC

A model wrapper queries a model with a list of text inputs.

Classification-based models return a list of lists, where each sublist represents the model’s scores for a given input.

Text-to-text models return a list of strings, where each string is the output – like a translation or summarization – for a given input.

__init__(model, tokenizer, task='SA', batch_size=1)

Parameters

• model – any model object

• tokenizer – support tokenize sentence and convert tokens to model input ids

• task (str) – task name

• batch_size (int) – batch size to apply evaluation

evaluate(data_samples, prefix='')

Parameters

• data_samples (list[Sample]) – list of Samples

• prefix (str) – name prefix to add to metrics

Returns

dict obj to save metrics result

get_grad(*inputs)

Get gradient of loss with respect to input tokens.

Parameters

inputs (tuple) – tuple of original texts

get_model_grad(*inputs)

Get gradient of loss with respect to input tokens.

Parameters

inputs (tuple) – list of original text

unzip_samples(data_samples)

Unzip sample to input texts and labels.

Parameters

data_samples (list) – sample list

Returns

(inputs_text), labels.

class textflint.input_layer.model.flint_model.flint_model.ABC

Bases: object

Helper class that provides a standard way to create an ABC using inheritance.

textflint.input_layer.model.flint_model.flint_model.Accuracy(y_true, y_pred, *, normalize=True, sample_weight=None)

Accuracy classification score.

In multilabel classification, this function computes subset accuracy: the set of labels predicted for a sample must exactly match the corresponding set of labels in y_true.

Read more in the User Guide.

y_true1d array-like, or label indicator array / sparse matrix

Ground truth (correct) labels.

y_pred1d array-like, or label indicator array / sparse matrix

Predicted labels, as returned by a classifier.

normalizebool, default=True

If False, return the number of correctly classified samples. Otherwise, return the fraction of correctly classified samples.

sample_weightarray-like of shape (n_samples,), default=None

Sample weights.

scorefloat

If normalize == True, return the fraction of correctly classified samples (float), else returns the number of correctly classified samples (int).

The best performance is 1 with normalize == True and the number of samples with normalize == False.

jaccard_score, hamming_loss, zero_one_loss

In binary and multiclass classification, this function is equal to the jaccard_score function.

>>> from sklearn.metrics import accuracy_score
>>> y_pred = [0, 2, 1, 3]
>>> y_true = [0, 1, 2, 3]
>>> accuracy_score(y_true, y_pred)
0.5
>>> accuracy_score(y_true, y_pred, normalize=False)
2

In the multilabel case with binary label indicators:

>>> import numpy as np
>>> accuracy_score(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
0.5