FlintModel

FlintModel is a necessary input to apply adversarial attack or generate robustness report. textflint allows practitioners to customize target model, practitioners just need to wrap their own models through FlintModel and implement the corresponding interfaces. Thanks to TextAttack for integrating various attack methods.

import sys
sys.path.append('/Users/wangxiao/code/python/RobustnessTool/TextRobustness/textrobustness')

How to customize targe model

• You need provide the tokenizer object corresponding to the model, which is used to process the sample into the input of the model. This may include tokenize the text, and convert tokens to ids.

• You need to provide your own model object to support the prediction function of the model.

e.g.

from textflint.model.flint_model.torch_model import TorchModel
from textflint.model.test_model.textcnn_torch_model import TextCNNTorchModel
from textflint.model.test_model.glove_embedding import GloveEmbedding
from textflint.model.tokenizers.glove_tokenizer import GloveTokenizer

class TextCNNTorch(TorchModel):
    r"""
    Model wrapper for TextCnn implemented by pytorch.

    """
    def __init__(self):
        glove_embedding = GloveEmbedding()
        word2id = glove_embedding.word2id

        super().__init__(
            model=TextCNNTorchModel(
                init_embedding=glove_embedding.embedding
            ),
            task='SA',
            tokenizer=GloveTokenizer(
                word_id_map=word2id,
                unk_token_id=glove_embedding.oovid,
                pad_token_id=glove_embedding.padid,
                max_length=30
            )
        )
        self.label2id = {"positive": 0, "negative": 1}

How to implement the automatic evaluation

For testing the robustness of the model, users can test the generated samples through their own code, not necessarily using FlintModel. FlintModel provides verification metrics for most tasks, and its verification results can be directly used as input for subsequent report generation.

Users have to implement two functions. * unzip_samples( ) function, which accept batch samples as input, and return (batch input features, batch labels), input features can directly pass to **call( )** to predict, while labels can be used to calculate metrics. * **call( )** function, which accept batch input features as input and predict target label .

e.g.

    def unzip_samples(self, data_samples):
        r"""
        Unzip sample to input texts and labels.

        :param list[Sample] data_samples: list of Samples
        :return: (inputs_text), labels.

        """
        x = []
        y = []

        for sample in data_samples:
            x.append(sample['x'])
            y.append(self.label2id[sample['y']])

        return [x], y

    def __call__(self, batch_texts):
        r"""
        Tokenize text, convert tokens to id and run the model.

        :param batch_texts: (batch_size,) batch text input
        :return: numpy.array()

        """
        model_device = next(self.model.parameters()).device
        inputs_ids = [self.encode(batch_text) for batch_text in batch_texts]
        ids = torch.tensor(inputs_ids).to(model_device)

        return self.model(ids).detach().cpu().numpy()

        def encode(self, inputs):
        r"""
        Tokenize inputs and convert it to ids.

        :param inputs: model original input
        :return: list of inputs ids

        """
        return self.tokenizer.encode(inputs)

How to implement adversarial attack

FlintModel is a necessary input for the generation of adversarial attack samples. Since textflint just support apply attack to four tasks, including ‘SA’, ‘SM’, ‘NLI’ and ‘TC’.

Users have to implement two functions. * unzip_samples( ) function, which accept batch samples as input, and return (batch input features, batch labels). * get_model_grad( ) function, which accept input features as input, and return gradient of loss with respect to input tokens.

e.g.

def get_model_grad(self, text_inputs, loss_fn=CrossEntropyLoss()):
        r"""
        Get gradient of loss with respect to input tokens.

        :param str|[str] text_inputs: input string or input string list
        :param torch.nn.Module loss_fn: loss function.
            Default is `torch.nn.CrossEntropyLoss`
        :return: Dict of ids, tokens, and gradient as numpy array.

        """
        if not hasattr(self.model, "get_input_embeddings"):
            raise AttributeError(
                f"{type(self.model)} must have method `get_input_embeddings` "
                f"that returns `torch.nn.Embedding` object that represents "
                f"input embedding layer"
            )

        if not isinstance(loss_fn, torch.nn.Module):
            raise ValueError("Loss function must be of type `torch.nn.Module`.")

        self.model.train()

        embedding_layer = self.model.get_input_embeddings()
        original_state = embedding_layer.weight.requires_grad
        embedding_layer.weight.requires_grad = True

        emb_grads = []

        def grad_hook(module, grad_in, grad_out):
            emb_grads.append(grad_out[0])

        emb_hook = embedding_layer.register_backward_hook(grad_hook)
        self.model.zero_grad()
        model_device = next(self.model.parameters()).device

        inputs_ids = self.encode(text_inputs)
        ids = [torch.tensor(ids).to(model_device) for ids in inputs_ids]

        predictions = self.model(text_inputs)

        output = predictions.argmax(dim=1)
        loss = loss_fn(predictions, output)
        loss.backward()

        # grad w.r.t to word embeddings
        grad = torch.transpose(emb_grads[0], 0, 1)[0].cpu().numpy()

        embedding_layer.weight.requires_grad = original_state
        emb_hook.remove()
        self.model.eval()

        output = {"ids": ids[0].tolist(), "gradient": grad}

        return output

textflint provides a base class for PyTorch model which has implemented get_model_grad( ) function. Take TextCNN’s pytorch implementation as an example, and give a complete FlintModel example implementation.

import torch

from textflint.model.flint_model.torch_model import TorchModel
from textflint.model.test_model.textcnn_torch_model import TextCNNTorchModel
from textflint.model.test_model.glove_embedding import GloveEmbedding
from textflint.model.tokenizers.glove_tokenizer import GloveTokenizer


class TextCNNTorch(TorchModel):
    r"""
    Model wrapper for TextCnn implemented by pytorch.

    """
    def __init__(self):
        glove_embedding = GloveEmbedding()
        word2id = glove_embedding.word2id

        super().__init__(
            model=TextCNNTorchModel(
                init_embedding=glove_embedding.embedding
            ),
            task='SA',
            tokenizer=GloveTokenizer(
                word_id_map=word2id,
                unk_token_id=glove_embedding.oovid,
                pad_token_id=glove_embedding.padid,
                max_length=30
            )
        )
        self.label2id = {"positive": 0, "negative": 1}

    def __call__(self, batch_texts):
        r"""
        Tokenize text, convert tokens to id and run the model.

        :param batch_texts: (batch_size,) batch text input
        :return: numpy.array()

        """
        model_device = next(self.model.parameters()).device
        inputs_ids = [self.encode(batch_text) for batch_text in batch_texts]
        ids = torch.tensor(inputs_ids).to(model_device)

        return self.model(ids).detach().cpu().numpy()

    def encode(self, inputs):
        r"""
        Tokenize inputs and convert it to ids.

        :param inputs: model original input
        :return: list of inputs ids

        """
        return self.tokenizer.encode(inputs)

    def unzip_samples(self, data_samples):
        r"""
        Unzip sample to input texts and labels.

        :param list[Sample] data_samples: list of Samples
        :return: (inputs_text), labels.

        """
        x = []
        y = []

        for sample in data_samples:
            x.append(sample['x'])
            y.append(self.label2id[sample['y']])

        return [x], y