textflint.report_layer.analyzer.analyzer

class textflint.report_layer.analyzer.analyzer.Analyzer

Bases: object

Convert evaluate result json to DataFrame for report generator, and analysis model robustness according to linguistic classification.

Example:

{
    "model_name": "BERT",
    "dataset_name": "medical data",
    "transformation": {
        "Case": {
            "ori_precision": 0.70,
            "trans_precision": 0.65,
            "ori_f1": 0.63,
            "trans_f1": 0.60,
            "size": 5000,
        },
        "Ocr": {
            "ori_precision": 0.72,
            "trans_precision": 0.43,
            "ori_f1": 0.62,
            "trans_f1": 0.41,
            "size": 5000,
        }
    },
    "subpopulation": {
        "LengthLengthSubPopulation-0.0-0.1": {
            "trans_precision": 0.68,
            "trans_f1": 0.63,
            "size": 500
        }
    },
    "attack": {
        "Bert-Attack": {
            "ori_precision": 0.72,
            "trans_precision": 0.43,
            "ori_f1": 0.62,
            "trans_f1": 0.41,
            "size": 400,
        }
    }
}

static json_to_bar_chart(evaluate_json)

Parsing evaluate json and convert to bar chart input format.

Parameters

evaluate_json (dict) – evaluate result of specific model.

Returns

pandas.DataFrame, list[ReportColumn]

static json_to_sunburst(evaluate_json, metric=None)

Parsing evaluate json and classify each transformation.

Parameters

• evaluate_json (dict) – evaluate result of specific model.

• metric (str) – key metric to plot subburst figure.

Returns

pandas.DataFrame, dict

static get_metric(transformations, metric=None)

Get key metric of given transformations.

Parameters

• transformations (dict) – evaluation result of transformation

• metric (str) – key metric to plot subburst figure.

Returns

str legal metric name

static get_parent(transformation_str)

Find linguistic classification of given transformation, if not found, return Other label.

Parameters

transformation_str (str) – transformation name

Returns

str linguistic classification name

static json_to_linguistic_radar(evaluate_json)

Parsing evaluation result and calculate linguistic robustness scores.

Parameters

evaluate_json (dict) – evaluate result of specific model.

Returns

pandas.DataFrame

static radar_score(trans_json)

Get radar score by calculate average metric decreasing ratio.

Parameters

trans_json (dict) – evaluation result of specific transformation.

Returns

pandas.DataFrame

static get_metrics(trans_json)

Parsing and checking evaluation result of specific transformation.

Parameters

trans_json (dict) – evaluation result.

Returns

dict, dict

class textflint.report_layer.analyzer.analyzer.ReportColumn(title)

Bases: object

A single column in the Robustness Report.

class textflint.report_layer.analyzer.analyzer.ScoreColumn(title, min_val, max_val, is_0_to_1=False)

Bases: textflint.report_layer.analyzer.analyzer.ReportColumn

A column for numeric scores in the Robustness Report, displayed as a bar chart.

class textflint.report_layer.analyzer.analyzer.NumericColumn(title)

Bases: textflint.report_layer.analyzer.analyzer.ReportColumn

A column for numeric data in the Robustness Report, displayed as the raw value.

class textflint.report_layer.analyzer.analyzer.OrderedDict

Bases: dict

Dictionary that remembers insertion order

__init__(*args, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

clear() → None.  Remove all items from od.

popitem(last=True)

Remove and return a (key, value) pair from the dictionary.

Pairs are returned in LIFO order if last is true or FIFO order if false.

move_to_end(key, last=True)

Move an existing element to the end (or beginning if last is false).

Raise KeyError if the element does not exist.

pop(k[, d]) → v, remove specified key and return the corresponding

value. If key is not found, d is returned if given, otherwise KeyError is raised.

setdefault(key, default=None)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

copy() → a shallow copy of od

fromkeys(value=None)

Create a new ordered dictionary with keys from iterable and values set to value.

textflint.report_layer.analyzer.analyzer.deepcopy(x, memo=None, _nil=[])

Deep copy operation on arbitrary Python objects.

See the module’s __doc__ string for more info.

textflint.report_layer.analyzer.analyzer.reduce(function, sequence[, initial]) → value

Apply a function of two arguments cumulatively to the items of a sequence, from left to right, so as to reduce the sequence to a single value. For example, reduce(lambda x, y: x+y, [1, 2, 3, 4, 5]) calculates ((((1+2)+3)+4)+5). If initial is present, it is placed before the items of the sequence in the calculation, and serves as a default when the sequence is empty.