textflint.input_layer.component.sample.ner_sample

NER Sample class to hold the necessary info and provide atomic operations.

class textflint.input_layer.component.sample.ner_sample.NERSample(data, origin=None, sample_id=None, mode='BIO')

Bases: textflint.input_layer.component.sample.sample.Sample

NER Sample class to hold the necessary info and provide atomic operations.

__init__(data, origin=None, sample_id=None, mode='BIO')

Parameters

• data (dict) – The dict obj that contains data info

• origin (~BaseSample) – Original sample obj

• sample_id (int) – the id of sample

• mode (str) – The sequence labeling mode for NER samples.

check_data(data)

Check rare data format.

Parameters

data (dict) – rare data input.

load(data)

Parse data into sample field value.

Parameters

data (dict) – rare data input.

dump()

Convert sample info to input data json format.

Return json

the dict of sentences and labels

delete_field_at_indices(field, indices)

Delete tokens and their NER tag.

Parameters

• field (str) – field str

• indices (list) –

  list of int/list/slice shape：indices_num each index can be int indicate delete single item or

  their list like [1, 2, 3],

  can be list like (0,3) indicate replace items

  from 0 to 3(not included),

  can be slice which would be convert to list

Returns

Modified NERSample.

delete_field_at_index(field, index)

Delete tokens and their NER tag.

Parameters

• field (str) – field string, normally ‘x’

• index (int|list|slice) –

  int/list/slice can be int indicate delete single item

  or their list like [1, 2, 3],

  can be list like (0,3) indicate replace items

  from 0 to 3(not included),

  can be slice which would be convert to list

Returns

Modified NERSample

insert_field_before_indices(field, indices, items)

Insert tokens and ner tags.Assuming the tag of new_item is O.

:param str field:field string :param list indices: list of int

shape：indices_num, list like [1, 2, 3]

Parameters

items (list) – list of str/list shape: indices_num, correspond to indices

Returns

Modified NERSample

insert_field_before_index(field, ins_index, new_item)

Insert tokens and ner tags.Assuming the tag of new_item is O

Parameters

• field (str) – field str

• ins_index (int) – indicate which index to insert items

• new_item (str/list) – items to insert

Returns

Modified NERSample

insert_field_after_indices(field, indices, items)

Insert tokens and ner tags.Assuming the tag of new_item is O.

Parameters

• field (str) – field string

• indices (list) – list of int shape：indices_num, like [1, 2, 3]

• items (list) – list of str/list shape: indices_num, correspond to indices

Returns

Modified NERSample

insert_field_after_index(field, ins_index, new_item)

Insert tokens and ner tags.Assuming the tag of new_item is O.

Parameters

• field (str) – field string

• ins_index (int) – indicate where to apply insert

• new_item (str|list) – shape: indices_num, correspond to field_sub_items

Returns

Modified NERSample

find_entities_BIO(word_seq, tag_seq)

find entities in a sentence with BIO labels.

Parameters

• word_seq (list) – a list of tokens representing a sentence

• tag_seq (list) – a list of tags representing a tag sequence labeling the sentence

Return list entity_in_seq

a list of entities found in the sequence, including the information of the start position & end position in the sentence, the category, and the entity itself.

find_entities_BIOES(word_seq, tag_seq)

find entities in a sentence with BIOES labels.

Parameters

• word_seq (list) – a list of tokens representing a sentence

• tag_seq (list) – a list of tags representing a tag sequence labeling the sentence

Return list entity_in_seq

a list of entities found in the sequence, including the information of the start position & end position in the sentence, the category, and the entity itself.

entities_replace(entities_info, candidates)

Replace multi entity in once time.Assume input entities with reversed sequential.

Parameters

• entities_info (list) – list of entity_info

• candidates (list) – candidate entities

Returns

Modified NERSample

entity_replace(start, end, entity, label)

Replace one entity and update entities info.

Parameters

• start (int) – the start position of the entity to be replaced

• end (int) – the end position of the entity to be replaced

• entity (str) – the entity to be replaced with

• label (str) – the category of the entity

Returns

Modified NERSample

class textflint.input_layer.component.sample.ner_sample.ListField(field_value, **kwargs)

Bases: textflint.input_layer.component.field.field.Field

A helper class that represents input list values that to be modified.

Operations which modify field_value would generate new Field instance.

__init__(field_value, **kwargs)

Parameters

field_value ([str]) – The list that ListField represents.

replace_at_indices(indices, new_items)

Replace items at indices.

Notice: just support isometric replace.

Parameters

• indices (list[int|list|slice]) – each index can be int indicate replace single item or their list like [1, 2, 3]. each index can be list like (0,3) indicate replace items from 0 to 3(not included) or their list like [(0, 3), (5,6)] each index can be slice which would be convert to list.

• new_items (list) – items corresponding indices.

Returns

new field object.

replace_at_index(index, new_items)

Replace item at index.

Parameters

• index (int|list|slice) –

  can be int indicate replace single item or their list like [1, 2, 3] can be list like (0,3) indicate replace items from 0

  to 3(not included) or their list like [(0, 3), (5,6)]

  can be slice which would be convert to list.

• new_items (list) – items corresponding index.

Returns

new field object.

delete_at_indices(indices)

Delete items at indices.

Parameters

indices (list[int|list|slice]) – each index can be int indicate delete single item or their list like [1, 2, 3]. each index can be list like (0,3) indicate replace items from 0 to 3(not included) or their list like [(0, 3), (5,6)] each index can be slice which would be convert to list.

Returns

new field object.

delete_at_index(index)

Delete item at index.

Parameters

index (int|list|slice) –

can be int indicate delete single item or their list like [1, 2, 3] can be list like (0,3) indicate replace items from 0

to 3(not included) or their list like [(0, 3), (5,6)]

can be slice which would be convert to list.

Returns

new field object.

insert_before_indices(indices, new_items)

Insert items before indices.

Parameters

• indices (list[int|list|slice]) – each index can be int indicate insert single item or their list like [1, 2, 3]. each index can be list like (0,3) indicate replace items from 0 to 3(not included) or their list like [(0, 3), (5,6)] each index can be slice which would be convert to list.

• new_items (list) – items corresponding indices.

Returns

new field object.

insert_before_index(index, new_items)

Insert items before index.

Parameters

• index (int|list|slice) –

  can be int indicate insert single item or their list like [1, 2, 3] can be list like (0,3) indicate replace items from 0

  to 3(not included) or their list like [(0, 3), (5,6)]

  can be slice which would be convert to list.

• new_items (list) – items corresponding index.

Returns

new field object.

insert_after_indices(indices, new_items)

Insert item after index.

Parameters

• indices (list[int|list|slice]) – each index can be int indicate insert single item or their list like [1, 2, 3]. each index can be list like (0,3) indicate replace items from 0 to 3(not included) or their list like [(0, 3), (5,6)] each index can be slice which would be convert to list.

• new_items (list) – items corresponding indices.

Returns

new field object.

insert_after_index(index, new_items)

Insert item after index.

Parameters

• index (int|list|slice) –

  can be int indicate insert single item or their list like [1, 2, 3] can be list like (0,3) indicate replace items from 0

  to 3(not included) or their list like [(0, 3), (5,6)]

  can be slice which would be convert to list.

• new_items (list) – items corresponding index

Returns

new field object.

swap_at_index(first_index, second_index)

Swap item between first_index and second_index.

Parameters

• first_index (int) – index of first item

• second_index (int) – index of second item

Returns

new field object.

class textflint.input_layer.component.sample.ner_sample.Sample(data, origin=None, sample_id=None)

Bases: abc.ABC

Base Sample class to hold the necessary info and provide atomic operations

text_processor = <textflint.common.preprocess.en_processor.EnProcessor object>

__init__(data, origin=None, sample_id=None)

Parameters

• data (dict) – The dict obj that contains data info.

• origin (sample) – original sample obj.

• sample_id (int) – sampleindex

get_value(field)

Get field value by field_str.

Parameters

field (str) – field name

Returns

field value

get_words(field)

Get tokenized words of given textfield

Parameters

field (str) – field name

Returns

tokenized words

get_text(field)

Get text string of given textfield

Parameters

field (str) – field name

Return string

text

get_mask(field)

Get word masks of given textfield

Parameters

field (str) – field name

Returns

list of mask values

get_sentences(field)

Get split sentences of given textfield

Parameters

field (str) – field name

Returns

list of sentences

get_pos(field)

Get text field pos tags. :param str field: field name :return: pos tag list

get_ner(field)

Get text field ner tags

Parameters

field (str) – field name

Returns

ner tag list

replace_fields(fields, field_values, field_masks=None)

Fully replace multi fields at the same time and return new sample. Notice: Not suggest use this API as it will set mask values of TextField to MODIFIED_MASK.

Parameters

• fields (list) – field str list

• field_values (list) – field value list

• field_masks (list) – indicate mask values, useful for printable text

Returns

Modified Sample

replace_field(field, field_value, field_mask=None)

Fully replace single field and return new sample. Notice: Not suggest use this API as it will set mask values of TextField to MODIFIED_MASK.

Parameters

• field (str) – field str

• field_value – field_type

• field_mask (list) – indicate mask value of field

Returns

Modified Sample

replace_field_at_indices(field, indices, items)

Replace items of multi given scopes of field value at the same time. Stay away from the complex function !!!

Be careful of your input list shape.

Parameters

• field (str) – field name

• of int|list|slice indices (list) –

  each index can be int indicate replace single item or their list

  like [1, 2, 3],

  can be list like (0,3) indicate replace items from

  0 to 3(not included),

  can be slice which would be convert to list.

• items –

Returns

Modified Sample

replace_field_at_index(field, index, items)

Replace items of given scope of field value.

Be careful of your input list shape.

Parameters

• field (str) – field name

• index (int|list|slice) –

  can be int indicate replace single item or list like [1, 2, 3], can be list like (0,3) indicate replace items

  from 0 to 3(not included),

  can be slice which would be convert to list.

• items (str|list) – shape: indices_num, correspond to field_sub_items

Returns

Modified Sample

unequal_replace_field_at_indices(field, indices, rep_items)

Replace scope items of field value with rep_items which may not equal with scope.

Parameters

• field – field str

• indices – list of int/tupe/list

• rep_items – list

Returns

Modified Sample

delete_field_at_indices(field, indices)

Delete items of given scopes of field value.

Parameters

• field (str) – field name

• of int|list|slice indices (list) –

  shape：indices_num each index can be int indicate delete single item or their list

  like [1, 2, 3],

  can be list like (0,3) indicate replace items

  from 0 to 3(not included),

  can be slice which would be convert to list.

Returns

Modified Sample

delete_field_at_index(field, index)

Delete items of given scopes of field value.

Parameters

• field (str) – field value

• index (int|list|slice) –

  can be int indicate delete single item or their list like [1, 2, 3], can be list like (0,3) indicate replace items

  from 0 to 3(not included),

  can be slice which would be convert to list.

Returns

Modified Sample

insert_field_before_indices(field, indices, items)

Insert items of multi given scopes before indices of field value at the same time.

Stay away from the complex function !!! Be careful of your input list shape.

Parameters

• field (str) – field name

• indices – list of int, shape：indices_num, list like [1, 2, 3]

• items – list of str/list, shape: indices_num, correspond to indices

Returns

Modified Sample

insert_field_before_index(field, index, items)

Insert items of multi given scope before index of field value.

Parameters

• field (str) – field name

• index (int) – indicate which index to insert items

• items (str|list) – items to insert

Returns

Modified Sample

insert_field_after_indices(field, indices, items)

Insert items of multi given scopes after indices of field value at the same time.

Stay away from the complex function !!! Be careful of your input list shape.

Parameters

• field (str) – field name

• indices – list of int, shape：indices_num, like [1, 2, 3]

• items – list of str/list shape: indices_num, correspond to indices

Returns

Modified Sample

insert_field_after_index(field, index, items)

Insert items of multi given scope after index of field value

Parameters

• field (str) – field name

• index (int) – indicate where to apply insert

• items (str|list) – shape: indices_num, correspond to field_sub_items

Returns

Modified Sample

swap_field_at_index(field, first_index, second_index)

Swap items between first_index and second_index of field value.

Parameters

• field (str) – field name

• first_index (int) –

• second_index (int) –

Returns

Modified Sample

abstract check_data(data)

Check rare data format

Parameters

data – rare data input

Returns

abstract load(data)

Parse data into sample field value.

Parameters

data – rare data input

abstract dump()

Convert sample info to input data json format.

Returns

dict object.

classmethod clone(original_sample)

Deep copy self to a new sample

Parameters

original_sample – sample to be copied

Returns

Sample instance

property is_origin

Return whether the sample is original Sample.

class textflint.input_layer.component.sample.ner_sample.TextField(field_value, mask=None, is_one_sent=False, split_by_space=False, **kwargs)

Bases: textflint.input_layer.component.field.field.Field

A helper class that represents input string that to be modified.

Text that Sample contains parsed in data set, TextField provides multiple methods for Sample to modify.

Support sentence level and word level modification, default using word level API.

text_processor = <textflint.common.preprocess.en_processor.EnProcessor object>

__init__(field_value, mask=None, is_one_sent=False, split_by_space=False, **kwargs)

Parameters

• field_value (str|list) – Sentence string or tokenized words.

• mask (list) – list of mask values

• is_one_sent (bool) – whether input is a sentence

• split_by_space (boo) – whether tokenize sentence by split space

• kwargs –

pos_of_word_index(desired_word_idx)

Get pos tag of given index.

Parameters

desired_word_idx (int) – desire index to get pos tag

Returns

pos tag of word of desired_word_idx.

replace_at_indices(indices, new_items)

Replace words at indices and set their mask to MODIFIED_MASK.

Parameters

• indices ([int|listslice]) –

  each index can be int indicate replace single item

  or their list like [1, 2, 3].

  each index can be list like (0,3) indicate replace items

  from 0 to 3(not included) or their list like [(0, 3), (5,6)]

  each index can be slice which would be convert to list.

• new_items ([str|list|tuple]) – items corresponding indices.

Returns

Replaced TextField object.

replace_at_index(index, new_items)

Replace words at indices and set their mask to MODIFIED_MASK.

Parameters

• index (intlistslice) –

  can be int indicate replace single item or their list like [1, 2, 3] can be list like (0,3) indicate replace items

  from 0 to 3(not included) or their list like [(0, 3), (5,6)]

  can be slice which would be convert to list.

• new_items (str|listtuple) – items corresponding index.

Returns

Replaced TextField object.

delete_at_indices(indices)

Delete words at indices and remove their mask value.

Parameters

indices ([int|list|slice]) –

each index can be int indicate replace single item

or their list like [1, 2, 3].

each index can be list like (0,3) indicate replace items

from 0 to 3(not included) or their list like [(0, 3), (5,6)]

each index can be slice which would be convert to list.

Returns

Modified TextField object.

delete_at_index(index)

Delete words at index and remove their mask value.

Parameters

index (int|list|slice) –

can be int indicate replace single item or their list like [1, 2, 3] can be list like (0,3) indicate replace items

from 0 to 3(not included) or their list like [(0, 3), (5,6)]

can be slice which would be convert to list.

Returns

Modified TextField object.

insert_before_indices(indices, new_items)

Insert words before indices.

Parameters

• indices ([int]) –

  can be int indicate replace single item or their list like [1, 2, 3] can be list like (0,3) indicate replace items

  from 0 to 3(not included) or their list like [(0, 3), (5,6)]

  can be slice which would be convert to list.

• new_items ([str|list|tuple]) – items corresponding index.

Returns

new TextField object.

insert_before_index(index, new_items)

Insert words before index and remove their mask value.

Parameters

• index (int) –

  can be int indicate replace single item or their list like [1, 2, 3] can be list like (0,3) indicate replace items

  from 0 to 3(not included) or their list like [(0, 3), (5,6)]

  can be slice which would be convert to list.

• new_items (str|list|tuple) – items corresponding index.

Returns

new TextField object.

insert_after_indices(indices, new_items)

Insert words after indices.

Parameters

• indices ([int]) –

  can be int indicate replace single item or their list like [1, 2, 3] can be list like (0,3) indicate replace items

  from 0 to 3(not included) or their list like [(0, 3), (5,6)]

  can be slice which would be convert to list.

• new_items ([str|list|tuple]) – items corresponding index.

Returns

new TextField object.

insert_after_index(index, new_items)

Insert words before index and remove their mask value.

Parameters

• index (int) –

  can be int indicate replace single item or their list like [1, 2, 3] can be list like (0,3) indicate replace items

  from 0 to 3(not included) or their list like [(0, 3), (5,6)]

  can be slice which would be convert to list.

• new_items (str|list|tuple) – items corresponding index.

Returns

new TextField object.

swap_at_index(first_index, second_index)

Swap items between first_index and second_index of origin_list

Parameters

• first_index (int) – index of first item

• second_index (int) – index of second item

Returns

Modified TextField object.

property pos_tagging

Get POS tags.

Example:

given sentence 'All things in their being are good for something.'

>> [('All', 'DT'),
    ('things', 'NNS'),
    ('in', 'IN'),
    ('their', 'PRP$'),
    ('being', 'VBG'),
    ('are', 'VBP'),
    ('good', 'JJ'),
    ('for', 'IN'),
    ('something', 'NN'),
    ('.', '.')]

Returns

Tokenized tokens with their POS tags.

property ner

Get NER tags.

Example:

given sentence 'Lionel Messi is a football player from Argentina.'

>>[('Lionel Messi', 0, 2, 'PERSON'),
   ('Argentina', 7, 8, 'LOCATION')]

Returns

A list of tuples, (entity, start, end, label)

property dependency_parsing

Dependency parsing.

Example:

given sentence: 'The quick brown fox jumps over the lazy dog.'

>>
    The     DT      4       det
    quick   JJ      4       amod
    brown   JJ      4       amod
    fox     NN      5       nsubj
    jumps   VBZ     0       root
    over    IN      9       case
    the     DT      9       det
    lazy    JJ      9       amod
    dog     NN      5       obl

Returns

A list of tuples, (token, pos, target, type)

textflint.input_layer.component.sample.ner_sample.get_align_seq(align_items, value)

Get values which shape align with align items.

Parameters

• align_items (list) –

• value (str) –

Return list

list which align with align_items.