API Reference

TextDirectory

Main module.

class textdirectory.textdirectory.AggregationState(aggregation: list[int], applied_filters: list[str])

A saved aggregation state (checkpoint).

Behaves like the [aggregation, applied_filters] list used before 0.4.1, so state[0] and state[1] keep working.

aggregation: list[int]

Alias for field number 0

applied_filters: list[str]

Alias for field number 1

class textdirectory.textdirectory.TextDirectory(directory: str | Path, encoding: str = 'utf8', autoload: bool = False, disable_tqdm: bool = False)

aggregate_to_file(filename: str | Path = 'aggregated.txt') → None

Parameters:

filename (str) – the path/filename to write to

aggregate_to_memory() → str

Returns:

a string containing the aggregated text files

Type:

str

aggregation: list[int]

aggregation_states: list[AggregationState]

applied_filters: list[str]

clear_transformation() → None

Destage all transformations and clear memory.

destage_transformation(transformation: list[Any]) → None

Parameters:

transformation (list) – the transformation that should be de-staged and its parameters

filenames: list[str]

files: list[dict[str, Any]]

filter() → Callable[[...], Any]

A wrapper for filters.

filter_by_chars_outliers(sigmas: int = 2) → tuple[float, float, float, float]

Parameters:

sigmas (int) – The number of stds that qualifies an outlier.

Human_name:

Character outliers

filter_by_contains(contains: str) → None

Parameters:

contains (str) – A string that needs to be present in the file

Human_name:

Contains string

filter_by_filename_contains(contains: str) → None

Parameters:

contains (str) – A string that needs to be present in the filename

Human_name:

Filename contains string

filter_by_filename_not_contains(not_contains: str) → None

Parameters:

not_contains (str) – A string that needs not to be present in the filename

Human_name:

Filename does not contain string

filter_by_filenames(filenames: list[str]) → None

Parameters:

filenames (list) – A list of filenames to include

filter_by_max_chars(max_chars: int = 100) → None

Parameters:

max_chars (int) – the maximum number of characters a file can have

Human_name:

Maximum characters

filter_by_max_filesize(max_kb: int = 100) → None

Parameters:

max_mb (int) – The maximum number of kB a file is allowed to have.

Human_name:

Maximum filesize

filter_by_max_tokens(max_tokens: int = 100) → None

Parameters:

max_tokens (int) – the maximum number of tokens a file can have

Human_name:

Maximum tokens

filter_by_min_chars(min_chars: int = 100) → None

Parameters:

min_chars (int) – the minimum number of characters a file can have

Human_name:

Minimum characters

filter_by_min_filesize(min_kb: int = 10) → None

Parameters:

max_mb (int) – The minimum number of kB a file is allowed to have.

Human_name:

Minimum Filesize

filter_by_min_tokens(min_tokens: int = 1) → None

Parameters:

min_tokens (int) – the minimum number of tokens a file can have

Human_name:

Minimum tokens

filter_by_not_contains(not_contains: str) → None

Parameters:

not_contains (str) – A string that is not allowed to be present in the file

Human_name:

Does not contain string

filter_by_random_sampling(n: int | str, replace: bool = False) → None

Parameters:

• n (int) – the number of documents in the sample

• replace (bool) – Should valued be replaced

Human_name:

Random sampling

filter_by_similar_documents(reference_file: str | Path, threshold: float = 0.8) → None

Parameters:

• reference_file (str) – Path to the reference file

• threshold (float) – A value between 0.0 and 1.0 indicating the max. difference between the file and the reference.

Human_name:

Similar documents

filter_by_type_token_ratio(min_ttr: float = 0.0, max_ttr: float = 1.0) → None

Parameters:

• min_ttr (float) – The minimum TTR

• max_ttr (float) – The maximum TTR

Human_name:

Type-Token Ratio

get_aggregation() → Iterator[dict[str, Any]]

A generator that provides the current aggregation.

get_file_length(path: Path) → int

Parameters:

path – path to a textfile

Returns:

the files length in characters

get_file_tokens(path: Path) → int

Parameters:

path – path to a textfile

Returns:

the files length in tokens

get_text(file_id: int) → str

Parameters:

file_id – the file_id in files

Returns:

the (transformed) text of the given file

load_aggregation_state(state: int = 0) → None

Parameters:

state (int) – the state to go back to

load_files(recursive: bool = True, sort: bool = True, filetype: str = 'txt', fast: bool = False, skip_checkpoint: bool = False) → None

Parameters:

• recursive (bool) – recursive search

• sort (bool) – sort the files by name

• filetype (str) – filetype to look for (e.g. txt)

• fast (bool) – load files faster without getting metadata

print_aggregation() → None

Print the aggregated files as a table.

print_pipeline() → None

Print the current pipeline.

print_saved_states() → None

Print all saved states.

run_filters(filters: list[Any]) → None

Parameters:

filters (list) – A list of tuples with filters and their arguments.

run_transformations(text: str) → str

Parameters:

text (str) – the text to run staged transformations on

Returns:

the transformed text

save_aggregation_state() → None

Saves the current self.aggregation state.

set_aggregation(aggregation: list[dict[str, Any]]) → None

Set the aggregation.

stage_transformation(transformation: list[Any]) → None

Parameters:

transformation (list) – the transformation that should be staged and its parameters

staged_transformations: list[list[Any]]

transform_to_files(output_directory: str | Path) → None

Runs all transformations and stores the transformed texts in individual files.

Parameters:

output_directory (str) – the path/filename to write to

transform_to_memory() → None

Runs all transformations and stores the transformed texts in memory.

Transformations

Transformation module.

textdirectory.transformations.transformation_crude_spellchecker(text: str, language_model: str = 'crudesc_lm_en', *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_eebop4_to_plaintext(text: str, *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_expand_english_contractions(text: str, *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_ftfy(text: str, *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_lemmatize(text: str, spacy_model: str = 'en_core_web_sm', *args: Any) → str

Parameters:

• text (str) – the text to run the transformation on

• spacy_model (str) – the spaCy model we want to use

Returns:

the transformed text

Human_name:

Lemmatizer

textdirectory.transformations.transformation_lowercase(text: str, *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_postag(text: str, spacy_model: str = 'en_core_web_sm', *args: Any) → str

Parameters:

• text (str) – the text to run the transformation on

• spacy_model (str) – the spaCy model we want to use

Returns:

the transformed text

Human_name:

Add pos-tags

textdirectory.transformations.transformation_remove_htmltags(text: str, *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_remove_nl(text: str, *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_remove_non_alphanumerical(text: str, *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_remove_non_ascii(text: str, *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_remove_stopwords(text: str, stopwords_source: str = 'internal', stopwords: str = 'en', spacy_model: str = 'en_core_web_sm', custom_stopwords: str | None = None, *args: Any) → str

Parameters:

• text (str) – the text to run the transformation on

• stopwords_source (str) – [internal, file] where are stopwords loaded from

• stopwords (str) – filename of a list containing stopwords

• spacy_model (str) – the spaCy model we want to use

• custom_stopwords (str) – a comma separated list of additional stopwords to consider:

Returns:

the transformed text

textdirectory.transformations.transformation_remove_weird_tokens(text: str, spacy_model: str = 'en_core_web_sm', remove_double_space: bool = False, *args: Any) → str

Parameters:

• text (str) – the text to run the transformation on

• spacy_model (str) – the spaCy model we want to use

• remove_double_space – remove duplicated spaces

Type:

remove_double_space: bool

Returns:

the transformed text

textdirectory.transformations.transformation_replace_digits(text: str, replacement_character: str = '%', *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_replace_string(text: str, replace: str, replace_with: str, *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_to_leetspeak(text: str, *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_uppercase(text: str, *args: Any) → str

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

textdirectory.transformations.transformation_usas_en_semtag(text: str, *args: Any) → str

Tag the text semantically using the web version of the UCREL USAS tagger.

This uploads the text to a third-party server operated by Lancaster University. Do not use it with confidential, personal, or licensed data.

Parameters:

text (str) – the text to run the transformation on

Returns:

the transformed text

Helpers

Helpers module.

textdirectory.helpers.chunk_text(string: str, chunk_size: int = 50000) → list[str]

Parameters:

• string (str) – a string

• chunk_size (int) – the max characters of one chunk

Returns:

a list of chunks

textdirectory.helpers.coerce_args_by_signature(func: Callable[[...], Any], args: list[Any]) → list[Any]

Coerce string arguments to the types suggested by a function’s signature.

String values are converted to int, float, or bool when the corresponding parameter’s annotation or default value indicates such a type; all other values are passed through unchanged.

Parameters:

• func (callable) – the function or method the arguments are meant for

• args (list) – the arguments to coerce

Returns:

a list of coerced arguments

textdirectory.helpers.count_non_alphanum(string: str) → int

Parameters:

string (str) – a string

Returns:

the number of non-alphanumeric characters in the string

textdirectory.helpers.estimate_spacy_max_length(override: float | bool = False, tokenizer_only: bool = False) → float

Returns a somewhat sensible suggestions for max_length.

textdirectory.helpers.get_available_filters(get_human_name: bool = False) → list[Any]

Parameters:

get_human_name (bool) – if True, also return the ‘human name’

Returns:

a list of functions; if get_human_name a list of tuples

textdirectory.helpers.get_available_transformations(get_human_name: bool = False) → list[Any]

Parameters:

get_human_name – if True, also return the ‘human name’

Returns:

a list of functions; if get_human_name a list of tuples

textdirectory.helpers.get_human_from_docstring(doc: str) → dict[str, str]

Parameters:

doc (string) – if True, also return the ‘human name’

Returns:

a dictionary of name_* keys/values from the docstring.

textdirectory.helpers.simple_tokenizer(string: str, regular_expression: str = '\\w+') → list[str]

Parameters:

string (str) – a string

Returns:

a list of tokens

textdirectory.helpers.tabulate_flat_list_of_dicts(list_of_dicts: list[dict[str, Any]], max_length: int = 25) → str

Parameters:

• list_of_dicts (list) – a list of dictionaries; each list is a row

• max_length (int) – the maximum length of a cell

Returns:

a table

textdirectory.helpers.type_token_ratio(text: str) → float

Returns a simple rounded type-token ratio of a text.

CrudeSpellChecker

Spellchecker module.

textdirectory.crudespellchecker.AVAILABLE_LANGUAGE_MODELS = frozenset({'crudesc_lm_ame', 'crudesc_lm_amehistorical', 'crudesc_lm_en'})

The language models shipped with the package. The models are pickled, so only these names are accepted: an arbitrary name would allow loading (and executing) any file.

class textdirectory.crudespellchecker.CrudeSpellChecker(caching: bool = True, language_model: str = 'crudesc_lm_en')

A very simple and crude spellchecker based on Peter Norvig’s design. Simple Language Models: crudesc_lm_en.gz.lm English based on COCA (sample), OANC (written), BNC crudesc_lm_ame.lm American English based on COCA (sample) and OANC (written) crudesc_lm_amehistorical.lm American English based on COHA (sample)

cache: dict[str, str]

candidates(word: str) → set[str] | list[str]

Parameters:

word (str) – a word

Returns:

a list of candidates

correct_string(string: str, return_corrections: bool = False) → Any

Parameters:

• string (str) – the string to correct.

• return_corrections (bool) – include the corrections in the result

Returns:

the corrected string

correction(word: str) → str

Parameters:

word (str) – a word

Returns:

most probable spelling correction for word

edit_distance_1(word: str) → set[str]

Parameters:

word (str) – a word

Returns:

all edits one edit away from the word

edit_distance_2(word: str) → Iterator[str]

Parameters:

word (str) – a word

Returns:

all edits two edits away from the word

frequencies: Counter[str]

known(words: Iterable[str]) → set[str]

Parameters:

word (str) – a word

Returns:

a subset of words in the dictionary of frequencies

p_word(word: str) → float

Parameters:

word (str) – a word

textdirectory.crudespellchecker.generate_crudespellchecker_lm(corpus_directory: str | Path, model_name: str, strip_xml: bool = False) → None

Parameters:

• corpus_directory (str) – path the folder containing the files.

• model_name (str) – th name of the model

• strip_xml (bool) – stripping XML tags with bs4

CLI

Console script for textdirectory.