Class: NGramTokenizer

nanosearch~NGramTokenizer(length) → {this}

An n-gram tokenizer. Takes a word & generates a list of tokens to index.

Constructor

new NGramTokenizer(length) → {this}

Creates a new tokenizer.
Parameters:
Name Type Description
length int The minimum length of the n-grams.
Source:
Returns:
Type
this

Methods

tokenize(word) → {array}

Processes a word into a list of tokens.
Parameters:
Name Type Description
word string The word to process.
Source:
Returns:
Type
array