Package opennlp.tools.tokenize

# Class TokenizerFactory

java.lang.Object
opennlp.tools.util.BaseToolFactory
opennlp.tools.tokenize.TokenizerFactory

public class TokenizerFactory extends BaseToolFactory
The factory that provides `Tokenizer` default implementation and resources. Users can extend this class if their application requires overriding the `TokenContextGenerator`, `Dictionary` etc.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `TokenizerFactory()`
  Instantiates a `TokenizerFactory` that provides the default implementation of the resources.
  `TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, Pattern alphaNumericPattern)`
  Instantiates a `TokenizerFactory`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `static TokenizerFactory`
  `create(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, Pattern alphaNumericPattern)`
  Factory method the framework uses instantiate a new `TokenizerFactory`.
  `Map<String,Object>`
  `createArtifactMap()`
  A model's implementation should call this constructor that creates a model programmatically.
  `Map<String,String>`
  `createManifestEntries()`
  `Dictionary`
  `getAbbreviationDictionary()`
  `Pattern`
  `getAlphaNumericPattern()`
  `opennlp.tools.tokenize.TokenContextGenerator`
  `getContextGenerator()`
  `String`
  `getLanguageCode()`
  `boolean`
  `isUseAlphaNumericOptimization()`
  `void`
  `validateArtifactMap()`
  Validates the parsed artifacts.

  ### Methods inherited from class opennlp.tools.util.BaseToolFactory

  `create, create, createArtifactSerializersMap`

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### TokenizerFactory

    public TokenizerFactory()
    Instantiates a `TokenizerFactory` that provides the default implementation of the resources.

  - ### TokenizerFactory

    public TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, Pattern alphaNumericPattern)
    Instantiates a `TokenizerFactory`. Use this constructor to programmatically create a factory.
    Parameters:
      `languageCode` - The ISO language code to be used for this factory.
      `abbreviationDictionary` - The `Dictionary` which holds abbreviations.
      `useAlphaNumericOptimization` - Whether alphanumerics are skipped, or not.
      `alphaNumericPattern` - `null` or a custom alphanumeric `Pattern` (default is: `"^[A-Za-z0-9]+$"`, provided by `Factory.DEFAULT_ALPHANUMERIC`.

- ## Method Details

  - ### validateArtifactMap

    public void validateArtifactMap() throws opennlp.tools.util.InvalidFormatException
    Description copied from class: `BaseToolFactory`
    Validates the parsed artifacts.

    Note: Subclasses should generally invoke `super.validateArtifactMap` at the beginning of this method.

    Specified by:
      `validateArtifactMap` in class `BaseToolFactory`
    Throws:
      `opennlp.tools.util.InvalidFormatException` - Thrown if validation found invalid states.

  - ### createArtifactMap

    public Map<String,Object> createArtifactMap()
    Description copied from class: `BaseToolFactory`
    A model's implementation should call this constructor that creates a model programmatically.

    The base implementation will return a `HashMap` that should be populated by subclasses.

    Overrides:
      `createArtifactMap` in class `BaseToolFactory`
    Returns:
      Retrieves a `Map` with pairs of keys and objects.

  - ### createManifestEntries

    public Map<String,String> createManifestEntries()
    Overrides:
      `createManifestEntries` in class `BaseToolFactory`
    Returns:
      Retrieves the manifest entries to be added to the model manifest.

  - ### create

    public static TokenizerFactory create(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, Pattern alphaNumericPattern) throws opennlp.tools.util.InvalidFormatException
    Factory method the framework uses instantiate a new `TokenizerFactory`.
    Parameters:
      `subclassName` - The name of the class implementing the `TokenizerFactory`.
      `languageCode` - The ISO language code the `Tokenizer` should use.
      `abbreviationDictionary` - An optional `Dictionary` containing abbreviations, or `null` if not present.
      `useAlphaNumericOptimization` - Whether the alphanumeric optimization is be enabled or not.
      `alphaNumericPattern` - The `Pattern` the alphanumeric optimization should use, if enabled.
    Returns:
      A valid `TokenizerFactory` instance.
    Throws:
      `opennlp.tools.util.InvalidFormatException` - Thrown if one of the input parameters doesn't comply the expected format.

  - ### getAlphaNumericPattern

    public Pattern getAlphaNumericPattern()
    Returns:
      Retrieves the (user-)specified alphanumeric `Pattern` or a default.

  - ### isUseAlphaNumericOptimization

    public boolean isUseAlphaNumericOptimization()
    Returns:
      `true` if the alphanumeric optimization is enabled, otherwise `false`.

  - ### getAbbreviationDictionary

    public Dictionary getAbbreviationDictionary()
    Returns:
      The abbreviation `Dictionary` or `null` if none is active.

  - ### getLanguageCode

    public String getLanguageCode()
    Returns:
      Retrieves the ISO language code in use.

  - ### getContextGenerator

    public opennlp.tools.tokenize.TokenContextGenerator getContextGenerator()
    Returns:
      Retrieves a `TokenContextGenerator` instance.