Package opennlp.tools.tokenize

# Class TokenizerModel

java.lang.Object
opennlp.tools.util.model.BaseModel
opennlp.tools.tokenize.TokenizerModel

All Implemented Interfaces:
  `Serializable`, `opennlp.tools.util.model.ArtifactProvider`

public final class TokenizerModel extends BaseModel
The `TokenizerModel` is the model used by a learnable `Tokenizer`.

See Also:
  - `TokenizerME`
  - `TokenizerFactory`
  - Serialized Form

- ## Field Summary

  ### Fields inherited from class opennlp.tools.util.model.BaseModel

  `TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `TokenizerModel(File modelFile)`
  Initializes a `TokenizerModel` instance via a valid `File`.
  `TokenizerModel(InputStream in)`
  Initializes a `TokenizerModel` instance via a valid `InputStream`.
  `TokenizerModel(URL modelURL)`
  Initializes a `TokenizerModel` instance via a valid `URL`.
  `TokenizerModel(Path modelPath)`
  Initializes a `TokenizerModel` instance via a valid `Path`.
  `TokenizerModel(opennlp.tools.ml.model.MaxentModel tokenizerModel, Map<String,String> manifestInfoEntries, TokenizerFactory tokenizerFactory)`
  Initializes a `TokenizerModel` instance via a `MaxentModel` and related resources.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `boolean`
  `equals(Object obj)`
  `Dictionary`
  `getAbbreviations()`
  `TokenizerFactory`
  `getFactory()`
  `opennlp.tools.ml.model.MaxentModel`
  `getMaxentModel()`
  `int`
  `hashCode()`
  `boolean`
  `useAlphaNumericOptimization()`

  ### Methods inherited from class opennlp.tools.util.model.BaseModel

  `getArtifact, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serialize, serialize, serialize`

  ### Methods inherited from class java.lang.Object

  `getClass, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### TokenizerModel

    public TokenizerModel(opennlp.tools.ml.model.MaxentModel tokenizerModel, Map<String,String> manifestInfoEntries, TokenizerFactory tokenizerFactory)
    Initializes a `TokenizerModel` instance via a `MaxentModel` and related resources.
    Parameters:
      `tokenizerModel` - The `model` to be used.
      `manifestInfoEntries` - Additional information kept in the manifest.
      `tokenizerFactory` - The `TokenizerFactory` to be used internally.

  - ### TokenizerModel

    public TokenizerModel(InputStream in) throws IOException
    Initializes a `TokenizerModel` instance via a valid `InputStream`.
    Parameters:
      `in` - The `InputStream` used for loading the model.
    Throws:
      `IOException` - Thrown if IO errors occurred during initialization.

  - ### TokenizerModel

    public TokenizerModel(File modelFile) throws IOException
    Initializes a `TokenizerModel` instance via a valid `File`.
    Parameters:
      `modelFile` - The `File` used for loading the model.
    Throws:
      `IOException` - Thrown if IO errors occurred during initialization.

  - ### TokenizerModel

    public TokenizerModel(Path modelPath) throws IOException
    Initializes a `TokenizerModel` instance via a valid `Path`.
    Parameters:
      `modelPath` - The `Path` used for loading the model.
    Throws:
      `IOException` - Thrown if IO errors occurred during initialization.

  - ### TokenizerModel

    public TokenizerModel(URL modelURL) throws IOException
    Initializes a `TokenizerModel` instance via a valid `URL`.
    Parameters:
      `modelURL` - The `URL` used for loading the model.
    Throws:
      `IOException` - Thrown if IO errors occurred during initialization.

- ## Method Details

  - ### getFactory

    public TokenizerFactory getFactory()
    Returns:
      Retrieves the active `TokenizerFactory`.

  - ### getMaxentModel

    public opennlp.tools.ml.model.MaxentModel getMaxentModel()
    Returns:
      Retrieves the model as `MaxentModel` instance.

  - ### getAbbreviations

    public Dictionary getAbbreviations()
    Returns:
      Retrieves the active abbreviation `Dictionary`.

  - ### useAlphaNumericOptimization

    public boolean useAlphaNumericOptimization()
    Returns:
      `true` if alphanumeric optimization is active, `false` otherwise.

  - ### hashCode

    public int hashCode()
    Overrides:
      `hashCode` in class `Object`

  - ### equals

    public boolean equals(Object obj)
    Overrides:
      `equals` in class `Object`