Package opennlp.tools.tokenize

# Class TokenSampleStream

java.lang.Object
opennlp.tools.util.FilterObjectStream<String,TokenSample>
opennlp.tools.tokenize.TokenSampleStream

All Implemented Interfaces:
  `AutoCloseable`, `ObjectStream<TokenSample>`

public class TokenSampleStream extends FilterObjectStream<String,TokenSample>
This class is a `stream filter` which reads in string encoded samples and creates `samples` out of them. The input string sample is tokenized if a whitespace or the special separator chars occur.

Sample:
"token1 token2 token3<SPLIT>token4"
The tokens `token1` and `token2` are separated by a whitespace, `token3` and `token4` are separated by the special character sequence. In this case, the default split sequence applies.

Note: The sequence must be unique in the input string and is not escaped.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `TokenSampleStream(ObjectStream<String> sentences)`
  Initializes a `instance`.
  `TokenSampleStream(ObjectStream<String> samples, String separatorChars)`
  Initializes a `instance`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `TokenSample`
  `read()`
  Returns the next `ObjectStream` object.

  ### Methods inherited from class opennlp.tools.util.FilterObjectStream

  `close, reset`

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### TokenSampleStream

    public TokenSampleStream(ObjectStream<String> samples, String separatorChars)
    Initializes a `instance`.
    Parameters:
      `samples` - A plain text `line stream`. Must not be `null`.
      `separatorChars` - The characters to be considered separators. See `TokenSample.DEFAULT_SEPARATOR_CHARS`. Must not be `null`.

  - ### TokenSampleStream

    public TokenSampleStream(ObjectStream<String> sentences)
    Initializes a `instance`.
    Parameters:
      `sentences` - A plain text `line stream`. Must not be `null`.

- ## Method Details

  - ### read

    public TokenSample read() throws IOException
    Description copied from interface: `ObjectStream`
    Returns the next `ObjectStream` object. Calling this method repeatedly until it returns `null` will return each object from the underlying source exactly once.
    Returns:
      The next object or `null` to signal that the stream is exhausted.
    Throws:
      `IOException` - Thrown if there is an error during reading.