# Uses of Class
opennlp.tools.tokenize.TokenSample

Packages that use TokenSample
Package
Description
opennlp.tools.cmdline.tokenizer
opennlp.tools.formats
Experimental package related to converting various corpora to OpenNLP Format.
opennlp.tools.formats.ad
Experimental package related to the `Arvores Deitadas corpus` format.
opennlp.tools.formats.conllu
Experimental package related to the CoNNL-U format.
opennlp.tools.formats.convert
opennlp.tools.formats.irishsentencebank
Experimental package related to the Irish Sentence Bank format.
opennlp.tools.formats.masc
Experimental package related to the `MASC` corpus format.
opennlp.tools.tokenize
Contains classes related to finding token or words in a string.
opennlp.tools.tokenize.lang.en

- ## Uses of TokenSample in opennlp.tools.cmdline.tokenizer

  Methods in opennlp.tools.cmdline.tokenizer with parameters of type TokenSample
  Modifier and Type
  Method
  Description
  `void`
  DetokenEvaluationErrorListener.`misclassified(TokenSample reference, TokenSample prediction)`
  `void`
  TokenEvaluationErrorListener.`misclassified(TokenSample reference, TokenSample prediction)`

- ## Uses of TokenSample in opennlp.tools.formats

  Methods in opennlp.tools.formats that return types with arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `ObjectStream<TokenSample>`
  ConllXTokenSampleStreamFactory.`create(String[] args)`
  `ObjectStream<TokenSample>`
  TokenSampleStreamFactory.`create(String[] args)`

- ## Uses of TokenSample in opennlp.tools.formats.ad

  Methods in opennlp.tools.formats.ad that return types with arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `ObjectStream<TokenSample>`
  ADTokenSampleStreamFactory.`create(String[] args)`

- ## Uses of TokenSample in opennlp.tools.formats.conllu

  Methods in opennlp.tools.formats.conllu that return TokenSample
  Modifier and Type
  Method
  Description
  `TokenSample`
  ConlluTokenSampleStream.`read()`
  Methods in opennlp.tools.formats.conllu that return types with arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `ObjectStream<TokenSample>`
  ConlluTokenSampleStreamFactory.`create(String[] args)`

- ## Uses of TokenSample in opennlp.tools.formats.convert

  Methods in opennlp.tools.formats.convert that return TokenSample
  Modifier and Type
  Method
  Description
  `TokenSample`
  NameToTokenSampleStream.`read()`
  `TokenSample`
  POSToTokenSampleStream.`read()`
  Methods in opennlp.tools.formats.convert that return types with arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `ObjectStream<TokenSample>`
  NameToTokenSampleStreamFactory.`create(String[] args)`
  `ObjectStream<TokenSample>`
  ParseToTokenSampleStreamFactory.`create(String[] args)`
  `ObjectStream<TokenSample>`
  POSToTokenSampleStreamFactory.`create(String[] args)`

- ## Uses of TokenSample in opennlp.tools.formats.irishsentencebank

  Methods in opennlp.tools.formats.irishsentencebank that return TokenSample
  Modifier and Type
  Method
  Description
  `TokenSample`
  IrishSentenceBankDocument.IrishSentenceBankSentence.`getTokenSample()`
  Methods in opennlp.tools.formats.irishsentencebank that return types with arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `ObjectStream<TokenSample>`
  IrishSentenceBankTokenSampleStreamFactory.`create(String[] args)`

- ## Uses of TokenSample in opennlp.tools.formats.masc

  Methods in opennlp.tools.formats.masc that return TokenSample
  Modifier and Type
  Method
  Description
  `TokenSample`
  MascTokenSampleStream.`read()`
  Methods in opennlp.tools.formats.masc that return types with arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `ObjectStream<TokenSample>`
  MascTokenSampleStreamFactory.`create(String[] args)`

- ## Uses of TokenSample in opennlp.tools.tokenize

  Methods in opennlp.tools.tokenize that return TokenSample
  Modifier and Type
  Method
  Description
  `static TokenSample`
  TokenSample.`parse(String sampleString, String separatorChars)`
  Parses a string sample.
  `protected TokenSample`
  DetokenizerEvaluator.`processSample(TokenSample reference)`
  `protected TokenSample`
  TokenizerEvaluator.`processSample(TokenSample reference)`
  `TokenSample`
  TokenizerStream.`read()`
  `TokenSample`
  TokenSampleStream.`read()`
  Methods in opennlp.tools.tokenize with parameters of type TokenSample
  Modifier and Type
  Method
  Description
  `protected Iterator<Event>`
  TokSpanEventStream.`createEvents(TokenSample tokenSample)`
  Adds training events to the event stream for each of the specified `sample`.
  `protected TokenSample`
  DetokenizerEvaluator.`processSample(TokenSample reference)`
  `protected TokenSample`
  TokenizerEvaluator.`processSample(TokenSample reference)`
  Method parameters in opennlp.tools.tokenize with type arguments of type TokenSample
  Modifier and Type
  Method
  Description
  `void`
  TokenizerCrossValidator.`evaluate(ObjectStream<TokenSample> samples, int nFolds)`
  Starts the evaluation.
  `static TokenizerModel`
  TokenizerME.`train(ObjectStream<TokenSample> samples, TokenizerFactory factory, TrainingParameters mlParams)`
  Trains a model for the `TokenizerME`.
  Constructor parameters in opennlp.tools.tokenize with type arguments of type TokenSample
  Modifier
  Constructor
  Description
  ``
  `TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics)`
  Initializes a new event stream based on the data stream using a `TokenContextGenerator` that relies on a `DefaultTokenContextGenerator`.
  ``
  `TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, Pattern alphaNumeric, TokenContextGenerator cg)`
  Initializes a new event stream based on the data stream using a `TokenContextGenerator`.
  ``
  `TokSpanEventStream(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, TokenContextGenerator cg)`
  Initializes a new event stream based on the data stream using a `TokenContextGenerator`.
  ``
  `WhitespaceTokenStream(ObjectStream<TokenSample> tokens)`
  Initializes a `WhitespaceTokenStream`.

- ## Uses of TokenSample in opennlp.tools.tokenize.lang.en

  Methods in opennlp.tools.tokenize.lang.en that return TokenSample
  Modifier and Type
  Method
  Description
  `TokenSample`
  TokenSampleStream.`next()`