The correct choice is Tokenization segments input text into manageable units such as words or subwords so models can learn and infer. In generative AI workflows this preprocessing step converts raw text into discrete tokens that models can embed and process efficiently for both training and inference.

Tokenization maps characters and character sequences into token ids so embeddings and attention mechanisms can operate on fixed units. Using subword methods helps handle rare words and reduces vocabulary size which improves model efficiency for encoding and decoding.

Tokenization calculates probabilities to pick the next token during text generation is incorrect because computing next token probabilities is performed by the model during decoding and not by the preprocessing step.

Tokenization masks sensitive attributes by replacing them with nonreversible stand-ins is incorrect because that describes data masking or anonymization which is a separate data processing activity and not the typical role of NLP tokenization.

AWS Key Management Service is incorrect because KMS manages encryption keys and related cryptographic operations and it does not perform text tokenization for models.

Full AWS Practitioner Certification Question