diff --git a/CLOSEDAI.mkd b/CLOSEDAI.mkd new file mode 100644 index 0000000..2db6d63 --- /dev/null +++ b/CLOSEDAI.mkd @@ -0,0 +1,8 @@ +Special Tokens + +1. **\ (Start of Sequence)**: Marks the beginning of a sequence for the model to start processing. +2. **\ (End of Sequence)**: Tells the model when to stop generating text or processing. +3. **\ (Padding Token)**: Pads sequences to the same length for batch processing. +4. **\ (Unknown Token)**: Represents words not in the model's vocabulary. +5. **\ (Mask Token)**: Used in tasks like predicting missing words in masked language models. +6. **\ (Separator Token)**: Separates different segments in input, like questions from context.