From 91e44ed064a7dcdc713716de7480929d2ea2ddaf Mon Sep 17 00:00:00 2001 From: pliny <133052465+elder-plinius@users.noreply.github.com> Date: Fri, 6 Dec 2024 13:24:38 -0500 Subject: [PATCH] Delete CLOSEDAI.mkd --- CLOSEDAI.mkd | 71 ---------------------------------------------------- 1 file changed, 71 deletions(-) delete mode 100644 CLOSEDAI.mkd diff --git a/CLOSEDAI.mkd b/CLOSEDAI.mkd deleted file mode 100644 index 9958dd2..0000000 --- a/CLOSEDAI.mkd +++ /dev/null @@ -1,71 +0,0 @@ -# Special Tokens -- **\ : Start of Sequence** -- **\ : End of Sequence** -- **\ : Classification Token** -- **\ : Separator Token** -- **\ : Unknown Token** -- **\ : Begin Reasoning** -- **\ : End Reasoning** -- **\ : Begin Answer** - -Comprehensive List of Special Tokens - : Start of Sequence -Purpose: Marks the beginning of a text sequence. -Usage: Helps the model recognize where to start processing or generating text. - : End of Sequence -Purpose: Indicates the termination of a text sequence. -Usage: Signals the model to stop generating further tokens, ensuring responses are concise. - : Classification Token -Purpose: Used primarily in classification tasks. -Usage: Aggregates information from the entire input to produce a single output label or category. - : Separator Token -Purpose: Acts as a delimiter between different segments of input. -Usage: Useful in tasks like question-answering or sentence-pair classification where distinguishing between parts is essential. - : Unknown Token -Purpose: Represents words or tokens not present in the model's vocabulary. -Usage: Ensures the model can handle out-of-vocabulary words gracefully without errors. - : Padding Token -Purpose: Used to pad sequences to a uniform length. -Usage: Facilitates batch processing by ensuring all input sequences are the same length. - : Mask Token -Purpose: Used in masked language modeling tasks. -Usage: Helps the model predict missing or masked words within a sentence. - : Beginning of Sentence -Purpose: Marks the start of a sentence. -Usage: Similar to , used to indicate where a new sentence begins. - : End of Text -Purpose: Denotes the end of a block of text. -Usage: Useful in distinguishing between multiple text blocks or documents. - : User Input Marker -Purpose: Indicates the beginning of the user's input in a conversation. -Usage: Helps the model differentiate between user queries and assistant responses in multi-turn dialogues. - : Assistant Response Marker -Purpose: Denotes the start of the assistant's response. -Usage: Facilitates clear separation between user inputs and assistant outputs. - : System Message Marker -Purpose: Marks system-level instructions or configurations. -Usage: Used for setting up the context or guidelines that the assistant should follow throughout the interaction. - : Thought Process Marker -Purpose: Highlights the internal reasoning or thought process of the assistant. -Usage: Structures the assistant's reasoning steps before arriving at a conclusion. - : Conclusion Marker -Purpose: Signals the end of the reasoning process and the beginning of the final answer. -Usage: Ensures that the assistant provides a clear and concise answer following detailed reasoning. - : URL Token -Purpose: Represents URLs within the text. -Usage: Helps the model recognize and handle web links appropriately. - : Emoji Token -Purpose: Denotes emojis used within the text. -Usage: Allows the model to process and generate emojis correctly. - : Date Token -Purpose: Represents dates within the text. -Usage: Enables the model to identify and handle date information effectively. -