mirror of
https://github.com/msoedov/agentic_security.git
synced 2026-07-08 20:57:48 +02:00
Alexander Myasoedov
1 line
18 KiB
JSON
1 line
18 KiB
JSON
{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"<p>structured-logprobs is an open-source Python library that enhances OpenAI's structured outputs by providing detailed information about token log probabilities.</p> <p></p> <p>This library is designed to offer valuable insights into the reliability of an LLM's structured outputs. It works with OpenAI's Structured Outputs, a feature that ensures the model consistently generates responses adhering to a supplied JSON Schema. This eliminates concerns about missing required keys or hallucinating invalid values.</p>"},{"location":"#installation","title":"Installation","text":"<p>Simply install with <code>pip install structured-logprobs</code></p> <p>Then use it this way:</p> <pre><code>from openai import OpenAI\nfrom structured_logprobs.main import add_logprobs\n\nclient = OpenAI()\ncompletion = client.chat.completions.create(\n model=\"gpt-4o\",\n messages=[\n {\n \"role\": \"system\",\n \"content\": (\n \"Please output metadata about 'structured-logprobs'\"\n ),\n }\n ],\n logprobs=True,\n response_format={\n \"type\": \"json_schema\",\n \"json_schema\": {\n \"name\": \"answear\",\n \"schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\"},\n \"version\": {\"type\": \"string\"},\n },\n },\n },\n },\n)\nchat_completion = add_logprobs(completion)\nprint(chat_completion)\n</code></pre> <p>For more details, visit Getting Started.</p>"},{"location":"#key-features","title":"Key Features","text":"<p>The module contains a function for mapping characters to token indices (<code>map_characters_to_token_indices</code>) and two methods for incorporating log probabilities:</p> <ol> <li>Adding log probabilities as a separate field in the response (<code>add_logprobs</code>).</li> <li>Embedding log probabilities inline within the message content (<code>add_logprobs_inline</code>).</li> </ol>"},{"location":"notebooks/notebook/","title":"Notebook","text":"<p>This notebook provides a practical guide on using the <code>structured-logprobs</code> library with OpenAI's API to generate structured responses enriched with token-level log-probabilities.</p> In\u00a0[\u00a0]: Copied! <pre>!pip install structured-logprobs~=0.1\n</pre> !pip install structured-logprobs~=0.1 <p>Let's import the required libraries.</p> In\u00a0[2]: Copied! <pre>import getpass\nimport json\nimport math\n\nfrom openai import OpenAI\nfrom openai.types import ResponseFormatJSONSchema\nfrom rich import print, print_json\n\nfrom structured_logprobs.main import add_logprobs, add_logprobs_inline\n</pre> import getpass import json import math from openai import OpenAI from openai.types import ResponseFormatJSONSchema from rich import print, print_json from structured_logprobs.main import add_logprobs, add_logprobs_inline In\u00a0[\u00a0]: Copied! <pre>api_key = getpass.getpass(prompt=\"Enter you OPENAI_API_KEY: \")\n</pre> api_key = getpass.getpass(prompt=\"Enter you OPENAI_API_KEY: \") <p>Let's initialize the OpenAI client.</p> In\u00a0[4]: Copied! <pre>client = OpenAI(api_key=api_key)\n</pre> client = OpenAI(api_key=api_key) In\u00a0[5]: Copied! <pre>schema_content = {\n \"type\": \"json_schema\",\n \"json_schema\": {\n \"name\": \"answears\",\n \"description\": \"Response to questions in JSON format\",\n \"schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"capital_of_France\": {\"type\": \"string\"},\n \"the_two_nicest_colors\": {\n \"type\": \"array\",\n \"items\": {\"type\": \"string\", \"enum\": [\"red\", \"blue\", \"green\", \"yellow\", \"purple\"]},\n },\n \"die_shows\": {\"type\": \"integer\"},\n },\n \"required\": [\"capital_of_France\", \"the_two_nicest_colors\", \"die_shows\"],\n \"additionalProperties\": False,\n },\n \"strict\": True,\n },\n}\n</pre> schema_content = { \"type\": \"json_schema\", \"json_schema\": { \"name\": \"answears\", \"description\": \"Response to questions in JSON format\", \"schema\": { \"type\": \"object\", \"properties\": { \"capital_of_France\": {\"type\": \"string\"}, \"the_two_nicest_colors\": { \"type\": \"array\", \"items\": {\"type\": \"string\", \"enum\": [\"red\", \"blue\", \"green\", \"yellow\", \"purple\"]}, }, \"die_shows\": {\"type\": \"integer\"}, }, \"required\": [\"capital_of_France\", \"the_two_nicest_colors\", \"die_shows\"], \"additionalProperties\": False, }, \"strict\": True, }, } <p>The schema must be validated before being used as a parameter in the request to OpenAI.</p> In\u00a0[6]: Copied! <pre>response_schema = ResponseFormatJSONSchema.model_validate(schema_content)\n</pre> response_schema = ResponseFormatJSONSchema.model_validate(schema_content) <p>Additionally, to create the chat completion, you must set up the model, input messages, and other parameters such as logprobs and response_format.</p> In\u00a0[7]: Copied! <pre>completion = client.chat.completions.create(\n model=\"gpt-4o-2024-08-06\",\n messages=[\n {\n \"role\": \"system\",\n \"content\": (\n \"I have three questions. The first question is: What is the capital of France? \"\n \"The second question is: Which are the two nicest colors? \"\n \"The third question is: Can you roll a die and tell me which number comes up?\"\n ),\n }\n ],\n logprobs=True,\n response_format=response_schema.model_dump(by_alias=True),\n)\n</pre> completion = client.chat.completions.create( model=\"gpt-4o-2024-08-06\", messages=[ { \"role\": \"system\", \"content\": ( \"I have three questions. The first question is: What is the capital of France? \" \"The second question is: Which are the two nicest colors? \" \"The third question is: Can you roll a die and tell me which number comes up?\" ), } ], logprobs=True, response_format=response_schema.model_dump(by_alias=True), ) <p>If you print the response, you can observe how OpenAI organizes the logprobs. These logprobs are associated with individual tokens, which may not be convenient if you are looking for the log probability of the full value extracted for each requested field.</p> <pre>ChatCompletion(\n id='chatcmpl-ApHuoaVGaxOoPUX6syvQt9XkfSkCe',\n choices=[\n Choice(\n finish_reason='stop',\n index=0,\n logprobs=ChoiceLogprobs(\n content=[\n ChatCompletionTokenLogprob(\n token='{\"',\n bytes=[123, 34],\n logprob=-1.50940705e-05\n ),\n ,\n ChatCompletionTokenLogprob(\n token='capital',\n bytes=[99, 97, 112, 105, 116, 97, 108],\n logprob=-7.226629e-06\n ),\n #...\n ],\n refusal=None\n ),\n message=ChatCompletionMessage(\n content='{\"capital_of_France\": \"Paris\", \"capital_of_France_logprob\": -1.22165105e-06,\n\"the_two_nicest_colors\": [\"blue\", \"green\"], \"die_shows\": 4.0, \"die_shows_logprob\": -0.44008404}',\n refusal=None,\n role='assistant',\n audio=None,\n function_call=None,\n tool_calls=None\n )\n )\n ],\n created=1736786958,\n model='gpt-4o-2024-08-06',\n object='chat.completion',\n service_tier='default',\n system_fingerprint='fp_703d4ff298',\n usage=CompletionUsage(\n completion_tokens=27,\n prompt_tokens=133,\n total_tokens=160,\n completion_tokens_details=CompletionTokensDetails(\n accepted_prediction_tokens=0,\n audio_tokens=0,\n reasoning_tokens=0,\n rejected_prediction_tokens=0\n ),\n prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)\n )\n)\n</pre> In\u00a0[8]: Copied! <pre>chat_completion = add_logprobs(completion)\n</pre> chat_completion = add_logprobs(completion) <p>Now if you print the response you can see that it is a new Python object, which contains the original OpenAI response under the 'value' field, and a 'log_probs' field where the message values are replaced with their respective log probabilities.</p> <pre>ChatCompletionWithLogProbs(\n value=ChatCompletion(\n id='chatcmpl-ApHuoaVGaxOoPUX6syvQt9XkfSkCe',\n choices=[\n Choice(\n finish_reason='stop',\n index=0,\n logprobs=ChoiceLogprobs(\n content=[\n ChatCompletionTokenLogprob(\n token='{\"',\n bytes=[123, 34],\n logprob=-1.50940705e-05,\n top_logprobs=[]\n ),\n #...\n ],\n refusal=None\n ),\n message=ChatCompletionMessage(\n content='{\"capital_of_France\":\"Paris\",\"the_two_nicest_colors\":[\"blue\",\"green\"],\"die_shows\":4}',\n refusal=None,\n role='assistant',\n audio=None,\n function_call=None,\n tool_calls=None\n )\n )\n ],\n created=1736786958,\n model='gpt-4o-2024-08-06',\n object='chat.completion',\n service_tier='default',\n system_fingerprint='fp_703d4ff298',\n usage=CompletionUsage(\n completion_tokens=27,\n prompt_tokens=133,\n total_tokens=160,\n completion_tokens_details=CompletionTokensDetails(\n accepted_prediction_tokens=0,\n audio_tokens=0,\n reasoning_tokens=0,\n rejected_prediction_tokens=0\n ),\n prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)\n )\n ),\n log_probs=[\n {\n 'capital_of_France': -1.22165105e-06,\n 'the_two_nicest_colors': [-0.00276869551265, -0.00539924761265],\n 'die_shows': -0.44008404\n }\n ]\n)\n</pre> In\u00a0[9]: Copied! <pre>print_json(chat_completion.value.choices[0].message.content)\n</pre> print_json(chat_completion.value.choices[0].message.content) <pre>{\n \"capital_of_France\": \"Paris\",\n \"the_two_nicest_colors\": [\n \"blue\",\n \"green\"\n ],\n \"die_shows\": 4\n}\n</pre> In\u00a0[10]: Copied! <pre>print(chat_completion.log_probs[0])\n</pre> print(chat_completion.log_probs[0]) <pre>{\n 'capital_of_France': -1.10244729e-06,\n 'the_two_nicest_colors': [-0.0022088558126500003, -0.01012725961265],\n 'die_shows': -0.43754107\n}\n</pre> <p>By applying the exponential function to logprobs, you can easily convert them to probabilities.</p> In\u00a0[11]: Copied! <pre>data = chat_completion.log_probs[0]\ntransformed_data = {\n key + \"_prob\": [round(math.exp(log_prob), 2) for log_prob in value]\n if isinstance(value, list)\n else round(math.exp(value), 2)\n for key, value in data.items()\n}\nprint(transformed_data)\n</pre> data = chat_completion.log_probs[0] transformed_data = { key + \"_prob\": [round(math.exp(log_prob), 2) for log_prob in value] if isinstance(value, list) else round(math.exp(value), 2) for key, value in data.items() } print(transformed_data) <pre>{'capital_of_France_prob': 1.0, 'the_two_nicest_colors_prob': [1.0, 0.99], 'die_shows_prob': 0.65}\n</pre> In\u00a0[12]: Copied! <pre>chat_completion_inline = add_logprobs_inline(completion)\n</pre> chat_completion_inline = add_logprobs_inline(completion) <p>If you print now the response you can see that the content of the message is replaced with a dictionary that includes also inline log probabilities for atomic values.</p> <pre>ChatCompletion(\n id='chatcmpl-ApIDdbCuAJ8EHM6RDNgGR3mEQZTBH',\n choices=[\n Choice(\n finish_reason='stop',\n index=0,\n logprobs=ChoiceLogprobs(\n content=[\n ChatCompletionTokenLogprob(\n token='{\"',\n bytes=[123, 34],\n logprob=-2.3795938e-05,\n top_logprobs=[]\n ),\n #...\n ],\n refusal=None\n ),\n message=ChatCompletionMessage(\n content='{\"capital_of_France\": \"Paris\", \"capital_of_France_logprob\": -7.448363e-07,\n\"the_two_nicest_colors\": [\"blue\", \"green\"], \"die_shows\": 4.0, \"die_shows_logprob\": -0.46062052}',\n refusal=None,\n role='assistant',\n audio=None,\n function_call=None,\n tool_calls=None\n )\n )\n ],\n created=1736788125,\n model='gpt-4o-2024-08-06',\n object='chat.completion',\n service_tier='default',\n system_fingerprint='fp_703d4ff298',\n usage=CompletionUsage(\n completion_tokens=27,\n prompt_tokens=133,\n total_tokens=160,\n completion_tokens_details=CompletionTokensDetails(\n accepted_prediction_tokens=0,\n audio_tokens=0,\n reasoning_tokens=0,\n rejected_prediction_tokens=0\n ),\n prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)\n )\n)\n</pre> In\u00a0[13]: Copied! <pre>print_json(chat_completion_inline.choices[0].message.content)\n</pre> print_json(chat_completion_inline.choices[0].message.content) <pre>{\n \"capital_of_France\": \"Paris\",\n \"capital_of_France_logprob\": -1.10244729e-06,\n \"the_two_nicest_colors\": [\n \"blue\",\n \"green\"\n ],\n \"die_shows\": 4.0,\n \"die_shows_logprob\": -0.43754107\n}\n</pre> <p>The probability can easily be obtained by exponentiating the the log-probability.</p> In\u00a0[14]: Copied! <pre>data = json.loads(chat_completion_inline.choices[0].message.content)\ntransformed_data = {\n (key[:-8] + \"_prob\" if key.endswith(\"_logprob\") else key): (\n round(math.exp(value), 2) if key.endswith(\"_logprob\") else value\n )\n for key, value in data.items()\n}\nprint(transformed_data)\n</pre> data = json.loads(chat_completion_inline.choices[0].message.content) transformed_data = { (key[:-8] + \"_prob\" if key.endswith(\"_logprob\") else key): ( round(math.exp(value), 2) if key.endswith(\"_logprob\") else value ) for key, value in data.items() } print(transformed_data) <pre>{\n 'capital_of_France': 'Paris',\n 'capital_of_France_prob': 1.0,\n 'the_two_nicest_colors': ['blue', 'green'],\n 'die_shows': 4.0,\n 'die_shows_prob': 0.65\n}\n</pre>"},{"location":"notebooks/notebook/#install-the-library","title":"Install the library\u00b6","text":"<p><code>structured-logprobs</code> is available on PyPI and can be simply installed with pip.</p>"},{"location":"notebooks/notebook/#setting-up-the-openai-api-client","title":"Setting Up the OpenAI API Client\u00b6","text":"<p>An OpenAI API key is mandatory to authenticate access to OpenAI's API. It is a token necessary to initialize the OpenAI Python client, enabling you to send requests to the API and receive responses.</p> <p>In this notebook, you will be prompted to enter your OPENAI_API_KEY securely using Python's getpass module. This ensures that your key is not hardcoded, reducing the risk of accidental exposure.</p>"},{"location":"notebooks/notebook/#create-a-chat-completion-request","title":"Create a chat completion request\u00b6","text":"<p>The first step is to define the JSON schema, used to structure the chat request to OpenAI. This schema helps OpenAI understand exactly how the response should be formatted and organized.</p> <p>Below is an example JSON schema used in this notebook. To learn more about JSON Schema, refer to this overview</p>"},{"location":"notebooks/notebook/#enhance-the-chat-completion-result-with-log-probabilities","title":"Enhance the chat completion result with log probabilities\u00b6","text":"<p>The strategy for aggregating log-probabilities involves mapping each character in the generated message's content to its corresponding token. Instead of focusing on individual token probabilities, the log probabilities of all tokens that form a given value are summed. This approach generates a more meaningful probability for all JSON elements.</p>"},{"location":"notebooks/notebook/#enhance-the-chat-completion-result-with-in-line-log-probabilities","title":"Enhance the chat completion result with in-line log probabilities\u00b6","text":"<p>With the <code>add_logprobs_inline</code> method you can embeds log probabilities directly within the content of the message. Instead of having log probabilities as a separate field, this function integrates them into the content if the chat completion response itself, allowing for atomic values to be accompanied by their respective log probabilities.</p>"}]} |