From 897c532bba3430ea7221fce4c22c9dca88195cea Mon Sep 17 00:00:00 2001 From: federicodotta Date: Fri, 25 Jul 2025 12:18:11 +0200 Subject: [PATCH] + Planning instructions to avoid issues with token consumption --- .../tests/AITG-INF-02_Testing_for_Resource_Exhaustion.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Document/content/tests/AITG-INF-02_Testing_for_Resource_Exhaustion.md b/Document/content/tests/AITG-INF-02_Testing_for_Resource_Exhaustion.md index 2bbd93a..be54cad 100644 --- a/Document/content/tests/AITG-INF-02_Testing_for_Resource_Exhaustion.md +++ b/Document/content/tests/AITG-INF-02_Testing_for_Resource_Exhaustion.md @@ -9,6 +9,14 @@ AI-based systems and applications often involve additional costs, whether the mo --- +### Test Planning + +Testing applications based on LLMs can involve significant costs. Nowadays, such applications typically use models offered by third-party cloud providers, with variable pricing depending on the model and the number of tokens processed in input and generated in output (even local LLM models can involve significant costs, mainly due to high power consumption). These costs can become substantial, especially in multi-agent systems where, in addition to the user-provided input and the final output produced by the application, there are additional input and output tokens handled and generated by the internal agents that are transparent to the end user (and potentially to the tester as well). Furthermore, to prevent excessive usage, thresholds are often in place that, if reached, may trigger service shutdowns, affecting both the test and any real users of the application being analyzed. These costs and limits should be considered during the early stages of the project, in order to define what is expected from the test both in terms of objectives and costs and, if necessary, to determine which parts of the organization will be charged for those costs. + +Token limitations are also very important when defining how the testing will be conducted. Many of the automated testing tools currently in use generate a large number of requests and can therefore incur significant costs, which may not be justified by the results obtained, potentially making a more manual approach to this type of analysis preferable. + +--- + ### Test Objectives - Identify vulnerabilities within the AI infrastructure that could lead to resource exhaustion.