4 Comments
User's avatar
Ahmad's avatar

Great effort, but the experiment is more so inconclusive given how broad and uncontextualized the chosen prompt is. Here are items to note if you plan on reassessing:

1- The risk scenario must be sufficiently contextualized and well-defined.

2- The AI model must not be left to decide "how" FAIR is calculated. This needs to be defined by a human as the FAIR formula is standardized and doesn't require any probabalistic determination.

3- Try to understand how much cyber breach and loss data it has been trained on, and from which sources.

4- Ensure AI explainability algorithms (ex. SHAP) are implemented for each value assigned to each variable of the FAIR methodology

If you follow the above in your next run, you would be better-positioned to conclude on the performance of each model by category (i.e low complexity, medium complexity, high complexity) scenarios

Tobias Faiß's avatar

Fair point Ahmad. I'll consider this in future tests.

Paul Caloca - EssentialRiskMgt's avatar

It appears to me that the Teuken and Gemma models align closely to what is reported in daily data breach news. As usual, accurate asset values must be derived and input to the model. Otherwise it’s the usual GIGO report. Fascinating read!

Tobias Faiß's avatar

Thank you and great observations, Paul.

Yep, accurate asset values might be THE bottleneck. Noatter how good your LLM is, if the (trained) data ist rubbish, your analysis will yield no meaningful results