Every AI agent needs context in order to perform its tasks. Practically always the organization cannot train and give the agent the whole context needed.
This incomplete context problem is relevant in process automation of any domain. In procurement and supply chain it is much more acute. Considerable knowledge is not in the organization but within other players in the supply chain, such as suppliers, n-tier suppliers, distributors, forwarders. Because of this, mitigation mechanisms such as HIL – Humans In the Loop, will be less effective. In many cases the humans in the company will lack the relevant knowledge to correct the agent.
The AI Agent procurement context ratio is the ratio between the procurement context we entered and trained the agent with, and the whole relevant context needed for the task the agent performs. The range of values, expressed in percentage, is of course between 0 to 100%.
We cannot measure the context ratio directly but there are good proxies. One proxy is the share of agent decisions that did not need human corrections, say in 90 days, to the total number of agent decisions in the same period.
You should measure the ratio across time. If it is getting better it usually means that we added more context to the agent and trained it better. If it is getting worse, it can indicate major changes in the process behavior which reduce the relevance of the “old” rules.
How can we improve the context ratio, and do it economically?
Integrating suppliers and other supply chain players into the training and assessment process.
We can test how the ratio changes after including context from relevant supply chain players involved in the process, e. g. suppliers. The greater the positive change the more we should include other supply chain members in the training and assessment of the agent.
An analysis of the wrong decisions will give us directions regarding which layer of the supply chain the missing knowledge might be.
Because in procurement and supply chain considerable knowledge is not in the organization, we should actively recruit other players’ assistance in improving the ratio. Examples can be supplier validated test cases, doing shared incidents reviews and in general involving suppliers with all other improvement efforts.
In reverse, we need to be involved in the assessment of context of important agents of our critical suppliers. We need to be major customers of course to obtain this.
Stress tests – Performing stress tests on the agents, simulating extreme conditions will give us some insights on the resilience of the agent and of the system employing it in radical environments, even those we have not tested.
Supply Chain Risk Management (SCRM) – Understanding the exposure and impact of missing context. Understanding behavior and risks of agents, ours and of critical suppliers, will be a major discipline in SCRM.
Debriefing of failures and near failures. We need to communicate this to the organization and employees that failures and near failures happen and should be analyzed, not hidden. The process owner should send a periodic report of wrong agents decisions and the corrective actions taken as a result.
Warning list – For each process we can prepare a list of signs which indicate potential problems.
Measuring the context ratio of your procurement AI agents before go-live and during operations, actively conducting efforts to increase this ratio, and collaborating with supply chain players for these purposes should be standard procedures for leading procurement teams.
How do you currently test what your agents do not know and how do you improve the context ratio?