Can a human be bound by a decision made by an AI computer program?
Automated decision making is not a new phenomenon in Australia. Computer programs in various forms have been used to augment and to automate administrative decision making since the 1990’s with over 20 pieces of Commonwealth legislation currently providing for decisions made by computer programs to be taken to be an official decision of a Minister, Secretary or Regulator (as applicable).
So what is new about automated decision making being undertaken by machine learning systems (a form of AI)?
Unlike computer programs used in traditional automated decision making – machine learning systems do not necessarily follow explicit rules authored by humans, rather the machine derives its own rules based on the data and algorithms it has been trained on. That is – machine learning continually “learns” from the correlations and patterns it identifies in data.
This does not mean (for the moment at least) that automated decisions made by machine learning systems are entirely devoid of human input – for example, humans will decide what decisions are to be automated, humans will design the base system and humans will decide which datasets to train the system on.
However, as the output will not be the result of the system following measurable and pre-programmed rules, significant questions arise around transparency, accountability, fairness, accuracy, predictability and consistency.
That is – if a human does not preprogram the logic of the decisionmaking process – do humans know how the decision-making process is undertaken? Does the decision maker understand what data was used by the system to inform the decision? Does the decision maker understand how that data interacted with the algorithms to reach the decision? Can the decision maker justify why the decision was made? Are there any concerns about the accuracy, or inherent bias, in the data upon which the system was trained? Are there methods in place to ascertain how the system changes (or evolves) its decisions based on what is has “learnt” in the past?
So what can be done? Is the answer to prohibit automated decision making by machine learning systems? Or to introduce methods that help produce answers to these questions? The pendulum is currently swinging to the later – with concepts such as explainable artificial intelligence (XAI) and qualified transparency being increasingly explored by companies and research institutions around the world. Similarly, there is an increased focus on introducing processes that allow automated decisions to be challenged and for individuals to be able to correct the information about them that is used by the system.
For example, in the United States, automated decision making has been used for a number of years in the judicial system to assess a defendant’s risk of re-offending. Although the use of these algorithmic risk assessments in sentencing decisions was upheld by the Supreme Court of Wisconsin, the decision-making process cannot be delegated to the algorithmic risk assessment. In fact the court urged caution when judges use such risk assessments as there is a lack of transparency, and understanding by the decision maker, over how the algorithmic risk assessments work. These concerns about transparency (and the consequential impact on accuracy) has been strengthened in light of independent testing of the algorithmic risk assessments that suggest that African Americans were more likely to receive high risk ratings than white offenders and that the results are often no more accurate than that of human estimates.
Similarly, Article 22 of Europe’s General Data Protection Regulations provides that a data subject (with a few limited exceptions such as the consent of the data subject) has a right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.
AI can, by removing human biases and arbitrariness from a decision, assist in improving the quality and predictability of decisions. However, where the AI continually “learns” and develops its own rules, the benefits of machine learning driven automated decision making can only be realised if steps are undertaken to limit the risks of relying upon a non-human decision maker…
Credit: Google News