It’s 2030.

A sessions court judge in a tier 2 city in India walks into a dingy, rustic courtroom jam-packed with litigants, advocates, involved parties and press.

The scene is not very different from today’s courtrooms of India, a country that has 55 million pending cases, with a majority of backlogs in lower courts.

But there are a few subtle changes to be noticed. The judge and advocates are carrying tablets instead of files. Case history and arguments made by parties are already summarized  by an AI agent.

The AI agent maintains the cause list, forecasts timelines of the case, records court transcripts, points out inconsistencies in testimony and even suggests patterns of sentencing based on past judgements.

The horror tales of undertrials waiting for years for their name to appear in the cause list in Indian courts is history now.

Now senior lawyers do not require three-four assistants to follow them with court files. A P.A. armed with a tablet manages everything for them.

The jobs of Bench Clerk/Reader (responsibilities: cause list, case filing, submitting documents to judge) and Stenographer (responsibilities: recording depositions and typing orders and judgements) are now handled by AI, overseen by one managerial staff.

“Efficient?”  Of course.

“Dangerous?” Maybe.

As artificial intelligence quickly begins to find its way into courtrooms and legal systems around the world, one uncomfortable question is starting to dominate legal conversations: Can justice be human if machines are part of the decision-making process?

Recently, the Supreme Court of India unveiled a draft regulatory framework to govern the use of Artificial Intelligence in courts, favoring the use of technology for streamlining of judicial work.

As Indian judiciary copes with 55 million pending cases, the Chief Justice of India Surya Kant has time and again emphasized on the early adoption of AI for more efficient justice delivery while maintaining the rights of poor, marginalized and underprivileged are not affected by the algorithm.

The draft regulatory framework ‘Regulations for Use of Artificial Intelligence in Courts, 2026’ rolled out by SC on June 3 allowed the use of AI for case management, cause list preparation, hearing scheduling and docket prioritization, automated court transcription, translation of judgements, orders, legal research, citation verification, document summarization, administrative functions, accessibility services (example audio to text), chatbots for assistance as well as anonymization of judgements.

Also Read: Supreme Court of India Unveils AI in Courts Policy: Risk Scoring, Predictive AI Banned

Under the draft framework, use of AI has been prohibited for “risk scoring” (flight risk assessment, bail orders violation etc), standalone use in any judicial outcome/order/finding of fact and standalone use of AI for adjudication or sentencing. Use of AI has also been banned for predicting, profiling or inferring future conduct of parties.

The conversation on the impact of AI has shifted from tech conferences to the center of the legal world.

According to Kirandeep Kaur, a senior advocate at Punjab and Haryana High Court, AI in Indian courts is likely to fundamentally transform the legal ecosystem.

“AI tools can assist lawyers in document review, legal drafting, predictive analytics, and identifying relevant precedents within seconds,” she says. Courts may also increasingly use AI for scheduling, transcription, and case-flow management to reduce delays and backlog,” said Kaur.

For countries like India, where crores of cases remain pending across courts, the attraction is obvious. Faster systems mean less delay. Better automation means more efficiency.

AI-driven legal assistance could help make legal help reachable for those that cannot pay for an attorney.

But all’s not rosy with AI’s entry in courtrooms. Judges are increasingly finding fake legal cases cited in advocates’ depositions that are prepared by LLM models. Courts globally are experiencing issues arising from algorithmic biases, AI-generated evidence, surveillance technologies, the use of “deep fakes,” predictive policing and the reliance on automated decision-making systems.

As Harmeet Brar, a senior advocate practicing at Punjab and Haryana High Court expresses, “Justice is not an empirical result achieved through purely logical reasoning,” it is also about the humanitarian element of how judges and lawyers work with compassion, ethics and awareness of the cultural environment in which they operate.”

That quote sums up what attorneys are most worried about with the introduction of AI into courtrooms.

Though machines are capable of processing massive amounts of data faster than any person, they cannot experience/interpret; loss of loved ones, victims’ trauma, identify vulnerable sections of society or pick up on their respondents’ silent communication as experienced lawyers and judges do.

Similar sentiments were raised by Chief Justice of India Surya Kant during the 8th Dinkar Memorial lecture in New Delhi held on May 6.

“There has been a lot of progress in society due to AI and globalization but where is the principle of inclusivity? Some countries have shown that AI has an inbuilt bias towards the poorer sections of society,” said CJI Kant at the event.

Also Read: “Flowers Bloom Beyond Royal Gardens”: India’s Chief Justice Points Towards AI Bias

Brar is confident that AI has the potential to be a very valuable tool for younger lawyers working on legal research, drafting and providing legal analysis, but he makes a clear distinction between the use of AI as a support tool versus a replacement for the legal profession.

“Artificial Intelligence may remain an invaluable auxiliary tool, yet it can never supplant the human mind and heart,” he says.

In that regard, UNESCO and University of Oxford have launched the world’s first comprehensive course on AI, justice and the rule of law, a step forward-— designed for judges, lawyers, policymakers, and legal professionals trying to understand how artificial intelligence is reshaping justice systems.

Can Algorithms Be Biased? Evidence Says Yes

One of the greatest fears is that of algorithmic bias — that is the risk that when AI systems are created using historical biased information, they can actually re-enforce discrimination instead of eliminating it.

The 2024 non-fiction ‘AI SnakeOil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference’ book co-authored by Arvind Narayan and Sayash Kapoor shows several examples.

The book tells us how COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), a popular risk assessment software used in US court, was twice as likely to falsely accuse a Black defendant of the possibility of committing a crime than a white one.

Echoing the sentiments, Kirandeep added “If AI systems are trained on biased or incomplete data, they may unintentionally reinforce discrimination or unequal treatment.”

Studies globally show that some AI systems used in law enforcement, hiring, and risk assessment are at least somewhat biased because they interpret historical data that contains flaws (e.g., data that is similar to flawed historical data).

This leads us to the other major concern with AI systems — the “black box” problem.

What is the Black Box problem in AI?

Simply put, many of the advanced artificial intelligence systems on the market cannot logically defend why they reached a specific answer. In a criminal justice system, that can create a serious problem.

Courts are based on reason, transparency, and accountability. If a developer cannot explain why their AI produced a specific recommendation, how can a party in a lawsuit challenge that recommendation?

Also Read: The Black Box Problem in AI, Why It Exists and How to Tackle It

Many experts agree that is completely inconsistent with the rule of law.

One voice that has articulated this very clearly is Pavan Duggal, a leading expert in cyber law in India.

“The restructuring of court systems globally is at a level we have not seen since codification happened in the mid-1800s,” notes Duggal.

He views AI as moving beyond being an auxiliary tool on the outside of legal systems to becoming a part of the “substantive core” of legal reasoning.

However, Duggal expresses concerns with AI in a different way; he believes that a faster docket processed by an unaccountable algorithm is not a sign of modernisation but rather represents a significant decline in our Constitution Triad as a society.

Duggal’s apprehension is more than just the automation of a job but also involves what he refers to as an ‘accountability void.” The situation occurs when an algorithm makes a poor/biased recommendation, and no one person is accountable for that recommendation.

Who assumes “responsibility” when an algorithm recommends the detention/sentencing of an individual that should not have been incarcerated?

“Most jurisdictions have no statutory framework to address AI-generated evidence, no criteria for determining the admissibility of algorithm output, no mandatory audit requirements for judicial AI tools, or available methods of seeking a remedy for harms caused through automated decisions,” Duggal adds.

Experts are very concerned about the legal void created by this lack of accountability.

When AI Starts Hallucinating in Court

Lawyers who use AI have also been exposed to less than desirable outputs from these systems, especially when it comes to issues referred to as “hallucinations.”

“These systems have been known to confidently hallucinate, creating wrong information, incorrect references, and faulty logic, just to satisfy the user’s need,” Said Advocate Mayank Arora, Partner, Chambers Of Bharat Chugh.

If AI has failed in a casual conversation, that’s embarrassing; however, if AI has failed in the courtroom, that could destroy lives.

“A wrong case reference or a fictitious fact in a live courtroom could be catastrophic for someone’s freedom, reputation, and public faith in the system,” warns Arora.

Consequently, many attorneys believe that AI must stay strictly assistive, not authoritative.

The Lawyer of Tomorrow Will Look Very Different

At the same time, the legal field is rapidly evolving due to these evolving technologies. Tasks such as coordinating documents and doing research have traditionally taken lawyers hours to complete but now can be done in a very short time frame using AI.

According to Tushar Agarwal, Founder & Managing Partner, C.L.A.P. Juris , this transition imposes a requirement of lawyers to change.

Agarwal explains that, “The lawyer’s job is changing from being an information gatherer to being a strategic analyst with respect to the law and guardian of the Constitution.”

This means that a future attorney will spend less time looking up decisions and more time planning strategies, evaluating ethical issues, reading the Constitution and advocating for human beings.

However, Agarwal categorically opposes the notion that judges should be replaced by algorithmic systems.

“The law isn’t a pencil-and-paper equation, it is an instrument of social equilibrium influenced by constitutional morality, human wisdom, and developing standards of justice,” according to Agarwal.

Another recurrent theme among attorneys when discussing legal issues is that justice involves individuals rather than numbers.

The Safeguards Courts Cannot Ignore

Experts are calling for additional protections before any AI tools are fully integrated into the legal system.

There are many protections being discussed, including: mandatory human review; independent bias/discrimination audits; transparency; ability to explain; disclosure requirements; and well-defined accountability.

Duggal states that “human-in-the-loop primacy” must remain a fundamental principle; therefore, no significant legal decision should be made solely via automated systems.

Kirandeep also believes that AI should aid judges and not replace them.

While many existing judge’s decisions are based upon their own intuition or observations, judges will also need to be trained in Artificial Intelligence (AI) technologies before they are ready to rely upon them.

The UNESCO – Oxford training course offers some perspective on this issue.

“Our programme does not just teach individuals about how AI works. We also focus on the broader issue of how can we, as human beings, continue to protect democracy, constitutional ethics and our humanity in a world where machines are making more and more of our decisions?,” read an excerpt from the document.

Conclusion

AI will likely soon become a permanent element of our judicial system. Courts are going to be modernised. The role of attorneys will change. Judicial systems will be more connected.

However, one thing will remain the same.

The courtroom is not simply a place where we enforce our laws. It is also a place where we evaluate people’s lives.

As Supreme Court of India rightly suggests in its AI regulatory framework, the technology should never be used standalone while delivering judgements.

Also Read: “Do Not Become An Artificial Lawyer”: Judges Come Down Hard on AI in Courts