California SB 574: America's First Law Regulating Lawyers' Use of AI

A Legislation Sparked by "Hallucinations"

Over the past two years, generative AI has rapidly entered the daily work of law firms: drafting documents, searching for cases, and organizing evidence summaries. But behind the convenience, a recurring problem has begun to plague the entire legal industry—AI "fabricating" case law.

Lawyers have found that even well-trained, high-performing large models will, in the course of writing, "spout nonsense with a straight face," fabricating cases, citations, and legal bases that appear real but do not exist at all. This phenomenon is called "AI hallucination." According to statistics from legal research institutions, as of early 2026, more than 600 court cases across the United States have involved lawyers submitting documents containing AI-hallucinated citations, and this trend is still accelerating—from a few sporadic cases per month to new cases appearing almost every day.

California has not been immune either. State Farm's attorney admitted in a Carson City fire lawsuit that AI fabricated non-existent case law; a San Francisco lawyer came under scrutiny for using AI-generated false quotations in court filings in a Trump administration layoff case; in a San Diego custody appeal, the court identified fabricated citations and imposed a $5,000 fine, while referring the lawyer involved to the state bar association.

SB 574 was proposed in an attempt to draw clear regulatory boundaries for lawyers' use of generative AI.

Practical Impact of the Bill on Practice Compliance

SB 574 is not intended to prohibit lawyers from using AI—supporters of the bill have repeatedly emphasized this point. The core idea of the bill is to wrap the act of "using AI" in a framework of professional responsibility, mainly including the following obligations:

Obligation to isolate confidential information. Confidential information, personally identifiable information, and other non-public information must not be input into any "public" generative AI system, unless access to the relevant information within that system is restricted to the lawyer and authorized personnel who are subject to confidentiality obligations. It is particularly worth noting that commentators generally point out that the wording of this restriction explicitly targets "public" generative AI systems and does not expressly regulate the use by lawyers of privately deployed or closed (private/closed) AI tools—this leaves room for law firms to circumvent this restriction by subsequently procuring enterprise-grade, data-isolated AI products.

Obligation to verify and correct outputs. Lawyers must take "reasonable measures" to verify the accuracy of generative AI output, and this obligation expressly "includes but is not limited to" verifying the accuracy of all case citations and statutory citations, and must correct erroneous or "hallucinated" outputs.

Professional judgment cannot be delegated. The bill expressly prohibits lawyers from "outsourcing" core legal judgment to AI systems—the substantive decision-making power of legal practice should always remain in the hands of lawyers themselves.

Court disclosure obligation. Lawyers must disclose to the court the use of generative AI for all documents submitted to the court; if generative AI is used to produce content intended for the public, lawyers must "consider" whether to disclose.

Anti-discrimination provision. The bill also requires that the use of generative AI must not cause unlawful discrimination or disproportionate impact on relevant groups based on a series of protected categories such as age, race, gender, sexual orientation, religious belief, and socioeconomic status.

It is worth mentioning that SB 574 itself does not establish an independent penalty mechanism—violations of the above obligations will be pursued through existing disciplinary procedures (such as the California State Bar's professional conduct rules system), rather than creating a separate new set of penalties.

Where the Legislative Process Stands

The bill has not advanced slowly. On January 30, 2026, SB 574 passed the California Senate by a vote of 39 to 0, with no opposition.

Thereafter, the bill entered committee review in the State Assembly. By the end of California's legislative session on August 31, 2026, SB 574 had been passed by the entire legislature, formally "enrolled" (finalized), and sent to the desk of Governor Gavin Newsom.

Under California law, because the bill was sent to the governor at the end of the session, the 30-day review window begins on the date of adjournment—meaning Newsom's actual deadline for a decision is around October 12, 2026. If he neither signs nor vetoes it within this period, the bill will automatically be deemed vetoed.