The popularity and accessibility of artificial intelligence (“AI”) tools, such as ChatGPT and Bard, have grown exponentially this year. However, the careless use of these tools in legal settings can lead to embarrassment and even sanctions. To test how reliable AI is for even basic legal research, we asked four popular AI search engines to write a blog about “the most important employment laws that a California business needs to know.”
The full text of the blogs written by ChatGPT 3.5, Bard, Bing, and Perplexity can be found here. The substance of each blog is fairly similar – and each blog contains information that is incorrect and/or out of date. Some interesting aspects of the AI search engines are discussed below, but the main takeaway is that you should NOT rely on these tools for guidance for any issue as complex as California employment law.
Each of the search engines raised some key issues that California employers face, including overtime, harassment, discrimination, and leave laws, but they varied somewhat in the amount of detail and accuracy.
ChatGPT
ChatGPT 3.5 stands apart from the other search engines due to its more colorful writing style – for example, its description of the “stunning landscapes and innovative tech hubs” of “the Golden State” – and its inclusion the most information specific to California, as opposed to general employment law requirements found in federal law and/or the laws of many states. In addition, ChatGPT 3.5 was the only one of the four search engines to include in its output an indication that its information may be out of date. Specifically, ChatGPT 3.5 blog post stated that, “As of my last knowledge update in September 2021, the minimum wage varies depending on the size of your business and location within the state.” Both Bard and Perplexity incorrectly stated that the current minimum wage $14.00 per hour for the employers with 25 or fewer employees and $15.00 per hour for larger employers. (Oddly, Bing did not even mention California’s minimum wage.)
Bard
In contrast, Bard provided very little California-specific information. It correctly noted that California’s minimum wage is higher than the federal minimum wage (but, as noted, provided out-of-date information concerning how much higher) and that California law gives employees a right to review their personnel records. The rest of the Bard blog post contained very broad, generally-applicable statements, such as that harassment and discrimination based on characteristics such as age, race, and sex are prohibited. And some of those statements are inaccurate, such as the suggestion that unemployment benefits are available only to “workers who have lost their jobs through no fault of their own.”
Bing
Bing’s AI search engine provided the least detailed information, and provided some incorrect information, such as that “Employers must provide accurate and truthful job references for former employees.” One notable feature of Bing is that it cites the source(s) of its information. In response to our prompt, though, Bing only referenced a single website. It did, however, provide links to other sources for anyone who wants to “learn more,” including the California Department of Industrial Relations and the California Chamber of Commerce.
Perplexity
Perplexity’s blog post was the shortest, but it included more detail about California’s overtime law than any of the others. In addition, Perplexity and Bing were the only ones to mention California’s paid sick leave requirement.
Three of the AI search engines we experimented with got one thing right: California businesses should consult qualified employment counsel to make sure they are in compliance with our state’s laws. ChatGPT missed that important point!
