Many people would be surprised to know that the foundational algorithm for Generative AI and Large Language Models (LLMs) was first ‘invented’ at Google, with eight researchers there writing a landmark 2017 paper ‘Attention is All You Need’. This laid down the basis of the Transformer algorithm, one that made generative AI possible more efficiently and effectively. Surprisingly, however, Google dithered with this amazing discovery, and let a small startup called OpenAI take the lead in leveraging this to create its series of GPTs, culminating with GPT3.5 or ChatGPT. The rest, as they say, is history.

Speculation abounds as to why the world’s most innovative company did not capitalize on its discovery. People point to reputational risk arising from hallucinating LLMs; some shrug it off as corporate inertia. Some believe that it had to do with the fear of cannibalization of this new technology denting the astoundingly profitable business model of Search. AdWords and AdSense driven advertising revenue works very well in a deterministic, database-driven model like Search, but would fail in a probabilistic, language-driven LLM construct. A score of companies have tried to topple Google from its lofty perch – Microsoft’s Bing and startups like DuckDuckGo and Neeva, but Google has been unassailable. The hallucinatory behaviour of even the best LLMs like GPT4 seems to keep the status quo, with Google still being the go-to place for factual information. LLMs like ChatGPT are not optimized for the truth, but for factual believability.

This seems to be changing though, with a couple of GenAI based startups threatening Google’s mighty search engine. The best known is Perplexity, built on top of GPT and a proprietary LLM built on Meta’s open-sourced LlaMA, Perplexity takes a different approach. When you ask its ‘search’ bar a question, it does not return with the famous ‘10 blue links’, plus a number of advertising-pushed, generally low-quality websites that Google pushes at you. On the other hand, Perplexity hunts the web and uses AI to write a succinct summary of its findings and annotates the same with the sources that it used to give you the answer. It also gives you a few suggested follow-up questions. So, when I asked Perplexity to give me the name of its founder, it spouted out ‘Aravind Srinivas,’ told me that the sources were LinkedIn, Wikipedia, YouTube, and Wired, and then checked whether it wanted me to know Perplexity’s mission or the background of its cofounders. Precise and useful. If you pay $20 a month, it also opens up its Copilot, which helps you arrive at a precise query, by asking you clarifying questions. As an example, if a student researching climate change on marine ecosystems asked it a question, the Copilot would ask her for regions or species, and then scour scientific journals and environmental reports for them. It then summarises the answer with deep insights and helps her build a comprehensive literature review. Copilot also allows you to search within a specified set of sources – like YouTube or Reedit or academic papers.

Another impressive contender is Arc Search from New York based The Browser Company. It adopts a slightly different approach and philosophy- “that a browser, a search engine, an AI chatbot, and a website aren’t different things. They’re all just parts of an internet information finder, and they might as well exist inside the same app” (https://bit.ly/4a55DVC ). So, again rather than return 10 blue links, Arc builds you a kind of a webpage with all the relevant information around the query you asked. Arc is also built on OpenAI and other LLMs and will

improve as the models do. These upstart Google competitors are not perfect. They still do hallucinate, though much lesser than bare boned LLMs. They do not have a business model yet, though it could evolve. Publications fear that they will not send traffic to their sites, as Google does. However, Google should be worried; its search engine seems to have lost its simplicity and purity along the way. It has shifted it from being made for the user to be optimized for an advertiser, and it’s monopoly has made it start to squeeze the advertiser now – the classical ‘enshittification’ doctrine that Cory Doctorow famously wrote about (https://on.ft.com/4asXeeo ).

I believe that newer search constructs like Perplexity and Google will live together; I now have a browser tab permanently open for Perplexity and another for Google, and depending on what I am looking for, I choose what to use. It will be good for Google to have some semblance of competition; hopefully, it makes them pivot towards the consumer again, bring back the purity of its search, and shake themselves out of the corporate sclerosis which has bogged it down. Shaan Puri, the tech thought leader, differentiated the two in an X post: “Google is a search engine. ChatGPT is a thought engine.” Innovators like Perplexity and Arc seem to be blurring those boundaries.