Who's Better At Generating Innovative Ideas, ChatGPT Or MBA Students

Wharton professors say Ideas Are Dimes a Dozen and they put that theory to a test. But how does one determine a good idea? And what does better mean?

Wharton notes the difference between consistency and better. For example, an airplane pilot who lands aircraft with average smoothness 10 times out of 10 is better than one who is brilliant 9 times out of 10 and crashes once. With ideas, one fantastic idea and 10 poor ones is better than 10 average ones. With that backdrop let’s dive into the article. Abstract: ChatGPT-4 can generate ideas much faster and cheaper than students, and the ideas are on average of higher quality (as measured by purchase-intent surveys) and exhibit higher variance in quality. More important, the vast majority of the best ideas in the pooled sample are generated by ChatGPT and not by the students. Providing ChatGPT with a few examples of highly rated ideas further increases its performance. We discuss the implications of these findings for the management of innovation. Introduction: Generative artificial intelligence has made remarkable advances in creating life-like images and coherent, fluent text. OpenAI’s ChatGPT chatbot, based on the GPT series of large language models (LLM) can equal or surpass human performance in academic examinations and tests for professional certifications (OpenAI, 2023). Despite their remarkable performance, LLMs sometimes produce text that is semantically or syntactically plausible but is, in fact, factually incorrect or nonsensical (i.e., hallucinations). The models are optimized to generate the most statistically likely sequences of words with an injection of randomness. They are not designed to exercise any judgment on the veracity or feasibility of the output. In what applications can we leverage artificial intelligence that is brilliant in many ways yet cannot be trusted to produce reliably accurate results? One possibility is to turn their weaknesses – hallucinations and inconsistent quality – into a strength (Terwiesch, 2023). ChatGPT can generate ides far faster than humans. This gives them a huge edge in coming up with a few great ideas. For this study the professors gave ChatGPT and the students the same prompt. I believe I would get a big zero in coming up with a truly innovative product idea. Could you do this? How fast? System Prompt “You are a creative entrepreneur looking to generate new product ideas. The product will target college students in the United States. It should be a physical good, not a service or software. I’d like a product that could be sold at a retail price of less than about USD 50. The ideas are just ideas. The product need not yet exist, nor may it necessarily be clearly feasible. Number all ideas and give them a name. The name and idea are separated by a colon.” User Prompt “Please generate ten ideas as ten separate paragraphs. The idea should be expressed as a paragraph of 40-80 words.” Do LLMs Enhance Productivity in Generating Ideas? The answer to this question is straightforward. ChatGPT-4 is very efficient at generating ideas. This question does not require much precision to answer. Two hundred ideas can be generated by one human interacting with ChatGPT-4 in about 15 minutes. A human working alone can generate about five ideas in 15 minutes (Girotra et al., 2010). Humans working in groups do even worse. In short, the productivity race between humans and ChatGPT is not even close. Still, the old saying that ideas are a dime a dozen is perhaps a tad optimistic. A professional working with ChatGPT-4 can generate ideas at a rate of about 800 ideas per hour. At a cost of USD 500 per hour of human effort, a figure representing an estimate of the fully loaded cost of a skilled professional, ideas are generated at a cost of about USD 0.63 each, or USD 7.50 (75 dimes) per dozen. At the time we used ChatGPT-4, the API fee for 800 ideas was about USD 20. For that same USD 500 per hour, a human working alone, without assistance from an LLM, only generates 20 ideas at a cost of roughly USD 25 each, hardly a dime a dozen. For the focused idea generation task itself, a human using ChatGPT-4 is thus about 40 times more productive than a human working alone. What Is The Quality Distribution of the Ideas Generated Using LLMs? A “stochastic parrot” can generate ideas, and LLMs do so shockingly productively. But we don’t care about quantity alone. More typically, the objective of idea generation is to generate at least a few truly exceptionally good ideas. In most innovation settings, we’d rather have 10 great ideas and 90 terrible ideas than 100 ideas of average quality. We, therefore, care about the quality distribution of the ideas, and in particular, the quality of the best few ideas in a sample. Of course, we might as well also measure the mean and standard deviation of the three sets of ideas, and we do so. Two useful measures of the extreme values are: What is the average quality of the ideas in the top decile of each of the three samples? Which sources provided the ideas comprising the top 10 percent of the ideas in the pooled sample? Chat-GPT generated the best-rated idea in our sample, with an 11% higher purchase probability than the best human idea. The average quality of the top decile in each of the three pools also follows the same pattern as average quality— seeded Chat-GPT ≻ ChatGPT ≻ Humans. Overall, we have 400 ideas, with an equal number generated by ChatGPT and humans. In the top 40 ideas (top decile) a full 35 (87.5%) are those generated by ChatGPT. > From <https://www.zerohedge.com/markets/whos-better-generating-innovative-ideas-chatgpt-or-mba-students>