New research from MIT Sloan affiliates reveals that improvements in generative artificial intelligence performance are not solely due to advancements in large language models. A large-scale experiment found that only half of the observed performance gains after transitioning to a more advanced AI model were attributed to the model itself.
The study underscores a crucial reality for businesses: investing in new AI tools will not yield their anticipated value unless employees also refine their usage. The researchers suggest that prompting is a learnable skill that individuals can quickly improve, even without formal instruction. David Holtz, an assistant professor at Columbia University and a research affiliate at the MIT Initiative on the Digital Economy, stated, “People often assume that better results come mostly from better models. The fact that nearly half the improvement came from user behavior really challenges that belief.”
The experiment involved nearly 1,900 participants who were randomly assigned to one of three versions of OpenAI’s DALL-E image generation system. Participants were tasked with recreating a reference image by typing instructions into the AI and were incentivized with a bonus payment for the top 20% of performers. The researchers reported several key findings, including that participants using the baseline version of DALL-E 3 produced images more similar to the target image compared to DALL-E 2 users.
Participants utilizing the baseline DALL-E 3 wrote prompts that were 24% longer than those of DALL-E 2 users, with greater similarity to each other and a higher proportion of descriptive words. Approximately half of the improvement in image similarity was attributed to the enhanced model, while the other half resulted from users adjusting their prompts to capitalize on the capabilities of the improved models.
The researchers believe the same pattern is likely to apply to other tasks, including writing and coding. The study demonstrated that the ability to adapt prompts over time was not exclusive to tech-savvy users. Holtz commented, “People often think that you need to be a software engineer to prompt well and benefit from AI. But our participants came from a wide range of jobs, education levels, and age groups — and even those without technical backgrounds were able to make the most of the new model’s capabilities.”
The data suggests that effective prompting is more about clear communication than coding. Holtz noted, “The best prompters weren’t software engineers. They were people who knew how to express ideas clearly in everyday language, not necessarily in code.” Eaman Jahani, an assistant professor at the University of Maryland and a digital fellow at the MIT Initiative on the Digital Economy, observed that generative AI has the potential to narrow performance gaps between users.
Jahani said, “People who start off at the lower end of the [performance] scale benefited the most, which means the differences in outcomes became smaller. Model advances can actually help reduce inequality in output.” However, Jahani clarified that the team’s findings are applicable to tasks with clear, measurable outcomes and an identifiable upper limit for a good result.
One unexpected finding was that rewriting prompts using generative AI led to a significant decrease in performance. The group that used DALL-E 3 with generative AI automatically rewriting their prompts experienced a 58% degradation in performance compared to the baseline DALL-E 3 group. Holtz explained that automatic prompt rewriting can introduce extraneous details or alter the intended meaning of the user’s input, causing the AI to produce an incorrect image.
The study’s implications for businesses are clear: beyond selecting the “right” AI model, leaders must prioritize enabling effective user learning and experimentation. Jahani emphasized that prompting is not a plug-and-play skill. “Companies need to continually invest in their human resources. People need to be caught up with these technologies and know how to use them well.”
To maximize the benefits of generative AI, the researchers offer several key priorities for business leaders, including investing in training and experimentation, designing for iteration, and being cautious with automation. The paper was co-authored by MIT Sloan PhD students Benjamin S. Manning, Hong-Yi TuYe, and Mohammed Alsobay, as well as Stanford University PhD student Joe Zhang, Microsoft computational social scientist Siddharth Suri, and University of Cyprus assistant professor Christos Nicolaides.
