LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writers
arXiv:2602.16162v1 Announce Type: new Abstract: We argue that uncertainty is a key and understudied limitation of LLMs' performance in creative writing, which is often characterized as trite and clich\'e-ridden. Literary theory identifies uncertainty as a necessary condition for creative expression, while current alignment strategies steer models away from uncertain outputs to ensure factuality and reduce hallucination. We formalize this tension by quantifying the "uncertainty gap" between human-authored stories and model-generated continuations. Through a controlled information-theoretic analysis of 28 LLMs on high-quality storytelling datasets, we demonstrate that human writing consistently exhibits significantly higher uncertainty than model outputs. We find that instruction-tuned and reasoning models exacerbate this trend compared to their base counterparts; furthermore, the gap is more pronounced in creative writing than in functional domains, and strongly correlates to writing q
arXiv:2602.16162v1 Announce Type: new Abstract: We argue that uncertainty is a key and understudied limitation of LLMs' performance in creative writing, which is often characterized as trite and clich\'e-ridden. Literary theory identifies uncertainty as a necessary condition for creative expression, while current alignment strategies steer models away from uncertain outputs to ensure factuality and reduce hallucination. We formalize this tension by quantifying the "uncertainty gap" between human-authored stories and model-generated continuations. Through a controlled information-theoretic analysis of 28 LLMs on high-quality storytelling datasets, we demonstrate that human writing consistently exhibits significantly higher uncertainty than model outputs. We find that instruction-tuned and reasoning models exacerbate this trend compared to their base counterparts; furthermore, the gap is more pronounced in creative writing than in functional domains, and strongly correlates to writing quality. Achieving human-level creativity requires new uncertainty-aware alignment paradigms that can distinguish between destructive hallucinations and the constructive ambiguity required for literary richness.
Executive Summary
This study presents a novel analysis of the uncertainty gap between human-authored creative writing and Large Language Model (LLM) generated content. Through a controlled information-theoretic analysis of 28 LLMs on high-quality storytelling datasets, the authors demonstrate that human writing consistently exhibits significantly higher uncertainty than model outputs. The results highlight the tension between ensuring factuality and reducing hallucination, and the need for creative expression. The study's findings have significant implications for the development of LLMs in creative writing, and the need for new uncertainty-aware alignment paradigms. The authors' conclusions underscore the importance of balancing model performance with the creative potential of human writers.
Key Points
- ▸ LLMs exhibit significantly lower uncertainty in creative writing than professional writers
- ▸ Human writing consistently exhibits higher uncertainty than model outputs
- ▸ Instruction-tuned and reasoning models exacerbate the trend compared to base counterparts
Merits
Insight into the Uncertainty Gap
The study provides a novel analysis of the uncertainty gap between human-authored creative writing and LLM generated content, shedding light on a previously understudied limitation of LLMs' performance in creative writing.
Methodological Rigor
The study employs a controlled information-theoretic analysis of 28 LLMs on high-quality storytelling datasets, ensuring a robust and reliable methodology for evaluating the uncertainty gap.
Demerits
Limited Generalizability
The study's findings may not be generalizable to other domains or applications, and further research is needed to confirm the results in different contexts.
Methodological Assumptions
The study relies on certain methodological assumptions, such as the definition of uncertainty and the selection of storytelling datasets, which may limit the interpretability of the results.
Expert Commentary
The study's findings are significant because they highlight the tension between ensuring factuality and reducing hallucination, and the need for creative expression. The results suggest that current alignment strategies may be too restrictive, and that new uncertainty-aware alignment paradigms are needed to distinguish between destructive hallucinations and the constructive ambiguity required for literary richness. The study's methodology is robust and reliable, and the results are well-supported by the data. However, further research is needed to confirm the findings in different contexts and to explore the implications for the development of LLMs in creative writing.
Recommendations
- ✓ Develop new uncertainty-aware alignment paradigms that can distinguish between destructive hallucinations and the constructive ambiguity required for literary richness.
- ✓ Conduct further research to confirm the findings in different contexts and to explore the implications for the development of LLMs in creative writing.
