Generation of Humorous Riddles with Prompt Chaining

This paper focuses on the generation of punning riddles with MariTalk via a prompt chaining pipeline, following the rules of SECO. We provide a comparison between the two systems through a human preference study (Comparative Mean Opinion Score, CMOS, with 10 evaluators) and statistical significance testing, showing that they do not differ significantly in terms of funniness (Wilcoxon signed-rank test, p = 0.41; Krippendorf's alpha = 0.341).

Methodology

1. Generation: for each SECO concept/relation pair, a 3-step prompt chain asks MariTalk (sabia-2-medium) to (i) find related words for each half of the pun, (ii) generate joke candidates, (iii) select the funniest one.
2. Sampling: 100 generated jokes are randomly sampled for evaluation.
3. Human evaluation: a Gradio interface presents each generated joke against its SECO counterpart and the evaluator rates their preference in a scale from -2 to +2.
4. Analysis: scores are tested for normality (Shapiro-Wilk), central tendency (Wilcoxon signed-rank), and inter-rater agreement (Krippendorff's alpha); a two-way ANOVA checks differences in evaluators' distributions.

Install

To install the project, you can use pipenv (Python 3.10).

pipenv install

Or by using pip.

pip install -r requirements.txt

Environment

You need a .env file in the repository root with your MariTalk API key:

MARITALK_API_KEY=your-key-here

External data (not in this repository)

generate_llm.py and evaluation scripts depend on two files from the original SECO project that are not distributed here:

• aglut_lexico_v2.txt: SECO's compounds lexicon.
• inlg_f1281359_300_man.csv: SECO's human annotated riddles with crowdsourced funniness ratings.

To obtain these files, we recommend contacting the original SECO authors.

Run

The entire project is in the scripts/ folder and is expected to be run from the repository root. The scripts are as follows:

• generate_llm.py: Provides the humor generation prompt chaining, by using MariTalk with an active API key stored in .env.
• sample_evaluation.py: Random sampling of 100 jokes for evaluation.
• evaluation_interface.py: CMOS evaluation interface created with Gradio. Note: the interface is set for our institutional server, change server to 127.0.0.1 to run locally.
• visualize_evaluation.py: Compute metrics and images for analysing the evaluation results.
• stat_test.R: Do statistical significance testing.

Results

All results can be found in the results/ folder. The folder is organized as follows:

• evaluation/evaluation_2024_04_16_15_55_49.jsonl: Contains all evaluation results. Each evaluator has a single uuid and evaluates joke pairs with unique joke_id. The score sign is already corrected (negative values always represent a preference for SECO).
• img/: Images created for the paper.
• jokes.jsonl: All results from the LLM. Contains the input concept, its sub-words, relations used, all prompts and responses, and the final LLM joke in column joke.
• evaluation_sample.jsonl: Sample of 100 jokes used for evaluation.
• evaluation_scores.jsonl: Evaluation result. Contains the LLM joke in column joke, and SECO's original riddle in adivinha. The final score already aggregates all evaluators' opinions.
• stat_test.txt: Significance testing results.

How to cite

@inproceedings{InacioMarcioLimaGoncaloOliveira2024,
  title     = {Generation of {{Punning Riddles}} in {{Portuguese}} with {{Prompt Chaining}}},
  booktitle = {Proceedings of the 15th {{International Conference}} on {{Computational Creativity}} ({{ICCC}}'24)},
  author    = {In{\'a}cio, Marcio Lima and Gon{\c c}alo Oliveira, Hugo},
  year      = {2024},
  month     = jun,
  publisher = {Association for Computational Creativity (ACC)},
  address   = {J{\"o}nk{\"o}ping},
  url       = {https://computationalcreativity.net/iccc24/full-papers/ICCC24_paper_143.pdf}
}

Inácio, M. L., & Gonçalo Oliveira, H. (2024, June). Generation of Punning Riddles in Portuguese with Prompt Chaining. Proceedings of the 15th International Conference on Computational Creativity (ICCC’24). 15th International Conference on Computational Creativity (ICCC’24), Jönköping. https://computationalcreativity.net/iccc24/full-papers/ICCC24_paper_143.pdf