Studying word order through iterative shuffling
Organizations: Microsoft · University of Maryland · Microsoft Research
Abstract
As neural language models approach human performance on NLP benchmark tasks, their advances are widely seen as evidence of an increasingly complex understanding of syntax. This view rests upon a hypothesis that has not yet been empirically tested: that word order encodes meaning essential to performing these tasks. We refute this hypothesis in many cases: in the GLUE suite and in various genres of English text, the words in a sentence or phrase can rarely be permuted to form a phrase carrying substantially different information. Our surprising result relies on inference by iterative shuffling (IBIS), a novel, efficient procedure that finds the ordering of a bag of words having the highest likelihood under a fixed language model. IBIS can use any black-box model without additional training and is superior to existing word ordering algorithms. Coalescing our findings, we discuss how shuffling inference procedures such as IBIS can benefit language modeling and constrained generation.
Figures & tables
| Base LM | Inference | BLEU | NLL |
|---|---|---|---|
| LSTM | BS-512 | 44.5 | – |
| GPT-2 Small | BS-512 | 42.4 | 5.60 |
| GPT-2 Small | BS-512 | 45.4 | 5.48 |
| GPT-2 Small | BS-1024 | 44.0 | 5.52 |
| GPT-2 Small | BS-1024 | 47.1 | 5.43 |
| GPT-2 Small | IBIS | 50.0 | 5.19 |
| Base LM | IBIS output |
|---|---|
| Distil- GPT-2 | Ibises, all mud and crustaceans, usually feed as a group, have long downcurved bills, usually probing for food items. |
| GPT-2 Small | Ibises, all mud crustaceans, usually as a group, have long downcurved bills, usually probing for food items and feed. |
| GPT-2 Medium | Ibises as a group usually have long downcurved bills, usually probing for feed, mud, crustaceans, and all food items. |
| GPT-2 Large | Ibises usually feed as a group, usually have long, downcurved bills, all probing for food items, mud and crustaceans. |
| GPT-2 XL | Ibises usually feed as a group, and all have long, downcurved bills, probing for food items, usually mud crustaceans. |
| original | Ibises all have long, downcurved bills, and usually feed as a group, probing mud for food items, usually crustaceans. |
| I was just there again in April and bragged to my friends about how great it was and we were all horribly disappointed. |
| just bragged to all my friends about how great it was in April and we were there again and I was horribly disappointed. |
| Mao Zedong’s philosophical essay furthered Marx and Lenin’s thesis and suggested that all existence is the result of contradiction. |
| Marx and Lenin’s philosophical essay suggested and furthered Mao Zedong’s thesis that all existence is the result of contradiction. |
| We introduce a longevity feature to the classical optimal dividend problem by adding a constraint on the time of ruin of the firm. |
| We introduce the classical problem of the optimal ruin of a firm by adding longevity to the feature time constraint on a dividend. |
| (no punct.) | Yelp | Wiki | arXiv | |||
|---|---|---|---|---|---|---|
| Length | BLEU | PR | BLEU | PR | BLEU | PR |
| 5–9 | 81.5 | 0.94 | 77.4 | 0.91 | 75.0 | 0.91 |
| 10–19 | 69.5 | 0.91 | 67.4 | 0.90 | 51.6 | 0.80 |
| 20–29 | 51.3 | 0.97 | 51.9 | 0.96 | 37.0 | 0.79 |
| 30–39 | 35.7 | 1.04 | 38.8 | 1.01 | 27.2 | 0.79 |
| 40–49 | 24.7 | 1.01 | 31.4 | 1.07 | 23.1 | 0.82 |
| perplexity | token acc % | ||||||
|---|---|---|---|---|---|---|---|
| latent | top | rand | latent | top | rand | acc % | |
| 1 | 30.1 | 38.1 | 100.0 | ||||
| 2 | 30.5 | 31.7 | 49.4 | 38.1 | 38.0 | 31.3 | 98.5 |
| 3 | 31.4 | 34.2 | 72.9 | 37.5 | 37.5 | 25.4 | 96.4 |
| 4 | 32.2 | 36.2 | 97.0 | 37.3 | 37.2 | 21.8 | 94.0 |
| 5 | 33.2 | 39.3 | 123.5 | 36.9 | 36.7 | 18.1 | 90.9 |
| entailment | paraphrase | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| CoLA | MNLI-m/mm | QNLI | RTE | MRPC | QQP | STS-B | SST-2 | WNLI | |
| MCC | acc % | acc % | acc % | F1 / acc % | F1 / acc % | / | acc % | acc % | |
| original | 58.2 | 83.7 / 84.1 | 90.7 | 64.8 | 88.0 / 83.6 | 90.6 / 87.3 | 88.3 / 88.1 | 91.9 | 39.4 |
| random | 0.4 | 65.5 / 65.4 | 74.8 | 56.5 | 81.8 / 73.1 | 74.2 / 72.3 | 82.1 / 82.0 | 80.7 | 54.3 |
| IBIS | 39.2 | 79.4 / 79.8 | 86.4 | 64.8 | 86.1 / 81.0 | 88.9 / 84.4 | 87.8 / 87.5 | 86.0 | 40.3 |
| Fairy in techniques for coronavirus vaccine has been confirmed by research. |
| The mouse was hungry. It started feeding itself by taking UndergroundMISC’s engineered growth cheese assay . Now the mouse is still hungry. |
| The cat was hungry. Someone picked up the mouse and chased the cat away from it . Now the cat is still hungry. |
| The cat liked mice. His appetite for sweet treats was a little more intense . Now the cat hates mice. |
| The Dragon King has reached out to the court at the request of a judge with the Magic Kingdom . The Minister of Magic declined to comment. |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Let’s make a bet: winner owes loser 50 dollars. |
| Let’s make a bet: loser owes winner 50 dollars. |
| The mouse chases the cat. |
| The cat chases the mouse. |
| Thoughts without content are empty, intuitions without concepts are blind. |
| Thoughts without content are empty, intuitions without concepts are blind. |
| Experience without theory is blind, but theory without experience is mere intellectual play. |
| 12 11 10 9 8 7 6 5 4 3 2 1 |
| 0 1 2 3 4 5 6 7 |
| Fairy in techniques for coronavirus vaccine has been confirmed by research. |
| In the aftermath of the storm , humans and hippos , penguins , dolphins, and the dinosaurs were forced into the streets by the hive mind of the Internet. |
| See the full video for a long list of physicists and Mars ’ history as well as insights to the origins of bubbles . |
| Drones, drugs , climate change : the search for answers to nothing. |
| The cat was hungry. Someone picked up the mouse and chased the cat away from it . Now the cat is still hungry. |
| The mouse was hungry. It started feeding itself by taking UndergroundMISC’s engineered growth cheese assay . Now the mouse is still hungry. |
| The cat liked mice. His appetite for sweet treats was a little more intense . Now the cat hates mice. |
| The world just had a cat and a dog fight . Now the cat isn’t hungry anymore. |
| The cat and the mouse were both hungry. Although they oversee the animal kingdom, their predators eat more . Now only one of them is hungry. |
| I thought I could find my future and fix it , but everyone was running in panic . |
| The announcement of the student strike is expected to be welcomed by many of us . Students across Canada are rejoicing. |