The Tower of Babel in artificial intelligence: who should speak, and when it pays to listen
Authors: Mohammad Salahshour
Organizations: Department of Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany · Centre for the Advanced Study of Collective Behaviour, University of Konstanz, Konstanz, Germany · Department of Biology, University of Konstanz, Konstanz, Germany
Two machines can each recognize an object well and still fail together. We study classifiers that see different parts of one object and send short messages to a receiver that identifies its class. Coordination asks how to design messages together; cooperation asks whether the team outperforms one sender given the whole budget. On three digit datasets, messages designed together beat messages designed apart by 18.07 to 22.68 percentage points on average with two-bit messages, with gains in all 24 configurations. A fixed agreement, you report the row and I the column, recovers about half this gain, with only the receiver learned. Senders that draw the same distinctions waste a narrow vocabulary, and senders whose distinctions cross multiply it. A team has no more possible messages than one sender with the same budget, so a partner must bring evidence the first lacks. A solvable model shows individual statistics alone cannot determine whether a partnership helps. On MNIST digits, six shared bits remove 49.54 percent of the strongest tested solo's errors on average. Paired measurements correctly predicted most sensor teams would reduce average accuracy on new participants. Who should speak is a property of the pair, and must be measured in pairs.
Figures & tables
Figure 1: Two senders, short messages and a learned receiver. A A handwritten digit is split into a top view xA and a bottom view xB . Each sender classifies its view, and its code ( fA or fB , labelled encoder in the panel) maps the predicted label to one of K=2b symbols. Sender A sends the symbol m and sender B the symbol n . The pair selects one cell of the receiver’s table q(y∣m,n) , which holds class probabilities learned from labelled examples, and the receiver answers with the most probable class. The displayed probabilities are illustrative. Digit image adapted from Alpaydin and Kaynak’s UCI Optical Recognition of Handwritten Digits dataset (CC BY 4.0), split into two views. B Real two-bit codes for one training split and one assignment of classifier types (logistic sender A, multilayer perceptron sender B) on optical digits [ 30 ] . Labels above each grid show how each sender groups its predicted classes into symbols. Each grid entry is the class that lands in that cell when both senders predict correctly. Classes nameable is the maximum number of classes the receiver can answer correctly, one per distinct message pair, when both senders predict correctly. (i) Independently designed codes produce 7 distinct pairs. Three shaded cells each merge two digits, of which the receiver can name only one. Test accuracy is 63.38%. (ii) Jointly designed codes give every digit its own pair, with test accuracy 88.31%. C Test accuracy at four total bits for independently designed 2+2 codes, the whole-budget solo and jointly designed 2+2 codes (legend). Markers show eight units per dataset (two assignments of classifier types to views, four training splits); horizontal lines, their means. Circles, optical digits; squares, pen digits [ 31 ] ; triangles, multiple-features digits [ 32 ] . On optical and pen digits the solo sends the multilayer perceptron’s label through a fitted receiver. On multiple-features digits it is the classifier trained on Zernike moments, using its own decision, a choice made after the results were seen (SI Appendix, section S13). The units of a dataset share its test objects (1,797, 3,498 and 1,000 objects, respectively).
Figure 2: Designing the codes together lets two short messages name every class. Class-label messages of b bits ( K=2b symbols) per sender. Joint design fits the two codes together. Independent design fits each code as if its sender were the receiver’s only source. In A, C and D these designs use vermilion filled and blue open markers, respectively. In A their lines are solid and dashed. Dataset shapes in A, B and D identify optical (circles) [ 30 ] , pen (squares) [ 31 ] and multiple-features digits (triangles) [ 32 ] . A Test accuracy against bits per sender for (i) optical, (ii) pen and (iii) multiple-features digits. Markers are means over eight units, with lowest-to-highest bars. The legends in (i) and (ii) apply to all three. The fusion reference multiplies the two senders’ full class-probability vectors with no message budget. The full-input model reads the whole input. Four possible message pairs allow at most four predicted classes at one bit per sender. The short black bar is their accuracy ceiling, the combined frequency of the four most common test classes. B Coordination gain at two bits per sender in each of the 24 units (filled, logistic sender on view A; open, multilayer perceptron (MLP) on view A); horizontal lines, dataset means; dashed line, zero. C Partition crossing, the number of classes the two codes can name when both senders are correct, against bits per sender, pooled over the 24 units (circle area, number of units; dashes, means); dotted line, min(K2,10) for crossed groupings; dashed line, min(K,10) for identical groupings. D Two-bit test accuracy of six ways of building the codes, one row per dataset within each scheme, from top to bottom: designed apart (mean, lowest-to-highest bar); the same 500 random code pairs in every unit, pooled over eight units (light boxes), and the 18 of them that name all ten classes (darker boxes); the row-and-column convention; the best random pair chosen on selection data; and designed together. Boxes span the 5th to 95th percentiles, with black median ticks and whiskers at the extremes. Filled and open dark-grey markers identify the convention and selection-chosen random pair. For the four data-blind schemes only the receiver is fitted.
Figure 3: Who should speak is a property of the pair. A The largest gain over the whole-budget solo that any two-bit team can achieve, G2∗ of Eq. ( 1 ), in percentage points (colour), against the coupling ρ of the two senders’ private bits and the solo error ϵ . Along every horizontal line both senders keep the same individual statistics and solo accuracy 1−ϵ . Left of the dashed boundary ρ=1−2ϵ no team can help. B Accuracy on fresh objects against the number of receiver-fitting objects in the same family at ϵ=0.1 and ρ=0.9 . Solid curves give exact expectations for the fitted receivers, and open markers show 200-simulation means (legend). The original-prediction solo requires no receiver fitting (black dashed, 1−ϵ ). The team with its receiver known scores (1+ρ)/2 (vermilion dotted). C Detecting a positive gain at ϵ=0.1 : the probability that a one-sided paired binomial test at the 5% level approves the team with its receiver known over the solo using its classifier’s original prediction, against ρ , for 32, 128, 512 and 2,048 calibration objects (legend). Curves give exact probabilities; points, frequencies over 200 synthetic repetitions. The dashed vermilion line marks ρ=1−2ϵ .
Figure 4: When the second sender pays: three outcomes on natural data. Gains above zero in A and B(ii) mean that the team beats one sender given the whole budget. The solo receives all T bits per object. In A (except MNIST) and B(ii), the team chooses its division of T on selection data. A Team minus solo, in percentage points of test accuracy, against T . For optical (circles) [ 30 ] and pen digits (squares) [ 31 ] the solo uses the stronger sender’s label code. It carries the label without loss from four bits, and its accuracy then stays constant. For multiple-features digits (triangles) [ 32 ] the solo is the Zernike sender’s label code, held at its lossless four-bit value from five bits on. MNIST halves (open diamonds) [ 39 ] use a team with equal bits per sender and the strongest tested solo, including encoders trained end to end, over 3 training splits. Points are means, with lowest-to-highest bars over units or splits. The separate column shows gains over the solo at its full label for the fusion reference and one model reading the whole input (legend). B Text halves (DBpedia; [ 40 ] ; means over 16 ordered language-model pairs), with class counts identified in the legend, against T : (i) coordination gain with equal bits per sender; (ii) the team with its division of T chosen on selection data, minus the solo. Dotted verticals mark T=log2C , beyond which the solo’s label is lossless. C Every studied team on one strip of test accuracy, showing the solo, team at its best budget and fusion reference (legend; budget in the row label, solo at the same budget; fusion undefined for text). Grey bars span the solo to the fusion reference. Rows are grouped by outcome, named at the right. The six-view row shows the best two of the six multiple-features views at two bits each. The three-sensor team from Daily and Sports Activities [ 33 ] , at two bits per sensor, failed the 80% competence requirement (only the left-leg sensor cleared it on development data) and is evaluated on recordings from four participants whose data were not used to fit or select the team. Group labels describe measured outcomes after the fact and do not identify the cause of a shortfall (SI Appendix, section S13).
Figure 5: Deciding before deployment: what transferred to new objects and new people. A Gain of the team over the whole-budget solo predicted before testing (horizontal) against the realized gain (vertical), in percentage points. Pentagons show body-worn sensors from Daily and Sports Activities [ 33 ] , predicted from paired calibration on development persons and realized as means over four held-out persons with codes and receivers frozen (filled, three to five sensors; open, two sensors). Hexagons show dry-bean classification [ 41 ] , with one sender reading size measurements and the other shape measurements, predicted from a joint table assembled from separate calibration cohorts (filled grey, three total bits; open grey, four; vermilion, the one design the frozen rule switched on). The inset enlarges the Dry Bean designs near the origin. Its vermilion dashed line marks the activation threshold of one predicted percentage point. Dashed diagonals mark equality. Dotted lines in the main panel mark zero. B Coordination gain on the held-out persons against total bits n×b for two to five sensors (filled circles, one bit per sensor; open squares, two bits), means with lowest-to-highest bars over persons. The vertical line marks Kn=19 , the start of the coordination window (shaded). The prediction for teams of three to five sensors, recorded before testing, was that the one-bit series peaks at five sensors, the first one-bit team inside the window. C Unseen message combinations are combinations of sensor messages that never occurred together during receiver fitting. The horizontal axis shows the predicted fraction of activity recordings producing these combinations, estimated from development data; the vertical axis shows the observed fraction, averaged over four new participants. Both axes are logarithmic ( 0.01=1% ). Filled symbols: codes and receiver learned from one person’s recordings and applied unchanged to four other people. Open symbols: each of those four people is evaluated using codes and a receiver learned from the other three people’s recordings. Labels give the number of sensors and bits per sensor. Dashed line: observed equals predicted; dotted line: observed is ten times predicted. Only conditions with nonzero observed fractions are shown (7 of 16). D Development accuracy of the best sender for each bird [ 42 , 43 ] and research-paper [ 44 ] view (dashed line: 80% competence threshold). Filled markers show selection accuracies in the first study, with bars spanning five split seeds. Open markers show qualification accuracies after source and readout selection on a separate subset of the same development cohort. The follow-up used a quantized seven-billion-parameter language model for captions, keywords and abstracts, and re-measured the photograph sender. Bird attributes, a third sender reading the archive’s binary attribute annotations and used only in three-sender teams, do not enter the pair requirement. The two text halves from DBpedia [ 40 ] show the procedure on a task whose outcome is known: one clears the threshold and the other falls just short. No pair reached it.