Tensorized Automata: Algebraically Informed Representation Learning for Sequential Decision Making
Organizations: Department of Computer Science, University of Texas at Austin
Abstract
Task representations for sequential decision making should describe both an objective and progress toward it. We present an algebraically informed pipeline that uses the relational structure and language semantics of alternating finite automata (AFAs) to learn control features and compile compact execution operators. AND-OR message passing learns task and state vectors; selected configurations define an event-driven Boolean matrix operator. We prove that initial and successor language coverage make its carrier represent the exact residual after every prefix, with a sufficient capacity bound in terms of reachable configurations. Conjunction and disjunction use the same pipeline. Fidelity and composition audits evaluate copied formulas and learned disjunct assignments. In five-policy-seed XLand experiments with larger training pools, pooled execution-state vectors outperform raw carrier bits and task-blind conditioning across three task pools under two rejection rewards. They achieve 77.3% success on multi-event tasks and 99.4% on a four-task stratum with 33--128-state minimal DFAs, using 16 operator slots. LetterWorld results show useful residual conditioning alongside a remaining gap to RAD. These results connect algebraically informed representation learning to compact task execution and useful policy conditioning.
Figures & tables
| Exactly equivalent compositions (out of 30) | |||
| Formula assignment | Copied | Learned | Learned |
| Transition construction | Syntactic | Semantic | Syntactic |
| Conjunction, | 30 | 25 | 20 |
| Disjunction, | 30 | 29 | 20 |
| Conjunction, | 6 | 6 | 5 |
| RAD-compatible | Multi-event | Plain | ||||
|---|---|---|---|---|---|---|
| Conditioning | Rejection reward | Rejection reward | Rejection reward | Rejection reward | Rejection reward | Rejection reward |
| Task-blind | 0.510 | 0.307 | 0.623 | 0.240 | 0.749 | 0.758 |
| Ours/Initial | 0.560 | 0.429 | 0.634 | 0.505 | 0.775 | 0.771 |
| Ours/Residual | 0.656 | 0.647 | 0.770 | 0.770 | 0.828 | 0.821 |
| Ours/Carrier bits | 0.551 | 0.457 | 0.662 | 0.634 | 0.759 | 0.752 |
| Ours/Carrier vectors | 0.653 | 0.640 | 0.773 | 0.755 | 0.833 | 0.833 |
Appendix figures & tables64 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning and domain |
|---|---|
| Input AFA, event alphabet, source states, accepting states. | |
| Source-state count and alphabet size . | |
| Positive Boolean formulas; initial formula and transition map . | |
| Subsets of appearing as terms in the irredundant disjunctive normal form of . | |
| Event words and empty word; is length. | |
| Language with initial formula ; residual . |
| Symbol | Meaning and domain |
|---|---|
| Shared encoder, decoder, and policy parameters. | |
| Learned formula-assignment map and its initial/transition formulas. | |
| Formula index and eligible input hyperedges. | |
| Numbers of disjunct and conjunct positions in each decoded formula. | |
| Nonnegative weight assigning edge to disjunct ; sums to one over eligible edges plus the false choice. | |
| Copied target symbol in conjunct position of edge , padded with . |
| Display name | Run/legend code | Conditioning input |
|---|---|---|
| Task-blind | none | Zero vector. |
| Ours/Initial | m-init | Fixed initial Morphata embedding. |
| RAD/Initial | r-init | Fixed initial RAD embedding. |
| Ours/Residual | m-res | Morphata embedding of the current residual language. |
| RAD/Residual | r-res | RAD embedding of the current residual language. |
| Ours/Carrier bits | m-op | Initial embedding concatenated with Boolean active-slot indicators. |
| Prefix | Residual language | Configuration-NFA pooling | Semantic pooling |
|---|---|---|---|
| Source | Four letters | Sixteen letters |
|---|---|---|
| Random DFA | 316 | 143 |
| Fresh random AFA | 719 | 146 |
| Reverse random DFA | 832 | 398 |
| -th from end | 181 | 91 |
| Modular counters | 108 | 70 |
| Eventually conjunction | 100 | 89 |
| Setting | Value or specification |
|---|---|
| Message passing | AND-OR, eight rounds, width 32. |
| Task readout | Initial-vertex state, width 32, without final normalization. |
| Source-state capacity | 16 pinned source-state vectors. |
| Operator capacity | 16 configuration slots. |
| Formula source | Copied conjuncts, learned disjunct assignments, copied acceptance. |
| Configuration source | Forward closure from thresholded initial terms. |
| Alphabet | Copied formulas | Learned assignments |
|---|---|---|
| 4 | c3c-s4 , 75k | c3lsem-s0 , 75k |
| 10 | c3c10-s0--s2 , 75k | c3lsem10-s0 , 75k |
| 16 | c3c16-s0 , 40k | c3lsem16-short-s0 , 45k |
| Membership accuracy | ||||
|---|---|---|---|---|
| Layer | States, | Initial, | States, | Initial, |
| AND-OR | .949/.951 | .882/.885 | .902/.903 | .841/.847 |
| Sum | .897/.898 | .843/.839 | .867/.868 | .810/.816 |
| Membership accuracy | ||||
|---|---|---|---|---|
| Layer | Initial, length 2 | Initial, length 3 | Random DFAs | Pooled, length 2 |
| AND-OR | .900/.897/.888 | .847/.848/.840 | .798/.823/.813 | .775/.772/.772 |
| Sum | .820/.842/.860 | .787/.788/.793 | .709/.702/.710 | .787/.780/.783 |
| Formulas | Configurations / transitions | Seeds | Product /119 | Plain /149 | /40 | FD |
|---|---|---|---|---|---|---|
| Learned | per slot / syntactic | 5 | 43–53 | 130–144 | 2–4 | .046–.270 |
| Learned | closure / syntactic | 5 | 95–100 | 120–132 | 2–4 | .031–.045 |
| Learned | closure / semantic | 3 | 114–115 | 139–143 | 4–6 | .019 |
| Copied | closure / syntactic | 5 | 119 | 149 | – | .000 |
| Copied | closure / semantic | 3 | 119 | 149 | 5–6 | .000 |
| Formulas / configurations | Seeds | Syntactic | Semantic re-step | Gain |
|---|---|---|---|---|
| Learned / per slot | 0–3 | 44–53 | 63–67 | 10–22 |
| Copied / per slot | 0–1 | 52–59 | 73–75 | 16–21 |
| Learned / closure | 0–2 | 98–100 | 105–107 | 5–9 |
| Model | AFA pair AUC | DFA pair AUC | Collapsed / seeds |
|---|---|---|---|
| Learned / per slot | .914–.960 | .774–.815 | 0/5 |
| Learned / closure, no identity | .504–.888 | .797–.936 | 1/5 |
| Learned / closure | .954–.967 | .758–.872 | 0/5 |
| Learned / semantic | .944–.962 | .801–.860 | 0/3 |
| Copied / closure | .845–.922 | .896–.951 | 0/5 |
| Model, seed | Product | Plain | Above capacity | FD | AFA AUC | DFA AUC |
|---|---|---|---|---|---|---|
| Copied/syntactic, 0 | 119 | 149 | – | .000 | .894 | .950 |
| Copied/syntactic, 1 | 119 | 149 | – | .000 | .845 | .896 |
| Copied/syntactic, 2 | 119 | 149 | – | .000 | .913 | .947 |
| Copied/syntactic, 3 | 119 | 149 | – | .000 | .865 | .938 |
| Copied/syntactic, 4 | 119 | 149 | – | .000 | .922 | .951 |
| Copied/semantic, 0 | 119 | 149 | 5 | .000 | .945 | .965 |
| Model, seed | All witness-length gaps | Gaps at least two |
|---|---|---|
| Learned/semantic, 0 | .645 (.612–.680) | .787 (.731–.832) |
| Copied/syntactic, 4 | .724 (.676–.773) | .895 (.862–.924) |
| Learned/syntactic, 0 | .665 (.634–.701) | .834 (.787–.878) |
| Axis | Recorded scope |
|---|---|
| Alphabet | 4, 10, and 16 letters; configurations differ across alphabets. |
| Representation | 16 source-state slots, 16 operator slots, at most 64 candidates. |
| DFA size | Within-budget four-letter rows include minimal DFAs with 144 states. |
| Training tasks | XLand: 25 versus 87 RAD-compatible, 139 multi-event, or 173 plain tasks, at fixed frames. |
| Policy samples | XLand: 1,000 updates at steps (16.384M); LetterWorld: 611 updates at steps (20.021M). |
| Compilation cost | Exact inclusion and bank construction; time and peak memory unmeasured. |
| Conditioning | RAD-compatible | Multi-event | Plain |
|---|---|---|---|
| Rejection reward 0 (validation-selected) | |||
| Task-blind | 0.510 (0.304–0.713) | 0.623 (0.471–0.768) | 0.749 (0.620–0.868) |
| Ours/Initial | 0.560 (0.354–0.759) | 0.634 (0.499–0.761) | 0.775 (0.648–0.886) |
| Ours/Residual | 0.656 (0.444–0.850) | 0.770 (0.637–0.888) | 0.828 (0.709–0.927) |
| Ours/Carrier bits | 0.551 (0.349–0.750) | 0.662 (0.521–0.795) | 0.759 (0.629–0.874) |
| Ours/Carrier vectors | 0.653 (0.448–0.841) | 0.773 (0.641–0.891) | 0.833 (0.716–0.930) |
| Conditioning | RAD-compatible | Multi-event | Plain |
|---|---|---|---|
| Rejection reward 0 (validation-selected) | |||
| Task-blind | 0.532 (0.319–0.738) | 0.602 (0.455–0.746) | 0.739 (0.608–0.859) |
| Ours/Initial | 0.526 (0.322–0.725) | 0.578 (0.442–0.710) | 0.755 (0.626–0.870) |
| Ours/Residual | 0.585 (0.378–0.782) | 0.700 (0.560–0.829) | 0.742 (0.616–0.855) |
| Ours/Carrier bits | 0.424 (0.237–0.619) | 0.492 (0.352–0.635) | 0.689 (0.555–0.813) |
| Ours/Carrier vectors | 0.601 (0.399–0.791) | 0.676 (0.542–0.803) | 0.657 (0.520–0.787) |
| Pool | Comparison | Reward | Reward |
|---|---|---|---|
| RAD-compatible | m-opv none | +0.143 (+0.031, +0.278) | +0.332 (+0.094, +0.609) |
| RAD-compatible | m-opv m-op | +0.101 (+0.015, +0.213) | +0.182 (+0.056, +0.338) |
| RAD-compatible | m-opv m-res | -0.003 (-0.066, +0.057) | -0.007 (-0.066, +0.047) |
| RAD-compatible | r-res m-res | +0.053 (-0.051, +0.171) | +0.053 (-0.051, +0.166) |
| Multi-event | m-opv none | +0.150 (+0.052, +0.262) | +0.515 (+0.239, +0.791) |
| Multi-event | m-opv m-op | +0.110 (+0.030, +0.205) | +0.121 (+0.034, +0.223) |
| Pool | Comparison | Reward | Reward |
|---|---|---|---|
| RAD-compatible | m-opv none | +0.069 (-0.043, +0.201) | +0.174 (-0.015, +0.404) |
| RAD-compatible | m-opv m-op | +0.178 (+0.048, +0.332) | +0.115 (-0.015, +0.249) |
| RAD-compatible | m-opv m-res | +0.016 (-0.068, +0.116) | +0.003 (-0.093, +0.112) |
| RAD-compatible | r-res m-res | +0.021 (-0.078, +0.129) | +0.022 (-0.094, +0.156) |
| Multi-event | m-opv none | +0.074 (-0.034, +0.189) | +0.092 (-0.022, +0.209) |
| Multi-event | m-opv m-op | +0.184 (+0.091, +0.287) | +0.166 (+0.076, +0.265) |
| Reward | Conditioning | DFA | DFA 11–32 | DFA 33–128 |
|---|---|---|---|---|
| Tasks | 17 | 11 | 4 | |
| 0 | Task-blind | 0.513 (0.294, 0.735) | 0.709 (0.463, 0.911) | 0.850 (0.425, 1.000) |
| 0 | Ours/Initial | 0.547 (0.338, 0.750) | 0.745 (0.550, 0.908) | 0.694 (0.250, 0.950) |
| 0 | Ours/Residual | 0.659 (0.439, 0.863) | 0.861 (0.710, 0.973) | 0.988 (0.900, 1.000) |
| 0 | Ours/Carrier bits | 0.535 (0.322, 0.748) | 0.802 (0.623, 0.948) | 0.819 (0.275, 1.000) |
| 0 | Ours/Carrier vectors | 0.651 (0.434, 0.855) | 0.880 (0.755, 0.969) | 0.994 (0.936, 1.000) |
| View/encoder | Conditioning | 25 training tasks | 87 training tasks |
|---|---|---|---|
| Local/linear | Task-blind | 0.313 (0.084, 0.587) | 0.100 (0.000, 0.327) |
| Local/linear | Ours/Residual | 0.631 (0.423, 0.825) | 0.647 (0.437, 0.843) |
| Local/linear | RAD/Residual | 0.671 (0.471, 0.850) | 0.724 (0.528, 0.893) |
| Local/conv | Task-blind | 0.512 (0.306, 0.715) | 0.104 (0.000, 0.347) |
| Local/conv | Ours/Residual | 0.619 (0.413, 0.810) | 0.643 (0.432, 0.840) |
| Local/conv | RAD/Residual | 0.632 (0.425, 0.822) | 0.603 (0.384, 0.809) |
| Conditioning | Linear/shared | Conv/separate |
|---|---|---|
| Task-blind | 0.290 (0.176–0.420) | 0.308 (0.201–0.430) |
| Ours/Residual | 0.693 (0.564–0.811) | 0.702 (0.557–0.830) |
| Ours/RAD-heavy s0 | 0.661 (0.518–0.795) | 0.671 (0.527–0.808) |
| Ours/RAD-heavy s1 | 0.671 (0.531–0.799) | 0.671 (0.525–0.804) |
| RAD/Residual | 0.876 (0.793–0.943) | 0.872 (0.786–0.942) |
| Conditioning | Distance 2 | Distance 3 | Distance 4–5 | Distance |
|---|---|---|---|---|
| Tasks | 17 | 1 | 7 | 3 |
| Task-blind | 0.434 (0.294–0.588) | 0.208 (0.000–0.500) | 0.137 (0.028–0.268) | 0.028 (0.000–0.150) |
| Ours/Residual | 0.926 (0.861–0.979) | 0.792 (0.000–0.958) | 0.417 (0.142–0.708) | 0.069 (0.000–0.250) |
| Ours/RAD-heavy s0 | 0.904 (0.815–0.975) | 0.333 (0.000–0.625) | 0.417 (0.167–0.688) | 0.056 (0.000–0.208) |
| Ours/RAD-heavy s1 | 0.882 (0.780–0.964) | 0.833 (0.000–1.000) | 0.369 (0.129–0.646) | 0.125 (0.000–0.375) |
| RAD/Residual | 0.949 (0.887–0.991) | 1.000 (0.000–1.000) | 0.738 (0.483–0.931) | 0.708 (0.000–1.000) |
| Encoder | Near-collisions | Test success | Plain/product (/112,/71) | Composition AND/OR (/30) |
|---|---|---|---|---|
| Baseline 75k | 3812 | .702 (.559–.830) | 104/66 | 29/24 |
| Discount .5 | 4474 | .695 (.551–.826) | 107/66 | 29/23 |
| Discount .9 | 174 | .686 (.540–.823) | 107/66 | 29/25 |
| Discount .9 + progress | 123 | .749 (.616–.869) | 102/65 | 26/23 |
| Pool | Conditioning | 25 tasks Reward | 25 tasks Reward | Large pool Reward | Large pool Reward |
|---|---|---|---|---|---|
| RAD-compatible | Task-blind | 0.507–0.551 | 0.000–0.537 | 0.471–0.529 | 0.000–0.515 |
| RAD-compatible | Ours/Initial | 0.507–0.544 | 0.456–0.522 | 0.515–0.581 | 0.360–0.478 |
| RAD-compatible | Ours/Residual | 0.515–0.654 | 0.485–0.640 | 0.625–0.676 | 0.632–0.662 |
| RAD-compatible | Ours/Carrier bits | 0.390–0.478 | 0.441–0.485 | 0.500–0.581 | 0.382–0.537 |
| RAD-compatible | Ours/Carrier vectors | 0.544–0.654 | 0.529–0.625 | 0.640–0.662 | 0.596–0.669 |
| RAD-compatible | RAD/Initial | 0.419–0.463 | 0.346–0.419 | 0.456–0.566 | 0.397–0.471 |
| Symbol | Meaning and domain |
|---|---|
| Input AFA, event alphabet, source states, accepting states. | |
| Source-state count and alphabet size . | |
| Positive Boolean formulas; initial formula and transition map . | |
| Subsets of appearing as terms in the irredundant disjunctive normal form of . | |
| Event words and empty word; is length. | |
| Language with initial formula ; residual . |
| Symbol | Meaning and domain |
|---|---|
| Shared encoder, decoder, and policy parameters. | |
| Learned formula-assignment map and its initial/transition formulas. | |
| Formula index and eligible input hyperedges. | |
| Numbers of disjunct and conjunct positions in each decoded formula. | |
| Nonnegative weight assigning edge to disjunct ; sums to one over eligible edges plus the false choice. | |
| Copied target symbol in conjunct position of edge , padded with . |
| Display name | Run/legend code | Conditioning input |
|---|---|---|
| Task-blind | none | Zero vector. |
| Ours/Initial | m-init | Fixed initial Morphata embedding. |
| RAD/Initial | r-init | Fixed initial RAD embedding. |
| Ours/Residual | m-res | Morphata embedding of the current residual language. |
| RAD/Residual | r-res | RAD embedding of the current residual language. |
| Ours/Carrier bits | m-op | Initial embedding concatenated with Boolean active-slot indicators. |
| Prefix | Residual language | Configuration-NFA pooling | Semantic pooling |
|---|---|---|---|
| Source | Four letters | Sixteen letters |
|---|---|---|
| Random DFA | 316 | 143 |
| Fresh random AFA | 719 | 146 |
| Reverse random DFA | 832 | 398 |
| -th from end | 181 | 91 |
| Modular counters | 108 | 70 |
| Eventually conjunction | 100 | 89 |
| Setting | Value or specification |
|---|---|
| Message passing | AND-OR, eight rounds, width 32. |
| Task readout | Initial-vertex state, width 32, without final normalization. |
| Source-state capacity | 16 pinned source-state vectors. |
| Operator capacity | 16 configuration slots. |
| Formula source | Copied conjuncts, learned disjunct assignments, copied acceptance. |
| Configuration source | Forward closure from thresholded initial terms. |
| Alphabet | Copied formulas | Learned assignments |
|---|---|---|
| 4 | c3c-s4 , 75k | c3lsem-s0 , 75k |
| 10 | c3c10-s0--s2 , 75k | c3lsem10-s0 , 75k |
| 16 | c3c16-s0 , 40k | c3lsem16-short-s0 , 45k |
| Membership accuracy | ||||
|---|---|---|---|---|
| Layer | States, | Initial, | States, | Initial, |
| AND-OR | .949/.951 | .882/.885 | .902/.903 | .841/.847 |
| Sum | .897/.898 | .843/.839 | .867/.868 | .810/.816 |
| Membership accuracy | ||||
|---|---|---|---|---|
| Layer | Initial, length 2 | Initial, length 3 | Random DFAs | Pooled, length 2 |
| AND-OR | .900/.897/.888 | .847/.848/.840 | .798/.823/.813 | .775/.772/.772 |
| Sum | .820/.842/.860 | .787/.788/.793 | .709/.702/.710 | .787/.780/.783 |
| Formulas | Configurations / transitions | Seeds | Product /119 | Plain /149 | /40 | FD |
|---|---|---|---|---|---|---|
| Learned | per slot / syntactic | 5 | 43–53 | 130–144 | 2–4 | .046–.270 |
| Learned | closure / syntactic | 5 | 95–100 | 120–132 | 2–4 | .031–.045 |
| Learned | closure / semantic | 3 | 114–115 | 139–143 | 4–6 | .019 |
| Copied | closure / syntactic | 5 | 119 | 149 | – | .000 |
| Copied | closure / semantic | 3 | 119 | 149 | 5–6 | .000 |
| Formulas / configurations | Seeds | Syntactic | Semantic re-step | Gain |
|---|---|---|---|---|
| Learned / per slot | 0–3 | 44–53 | 63–67 | 10–22 |
| Copied / per slot | 0–1 | 52–59 | 73–75 | 16–21 |
| Learned / closure | 0–2 | 98–100 | 105–107 | 5–9 |
| Model | AFA pair AUC | DFA pair AUC | Collapsed / seeds |
|---|---|---|---|
| Learned / per slot | .914–.960 | .774–.815 | 0/5 |
| Learned / closure, no identity | .504–.888 | .797–.936 | 1/5 |
| Learned / closure | .954–.967 | .758–.872 | 0/5 |
| Learned / semantic | .944–.962 | .801–.860 | 0/3 |
| Copied / closure | .845–.922 | .896–.951 | 0/5 |
| Model, seed | Product | Plain | Above capacity | FD | AFA AUC | DFA AUC |
|---|---|---|---|---|---|---|
| Copied/syntactic, 0 | 119 | 149 | – | .000 | .894 | .950 |
| Copied/syntactic, 1 | 119 | 149 | – | .000 | .845 | .896 |
| Copied/syntactic, 2 | 119 | 149 | – | .000 | .913 | .947 |
| Copied/syntactic, 3 | 119 | 149 | – | .000 | .865 | .938 |
| Copied/syntactic, 4 | 119 | 149 | – | .000 | .922 | .951 |
| Copied/semantic, 0 | 119 | 149 | 5 | .000 | .945 | .965 |
| Model, seed | All witness-length gaps | Gaps at least two |
|---|---|---|
| Learned/semantic, 0 | .645 (.612–.680) | .787 (.731–.832) |
| Copied/syntactic, 4 | .724 (.676–.773) | .895 (.862–.924) |
| Learned/syntactic, 0 | .665 (.634–.701) | .834 (.787–.878) |
| Axis | Recorded scope |
|---|---|
| Alphabet | 4, 10, and 16 letters; configurations differ across alphabets. |
| Representation | 16 source-state slots, 16 operator slots, at most 64 candidates. |
| DFA size | Within-budget four-letter rows include minimal DFAs with 144 states. |
| Training tasks | XLand: 25 versus 87 RAD-compatible, 139 multi-event, or 173 plain tasks, at fixed frames. |
| Policy samples | XLand: 1,000 updates at steps (16.384M); LetterWorld: 611 updates at steps (20.021M). |
| Compilation cost | Exact inclusion and bank construction; time and peak memory unmeasured. |
| Conditioning | RAD-compatible | Multi-event | Plain |
|---|---|---|---|
| Rejection reward 0 (validation-selected) | |||
| Task-blind | 0.510 (0.304–0.713) | 0.623 (0.471–0.768) | 0.749 (0.620–0.868) |
| Ours/Initial | 0.560 (0.354–0.759) | 0.634 (0.499–0.761) | 0.775 (0.648–0.886) |
| Ours/Residual | 0.656 (0.444–0.850) | 0.770 (0.637–0.888) | 0.828 (0.709–0.927) |
| Ours/Carrier bits | 0.551 (0.349–0.750) | 0.662 (0.521–0.795) | 0.759 (0.629–0.874) |
| Ours/Carrier vectors | 0.653 (0.448–0.841) | 0.773 (0.641–0.891) | 0.833 (0.716–0.930) |
| Conditioning | RAD-compatible | Multi-event | Plain |
|---|---|---|---|
| Rejection reward 0 (validation-selected) | |||
| Task-blind | 0.532 (0.319–0.738) | 0.602 (0.455–0.746) | 0.739 (0.608–0.859) |
| Ours/Initial | 0.526 (0.322–0.725) | 0.578 (0.442–0.710) | 0.755 (0.626–0.870) |
| Ours/Residual | 0.585 (0.378–0.782) | 0.700 (0.560–0.829) | 0.742 (0.616–0.855) |
| Ours/Carrier bits | 0.424 (0.237–0.619) | 0.492 (0.352–0.635) | 0.689 (0.555–0.813) |
| Ours/Carrier vectors | 0.601 (0.399–0.791) | 0.676 (0.542–0.803) | 0.657 (0.520–0.787) |
| Pool | Comparison | Reward | Reward |
|---|---|---|---|
| RAD-compatible | m-opv none | +0.143 (+0.031, +0.278) | +0.332 (+0.094, +0.609) |
| RAD-compatible | m-opv m-op | +0.101 (+0.015, +0.213) | +0.182 (+0.056, +0.338) |
| RAD-compatible | m-opv m-res | -0.003 (-0.066, +0.057) | -0.007 (-0.066, +0.047) |
| RAD-compatible | r-res m-res | +0.053 (-0.051, +0.171) | +0.053 (-0.051, +0.166) |
| Multi-event | m-opv none | +0.150 (+0.052, +0.262) | +0.515 (+0.239, +0.791) |
| Multi-event | m-opv m-op | +0.110 (+0.030, +0.205) | +0.121 (+0.034, +0.223) |
| Pool | Comparison | Reward | Reward |
|---|---|---|---|
| RAD-compatible | m-opv none | +0.069 (-0.043, +0.201) | +0.174 (-0.015, +0.404) |
| RAD-compatible | m-opv m-op | +0.178 (+0.048, +0.332) | +0.115 (-0.015, +0.249) |
| RAD-compatible | m-opv m-res | +0.016 (-0.068, +0.116) | +0.003 (-0.093, +0.112) |
| RAD-compatible | r-res m-res | +0.021 (-0.078, +0.129) | +0.022 (-0.094, +0.156) |
| Multi-event | m-opv none | +0.074 (-0.034, +0.189) | +0.092 (-0.022, +0.209) |
| Multi-event | m-opv m-op | +0.184 (+0.091, +0.287) | +0.166 (+0.076, +0.265) |
| Reward | Conditioning | DFA | DFA 11–32 | DFA 33–128 |
|---|---|---|---|---|
| Tasks | 17 | 11 | 4 | |
| 0 | Task-blind | 0.513 (0.294, 0.735) | 0.709 (0.463, 0.911) | 0.850 (0.425, 1.000) |
| 0 | Ours/Initial | 0.547 (0.338, 0.750) | 0.745 (0.550, 0.908) | 0.694 (0.250, 0.950) |
| 0 | Ours/Residual | 0.659 (0.439, 0.863) | 0.861 (0.710, 0.973) | 0.988 (0.900, 1.000) |
| 0 | Ours/Carrier bits | 0.535 (0.322, 0.748) | 0.802 (0.623, 0.948) | 0.819 (0.275, 1.000) |
| 0 | Ours/Carrier vectors | 0.651 (0.434, 0.855) | 0.880 (0.755, 0.969) | 0.994 (0.936, 1.000) |
| View/encoder | Conditioning | 25 training tasks | 87 training tasks |
|---|---|---|---|
| Local/linear | Task-blind | 0.313 (0.084, 0.587) | 0.100 (0.000, 0.327) |
| Local/linear | Ours/Residual | 0.631 (0.423, 0.825) | 0.647 (0.437, 0.843) |
| Local/linear | RAD/Residual | 0.671 (0.471, 0.850) | 0.724 (0.528, 0.893) |
| Local/conv | Task-blind | 0.512 (0.306, 0.715) | 0.104 (0.000, 0.347) |
| Local/conv | Ours/Residual | 0.619 (0.413, 0.810) | 0.643 (0.432, 0.840) |
| Local/conv | RAD/Residual | 0.632 (0.425, 0.822) | 0.603 (0.384, 0.809) |
| Conditioning | Linear/shared | Conv/separate |
|---|---|---|
| Task-blind | 0.290 (0.176–0.420) | 0.308 (0.201–0.430) |
| Ours/Residual | 0.693 (0.564–0.811) | 0.702 (0.557–0.830) |
| Ours/RAD-heavy s0 | 0.661 (0.518–0.795) | 0.671 (0.527–0.808) |
| Ours/RAD-heavy s1 | 0.671 (0.531–0.799) | 0.671 (0.525–0.804) |
| RAD/Residual | 0.876 (0.793–0.943) | 0.872 (0.786–0.942) |
| Conditioning | Distance 2 | Distance 3 | Distance 4–5 | Distance |
|---|---|---|---|---|
| Tasks | 17 | 1 | 7 | 3 |
| Task-blind | 0.434 (0.294–0.588) | 0.208 (0.000–0.500) | 0.137 (0.028–0.268) | 0.028 (0.000–0.150) |
| Ours/Residual | 0.926 (0.861–0.979) | 0.792 (0.000–0.958) | 0.417 (0.142–0.708) | 0.069 (0.000–0.250) |
| Ours/RAD-heavy s0 | 0.904 (0.815–0.975) | 0.333 (0.000–0.625) | 0.417 (0.167–0.688) | 0.056 (0.000–0.208) |
| Ours/RAD-heavy s1 | 0.882 (0.780–0.964) | 0.833 (0.000–1.000) | 0.369 (0.129–0.646) | 0.125 (0.000–0.375) |
| RAD/Residual | 0.949 (0.887–0.991) | 1.000 (0.000–1.000) | 0.738 (0.483–0.931) | 0.708 (0.000–1.000) |
| Encoder | Near-collisions | Test success | Plain/product (/112,/71) | Composition AND/OR (/30) |
|---|---|---|---|---|
| Baseline 75k | 3812 | .702 (.559–.830) | 104/66 | 29/24 |
| Discount .5 | 4474 | .695 (.551–.826) | 107/66 | 29/23 |
| Discount .9 | 174 | .686 (.540–.823) | 107/66 | 29/25 |
| Discount .9 + progress | 123 | .749 (.616–.869) | 102/65 | 26/23 |
| Pool | Conditioning | 25 tasks Reward | 25 tasks Reward | Large pool Reward | Large pool Reward |
|---|---|---|---|---|---|
| RAD-compatible | Task-blind | 0.507–0.551 | 0.000–0.537 | 0.471–0.529 | 0.000–0.515 |
| RAD-compatible | Ours/Initial | 0.507–0.544 | 0.456–0.522 | 0.515–0.581 | 0.360–0.478 |
| RAD-compatible | Ours/Residual | 0.515–0.654 | 0.485–0.640 | 0.625–0.676 | 0.632–0.662 |
| RAD-compatible | Ours/Carrier bits | 0.390–0.478 | 0.441–0.485 | 0.500–0.581 | 0.382–0.537 |
| RAD-compatible | Ours/Carrier vectors | 0.544–0.654 | 0.529–0.625 | 0.640–0.662 | 0.596–0.669 |
| RAD-compatible | RAD/Initial | 0.419–0.463 | 0.346–0.419 | 0.456–0.566 | 0.397–0.471 |
Representation Learning for RL
Representation learning for reinforcement learning builds state, task, and dynamics features that help agents learn, explore, and transfer.
See all 79 papers