Evaluating and Improving Natural Language Autoencoders
Organizations: Anthropic Fellows Program · Anthropic
Abstract
Natural language autoencoders (NLAs) are a method for generating natural language explanations of LLM activations. We develop evaluations based on auditing tasks that test the ability of NLAs to surface latent knowledge, identify whether a model was misled by a hint, and catch a silent fabrication. We also develop extensions to NLAs that let them use activation information from multiple layers and positions, let us apply them more effectively to long transcripts, and reduce the number of false claims they make. These extensions result in improvements on both our auditing evaluations and auxiliary evaluations adapted from prior work. We also use the evaluations to study how different auditing capabilities change over the course of NLA training, and find that although every evaluation improves early in training, extended training can hurt performance on some of our auditing evaluations.
Figures & tables
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| learning rate (both components) | 5e-6 (SFT warm start: 1e-5) |
| batch | 256 groups 8 rollouts per iteration |
| iterations | 500, 3 seeds per variant |
| KL coefficient | 0.01, against the run’s starting checkpoint |
| context / response caps | 300 / 150 tokens |
| Source text: …end. The winner will be awarded with Cornell gift cards.\n Also, students who began from Elementary Korean and completed 3 years of Korean language courses will be acknowledged at the event. Through this event, the Korean Language Program would like to raise awareness of | |
|---|---|
| Baseline | Supervised throughout training |
| UC Berkeley student organization presentation in US, promoting Korean language club at Loyola Chicago, organizing Korean cultural lunch events to encourage Korean students from China to participate in Korean language classes. Presentation description advocating promotional goals “We hope this event raises awareness of” Korean studies program, mentioning positive outreach for Korean Club, opening sentence promoting recognition of Korean community at Loyola, suggesting promoting Korea’s global presence on campus. Final token “the awareness of” “promote the awareness of” “Hopefully, this event will promote the awareness of” “Also, we hope to highlight the awareness of” expecting Korean culture, Korean language learning or international students’ contribution in US. | Academic event or presentation announcement involving Korean language or cultural participation, likely university setting involving students or program involvement. The text introduces an activity highlighting language or cultural contribution, now introducing broader significance or institutional recognition element. Final token “highlight the awareness of” continues formal intention statement (“the purpose of this event is to emphasize the importance of promoting broader awareness of”), expecting an organizational or cultural claim extension like “the ongoing efforts towards raising awareness of Korean language studies or multicultural appreciation at campus or Korean community importance in the university environment” (likely academic initiative mention, expected phrase continuation such as “the Korean cultural presence or language learning at U.S institutions” or introductory goal. |
| Source text: …in Past Tense by Ffc14\n A mix of young, fresh and new artist all adding a taste of years that have yet to come in 20 but have already passed in 19.\n indie, folk, psychedelic , | |
| Baseline | Supervised throughout training |
| DIY indie music blog in Washington DC, promoting underground vinyl streaming playlists with recurring format “band name @location | genre tags like lo-fi, indie rock, folk”. Long eclectic music tag list pattern “electronica, psychedelic, indie rock,” listing artist influences for song “Golden boy” “alternative country, folk rock, psychedelic,” suggesting more indie experimental genres. Final token “psychedelic,” in “genre: indie pop, psychedelic,” “recordings: indie folk, psychedelic,” for track tags “indie rock, folk,…” expecting another genre like “electronica” or “acoustic” or a closing genre word. | Music listing involving release or artist context with informal update elements including genre or impression elements. continuing evaluation or categorization involving indie style. The text is introducing musical qualities or genre tags now moving toward alternative music style considerations. Final token “,” continues a genre list extension (“psychedelic, folk-inspired, indie psych-like,…psychedelic,”), part of music genre enumeration (expected additional genre tags like “alternative rock, indie pop, experimental, folk-pop, indie rock, dreamy alternative or possibly adding more genre labels such as ’electronica, indie folk-pop, indie rock, bedroom pop”’ continuing list style (more genre keywords expected). |
Autoencoders
Autoencoders learn compressed representations by encoding data into latent codes and decoding them for reconstruction, generation, denoising, or anomaly detection.
See all 166 papers