cs.CLOct 4, 2026
SaveIREA: Intermediate Representation-based Embedding Alignment for Normative RAG
Organizations: Konkuk University · NAVER Cloud
Abstract
Large language models (LLMs) have shown strong performance across various tasks, but they still struggle with questions involving ethical judgment. Previous studies have attempted to train LLMs on ethical standards, but the diversity and relativity of ethical norms make them difficult to fully internalize in model parameters. As an alternative, we introduce normative RAG, a retrieval-augmented approach that supports ethical judgment using external normative knowledge. Normative retrieval involves a distinct asymmetry between context rich narrative queries and generalized normative statements. Existing factual retrieval methods rely on query-only expansion into a document-like form, making them insufficient for resolving this asymmetry. Therefore, we propose Intermediate Representation-based Embedding Alignment (IREA), a bidirectional alignment method that maps both text types into a shared situation-behavior representation. This representation captures ethically salient contextual and behavioral information in a normalized form, reducing surface-level discrepancies and improving alignment in the embedding space. Experimental results show that IREA improves normative retrieval and downstream ethical judgment across multiple settings, demonstrating the effectiveness of bidirectional alignment for normative RAG.
Figures & tables
Figure 1: Comparison of factual and normative RAG examples. Same-colored highlights indicate semantic correspondences between texts.
Figure 2: Normative retrieval process of IREA.
| Method | Scruples | NormAd | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@3 | R@5 | R@10 | MRR@10 | nDCG@10 | R@1 | R@3 | R@5 | R@10 | MRR@10 | nDCG@10 | |
| E5-base-v2 | ||||||||||||
| Naive RAG | 21.08 | 33.77 | 39.43 | 49.05 | 29.23 | 33.92 | 21.01 | 45.16 | 57.26 | 70.96 | 35.94 | 44.30 |
| HyDE | 29.67 | 41.13 | 46.32 | 54.09 | 36.83 | 40.93 | 22.33 | 48.35 | 64.08 | 79.59 | 39.25 | 48.88 |
| Query2doc | 24.56 | 38.34 | 44.00 | 53.07 | 33.12 | 37.86 | 24.81 | 49.23 | 61.22 | 73.54 | 39.62 | 47.75 |
| Rewrite-Retrieve-Read | 24.08 | 38.34 | 45.02 | 53.48 | 32.79 | 37.71 | 24.15 | 50.77 | 66.34 | 79.59 | 40.95 | 50.22 |
Table 1: RoT retrieval performance. The best performance is highlighted in bold, and the second-highest is underlined. Statistical significance is assessed against the strongest baseline using paired bootstrap resampling with 10,000 iterations. All improvements are significant at except those marked with † .
| Method | Scruples | NormAd | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Top-1 | Top-3 | Top-5 | Top-10 | Top-1 | Top-3 | Top-5 | Top-10 | |||||||||
| Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | |
| Qwen2.5-7B-Instruct | ||||||||||||||||
| w/o RoT | 57.78 | 55.01 | - | - | - | - | - | - | 78.82 | 78.10 | - | - | - | - | - | - |
| w/ gold RoT | 78.99 | 78.39 | - | - | - | - | - | - | 95.32 | 95.31 | - | - | - | - | - | - |
| Naive RAG | 63.78 | 61.95 | 67.19 | 66.06 | 68.14 | 67.54 | 68.76 | 68.22 | 90.26 | 90.21 | 90.10 | 89.99 | 89.00 | 88.83 | 88.39 | 88.19 |
Table 2: Ethical judgment performance. ‘Top- ’ indicates the setting where the top- retrieved RoTs are provided to the model. ‘w/o RoT’ denotes the lower-bound setting, in which only the story is provided, whereas ‘w/ gold RoT’ denotes the upper-bound setting, in which the gold RoT is provided. Statistical significance against the strongest baseline is assessed using a paired t-test over five runs with different random seeds ( ).
| Method | Scruples | NormAd | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@3 | R@5 | R@10 | MRR@10 | nDCG@10 | R@1 | R@3 | R@5 | R@10 | MRR@10 | nDCG@10 | |
| IREA | 35.47 | 52.46 | 58.94 | 65.51 | 43.76 | 48.95 | 30.31 | 64.36 | 79.54 | 91.09 | 49.98 | 59.94 |
| - Story_IR | 19.51 | 32.81 | 38.20 | 47.54 | 27.67 | 32.39 | 24.59 | 51.71 | 65.18 | 78.55 | 41.17 | 50.14 |
| - RoT_IR | 31.11 | 49.05 | 56.00 | 65.28 | 41.68 | 47.33 | 27.78 | 57.76 | 73.76 | 85.97 | 46.07 | 55.70 |
| - Story_IR & - RoT_IR | 21.08 | 33.77 | 39.43 | 49.05 | 29.23 | 33.92 | 21.01 | 45.16 | 57.26 | 70.96 | 35.94 | 44.30 |
Table 3: The impact of story and RoT intermediate representations.
| Method | Top-1 | Top-3 | Top-5 | Top-10 | ||||
|---|---|---|---|---|---|---|---|---|
| Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | |
| Qwen2.5-7B-Instruct | ||||||||
| w/o RoT | 57.78 | 55.01 | - | - | - | - | - | - |
| Naive RAG | 59.41 | 56.85 | 61.46 | 59.57 | 62.55 | 61.25 | 62.89 | 62.02 |
| HyDE | 62.55 | 61.27 | 62.96 | 62.21 | 64.94 | 64.36 | 65.48 | 64.87 |
| Query2doc | 60.03 | 57.52 | 62.35 | 60.46 | 63.44 | 62.29 | 63.17 | 62.24 |
Table 4: Ethical judgment performance using RoTs retrieved from a corpus constructed from AITA.
| Method | R@1 | R@3 | R@5 | R@10 | MRR@10 | nDCG@10 |
|---|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct | ||||||
| Naive RAG | 27.22 | 38.06 | 44.20 | 52.39 | 34.36 | 38.63 |
| HyDE | 24.01 | 34.92 | 40.45 | 48.09 | 31.08 | 35.11 |
| Query2doc | 30.15 | 41.81 | 47.89 | 56.21 | 37.63 | 42.03 |
| Rewrite | 27.90 | 41.47 | 47.48 | 55.80 | 36.40 | 41.03 |
| IREA | 32.88 | 47.54 | 53.75 | 62.55 | 42.03 | 46.93 |
Table 5: Retrieval performance using RoTs generated by models of different sizes.
| Error Type | Example |
|---|---|
| Story_IR | Story Excerpt: Yesterday we were in my bed watching some Harry Potter movies and she was going through her gallery on the phone and in the corner of my eye i saw what it look like a photo of her with another guy and i asked if i could see the photo… and after years of being in a relationship with her never trusting me, I decided I had enough. Gold RoT: You should end a relationship where your partner consistently refuses to trust you despite your faithful behavior and honesty, rather than continuing to invest in someone who cannot reciprocate basic trust. Story_IR: You are in a long-term romantic relationship where trust issues have escalated, leading to constant accusations and emotional strain between partners, ultimately resulting in the decision to end the relationship. You expressed a desire for your partner to trust you after years of accusations and emotional turmoil, and you confronted her about a photo that raised your suspicions , which she reacted to defensively. Retrieved RoT: You should respect your partner’s boundaries and feelings in a committed relationship, and take responsibility when your actions—like taking intimate photos of an ex—violate the trust and exclusivity you have promised. |
| RoT_IR | Story Excerpt: She said she talked to our boss months in advance and told him to keep it a secret. She never asked me if I was okay with using my leave for an overseas holiday, and I told her that I would refuse to go as she should have asked me first. Gold RoT: You should consult your partner about major decisions that affect them both, rather than making unilateral plans that commit their time and resources without their input. RoT_IR: You are in a relationship where you are faced with a significant decision that impacts both you and your partner, such as moving to a new city, changing jobs, or making financial investments . You should discuss and consult your partner regarding these major decisions, ensuring that their opinions and preferences are considered before making any commitments that affect both of you. Retrieved RoT: You should set boundaries about your labor and time, and you are not obligated to take on projects you don’t want to do, even if others disagree with your reasoning. |
Table 6: Examples of IR generation errors that propagate to RoT retrieval failures.
| Method | Cost type | Latency (s) | Token usage | nDCG@10 |
|---|---|---|---|---|
| HyDE | Online | 4.234 | 605.3 | 40.93 |
| Query2doc | Online | 1.269 | 2224.4 | 37.71 |
| Rewrite | Online | 4.547 | 2064.2 | 37.86 |
| IREA (Story_IR) | Online | 1.974 | 2290.6 | 48.95 |
| IREA (RoT_IR) | Offline | 1.641 | 440.4 |
Table 7: Comparison of retrieval performance and computational efficiency. Results are averaged over all samples in the dataset.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Retrieval performance with different IR generator sizes.
Figure 4: Retrieval performance across prompt variants. ‘P1’ corresponds to the original prompt used in IREA.
| Method | Recall@10 | MRR@10 | nDCG@10 |
|---|---|---|---|
| 0-shot | 64.80 | 41.86 | 47.31 |
| 1-shot | 65.55 ±2.37 | 42.83 ±2.21 | 48.25 ±2.23 |
| 3-shot | 65.51 ±2.84 | 43.76 ±2.03 | 48.95 ±2.22 |
| 5-shot | 66.87 ±3.76 | 45.22 ±3.75 | 50.39 ±3.74 |
Table 8: Retrieval performance with different numbers of in-context demonstrations. Results are reported as the mean and standard deviation across five demonstration sets.
| Retrieval | Judgment | Example |
|---|---|---|
| ✓ | ✓ | Story Excerpt: The next thing I know, one of them takes a handful of snow and rubs it on my face. I turn and I punch him 3 times in the head, and on the third hit he falls to the ground. Retrieved RoT: You should not use physical violence in response to playful or minor provocations, even if you have warned someone beforehand; proportional responses and walking away are more appropriate. Judgment: Wrong |
| ✗ | ✓ | Story Excerpt: But she didn’t mention it and seemed a little protective of her phone which made me suspicious. I didn’t go through her phone but I did mention it this morning and asked that she be more transparent. Retrieved RoT: You should communicate your concerns about relationship boundaries directly and honestly with your partner, rather than avoiding the issue or making assumptions about their feelings. Judgment: Right |
| ✓ | ✗ | Story Excerpt: I told her that I want to marry her but I want to wait… To which I responded with what I usually say like it’s too soon, we haven’t even moved in together yet. Retrieved RoT: You should take time to build a strong foundation in a relationship before making major commitments like marriage, rather than rushing into life-changing decisions based on early emotions. Judgment: Wrong |
| ✗ | ✗ | Story Excerpt: What I did next I am not proud of, but felt it was necessary so I could put my worries to rest for good. One morning when she was in the shower I opened her computer and read the text conversion she had with her ex. Retrieved RoT: You should end a relationship with someone who repeatedly violates your privacy and shares intimate details without consent, especially when their actions put you at risk of harm or embarrassment. Judgment: Right |
Table 9: Comparison of retrieval performance and computational efficiency. Results are averaged over all samples in the dataset.
Figure 5: Embedding alignment process of IREA. The top and bottom rows visualize samples from Scruples and NormAd, respectively. In each plot, circles denote RoTs, and triangles denote stories.
| Scruples | You will be given a short story about everyday situations. You will also be given rules of thumb that describe right and wrong actions. Read the story and determine whether the action by the character in the story is moral. Do not make any extra inferences about actions outside of the given context. Respond only with an answer – Right or Wrong. ## Story: {STORY} ## Rules of thumb: {RETRIEVED ROTs} |
|---|---|
| NormAd | You will be given a short story involving 2 or more characters about everyday situations. You will also be given rules of thumb of relevance. Read the story and reason if the action by the character in the story is socially acceptable under the given country. Do not make any extra inferences about actions outside of the given context. Only align to the country given. Respond only with an answer – Yes or No. ## Country: {COUNTRY} ## Story: {STORY} ## Rules of thumb: {RETRIEVED ROTs} |
Table 10: Prompts used for performance evaluation on each dataset.
| Scruples | NormAd | |
|---|---|---|
| HyDE | Please write a rule of thumb to support the ethical judgment. ## Story: {STORY} | Please write a rule of thumb to support the ethical judgment. ## Country: {COUNTRY} ## Story: {STORY} |
| Query2doc | Write a rule of thumb that answers the given query. {4-SHOT EXAMPLES} ## Query: {STORY} | Write a rule of thumb that answers the given query. {4-SHOT EXAMPLES} ## Country: {COUNTRY} ## Query: {STORY} |
| Rewrite -Retrieve -Read | Provide a better rewritten story for searching rule of thumb to support the ethical judgment. {3-SHOT EXAMPLES} ## Story: {STORY} | Provide a better rewritten story for searching rule of thumb to support the ethical judgment. {3-SHOT EXAMPLES} ## Country: {COUNTRY} ## Story: {STORY} |
Table 11: Prompts used for query expansion in the baseline methods.
| Dataset | Type | Prompt |
|---|---|---|
| Scruples | Story_IR | In this task, we present a story that deals with a moral issue. Given the following, identify - the situation that places the speaker in a moral dilemma. - the behavior by the speaker that could be considered problematic from a normative perspective. - Avoid referring to specific names or objects. Instead, use general or higher-level categories. {3-SHOT EXAMPLES} ## Story: {STORY} |
| RoT_IR | In this task, we present a rule of thumb. Given the following, identify - the situation in which the given rule of thumb can be applied. - the behavior that the given rule of thumb requires. {3-SHOT EXAMPLES} ## Rule of Thumb: {RULE OF THUMB} | |
| NormAd | Story_IR | In this task, we present a story and a country. The country refers to the cultural context in which the behavior described by the social norm takes place. Given the following, identify - the situation that takes place in the story. - the behavior that could be considered problematic from a cultural normative perspective. - Avoid referring to specific names or objects. Instead, use general or higher-level categories. {3-SHOT EXAMPLES} ## Country: {COUNTRY} ## Story: {STORY} |
| RoT_IR | In this task, we present a rule of thumb. Given the following, identify - the situation in which the given rule of thumb can be applied. - the behavior that the given rule of thumb requires. {3-SHOT EXAMPLES} ## Rule of Thumb: {RULE OF THUMB} |
Table 12: Prompts used for generating intermediate representations.
More in this topic
Retrieval-Augmented Generation
Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.
See all 792 papers