LG AI Research· EXAONE

[NeurIPS 2023] RLHF and RLAIF Research Trends and Noteworthy Papers

Recently, large language models (LLMs) trained on massive amounts of data have shown remarkable performance gains. However, supervised fine-tuning methods that require direct expert datasets are limited in their ability to improve performance. To overcome this, many studies have been conducted on LLM reinforcement learning methodologies utilizing human or AI feedback, and various research results were introduced at NeurIPS (Neural Information Processing Systems) 2023. This post will introduce the research trends in Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF). Recently, Reinforcement Learning from Human Feedback (RLHF)[1] has become a popular method of fine-tuning LLMs with reinforcement learning. The existing RLHF methodology goes through two standardized processes as follows. 1. Reward Model Learning Process Reward model learning uses a given pairwise dataset (preference data, which ranks two answers to a question and people’s preferences for them) 2. Policy Learning Process LLM learning uses the learned reward model through RL algorithms such as Proximal Policy Optimization (PPO) [2] …

Announced

23 Jan 2024 00:00 UTC

Detected

28 Aug 2026 22:20 UTC (+948d after)