Google DeepMind‘s “Internal RL” Shows Promise for More Efficient AI Reasoning
Recent research from Google DeepMind suggests a novel approach to AI reasoning – “internal reinforcement learning” (RL) – that may prove more efficient than current methods relying on verbose “chain of thought” prompting. Teh findings, published on arXiv, demonstrate that training a metacontroller to manage a frozen large language model (LLM) can lead to important improvements in long-horizon reasoning tasks.
The core idea involves a metacontroller learning to switch between different “checkpoints” within the LLM, effectively allowing the model to break down complex problems into subgoals without explicitly outputting intermediate reasoning steps. This contrasts with the current trend of prompting models to generate detailed, step-by-step explanations for their conclusions.
Key Findings:
* Internal RL outperforms other methods: Experiments showed that models trained with Internal RL rapidly improved on long-horizon reasoning tasks, while other baseline methods failed to learn. (See image: https://venturebeat.com/_next/image?url=https%3A%2F%2Fimages.ctfassets.net%2Fjdtwqhzvc2n1%2F6e37HwyGuwJTki8y1sO9lZ%2Fd54c4a139cbaf73d63a9f94f83da78ae%2FInternal_RL_performance.png%3Fw%3D1000%26q%3D100&w=3840&q=75)
* “Frozen” approach is crucial: The research team found that the most effective implementation involved applying the metacontroller to a frozen LLM – meaning the LLM’s weights were not updated during training. Co-training the base model and metacontroller from scratch proved unsuccessful. The metacontroller, when applied to a frozen model, was able to discover key checkpoints aligning with natural subgoal transitions without any human-provided labels.
* Efficiency and Modality Independence: Internal reasoning, as demonstrated by this research, appears to be more efficient than token-based chain-of-thought approaches.Furthermore, the “silent thoughts” generated internally are not tied to specific input modalities, possibly making it valuable for future multi-modal AI systems.
“our study joins a growing body of work suggesting that ‘internal reasoning’ is not only feasible but potentially more efficient than token-based approaches,” said David Schimpf, a researcher at Google DeepMind, as reported by VentureBeat.
Implications for the Future of AI:
This research suggests a potential shift in how AI agents are developed. Rather of focusing solely on prompting strategies to elicit reasoning from models, future efforts may prioritize understanding and steering the internal representations already present within LLMs.This is particularly relevant for enterprises developing autonomous systems that require long-term planning,adaptation,and action.
Source:
* VentureBeat: https://venturebeat.com/ai/google-deepminds-internal-rl-shows-promise-for-more-efficient-ai-reasoning/
Worth a look