GPT Reviews
GPT Reviews
Earkind
Multi-Task Language Understanding 📈 // Composable Interventions 🤝 // ARMT Sets Performance Record 💪
14 minutes Posted Jul 10, 2024 at 7:20 am.
Introduction
MNLU-Pro Release on HuggingFace Datasets
Extrinsic Hallucinations in LLMs
RouteLLM
Fake sponsor
Composable Interventions for Language Models
MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?
Associative Recurrent Memory Transformer
Outro
0:00
14:40
Download MP3
Show notes
The MNLU-Pro dataset is a more robust and challenging massive multi-task language understanding dataset that's tailored to more rigorously benchmark large language models' capabilities.
The Composable Interventions framework allows researchers to study the effects of using multiple interventions on a language model, and the order in which interventions are applied can have a significant impact on their effectiveness.
The MJ-Bench benchmark evaluates the effectiveness of different types of multimodal judges in providing feedback for text-to-image generation models, and the experiments reveal that close-source VLMs generally provide better feedback.
The Associative Recurrent Memory Transformer (ARMT) is an approach that combines transformer self-attention for local context with segment-level recurrence for storage of task-specific information distributed over a long context, and it sets a new performance record in the recent BABILong multi-task long-context benchmark.
Contact:  [email protected]
Timestamps:
01:32 MNLU-Pro Release on HuggingFace Datasets
03:48 Extrinsic Hallucinations in LLMs
04:53 RouteLLM
08:14 Composable Interventions for Language Models
09:45 MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?
11:31 Associative Recurrent Memory Transformer