Unique Research · Insight / AIGC monthly · October 23, 2023 · Full editorial rewrite of ~9,978 CJK — all ten highlighted papers and core claims preserved (not a stub)
September’s academic output was dense: agents, multimodal foundation models, RLAIF, multimodal LLMs, large multilingual datasets, and diffusion. Below are ten hot papers with abstracts distilled into core claims — from foundations to practice, unimodal to multimodal, text to image.
1. The Rise and Potential of Large Language Model Based Agents: A Survey
Humans have long sought AI that matches or exceeds human capability; agents — systems that perceive, decide, and act — are a classic path. Mid-20th-century work onward mostly improved algorithms or training for narrow tasks. What’s been missing is a comprehensive, powerful model as the substrate for agents that adapt across scenes.
This survey starts from the philosophy of agency, traces AI history, and argues why LLMs are a viable agent foundation. It proposes an LLM-agent framework with three parts — brain, perception, action — customizable for applications. It covers single-agent, multi-agent, and human–agent collaboration, plus “agent societies” (behavioral traits and emergent social phenomena), and open challenges toward AGI-adjacent agents that coexist with humans.
Core claims: (1) Brain–perception–action framework. (2) Brain as controller for memory, thinking, decisions. (3) Perception handles multimodal external input; action uses tools and affects the environment. (4) LLMs’ NLU/NLG enable rich interaction with humans and other agents. (5–8) Agents need intent understanding, multi-turn dialogue, uncertainty handling, trial-and-error adaptation, complex reasoning/planning, coherent responses, multilingual ability, and multimodal (vision/audio) perception. (9) LLMs are a spark toward AGI and high-value agent construction.
2. Multimodal Foundation Models: From Specialists to General-Purpose Assistants
Survey of multimodal foundation models along two arcs: (A) specialists pretrained for a purpose — (i) vision-backbone learning for understanding, (ii) text-to-image generation; (B) exploratory general-purpose assistants — (i) unified vision models inspired by LLMs, (ii) end-to-end multimodal LLMs, (iii) chaining multimodal tools via LLMs.
Core claims: Specialist multimodal FMs cover understanding and generation tracks. General-purpose assistants push toward LLM-like universality by unifying vision, training MM-LLMs end-to-end, or orchestrating tools — a path from narrow excellence to assistant-style breadth.
On specialists: vision backbones and text-to-image systems already deliver production-grade unimodal excellence. On assistants: the frontier is stitching those specialists — or training end-to-end MM-LLMs — so one system can act as a general multimodal aide rather than a single-task model. The survey is useful as a map of where research attention is migrating in late 2023.
3. RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
RLAIF lets AI systems improve recursively using feedback from other AI — used in RL fine-tuning. A fine-tuned large LM acts as judge, predicting human preferences (e.g. preferred summaries) and providing rewards. Training recipe: situation brief → optional few-shot → content to annotate → conclusion. Framework matches RLHF; the difference is AI judge vs human labelers.
Core claims: In one experiment, RLAIF summarization matched RLHF quality; beat SFT baseline in 71% of cases. Prompting without in-context examples plus chain-of-thought helped; larger judge models yielded better feedback and final summaries. Headroom remains via better prompting and stronger judges.
4. FLM-101B: An Open LLM and How to Train It with a $100K Budget
Training LLMs is so costly only a few firms can afford frontier runs; the race for more data raises costs further. Evaluation splits into knowledge tests vs NLP-task tests — knowledge-oriented scores may not reflect true capability. The paper’s answer: growth strategies that expand parameter count during training from small to large — first attempt to train a ~100B LLM this way. An aggressive growth strategy maximizes compute savings; growth operators inspired by MSG enable strict function-preserving growth. Releases open F(ree)LM-101B built on the FreeLM framework — notable for the claimed ~$100K training budget narrative.
Core claims: Cost is the bottleneck; growth training is a practical path; open 101B model + methodology for the community.
Why it mattered in September’s discourse: open weights plus a disclosed low-budget recipe challenge the assumption that only hyperscalers can participate at ~100B scale. Even if absolute quality trails frontier labs, the methodological signal — grow parameters while preserving function — is what Unique Research flagged for founders watching open-source economics.
5. Language Modeling Is Compression
Information theory and ML are tightly linked: probabilistic models and lossless compression are essentially equivalent. The source-coding theorem: expected bits under an optimal entropy coder equal the model’s negative log2 likelihood. Maximizing log likelihood = minimizing bits per message. Huffman, arithmetic coding, and asymmetric numeral systems implement model-based lossless compression; arithmetic coding is near-optimal in length. Large pretrained Transformers succeed at broad prediction — hence are strong arithmetic-coding compressors. Transformer + arithmetic coding hits SOTA in online and offline settings; offline uses fixed parameters with in-context compression. Foundation models can be viewed as universal compressors via in-context learning.
6. NExT-GPT: Any-to-Any Multimodal LLM
Text LLMs (Flan-T5, Vicuna, LLaMA, Alpaca) show strong reasoning; MM-LLMs (BLIP-2, Flamingo, MiniGPT-4, Video-LLaMA, LLaVA, PandaGPT, SpeechGPT) add perception of other modalities — but often lack multi-modal output. True AGI-leaning systems need any-to-any: arbitrary modality in, appropriate modality out. CoDi advances multimodal conversion but lacks LLM-level reasoning and is limited to simple pairwise generation; tool-augmented LLMs approximate any-to-any. NExT-GPT connects an LLM with multimodal adapters and diffusion decoders to perceive and generate combinations of text, image, video, and audio. Architecture: three layers — encoders for each modality, projections into language-like representations the LLM understands, then decoding back out.
7. CulturaX: A Cleaned, Enormous, Multilingual Dataset for LLMs (167 languages)
LLMs (encoder-only BERT/RoBERTa; decoder-only GPT; encoder–decoder BART/T5) scaled from hundreds of millions of parameters and ~13GB BERT data to trillion-scale models and hundreds of billions of tokens (Falcon, MPT, LLaMA, PolyLM, ChatGPT). Many model weights are public (APIs or open weights), but training data for SOTA models — even open ones like BLOOM, LLaMA, MPT, Falcon — often stays closed. That opacity blocks research on hallucination, bias, and toxicity, and concentrates capability among resource-rich players. CulturaX responds: a cleaned, document-refined, deduplicated, large-scale multilingual dataset spanning 167 languages, publicly oriented so high-quality pretraining data is not a private monopoly.
CulturaX’s cleaning pipeline (document refinement + dedup) is as important as raw scale: noisy multilingual crawls have historically limited non-English LLM quality. Publishing 167-language data is a transparency and democratization move aligned with Unique Research’s September theme that data openness — not only model openness — will shape who can train competitive LLMs.
8. DreamLLM: Synergistic Multimodal Comprehension and Creation
Content understanding and multimodal creation both matter for machine intelligence. Existing MLLMs reach strong visual understanding by feeding images (e.g. CLIP features) into LLMs for language output — but underuse the synergy between creation and comprehension. Diffusion models learn latent structure via iterative noise; multimodal signals have modality-specific structure with complementary semantics, so deep language understanding can aid cross-modal image generation. Prior work often glues diffusion to MLLMs by minimizing alignment loss; CLIP learns shared semantics and can ignore modality-specific info (modality gap). DreamLLM instead has MLLMs model distributions by sampling in pixel space; image posteriors are learned DeepDream-style with MLLM-conditioned parameterization — aiming for joint gains in understanding and generation.
9. FreeU: Free Lunch in Diffusion U-Net
Diffusion models differ from VAEs, GANs, and VQ methods: a fixed Markov chain maps data toward noise, then a learned reverse process denoises. A U-Net typically predicts noise at each step. Most work uses pretrained diffusion U-Nets for downstream tasks; internal behavior is underexplored. FreeU’s insight: strategically reweight skip-connection vs backbone feature-map contributions to exploit both U-Net components — a “free lunch” that improves generation quality without extra training or fine-tuning.
Practically: FreeU is attractive to product teams because it is inference-time / architectural reweighting rather than another fine-tune job — relevant when GPU budgets are the binding constraint (see also Sequoia’s $200B CapEx discussion in the same monthly cycle).
10. LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
LLMs are usually trained with fixed context (LLaMA 2048; LLaMA2 4096), limiting long-doc summarization and long QA. Training long context from scratch is expensive; full fine-tuning of pretrained models is also costly. Plain LoRA is memory-light but perplexity worsens as context grows. LongLoRA, with FlashAttention-2 and DeepSpeed stage 2 on LLaMA2-7B across context lengths, approaches full fine-tuning quality at much lower compute — an efficient path to long-context LLMs.
Long-context is a recurring product ask for RAG, legal/medical docs, and agent memory. LongLoRA’s pitch — LoRA-class memory with near full-FT quality at long sequence lengths — sits alongside FlashAttention-2 as part of the late-2023 efficiency toolkit Unique Research tracked for app builders who cannot afford full continued pretraining.
Images: paper figures and schematic diagrams from the CN edition — attach on import (9 assets).