Shu Yan

NVIDIA Research just published an intersting paper stating that SLMs aren't just "budget alternatives" to LLMs, but they're often superior for real-world applications. Their analysis of popular agents like MetaGPT, Open Operator, and Cradle shows that 40-70% of current LLM calls could be effectively replaced by specialized SLMs. Meanwhile, Andrej Karpathy envisions an even more radical future: a small "cognitive core" model that runs locally by default, with always-on capabilities, multimodal input/output, and the ability to dial up reasoning power when needed while delegating complex tasks to cloud-based oracles. The convergence is clear: we're moving from monolithic, cloud-dependent LLMs toward modular, efficient architectures. SLMs offer 10-30x better cost efficiency, lower latency, and can be fine-tuned faster and at a reasonable cost. For agentic systems performing repetitive and specialized task are better handled with careful context engineering and specialized SLMs. The future isn't about choosing between big and small models, but it's about intelligent orchestration of both. Paper: https://lnkd.in/gZAgH4Y6 Andrej Karpathy's X post: https://lnkd.in/g23777Mc

114

2 Comments

Whenever ML engineers concatenate vectors while building models, it usually makes me nervous; I wonder if they truly internalize the downstream implications, especially with respect to similarity measures such as cosine similarity. Let me explain. In linear algebra, vector space operations like addition and scalar multiplication are closed, but concatenation is not such an operation. Concatenation simply places vectors side by side to create a higher-dimensional vector, which lies outside the original vector space(s). As a result, properties of the original lower-dimensional spaces, such as semantic similarity, do not automatically carry over to the concatenated space. Concatenating vectors to form higher-dimensional representations can distort cosine similarity when corresponding subvectors are not well aligned (directional disagreement). Even if some pairs of original vectors are individually similar, differences in orientation, sign, or direction across others can reduce the overall similarity. This effect arises from the way contributions are weighted by the size of each subvector, not from any inherent distortion due to concatenation. As a result, naive concatenation does not reliably preserve the semantic relationships encoded in the original vector spaces. Magnitude imbalance between concatenated components is a critical issue. If one subvector has a much larger norm than the others, it dominates the cosine similarity calculation and diminishes the influence of smaller vectors. When additional heterogeneous features are appended, weakly related or noisy components can further reduce the discriminative power of cosine similarity. Although high-dimensional vectors are not always close to orthogonal, uncorrelated or random components often behave that way in practice, which makes similarity scores less meaningful when combining unrelated embeddings. Scaling subvectors before concatenation can reduce magnitude imbalance, but it introduces its own challenges. Choosing appropriate scaling factors is often arbitrary or task-specific, and the process may overemphasize or underemphasize certain features. Scaling also does not solve misalignment or the presence of noisy components. Instead, it simply redistributes influence across subvectors, which can sometimes make similarity measures worse rather than better. A robust way to mitigate these problems is to pass the concatenated vectors through a learned neural projection layer with normalization. By applying a linear transformation and optionally a non-linear activation, the network maps the high-dimensional vector into a joint embedding space where cosine similarity is more reliable. With sufficient training data, the network can learn to align semantically similar vectors while compensating for misalignment, magnitude imbalance, and noisy or uncorrelated components. This increases the likelihood that similarity metrics in the resulting space reflect the underlying semantic relationships.

381

30 Comments

What is Function calling & MCP for LLMs? (explained with visuals and code) Before MCPs became popular, AI workflows relied on traditional Function Calling for tool access. Now, MCP is standardizing it for Agents/LLMs. The visual below explains how Function Calling and MCP work under the hood. Today, let's learn: - Function calling by building custom tools for Agents. - How MCPs help by building a local MCP client with mcp-use and using tools from Browserbase MCP server. In Function Calling: - The LLM receives a prompt. - The LLM decides the tool. - The programmer implements a procedure to accept a tool call request from the LLM and prepare a function call. The tool call request is found in the LLM's response when you prompt it. - A backend service executes the tool. This Function Calling takes place within our stack: - We host the tool. - We implement a logic to determine the tool to invoke and its parameters. - We execute it. So Function Calling requires us to wire everything manually. MCP simplifies this! Instead of hard-wiring tools, MCP: - Standardizes defining, hosting, and exposing tools. - Makes it easy to discover tools, understand schemas, and use them. - Demands approval before invoking them. - Detaches implementation from consumption. For instance, whenever you integrate an MCP server, you never write a line of Python code to integrate the tools. Instead, you just integrate the MCP server and everything beyond this follows a standard protocol handled by the MCP client and the LLM: - They identify the MCP tool. - They prepare the input argument. - They invoke the tool. - They use the tool���s output to generate a response. Everything happens through a standard (but abstracted) protocol. So here’s the key point: MCP and Function Calling are not in conflict. They’re two sides of the same workflow. - Function Calling helps an LLM decide what it wants to do. - MCP ensures that tools are reliably available, discoverable, and executable, without you needing to custom-integrate everything. For example, an agent might say, “I need to search the web,” using function calling. That request can be routed through MCP to select from available web search tools, invoke the correct one, and return the result. Check the workflow in the diagram below. In this setup, to build a local MCP client, I used mcp-use because it lets us connect any LLM to MCP servers & build private MCP clients, unlike Claude/Cursor. - Compatible with Ollama & LangChain - Stream Agent output async - Built-in debugging mode, etc Find the mcp-use GitHub repo in the comments! ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

269

11 Comments

Truly Understanding Self-Attention in LLMs https://lnkd.in/gz_7M8Dg - Jeremy Jordan's excellent breakdown of attention in sequence modeling and how it solved the encoder-decoder bottleneck https://lnkd.in/geGwsY4A - Additive attention explained clearly https://lnkd.in/gY7aUPhK - The seminal 2014 Bahdanau paper that introduced attention https://lnkd.in/gbUpQfHn - Brandon Rohrer's masterpiece that breaks down each matrix operation in attention step-by-step https://lnkd.in/gV6uKPwD - Peter Bloem's blog with brilliant analogies that make transformers click https://lnkd.in/g2pjhCAD - Eugene Yan's focused exploration of building intuition behind attention https://lnkd.in/gYeb9Edn - Lilian Weng's technical deep-dive covering all aspects of attention mechanisms https://lnkd.in/gKu7ijvZ - Jay Alammar's visual masterpiece that made attention accessible to everyone https://lnkd.in/gKJxTbfC - Harvard's line-by-line annotated code implementation of the original transformer paper The key insight? Attention is essentially a differentiable database lookup that lets models dynamically retrieve relevant information from the entire sequence. #ML #AI #LLMs

164

3 Comments

I just published Vol. 109 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Zero-Shot Enhancement of LLM Rerankers through BM25 Score Integration, from Seetharaman et al. 📚 A Visual Analytics Framework for Debugging Graph-based RAG Systems, from Zhejiang University 📚 Production-Scale Generative Recommendations with Reinforcement Learning, from Kuaishou 📚 A Generative Approach to Cold-Start Recommendation via Sequential User Modeling, from ByteDance 📚 A Reference-Anchored Approach for Zero-Shot Document Ranking, from Li et al. 📚 Structure-Aware Code Chunking for Enhanced Retrieval-Augmented Generation, from CMU 📚 A Modular Open-Source Framework for Efficient Retrieval-Augmented Generation Research, from Zhang et al. 📚 A Learning-to-Rank Framework for Dynamic Retriever Selection in RAG Systems, from CMU 📚 A Comprehensive Review of Multi-Interest Recommendation, from Li et al. 📚 Hard Negative Mining through Hierarchical User Grouping in Recommendation Systems, from Credit Karma #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

46

2 Comments

More and more LLM models are released in native quantized formats. This blog (https://lnkd.in/gfTcp5yM) provides a brief overview of #LLM #quantization formats and methods, plus insights into the "native quantization" of DeepSeekV3.1 and GPT-OSS.

189

1 Comment

I just published Vol. 119 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Theoretical Limits of Single-Vector Embedding Models in Information Retrieval, from Google DeepMind 📚 Investigating Why Randomly Truncating Text Embeddings Barely Hurts Performance, from Takeshita et al. 📚 Vector Quantization Attention for Ultra-Long User Behavior Modeling in Recommender Systems, from Kuaishou 📚 Conditional Two-Tower Models for Bootstrapping User-to-Item Retrieval Systems, from Pinterest 📚 Computational Scaling Laws for Zero-Shot Information Retrieval with Decoder Models, from Databricks 📚 A Comprehensive Analysis of LLM-based Reranking Methods, from the University of Innsbruck 📚 Lazy Decoder-Only Architecture for Industrial-Scale Generative Recommendation, from Kuaishou 📚 Dynamic Multi-Task Learning for Scalable Recommendation Systems, from Kuaishou 📚 Enabling Compact Language Models for Agentic RAG Through Distillation-Guided Reinforcement Learning, from Kotoge et al. 📚 Combining ID and Content Embeddings Without Architectural Complexity, from Albatross AI #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

48

1 Comment

There’s a new RL algorithm in town… and it’s already shaking up the LLM research community. 🤯 Group Sequence Policy Optimization (GSPO) was created by the research team at Qwen and is purpose built for operating at the sequence level rather than the token level. This means it evaluates, clips, and optimizes entire response sequences by their overall likelihood. The important upshot is that it directly aligns the optimization process with how rewards are actually assigned in real world scenarios! Some key reasons to be excited! 1. Better stability. GSPO improves training stability and helps avoids model collapse that can happen with algorithms like GRPO. 2. Scalable RL for mixture of experts (MoE) models. GSPO inherently solves stability challenges teams face when scaling RL for MoE architectures. Large scale RL becomes more practically feasible! 3. Simplifies Infra. Sequence-level optimization is inherently less sensitive to small numerical differences between training and inference systems compared to token-level methods, which should lead to more robust RL pipelines! We can already see signs that this has paid off in big ways for the open-source community. Qwen3 models are doing really well on benchmarks, boasting competitive performance with top models like Gemini-2.5 Pro on GPQA, AIME25, and LiveCodeBench! https://lnkd.in/gCZMQvUj P.S. 👀 If you want to run RL on your own Qwen3 model with GRPO/GSPO, keep an eye on our open-source post-building repo 🧡 : https://lnkd.in/gXZ32qkm #AI #LLM #ReinforcementLearning

81

10 Comments

I just published Vol. 118 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Large Foundation Model for Ads Recommendation, from Tencent 📚 A Survey of Feedback-Driven Approaches in Retrieval-Augmented Generation, from Rathee et al. 📚 Scaling Laws for Click-Through Rate Prediction with Knowledge Distillation, from Meituan 📚 Closing the Performance Gap in Generative Recommenders with Collaborative Tokenization and Efficient Modeling, from Lepage et al. 📚 Can LLM-Generated QA Data Replace Human Benchmarks for RAG Systems?, from van Elburg et al. 📚 Reducing False Positives in Sequential Recommendation through Explicit Negative Feedback Modeling, from Ivanova et al. 📚 An Industry Study of RAG Implementation Challenges and Practices, from Brehme et al. 📚 A Framework for Ultra-Long Behavioral Modeling in Candidate Retrieval, from ByteDance 📚 Representation Quantization for Collaborative Filtering Augmentation, from Kuaishou 📚 Task-Specialized Co-training for Information Retrieval and Semantic Similarity, from Tencent #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

40

5 Comments

Understanding the Evolution of LLM Architectures: MCP vs Agentic AI vs RAG As enterprises rapidly adopt AI, it’s important to evaluate how models interact with data and execute tasks. Here's a quick breakdown of three leading approaches: 🟣 Model Context Protocol (MCP): A standardized protocol ensuring direct, real-time, secure access to dynamic data sources—without relying on vector storage. Ideal for low-latency enterprise applications. 🟧 Agentic AI: Multiple intelligent agents coordinate via an orchestrator (or crew system). Offers flexibility but brings high latency, resource intensiveness, and debugging challenges. 🩷 Retrieval Augmented Generation (RAG): Fetches context via embeddings from vector DBs before response generation. Powerful but suffers from static embeddings, semantic gaps, and context staleness. 💡 Each architecture has trade-offs. The future may not be “one size fits all” but a hybrid model that combines the best of each. --- Need an AI Consultant or help building your career in AI? Message me now

387

19 Comments

I just published Vol. 110 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Building Next-Generation Search with Multi-Agent LLM Systems, from Baidu Search 📚 Multi-Granularity Retriever Mixing for Robust Information Retrieval, from Kalra et al. 📚 Contrastive Item Tokenization for Multi-Modal Generative Recommendation, from Alibaba 📚 Self-Supervised Dense Retrieval through Language Modeling, from Cai et al. 📚 Nested Embeddings for Scalable E-commerce Information Retrieval, from Qian et al. 📚 Entity-Centric Inverted Indexing for Scalable Retrieval-Augmented Generation, from Zhang et al. 📚 jina-embeddings-v4: Unified Vision-Language Embeddings with Shared Semantic Space, from Jina AI 📚 End-to-End Learning of Search-Augmented Multimodal Reasoning, from ByteDance 📚 Hierarchical MLP-Mixer for Multi-Scale User Interest Modeling in Sequential Recommendation, from ByteDance 📚 Efficient and Scalable Retrieval Augmented Generation for Dynamic Corpora, from Zhang et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

34

1 Comment

I just published Vol. 112 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Multi-Query Parallel Retrieval-Augmented Generation with Reinforcement Learning, from Tan et al. 📚 Zero-Shot Text Embeddings for Recommendation and Search, from Attimonelli et al. 📚 Set-Wise Passage Selection for Retrieval-Augmented Generation, from LG AI Research 📚 Integrating User Intent Through Query Context in Transformer-Based Sequential Recommendation Systems, from Zalando 📚 Unified Text-Image Retrieval through Modified Vision-Language Architecture, from NVIDIA 📚 Achieving Pairwise Ranking Performance with Pointwise Efficiency Through Strategic Distillation, from Google 📚 Multi-Intent Session-Based Recommendation with Pluggable LLM Semantics, from Chen et al. 📚 Bridging Knowledge Graphs and Large Language Models for Enhanced Recommendation, from the University of Glasgow 📚 Rethinking ID Features in Multimodal Recommendation, from Li et al. 📚 Investigating RAG System Sensitivity to Real-World Query Variations, from Perçin et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

30

2 Comments

I explored in my blog post how reflective prompt optimization methods like GEPA can outperform reinforcement learners such as GRPO while using far fewer rollouts, yet they only nudge models superficially because the underlying weights remain untouched. I proposed closing that gap by pairing an open‑weight LLM with prompt generation and reinforcement/zeroth‑order weight tuning, feeding system outputs back to the model so it learns both from better instructions and from its own results. What we’re talking about is LLMs can be steered by clever prompts, but the state‑of‑the‑art in prompt optimization still behaves like we’re writing cheat sheets for a student. Techniques like GRPO grind through thousands of rollouts to sculpt a “perfect” prompt. Newer reflective methods like GEPA are more human‑like: they learn from a handful of tries, build a tree of candidate prompts, and can outperform GRPO with 35× fewer samples. GEPA even produces shorter prompts and has shown promise for inference‑time tasks like generating GPU kernels. Why that’s not enough: Prompts sit on the surface. They shape model behaviour, but they don’t change the model’s internal knowledge. You end up collecting a library of prompts and running separate optimization cycles for each new task. Worse, there’s no closed feedback loop; the model doesn’t learn from the consequences of its answers. It’s a bit like giving our student a cheat sheet but never helping them actually understand the material. The new idea: Start with an open‑weight LLM and let it propose candidate prompts. Then use reflective optimization (like GEPA) to evolve those prompts quickly. But don’t stop there: take the best prompts and use them as training signals to adjust the model’s weights. You can do this with lightweight reinforcement learning or zeroth‑order optimization. Crucially, feed the system’s actual outputs, successes and mistakes back into the LLM so it can internalize what works. In other words, merge the cheat sheet with the study session. Why this matters: Combining prompt and weight optimization closes the loop between instruction and understanding. It makes models more adaptive, reduces the need for endless prompt fiddling, and helps lessons learned in one task transfer to others. It also offers a practical path to improve efficiency: GEPA cuts rollouts by 35×, and embedding those gains into the weights could compound the savings. Think of it as turning prompt tricks into genuine learning, a step towards models that don’t just follow instructions better but actually become smarter as they go. https://lnkd.in/ewvxR2rv

37

5 Comments

The AI Engineering Stack As Chip Huyen explains in her book "AI Engineering", the AI Engineering stack has three layers. Most of the recent growth is observed in the Applications layer (as we have more and more AI powered applications built on top of foundation models) and Models layer (as we have new powerful models developed and exposed over APIs). Engineering AI applications using the foundational models is really powerful and you need strong domain understanding coupled with good prompt engineering skills. Without good prompt engineering skills and experience, you may still be able to make it work, but it may not be as efficient as it can be. Ultimately, cost is the aspect which you need to look at. Go deep into this stack as much as you can afford, but at least look into the Applications layer and see how you can leverage the other two layers for your use case. What layer(s) do you work at? Follow Kaivalya Apte for more.

43

1 Comment

Running PyTorch in production? Memory is most like an issue and also a silent bottleneck. I came across this blog that shows how they slashed inference latency and costs using lesser-known tricks. Here’s what stood out: 👉 Selective Gradient Checkpointing Checkpoint only memory-heavy layers → cuts peak memory by 40%. 👉 Dynamic Kernel Caching Cache common input shapes during warmup → avoids CUDA recompilation lag. 👉 Manual Precision Casting Control which tensors stay in FP32 vs. BF16 → stability + speed. 👉 Smart empty_cache() Scheduling Call it only during idle windows → avoids perf drops. 👉 Partial Quantization Quantize only safe layers like linear → preserve accuracy, save memory. 👉 Custom CUDA Streams Overlap compute and data loads → reduces GPU idle time. 👉 Shared Memory Tensors Zero-copy multiprocessing → boosts throughput for high RPS services. These aren’t just dev tips — they’re real production survival tactics. Full blog here - https://lnkd.in/gzJSccc8

46

2 Comments

I just published Vol. 120 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 An End-to-End Generative Framework for Industrial E-commerce Search, from Kuaishou 📚 A Model-Agnostic System for Customizable Deep Research Agent Deployment, from NVIDIA 📚 A Family of ModernBERT-Based Embedding Models for Enterprise Information Retrieval, from IBM 📚 Leveraging Uncertainty in Unlabeled Data for Improved Recommendations, from Zhao et al. 📚 Multi-Task Learning and Data Synthesis for State-of-the-Art Text Embeddings, from Kingsoft AI 📚 Fine-Grained Scoring and Reinforcement Learning for Efficient Text Reranking, from Alibaba 📚 Addressing Algorithm Adaptation Bias in Recommender System Evaluation, from Roblox 📚 Training LLMs to be Better Text Embedders through Bidirectional Reconstruction, from Su et al. 📚 Solving the RAG Inference Bottleneck with Adaptive Chunk Embeddings, from Meta 📚 Leveraging Code Generation Models for High-Performance Code Embeddings, from Jina AI #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

25

2 Comments