
So, you've probably heard about these new AI models like Gemini 1.5 Pro and GPT-4.1 that can handle a crazy amount of information at once due to their long context windows. It sounds amazing, right? Just dump a massive document into the AI and get perfect answers.

Well, it's not that simple. New research from Chroma reveals a major problem: these models start to struggle with real-world tasks as the amount of text increases. Even though they can technically process millions of tokens, their performance gets worse the longer the document gets.

---

## The Problem: Context Rot

Think of it like this: these AI models are supposed to handle the 10,000th word just as well as the 100th word. But in reality, they don't. As the text gets longer, the models get confused and start making more mistakes.

The Chroma team tested 18 different models, including the latest and greatest like GPT-4.1, Claude 4, Gemini 2.5, and Qwen3. What they found was pretty eye-opening: these models don't process long contexts uniformly at all.

### What is Context Rot?

Context rot is what happens when AI models become less reliable as the input gets longer. It's like trying to remember details from a 500-page book versus a 5-page summary - the more information there is, the harder it gets to find what you're looking for.

---

## What They Actually Tested

The Chroma team didn't just run simple tests. They put these models through some really interesting challenges to see how they'd perform in real-world scenarios:

### 1. The Needle in a Haystack Test (But Harder)

You know the classic "needle in a haystack" test? It's where you hide one specific sentence in a huge pile of text and ask the AI to find it. Well, the researchers made it much harder.

Instead of asking for exact matches, they tested semantic understanding. For example:
- **Question:** "Which character has been to Helsinki?"
- **Needle:** "Actually, Yuki lives next to the Kiasma museum."

The AI has to know that the Kiasma museum is in Helsinki, then figure out that Yuki is the character who's been there. This is way harder than just finding the word "Helsinki" in the text.

### 2. Long Conversations Test

They used something called LongMemEval to test how well models remember things from long chat histories. Think about having a conversation with an AI assistant that goes on for hours - can it still remember what you talked about at the beginning?

### 3. The Repeating Words Challenge

This one is wild. They gave the models a long list of repeating words with one unique word in the middle, like "apple apple apple apple... [unique word] ...apple apple apple." As the list got longer, the models started messing up - they'd miss the unique word, put it in the wrong place, or even start generating random nonsense.

---

## What They Found (And It's Pretty Surprising)

### 1. Finding the "Needle" Gets Trickier

When the question wasn't a direct match, it was much harder for the models to find the correct information. The longer the text, the worse they got. So if you ask a question about a long document, the model might get confused and give you the wrong answer.

### 2. Distractors Are a Big Problem

They also tried adding "distractors" - bits of information that are similar to the right answer but are actually wrong. The more of these distractors they added, the worse the models performed. This is a huge deal because real-world documents are full of similar but irrelevant information.

### 3. The Jumbled Text Surprise

Here's the weirdest finding: the models were actually better at finding the needle when the haystack was a complete mess of jumbled-up sentences! This was really surprising - you'd think organized text would be easier to process, but nope.

### 4. Long Conversations Don't Work Well

In the LongMemEval test, the models did a fantastic job when the context was very to-the-point. But with long, irrelevant context, the performance just dropped. Adding all the context isn't the solution.

### 5. Models Get Confused by Repetition

With the repeating words test, as the list got longer, the models started messing up. They would miss the unique word, put it in the wrong place, or even start generating random nonsense. It's like their attention span gets overwhelmed.

---

## Why This Matters for Real Apps

### 1. RAG Systems Are Affected

If you're building a RAG (Retrieval-Augmented Generation) system, this is bad news. These systems work by giving the AI a bunch of relevant documents to work with. But if the documents are too long, the AI might get confused and give wrong answers, even though all the right information is there.

### 2. Document Analysis Gets Harder

Think about legal documents, medical reports, or research papers. These are often hundreds of pages long. If you're building an AI to analyze these, it might miss important details just because there's too much text to process.

### 3. Chatbots Get Confused

Long conversations with AI assistants might become less reliable over time. The AI might forget what you talked about at the beginning of the conversation, even though all that information is technically still in its memory.

### 4. Code Generation Suffers

If you're trying to get an AI to understand a large codebase, it might miss important details or make mistakes because there's just too much code to process at once.

---

## The Problem with Current Testing

The research also revealed some major issues with how we've been testing these models:

### 1. The Standard Tests Are Too Simple

The classic "Needle in a Haystack" test only checks if the AI can find exact word matches. But real-world questions are much more complex - they require understanding context, making connections, and dealing with ambiguity.

### 2. We've Been Testing the Wrong Things

Most benchmarks mix up two different problems: the difficulty of the task itself versus the model's ability to handle long contexts. This makes it hard to figure out what's actually causing the performance drops.

### 3. Real-World Scenarios Are Missing

Current tests don't account for the messiness of real documents - they don't have distractors, ambiguous information, or conversational context that real applications have to deal with.

---

## So What's the Solution?

### 1. Context Engineering is Key

The big takeaway is that "context engineering" is super important. You can't just dump a huge amount of text into the AI and expect the best results. You need to be smart about how you present the information.

Think of it like cooking - you can't just throw all the ingredients into a pot and expect a great meal. You need to think about the order, timing, and presentation.

### 2. We Need Better Tests

The research shows we need more realistic benchmarks that:
- Test how models handle long contexts specifically
- Include semantic understanding, not just word matching
- Account for distractors and confusing information
- Test real conversational scenarios

### 3. Model Makers Need to Step Up

Instead of just making models that can handle more tokens, we need models that can handle long contexts more reliably. It's not about quantity - it's about quality.

### 4. Developers Need to Be Smarter

If you're building AI applications, you need to:
- Break down long documents into smaller, focused chunks
- Use summarization to create better context
- Think carefully about where you put important information
- Test your app with realistic document lengths

---

## The Bottom Line

So, what's the long and short of it?

This research shows that while AI models can technically handle millions of tokens, they don't actually process long contexts very well. The longer the document gets, the more confused they become. This means that anyone building apps with these models needs to be really careful about how they present information.

**Context engineering is super important.** You can't just dump a huge amount of text and expect the best results. You need to be smart about chunking, organizing, and presenting the information in a way that the AI can actually process effectively.

Context Rot: Why More Context Isn't Always Better for AI


Revolutionary AI agent that starts with minimal tools and evolves by creating its own capabilities on-the-fly, achieving state-of-the-art results on GAIA benchmark.

ALITA (Agent Generalist Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolution) represents a paradigm shift in AI agent architecture - instead of being a consumer of predefined tools, it becomes a creator of its own tools as needed.

---

## Executive Summary of ALITA

ALITA is a new type of AI agent designed to autonomously perform complex, open-ended tasks. Unlike many existing AI agents that rely on a large, manually predefined set of tools and workflows, ALITA is built on two core principles:

### 1. Minimal Predefinition
ALITA starts with very few built-in tools. Essentially, its main predefined capability is a "Web Agent" that can search the internet.

### 2. Maximal Self-Evolution
ALITA is designed to create its own tools as needed. When faced with a task it can't solve with its existing (limited) capabilities, it:

- **Brainstorms:** Figures out what kind of tool or capability it's missing
- **Searches:** Uses its Web Agent to find open-source code, libraries, or documentation that could help build that tool
- **Generates:** Writes new code (typically Python scripts) to implement the needed functionality  
- **Tests & Packages:** Runs the new code in an isolated environment. If it works, it packages this new script/tool as a "Model Context Protocol" (MCP)
- **Reuses:** Stores these self-created MCPs in an "MCP Box" for future use by itself or potentially other agents

This approach allows ALITA to be highly adaptable, scalable, and generalizable across different domains without developers needing to anticipate and pre-program every possible tool. The paper shows ALITA achieving top-ranking performance on the GAIA benchmark, outperforming more complex systems.

---

## Key Research Insights

### Core Philosophy
"Simplicity is the ultimate sophistication." The design emphasizes a lean starting point and dynamic capability generation.

### Architecture Components

**Manager Agent:** The central coordinator that analyzes tasks, decides if new MCPs are needed, and orchestrates the MCP creation process or uses existing MCPs.

**Web Agent:** Searches for information, libraries, and code.

**MCP Creation Component:** Includes sub-modules for brainstorming, script generation, code running, and environment management (e.g., creating isolated Conda environments).

### MCP (Model Context Protocol) Creation Process

MCPs are essentially self-generated, reusable tools/scripts that follow this process:
1. MCP Brainstorming 
2. Open-source Searching (via Web Agent)
3. Script Generation
4. Virtual Environment Execution & Testing
5. Encapsulation as MCP

This allows the agent to learn and expand its toolkit over time in a self-reinforcing cycle.

### Performance Results

- Achieves state-of-the-art results on the GAIA benchmark (75.15% pass@1), outperforming systems like OpenAI's DeepResearch
- Shows strong performance on Mathvista and PathVQA
- **Distillation Effect:** MCPs generated by ALITA (using powerful LLMs like GPT-4o) can be used by simpler agents, significantly boosting their performance

---

## What Makes This Game-Changing?

The revolutionary aspect is the shift from agents being consumers of predefined tools to agents being creators of their own tools on-the-fly.

### Key Advantages

**Scalability & Adaptability:** Traditional agents hit a wall because you can't pre-build a tool for every conceivable task. ALITA's approach means it can adapt to entirely new problems by finding and integrating existing open-source solutions.

**Reduced Development Overhead:** Developers don't need to spend time building and maintaining a vast library of tools. Instead, they focus on building the meta-capability of tool creation.

**Emergent Capabilities:** The agent can develop capabilities that the original designers might not have thought of, simply by combining information and code it finds in novel ways.

**Democratization of Tooling:** The MCPs, once created, are standardized and can potentially be shared and reused across different agent systems.

This "maximal self-evolution" through dynamic tool generation is a significant step towards more autonomous and truly "generalist" agents.

---

## Future Applications

### 1. More Powerful Generalist Assistants
As LLMs improve, ALITA-like agents will become even better at creating more complex and reliable tools, tackling sophisticated tasks like complex data analysis, automated scientific experimentation, and advanced software development assistance.

### 2. Personalized Agent Ecosystems
Users could have agents that build up a unique set of MCPs tailored to their specific needs, software, and workflows, becoming highly personalized and efficient assistants.

### 3. Automated Software Development
Agents could not only write code for specific functions but also find, adapt, and integrate existing libraries and APIs to build more complex applications with less human intervention.

### 4. Rapid Prototyping
Need a quick script to convert file formats, scrape a specific website, or automate a repetitive task? An ALITA-like agent could generate it for you.

### 5. Collaborative Agent Networks
Agents could share their self-generated MCPs, leading to a collective intelligence where the capabilities of the entire network grow much faster.

---

## Practical Implementation for Developers

Even without implementing the full ALITA framework, developers can adopt the core principles:

### 1. Embrace Dynamic Tool Creation
- Design your agent to recognize when it's missing a capability
- Have the LLM describe the tool it needs (inputs, outputs, function)
- Search for existing Python libraries, code snippets, or APIs
- Use LLMs to generate Python scripts based on task specifications
- Always test in isolated environments (Docker containers, virtual environments)

### 2. Build a Reusable "Toolbox"
- Save working scripts with descriptions
- Create a simple system to store these scripts (folder of Python files with JSON manifest)
- Teach your agent to check its "toolbox" before generating new tools

### 3. Focus on "Manager Agent" Logic
Design prompts that encourage the LLM to:
- Break down complex problems
- Identify when existing tools are sufficient
- Recognize when new tools are needed
- Plan the steps to create or use tools

### Example Workflow
```
User: "Find current weather in London and summarize top 3 AI news articles"

Agent Manager:
1. "Do I have a weather tool?" → If no, create one using weather API
2. "Do I have a news search tool?" → If no, create one
3. "Do I have a summarizer?" → If no, create one
4. Execute tools and combine results
```

By adopting this mindset of dynamic capability generation, agents can become more flexible and powerful without developers having to foresee every possible need.

---

## Key Limitation

The ability of ALITA to generate useful MCPs is highly dependent on the coding and reasoning capabilities of the underlying Large Language Model. When run with weaker LLMs, performance drops significantly, though this will improve as LLMs advance.

ALITA: Agent That Creates Its Own Tools


Providing context to the AI so that it gives out a magical response you want? That is now possible with RAG! (Retrieval Augmented Generation), upload your data source to AI and get better response every time.

---

## What is RAG?

In simple terms, RAG enhances large language models (LLMs) by integrating external knowledge sources, such as databases, documents, or even real-time web data, into their responses.

Traditional LLMs rely on static training data, RAG retrieves specific, relevant information before generating a response. This ensures outputs are more accurate, context-aware, and aligned with user intent.

---

## Architecture of RAG

Let's go through 3 simple steps and understand the underlying principle:

### 1. Ingestion

- **Data Source:** Upload a document or provide a data source on which you want AI to give a better response
- **Chunks:** Consider chunks as your data source broken down into small pieces
- **Embeddings:** Embeddings are vector representation of the chunks that we had in previous steps
- **Vector DB:** It is a type of database that stores the vectors, which are generated by embeddings, for future purpose

### 2. Retrieval

- **Query Embedding:** When a user asks a question, the query is converted into an embedding (vector representation)
- **Search in Vector DB:** The query embedding is compared with stored embeddings in the Vector DB using similarity measures
- **Retrieve Relevant Chunks:** Based on the similarity score, the most relevant document chunks are retrieved from the database

### 3. Augmentation and Generation

Combines the retrieved chunks with the user query to create an augmented input. Passes this augmented input to a LLM, which generates a contextually rich and accurate response by using both its pre-trained knowledge and the retrieved information.

---

## Why RAG Matters

Using RAG provides proper context and unlocks true capabilities of AI and allows for accurate and relevant answers without long explanations. It bridges the gap between static AI knowledge and dynamic, real-world information needs.

RAG Architecture


I slashed my Node.js application image size by 6 times! Docker often eats up a lot of memory (we all know that pain right?)

Here's where Docker Multi-Stage builds changes the game completely. Traditional Dockerfiles often lead to bloated images due to unnecessary dependencies carried over from the build process. Multi-stage builds solve this by using multiple intermediate images, ensuring only the essentials make it to the final image, resulting in faster deployments and improved security.

---

## How Multi-Stage Builds Work

Multi-stage builds work by defining multiple `FROM` statements in a single Dockerfile. The first stage includes everything needed to build the application, like dependencies and compilers. The second stage, which is the final one, extracts only the compiled output, significantly reducing the image size. This approach is particularly useful for Node.js applications where `node_modules` and build artifacts can be large.

Let's dive into a real-world example that reduces image size by 80% while improving security:

---

## Traditional Dockerfile (The Problem)

```dockerfile
FROM node:18
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm install
COPY . .
RUN npm run build
CMD ["node", "dist/index.js"]
```

**Issues with this approach:**
- This final image contains unnecessary build dependencies
- The image size is large (~500MB+)
- Security risks due to extra files being included

---

## Optimized Multi-Stage Build (The Solution)

```dockerfile
# First stage: build
FROM node:18 AS builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm install
COPY . .
RUN npm run build

# Second stage: production
FROM node:18-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
CMD ["node", "dist/index.js"]
```

**Benefits of this approach:**
- The final image only contains the necessary runtime files
- Reduced image size (under 100MB)
- Improved security by avoiding unnecessary dependencies

---

## The Results

With this method, we achieve:

- **Better Efficiency**: Significantly smaller image sizes mean faster deployments
- **Enhanced Security**: Fewer components in the final image reduce attack surface
- **Improved Performance**: Lighter images consume less memory and start faster

Whether deploying to the cloud or optimizing CI/CD pipelines, multi-stage builds in Docker are a must-have for any serious developer.

---

## Key Takeaways

1. **Use Alpine Images**: The `node:18-alpine` base image is much smaller than the standard `node:18`
2. **Copy Only What's Needed**: Use `COPY --from=builder` to selectively copy files from build stage
3. **Separate Build and Runtime**: Keep build tools in the first stage, runtime essentials in the final stage
4. **Security First**: Fewer dependencies mean fewer potential vulnerabilities

Diving into Multi-stage Builds with Docker


First things first while developing a VS Code extension, Don't use Normal VS Code, Instead use [VS Code Insiders](https://code.visualstudio.com/insiders/) which is a development version of VS Code specially built to test the changes when you want to build something that is related to VS Code.

Managing authentication in a VS Code extension can be frustrating. There aren't many tutorials that cover this topic in depth; most only explain basic concepts like registering a command and a small use case, without going into CRUD operations.

---

## The Challenge

I wanted to perform CRUD operations from my VS Code extension, which means I needed to send authenticated requests to the back-end. There are two main ways to do this:

- Using Cookies
- Using JWT token

Cookies is way too complicated when it comes to accessing it and storing the info in the global storage of VS Code. So, I decided to move with JWT token. After extensively researching about how to handle the authentication in the VS Code, I finally came up with a solution that works pretty well.

---

## Step-by-Step Solution

### Step 1: Create an Extension Project Ready for Development

```bash
npx --package yo --package generator-code -- yo code
```

### Step 2: Handling the Authentication with Express

We will fire up a local-server when there is a request to authenticate the user, (expecting a token from back-end in the URL parameters) we will get the token and extract it from URL and store it in VS Code.

The purpose of firing up the server is to just have a safe local server that will get the token without any tampering of security.

```javascript
//Authentication file
const vscode = require("vscode");
const express = require("express");
const { TokenManager } = require("./TokenManager.js"); // Token Manager explained below

const authenticate = (fn) => {
  const app = express();
  app.use(express.urlencoded({ extended: true }));
  
  app.get(`/auth/:token`, async (req, res) => {
    const { token } = req.params;
    if (!token) {
      res.send(`<h1>Something went wrong</h1>`);
      return;
    }
    
    await TokenManager.setToken(token);
    fn();
    res.send(`<h1>Authentication was successful, you can close this now</h1>`);
    server.close();
  });
  
  const server = app.listen(54321, (err) => {
    if (err) {
      vscode.window.showErrorMessage(err.message);
    } else {
      vscode.env.openExternal(vscode.Uri.parse(`http://localhost:54321`));
    }
  });
};
```

---

## Token Manager Implementation

So what is Token Manager that I am using? Don't worry, it's just a custom file that handles the token and stores it in VS Code.

```javascript
//TokenManager file
const KEY = "your_token_name";

class TokenManager {
  static globalState;
  
  static setToken(token) {
    return this.globalState.update(KEY, token);
  }
  
  static getToken() {
    return this.globalState.get(KEY);
  }
  
  static removeToken() {
    return this.globalState.update(KEY, null);
  }
}

module.exports = { TokenManager };
```

---

## How It Works

1. **Local Server Setup**: When authentication is needed, we spin up a local Express server on port 54321
2. **Token Extraction**: The server listens for a GET request with the token as a URL parameter
3. **Token Storage**: Once received, the token is stored using VS Code's global state management
4. **Cleanup**: The server automatically closes after successful authentication

This approach provides a secure way to handle JWT tokens in VS Code extensions without the complexity of cookie management.

About

Hey! I'm Dishant ^_^
I build things that people use, break and accidentally depend upon :P

Experience

Backend AI Engineer

Contributor under Google Summer of Code

Software Engineer Intern

Writing