Everyone's writing about agent context: sub-agents, context sharing, compaction, memory, dynamic tool loading. All great insights, but most of them assumes the agent's output is text — a plan, a code diff, a message.
Ours isn't. We build agents whose output are finished videos:
This one “constraint” reshapes every context engineering decision downstream, and it surfaces two problems the text-agent playbook doesn't really cover.
Internally, we named this, context reframing, you will see why
Step-1: Max-out VLM’s Multi-modal Intelligence
VLMs are the backbone of our agent harness. They have two jobs:
- Before the agent builds, they look at the ingredients: avatars, b-roll, logos, stock clips, AI images.
- After it renders, they look at the result: bad cut, wrong asset, logo collision, subtitle off the frame.
The easy move is to hand the model EVERYTHING and pray that they are intelligent enough to make every right decision.
Don't.
Cost explodes. Signal drowns. Agent fails. The clip / asset / fix you need is buried under fifty you don't. The bottleneck is attention. Everything here is the same move: spend it where it pays.
Librarian, not Museum
Is media context a museum? No. It is a library. Index it, retrieve it, load only what matters.
Anyone can spin up a vector db and claim their agent does smart retrieval, but that's not the real unlock.Naming was.
A filename has to do real work: what the asset is, what it is for, and when to reach for it. People forget that last one. Compare:
- avatar_reference_0473.mp4
- ceo-emily-avatar_3-4.mp4
- ceo-emily-avatar_neutral_front_facing_business_casual.mp4
The second set is reasoning material. "neutral_front_facing..." vs "avatar_3-4" is a casting decision the model can make from the string alone, no pixels loaded. The name is the metadata. Get this right and half your wrong-asset bugs vanish before any VLM runs.
A Frame speaks a thousand Frames
Even with good names, often the agent has to see: best frame, best take, the one scene in the render that broke. The naive way is nine image calls. Nine token bills, nine latency hits, nine guesses that never compare notes.Instead: collage them.
Put nine candidates into one 3×3 contact sheet and send one image.
- One pass, not nine.
- Still groundable. Row 2, column 3 is enough.
- Better comparison. The options sit side by side.
- Cheap. Boring. Works.
The deeper reason is that video already thinks this way. A contact sheet is a storyboard. YouTube's hover preview is a sprite sheet. Same idea: make the clip legible at a glance without paying for every frame.
Step-2: Manipulate Attention Through Numbering
Collage saves tokens, but a flat grid still has one problem. It shows the model what is there, not how to point at it. VLMs are good at what, weaker at when and where. So stop asking for fragile open-ended questions to VLMs. Write the reference system onto the pixels so that "when and where" become "what".
When -> What
Number the sequence.Ask a VLM which frames an event spans and it may give you a confident fiction, like "frames 0 to 580" for a clip only a few seconds long. Stamp an index on each frame. Let OCR read it. Map the number back to a timestamp in code. Temporal grounding becomes reading. This is the idea behind [Number-it / NumPro (CVPR 2025)](https://arxiv.org/abs/2411.10332), and it's training-free. It composes with the contact sheet: a numbered grid of frames is a manga page, so "the highlight runs 3 to 12" is read, not guessed.
Each frame's index ties it to a story beat and a time range, so the agent can localize an event to the second, not just describe it.
Where -> What
Draw the grid. Same move, in space. Pixel coordinates are the wrong interface: precise in format, fuzzy in meaning. Don't ask for coordinates, your VLM will give your this: x=412, y=231, w=380, h=94 - and most of the time, it's not what you want.
Instead, overlay a numbered grid and let the model answer in cells, like "rows 4 to 6, columns 5 to 8." Localization becomes reading again. That's the move behind RSVP (ACL 2025): make the regions explicit, then let the model name them. The grid also gives cheap layout checks. Does the lower-third hit the logo? Is the subtitle covering the product? Is the avatar crowding the edge?
Under a numbered grid, vlm labels each region (person, head, overlay, caption) as a block of cells, so structure and overlaps read straight off the grid, no segmentation model needed and Most importantly, you get exactly what you want: how to output a high quality video composition.
Step-3: Rethink the context for Agents & Humans (To Be Cont.)
We've probably exhausted X article readers attention & context window by now, so we will continue in the part-2 of this technical blog.
A short teaser:
When your agent's output needs to be a fully composed video, what exactly is the output format in its context window? MP4? JSON? Structured Text?
Our answer to this question is: @HyperFrames_ (HTML+CSS+JS)
Learn more about HyperFrames here.
In pt2, we will dive into the topic of building the proper context so that agents and humans can simultaneously operate on top and maximize LLM's intelligence and human's creativity and taste.