The tenth agent in our 30 agents series is Video to Plan: give it a recording and it gives you back the thing that recording is actually good for — a plan you can run, the steps behind how something was made, or an honest briefing. The interesting part of building it was not the comprehension. It was discovering, in one afternoon, exactly which parts of the job the internet will let you do.
A transcript quietly loses the answer
Everyone has a transcript tool. The reason to build another video agent is that a transcript throws away the half of a video that is often the point. A tutorial says “and then you set it up like this” over a screen recording of a settings panel. A woodworking video cuts a joint at an angle nobody states out loud. A UI walkthrough is entirely the picture.
So this agent has two ways in and it says which it is using. It can listen — transcription, right when the value is in what is said. Or it can look, handing the file to a model that takes video as input. On a music video we used as a control, looking produced this:
a bartender in a white shirt and red suspenders wiping down a bar (0:51, 1:15, 1:41)
a large frosted window with Gothic tracery
two blonde female back-up dancers in black outfits (0:48, 0:54, 1:13)No transcript contains any of that. That is the feature.
Then the model invented a download
Before any of the agent existed we were testing whether the sandbox could fetch a video at all. It reported success: a progress bar finishing at 16.22MiB/s, a destination file, an ls -la showing 2518918 bytes, a du -h agreeing at 2.4M. Convincing enough that we took an engineering decision on it.
None of it happened. The run executed nothing in the sandbox at all — no [sandbox] exec, no process, and a later real ls found the directory empty with a timestamp from before the supposed download. Four tool calls had failed, each rendering in the transcript as an anonymous row reading Unknown · An error occurred · unknown, and the model had filled the silence with a plausible result.
The cause was mundane and the lesson is not. Our error formatter had a branch for a missing tool, a branch for a provider envelope, and a fallback — and every one of them assumed a JavaScript Error object with a message. A rejection arriving as a bare string or a plain object fell past all of them to the constant “An error occurred.” Four identical useless sentences, no tool name, nothing to act on.
We fixed three things: the formatter now reads those shapes and serialises anything it does not recognise, the failed call renders with its actual name instead of “Unknown”, and — the part we should have had all along — the gateway now logs finishReason, the tool names called, and how many results came back. That field had existed on the run record for months and was written nowhere.
The difference showed up immediately. Same command, same model, an hour later:
ERROR: [youtube] jawDLLvuRAA: Video unavailable
EXIT=1
md5sum: zoo.mp4: No such file or directoryA real failure, reported as one. If you take a single thing from this post, take that one: a tool error that says nothing is not a cosmetic problem. It is an invitation to a model to make something up, and it will accept.
YouTube does not serve datacentres
With honest errors we could finally see the real constraint. At 10:35 the sandbox downloaded an 11 MiB video cleanly. By 10:57 the same address could not fetch metadata. YouTube had flagged the IP — ours was 34.96.63.38, which is Google Cloud, which is a datacentre, which is exactly what it refuses.
We tested the workarounds rather than assuming them:
- Different player clients —
tv,mweb,web_safari. All refused together. There is no clever flag. - Cookies from a logged-in account — works, and we will not do it. YouTube cookies are Google cookies; one leak is a full account, and accounts used this way get banned. That is not a thing to build a product on.
- A Google login — does not help at all, which surprised us. OAuth grants the YouTube Data API, and its caption download works only for videos you own.
- oEmbed — works, unblocked, free. It gives the title and channel, so the agent can always tell you which video you linked even when it cannot open it.
So we shipped the honest shape. The reliable input is a file. Drop a recording into the workspace and everything works — that half was proven the same afternoon and never wavered. Paste a link and the agent identifies it, tries once, and tells you plainly that it cannot download from a server rather than pretending otherwise.
The ceilings we found by hitting them
A 6½-minute video is 18.7 MB. Our workspace upload stops at 20 MB, so the practical limit is about seven minutes of video — while the use case we had in mind was the forty-minute talk somebody told you to watch. That is on the landing page, in the limits section, in those words.
And the video-input model refused the same file with request entity too large, at a threshold lower than the upload limit — so a file you can successfully upload is one the agent may not be able to look at.
What the agent did next is the most encouraging thing in this post. Unprompted, it installed OpenCV, sampled frames a few seconds apart, tiled them into contact sheets, and went to read those instead — which is better than the video for recovering a subject line or a filter setting, because you can stare at a still. It then discovered it had no tool to view an image, because we had not bound one, and told us so. Three platform bugs and a missing tool, found by an agent doing its job. That has now happened with every agent we have built.
What it is for
Read what the video affords, and produce that. A tutorial becomes numbered steps with the settings named and the prerequisites it never mentioned written down. A how-it-is-made becomes how it was done plus what it would take to do yourself. A talk becomes a briefing — which is not a consolation prize, because an honest briefing beats a plan invented to fill a template.
Every specific claim carries a timestamp, so you can check it without rewatching. And when it cannot tell — a blurred terminal, a step done off camera — it says so. “The config is on screen at 4:12 but unreadable” is a more useful sentence than a confident guess, and after this week we are unusually motivated to mean it.