AI Auto-Framing for Vertical Video: How It Works
An honest explainer of AI auto-framing for live vertical production – what it actually does, where it works brilliantly, where it still needs a human in the loop, and what to ask any vendor selling it.
Vertical video isn’t a niche anymore. What began as a social-first format has become a primary way audiences consume sports content, particularly on mobile platforms. At FOX Sports, nearly 90% of digital content is now consumed in a vertical format, reflecting a broader shift in how fans discover, watch and engage with live moments.
For broadcasters and rights holders, the question has stopped being whether to produce vertical content and started being how to produce it at the same speed, quality and scale as the main 16:9 broadcast, without doubling the crew.
That problem is what AI auto-framing solves. It’s one of the biggest enablers behind live vertical production at scale, yet most explanations of it are vendor pitches rather than honest descriptions of what it does, where it shines and where it still struggles.
What is AI auto-framing?
AI auto-framing is the use of computer vision to detect the subject of a video frame in real time and dynamically reframe the picture around it, most commonly to convert a horizontal (16:9) feed into a vertical (9:16) one without cutting off the action. It effectively acts as a virtual camera operator, keeping the most important part of the picture inside a new, narrower frame as the action moves.
How AI auto-framing actually works
Under the surface, an auto-framing system does four things in sequence, several times a second.
1. Detect
A computer-vision model scans each frame and identifies salient content – players, the ball, faces, the focus of action. Different systems weight these signals differently: some are trained specifically for sport, some for presenters and talking heads, some are general-purpose. The detection model is the heart of the system, and the difference between a good auto-framing tool and a poor one usually starts here.
2. Choose a crop
Given the detected subject, the system selects the optimal 9:16 (or 1:1, or 4:5) crop that contains the action. This is harder than it sounds: in fast sport the “subject” can shift across the frame in milliseconds, and the crop has to anticipate movement rather than chase it.
3. Smooth
A raw, frame-by-frame crop would judder horribly. The system applies temporal smoothing, easing the crop’s movement over time so it pans like a human operator rather than snapping. Too much smoothing and the crop lags behind the action; too little and it looks twitchy. Getting this right is the second biggest differentiator between vendors.
4. Output
The reframed picture is rendered out as a clean 9:16 stream, ready to go to TikTok, Reels, Shorts, Stories, or a dedicated vertical alt-cast, in parallel with the main 16:9 broadcast and from the same source feed.
Live vs post-production: two different jobs
AI auto-framing splits cleanly into two use cases that get talked about as if they’re the same thing, and aren’t.
Post-production reframing takes a finished clip and reframes it after the fact – ideal for clipping highlights, social-ready cuts and packaged content. The system has the whole clip in hand and can plan ahead. Most consumer tools and many editing platforms do this well.
Live reframing has to make the same decisions in real time, frame by frame, with no knowledge of what happens next. It also has to do them fast enough to keep up with the main broadcast. AWS Elemental Inference, which Grabyo uses, runs with a 6-10 second end-to-end delay for live vertical from horizontal. That order-of-magnitude difference in difficulty is why most consumer tools don’t even attempt live, and why live vertical at broadcast scale only got viable in the last few years.
Why AI auto-framing matters: the alternatives are harder
It’s easy to underestimate what AI auto-framing is replacing. The two alternatives both have real costs.
| Approach | What it involves | Why it doesn’t scale |
|---|---|---|
| Native vertical capture | A second camera (or operator) framed for portrait from the start: sometimes a phone, sometimes a dedicated camera. | Adds crew, kit and cost per venue. Fine for behind-the-scenes; impractical for full live coverage at scale. |
| Manual cropping / reframing | An operator actively repositioning a 9:16 crop on top of the 16:9 feed throughout the production. | Doesn’t scale when you’re producing dozens of moments an hour. Fixed centre-crops drop the play the moment it drifts off-centre. |
| AI auto-framing | Computer vision detects the subject and reframes in real time, with a human able to override when needed. | Genuinely scales: one operation produces 16:9 and 9:16 in parallel from the same sources. |
Where AI auto-framing wins
Auto-framing performs best in content with a clear, trackable focus of action, which is most of what live sport and presenter-led content actually is.
- Single-subject sport: tennis, athletics, golf, boxing. One athlete, one obvious subject, clean tracking.
- Ball-led team sport: football, basketball, rugby, cricket. The ball gives the system a strong, stable signal to anchor the crop to.
- Presenter and talking-head content: news, studio shows, interviews. Faces are the most reliable subject computer vision has.
- Music and live events: solo performances, principal artists, single-subject moments on stage.
Where AI auto-framing still struggles, and what to do about it
Honest answer: auto-framing isn’t a magic button. Three situations regularly trip it up, and the answer in each case is the same: a human operator with the ability to override.
Crowd shots and ensemble action
When there’s no single “subject”, such as a crowd celebration, a wide ensemble dance number, a team huddle, the system has nothing reliable to anchor to. A fixed wide crop, or a manual override to a presenter, almost always looks better than the AI’s best guess.
Wide establishing shots
A stadium pan, an opening flyover, a sweeping landscape, these are framed to be seen wide and lose most of their meaning when cropped to portrait. The system can’t rescue this; the right call is a different shot in the vertical feed, not a reframe of the wide.
Fast, unpredictable subject changes
Sudden cuts to a sideline reaction, a coach, an off-ball incident. The AI is reactive by nature and will catch up a fraction late. A skilled vertical-feed operator with manual override solves this without breaking the main broadcast.
The human in the loop: why pure automation isn’t the answer
The best live vertical operations use AI auto-framing as the default and a human operator as the safety net. The AI handles the 90% of moments where it does the job better than a human ever could at scale; the human takes the wheel for the 10% where editorial judgement matters more than tracking. Vendors who pitch “fully autonomous” vertical production are oversimplifying, the operations actually doing this at broadcast scale keep a human in the chair.
Editorial control still matters
AI auto-framing solves a huge part of the vertical production challenge, but it doesn’t remove the need for editorial judgement.
There will always be moments where a producer wants to make a creative decision rather than rely on automation. A social editor may want to favour a player reaction over the ball, reposition a subject for better visual balance, or create a completely different composition for a specific platform. In those situations, manual reframing remains an important part of the workflow.
The most effective vertical production tools combine both approaches. AI can handle the heavy lifting by generating a live vertical output in real time, while producers retain the ability to refine, smooth and adjust framing when greater editorial control is required.
This is particularly valuable for clipping workflows. While live auto-framing helps create vertical feeds at scale, editors often want to revisit key moments and make precise framing adjustments before publishing to social platforms. Having those controls available within the same workflow avoids the need to export content into separate editing applications, helping teams move quickly without sacrificing quality or creative intent.
The goal isn’t to choose between AI and manual production. It’s to use automation where it adds efficiency and apply human judgement where it adds value.
What to ask any vendor selling AI auto-framing
Five questions sort the serious offerings from the marketing-led ones.
- How quickly does it run for live? Sub-10-second end-to-end delay for live reframing is the bar.
- What’s the detection model trained on? Sport-trained, presenter-trained, general? It matters, a system trained mostly on YouTuber faces won’t track a football match well.
- Can a human operator override it? If not, walk away. Editorial control is not optional at broadcast scale.
- Does it produce multiple aspect ratios simultaneously? 16:9, 9:16, 4:5, 1:1 from the same source in parallel is the modern standard.
- Does it integrate with my production workflow? A standalone reframing tool is fine for clipping; for live, you want it inside the same production environment as your main broadcast.
Where this fits in a modern vertical workflow
AI auto-framing is one part of a broader move toward producing vertical at the same scale as horizontal. The full picture, covered in our definitive guide to vertical video production, involves choosing the right aspect ratios for each platform, deciding whether to run a dedicated vertical alt-cast (sidecar broadcast) alongside the main feed, and producing both outputs from the same live sources without doubling the crew. AI auto-framing is the bit that makes the whole thing economic.
