Adobe is extending Firefly generative AI to its video tools
Adobe has unveiled the Firefly Video Model, a new AI model that will bring generative AI to the company’s video production tools like Premiere Pro, and potentially After Effects.
Its capabilities include generating additional audio or video frames from an existing video clip, and generating new video clips from reference images or text prompts.
Below, we’ve compiled our FAQs on the Firefly Video Model.
What can Firefly’s generative AI models currently do?
Released last year, Firefly is a suite of generative AI art tools available both as a web application and integrated into Adobe Creative Cloud tools.
The underlying AI models were initially geared towards generating still images, and are now integrated into applications including Photoshop and the Substance 3D tools.
What are the key features of the Firefly Video Model?
The new Firefly Video Model extends those capabilities to moving images, synthesising new video footage from a range of sources in a similar way to platforms like Runway.
According to Adobe, its capabilities include Generative Extend, which extends existing video clips, generating new frames at the start or end, or generating extra audio ‘room tone’.
Image to Video generates clips based on reference stills, or even concept illustrations.
Text to Video generates video from text prompts, with the additional option to specify camera properties like camera position, zoom extent and common camera motion types.
What kinds of video and animation can the Firefly Video Model generate?
Although the Firefly Video Model can generate 2D and 3D animation – Adobe’s blog post shows examples of generated footage resembling 3D, hand-drawn and stop-motion animation – its main initial use case is more naturalistic video.
According to Adobe, it “excels at generating videos of the natural world … like landscapes, plants or animals”.
However, the firm also cites a range of other possible uses, including creating FX elements like fire, smoke and dust that could be composited into VFX or motion graphics projects.
How good is the Firefly Video Model?
Although the Firefly Video Model could potentially generate entire video sequences, current generative video tools struggle to maintain temporal and stylistic consistency in their output.
Adobe’s blog post suggests that its initial use will be to generate supplementary material for projects shot conventionally – essentially, generating AI stock footage – citing use cases like generating B-roll and missing establishing shots.
The Firefly Video Model’s Generative Extend feature in use in a Premiere Pro project.
Which Adobe applications will the Firefly Video Model be integrated in?
Although the Firefly tools are available online, Adobe is also increasingly integrating them into its Creative Cloud tools.
So far, the only one in which the Firefly Video Model will definitely be included is video editing software Premiere Pro, which is due to include Generative Extend in beta builds “later this year”.
However, Adobe’s blog post also namechecks compositing app After Effects. The firm has previously shown tech previews of generative AI tools suitable for After Effects.
What data is the Firefly Video AI model trained on?
Adobe describes the Firefly Video AI model as “commercially safe and is only trained on content we have permission to use”.
Earlier this year, the firm updated its Terms of Use to explicitly rule out the use of users’ content – aside from that uploaded to stock content marketplace Adobe Stock – to train AI models, following a widely publicised artist backlash.
Price, release date and system requirements
The Firefly Video Model is currently a technology preview. It is due to become available in public beta “later this year”.
Aside from Premiere Pro, Adobe hasn’t announced which Creative Cloud applications it will be integrated in, or in which releases.
Existing Firefly generative AI services are credit-based. You can find details of pricing here.
Read more about the Firefly Video model on Adobe’s blog
Have your say on this story by following CG Channel on Facebook, Instagram and X (formerly Twitter). As well as being able to comment on stories, followers of our social media accounts can see videos we don’t post on the site itself, including making-ofs for the latest VFX movies, animations, games cinematics and motion graphics projects.