Tuesday, March 21st, 2023 Posted by Jim Thacker

Nvidia and Shutterstock to build AI text-to-3D service

Nvidia is partnering with stock asset library Shutterstock to build new AI-based tools that will enable users to generate custom 3D models simply by typing in text descriptions.

Text-to-3D capabilities will be added to Shutterstock’s Creative Flow suite of AI apps, and will also be available on the Shutterstock-owned TurboSquid website.

Nvidia has also partnered with Adobe and Getty Images to create new text-to-image and text-to-video models for the launch of Nvidia Picasso, its new cloud service for building generative AI tools.

The announcements were made at GTC 2023, along with six new professional RTX GPUs based on Nvidia’s Ada Lovelace architecture, and updates to Omniverse.

Creating 3D models from text descriptions: the ‘Holy Grail’ of generative AI art tools
Speaking to CG Channel ahead of the launch, Shutterstock VP of 3D innovation Dade Orgeron described the creation of production-quality 3D models from text prompts as the “Holy Grail” of generative AI art tools.

Existing experimental tools like Google’s DreamFusion and Nvidia’s own Magic3D can generate 3D models from text descriptions, but their output is relatively low-quality.

Orgeron told us that recent months have seen a “spectacular leap forward” in the level of detail in generated 3D models, and that current AI models are “phenomenal” compared to those created even a year ago.

One of Nvidia’s demos of its Edify AI models, which will be used to develop Shutterstock’s text-to-3D service.

What will Shutterstock’s text-to-3D service be able to do?
On its launch, Shutterstock expects its new text-to-3D service to generate 3D models of a quality suitable for non-demanding hobbyist work, or as a base that can be refined manually for commercial projects.

Processing will be done online, and is expected to take around 15 minutes per model.

Initially, the service is expected to generate single meshes with single textures, although the generation of more complex multi-part models may become possible in future.

Its most likely initial use is to create content for industrial digital twins, recreating real-world objects like machinery, buildings and cars, but Shutterstock expects it to be adoped for entertainment work in future.

But what if I don’t want my 3D models used to train AI art tools?
The underlying AI models will be trained on assets from TurboSquid, the online marketplace that Shutterstock acquired in 2021, and which currently includes over 1.5 million 3D models.

The use of artists’ work to train AI tools is a contentious issue: something we’ve touched on in stories like this one on Glaze, a free tool intended to prevent unauthorised use of images to train “unethical” AI models.

Shutterstock describes its own policy – introduced following its partnership with DALL-E developer OpenAI to create its online text-to-image generator – as “responsible AI”.

Artists can opt out of having their content included in AI training datasets, although Shutterstock told us that only around 10% of its users had done so since the option was added to account settings last month.

Those that don’t opt out are paid for the use of their assets from a contributor fund, with payments made every six months, although Shutterstock hasn’t published details of how earnings are calculated.

Orgeron told CG Channel that the value of a 3D model to a training data set depended on a number of factors, including the quality of the mesh, how parts are named, and how well materials are represented.

What consequences will this have for the 3D industry?
The move is significant for the development of AI tools for generating 3D content, as opposed to 2D images.

Following last year’s wave of new text-to-image apps, Google identified the lack of large-scale labelled datasets of 3D assets for training AI models as a major obstacle in developing equivalent text-to-3D tools.

Being able to train AI models on assets from online marketplaces like TurboSquid removes that obstacle.

Here, Shutterstock and TurboSquid’s stance on generative AI differs from that of other online marketplaces, with Sketchfab having explicitly ruled out the use of its users’ content to train AI models.

Orgeron told CG Channel that Shutterstock’s “biggest goal [is] to enable creators” by creating a service that artists can themselves use to generate base meshes to refine manually, commenting that demand for 3D content on commercial projects outstrips the rate at which it can be created from scratch.

Nvidia’s promo for Nvidia Picasso, showing content of the type it expects to be created through the new cloud-based platform for building generative AI tools, used for Shutterstock’s text-to-3D service.

Developed using the new Nvidia Picasso generative AI tools platform
Shutterstock’s new text-to-3D service is being developed using Picasso, Nvidia’s new cloud-based platform for building and deploying generative AI tools, also announced at GTC 2023.

Picasso is aimed at developers rather than end users, with services being rolled out via Nvidia’s partners.

As well as Shutterstock, Nvidia has partnered with Getty Images to develop new text-to-image and text-to-video models trained on its stock images, and has extended its existing partnership with Adobe.

Pricing and release dates
Shutterstock and TurboSquid’s new text-to-3D capabilities are expected to enter beta in “Q4 2023”. On Shutterstock, they will be available via the Creative Flow suite of apps. Subscriptions cost $12.99/month.

Read Shutterstock’s announcement of its upcoming AI-based online text-to-3D service

Read Shutterstock’s FAQs on the use of users’ content in AI training data sets

Read more about Nvidia Picasso on Nvidia’s blog