Hello all! Here is a write-up about my R&D with AI for this project. I think it looks very promising indeed, but there are obvious limitations and workflow requirements. Forgive me if you all are already familiar with this stuff… just sharing my journey of discovery!

Workflow 1 - 3D model to AI image generation

STP Model:

Render from C4D:

Runway image generation based on previous image as an input source:

Runway video generation based on previous image as an input source:

tent_gen4.mp4

I told the prompt to fly around the tent but it didn’t do it.

Image source from brochure:

Runway video generation based on previous image as an input source:

Gen-4 camera dollies forward through the scene, looking through the windshield of a humvee 573367834.mp4

In the prompt I told it to dolly forward, looking through a humvee window.

Second generation with the same input. I prompted it to do a crane move up into the sky, then zoom in on the tent on the left. It didn’t really do it, but it did decided to make a cut to a closeup:

desert_gen4_V2.mp4

Another source image from client:

Runway video generation

prompted with:

camera rotates around the military tent as the scenery changes from green grass and trees, to tropical forest with heavy rainfall, to arid desert during a sandstorm, to snowy tundra during a blizzard, and finally back to an open grassy field during a sunset

So you don’t always get what you want. But I can see the potential here.

seasons_gen4.mp4

this one was without a source. I just asked for a jungle scene

Asia Pacific tropical forest environment with a clearing in the middle of the frame

Runway video generation

Note that I asked for a static shot, but it put a push forward in it anyway.

A static shot showing rain falling on the leaves in the forest as the sun shines through the canopy..mp4

After Effects composite

super duper rough proof of concept but with the right camera track you could add a cinema 4D render of just the tent and make it fit into the environment:

A static shot showing rain falling on the leaves in the forest as the sun shines through the canopy_AI_tests_001.mp4

Runway video generation

for this prompt I used the previous video as a source, but told it to generate a new camera angle looking through the trees, in cinematic photo-real style:

new_angle_gen4.mp4

second generation with the same video input, told to look top-town at the tent next to a humvee and people moving supplies:

topdown_gen4.mp4

With all of these you can see issues… mostly with people looking funny which makes sense. But some of it is passable, and some is damn impressive. And I know it can do better than what we see in these tests.

It is as they say… garbage in garbage out. The better I can prep my input images and videos the better result you can get, and conversely if the source input isn’t great it’s going to be a challenge. It seems that it doesn’t love using AI-generated imagery as the input for another AI video. Or rather… that because the AI generated imagery has weird non-realistc artifacts, that in turn messes with the video AI’s ability to interpret the scene correctly.

But this absolutely changes the way I can work on a project like this. This is very exciting and a little overwhelming.

a few more:

tundra..mp4

sandy wind.mp4

gravel and dirt road.mp4

xray.mp4

make the tent photo realistic.mp4