Discover and read the best of Twitter Threads about #img2img
Most recents (12)
I wanted to imagine how we’d better use #stablediffusion for video content / AR.
A major obstacle, why most videos are so flickery, is lack of temporal & viewing angle consistency, so I experimented with an approach to fix this
See 🧵 for process & examples
A major obstacle, why most videos are so flickery, is lack of temporal & viewing angle consistency, so I experimented with an approach to fix this
See 🧵 for process & examples
Ideally you want to learn a single representation of an object across time or different viewing directions to perform a *single* #img2img generation on.
For this I used layered-neural-atlases.github.io (2021)
For this I used layered-neural-atlases.github.io (2021)
This learns an "atlas" to represent an object and its background across the video.
Regularization losses during training help preserve the original shape, with a result that resembles a usable slightly "unwrapped" version of the object
Regularization losses during training help preserve the original shape, with a result that resembles a usable slightly "unwrapped" version of the object
Thread time 🧵
Here's how to precisely design a small building in a game (such as an isometric bunker) by fine-tuning #StableDiffusion
This example was inspired by #RedAlert, which I spent countless hours on (in 96-97 - pls don't call me old 😅)
CC scenario.gg
Here's how to precisely design a small building in a game (such as an isometric bunker) by fine-tuning #StableDiffusion
This example was inspired by #RedAlert, which I spent countless hours on (in 96-97 - pls don't call me old 😅)
CC scenario.gg
Style-consistency is paramount when it comes to designing #game assets.
I trained a fine-tune using @Scenario_gg (alpha), using 16 images (below), inspired by the Red Alert/Command & Conquer buildings.
Fun fact: I generated them all in... @midjourney.
I trained a fine-tune using @Scenario_gg (alpha), using 16 images (below), inspired by the Red Alert/Command & Conquer buildings.
Fun fact: I generated them all in... @midjourney.
I actually generated more than 200 buildings (and even some vehicles), from which I picked a smaller dataset, keeping enough variability within a certain consistent style.
I will keep the remaining data for new trainings in the coming days :)
I will keep the remaining data for new trainings in the coming days :)
I've explored a reliable method to create high-quality, style-consistent #game assets w. #StableDiffusion
We're just scratching the surface here & I believe this can become a very potent creation tool (like a "@procreate on AI-steroids"🤔)
A demonstration with... spellbooks 🧵
We're just scratching the surface here & I believe this can become a very potent creation tool (like a "@procreate on AI-steroids"🤔)
A demonstration with... spellbooks 🧵
Spellbooks are another ubiquitous asset in heroic #fantasy games, tabletop games, D&D, etc.
However (like for potions), the Unity/Unreal asset stores have limited choice, and most assets have a rather simplistic design.
Here's how #AI can solve such a situation.
However (like for potions), the Unity/Unreal asset stores have limited choice, and most assets have a rather simplistic design.
Here's how #AI can solve such a situation.
At first, I generated a set of 38 "random" spellbooks, which I used to train a finetune model of Stable Diffusion (Dreambooth). It took 1 hr.
I then explored a set of possibilities with simple prompts and or limited modifiers (intricate, detailed, beautiful, 3D render...)
I then explored a set of possibilities with simple prompts and or limited modifiers (intricate, detailed, beautiful, 3D render...)
I designed the most incredible pack of potions with #AI (and nothing but AI - #StableDiffusion)
Mega-thread 🧵
Follow the exploration below, esp. if you're in the #gaming industry (Game dev, Game Artist, Creative Director, etc.) Content production is about to be transformed 🤯
Mega-thread 🧵
Follow the exploration below, esp. if you're in the #gaming industry (Game dev, Game Artist, Creative Director, etc.) Content production is about to be transformed 🤯
The gaming industry always needs a lot of new content. 40% of their budget is in the art assets.
Take potions, a ubiquitous prop in RPG games.
Below are some of the “most popular” potion packs available on the Unity asset store (similar sets are on the Unreal marketplace).
Take potions, a ubiquitous prop in RPG games.
Below are some of the “most popular” potion packs available on the Unity asset store (similar sets are on the Unreal marketplace).
And these are some of the potions found on Artstation.
They look much more elaborate, and some designs are highly creative. However, there are only 740 results for the “potion” query, which seems limited.
They look much more elaborate, and some designs are highly creative. However, there are only 740 results for the “potion” query, which seems limited.
Sharing additional thoughts on #StableDiffusion to create #gaming content & #game assets 🧵
My last exploration was on golems; this one is with "Space Marines-like" heavy infantry.
This was a fun creation with lots of learnings 👨💻. Feel free to like/RT if you find it useful 🚀
My last exploration was on golems; this one is with "Space Marines-like" heavy infantry.
This was a fun creation with lots of learnings 👨💻. Feel free to like/RT if you find it useful 🚀
1/ First (as mentioned before - ) > curate a training dataset (such as pictures of figurines) to feed a #Dreambooth finetune.
Once the model is ready, compare different generic prompts (e.g. "low-poly," "3D rendering“, "ultra-detailed," "pixel art," etc.)
Once the model is ready, compare different generic prompts (e.g. "low-poly," "3D rendering“, "ultra-detailed," "pixel art," etc.)
2/ Once a prompt looks good, keep the modifiers (in this case: "3D rendering, highly detailed, trending on Artstation").
And start iterating around variations, such as colors (or pose). Don't over-engineer it, to keep the consistency. You should get this:
And start iterating around variations, such as colors (or pose). Don't over-engineer it, to keep the consistency. You should get this:
Let’s keep using #StableDiffusion to design or create #game assets, quickly and consistently.
In 15 min, generated an army of golems with similar shapes and sizes, but various “materials” (lava, rock, water, ice, forest, sand, gold…).
No in-painting is needed. Thread 🧵
In 15 min, generated an army of golems with similar shapes and sizes, but various “materials” (lava, rock, water, ice, forest, sand, gold…).
No in-painting is needed. Thread 🧵
Step 1 (as in my previous examples with chests - ), I made a Dreambooth finetune w. just 11 pics.
As soon as the model was trained, the first step was randomly generating a large set of images w. a simple prompt ("golem, detailed, realistic, 3D rendering")
As soon as the model was trained, the first step was randomly generating a large set of images w. a simple prompt ("golem, detailed, realistic, 3D rendering")
Each golem can be extracted by removing the images' backgrounds (reco: @photoroom_app).
Some of the designs are amazing. However, the golems all look very similar to each other. Let's separate them into categories.
Some of the designs are amazing. However, the golems all look very similar to each other. Let's separate them into categories.
People loved the "treasure chests" #AI experiment I posted yesterday! However, the chests were all closed 🤨.
So I quickly ran a second training on a different dataset, and this time, gold and diamonds are overflowing from the open chests! 🪙🤑💰
So I quickly ran a second training on a different dataset, and this time, gold and diamonds are overflowing from the open chests! 🪙🤑💰
I also used #img2img to instantly generate dozens of variants, "inspired“ by a single original photograph.
This provides consistent assets (similar shape, size, or materials) with some slight variations. It's up to the artist/user to select which one looks best.
This provides consistent assets (similar shape, size, or materials) with some slight variations. It's up to the artist/user to select which one looks best.
Same thing here, using #img2img - however, I prompted "steel chest" instead of "wooden chest"
While it's not 100% perfect, there's still more steel in these chests than in the previous ones.
Also, some of the assets are disjoint or show anomalies. Some fine-tuning is necessary
While it's not 100% perfect, there's still more steel in these chests than in the previous ones.
Also, some of the assets are disjoint or show anomalies. Some fine-tuning is necessary
#Dreambooth will change how 2D game assets are being designed or generated.
Today's exploration: treasure chests 💎💰.
I trained Stable Diffusion with some low-poly treasure chests (23 images). 1h later, here's what's possible (🧵):
Today's exploration: treasure chests 💎💰.
I trained Stable Diffusion with some low-poly treasure chests (23 images). 1h later, here's what's possible (🧵):
At first, I randomly generated a set of 64 items, without any specific prompt engineering ("a treasure chest").
Here's a close-up view:
Tonight's dreamboothing is about grenades 💣(!).
I made a #dreambooth finetune from a few Sci-Fi grenade illustrations.
Explosive thread 🧵👇
#AI #StableDiffusion
I made a #dreambooth finetune from a few Sci-Fi grenade illustrations.
Explosive thread 🧵👇
#AI #StableDiffusion
First, some random grenades, no specific adjectives.
But they look great in blue too!
1/ I created this with Stable Diffusion using image inpainting and “walking through the latent space”
Without using tweening, every frame is generated by an interpolated embedding and variable denoising strength, so keeping continuity was tricky
See 🧵for process
Without using tweening, every frame is generated by an interpolated embedding and variable denoising strength, so keeping continuity was tricky
See 🧵for process
2/ First off, finding the right combination of prompt, seed and denoising strength for an #img2img in-painting is a roll of the dice
Luckily it is easy to script large batches to cherrypick
Luckily it is easy to script large batches to cherrypick
3/ The first and last pairs were just regular #img2img ramped through a range of denoising strength of 0 to 0.8
Took a face made in #stablediffusion driven by a video of a #metahuman in #UnrealEngine5 and animated it using Thin-Plate Spline Motion Model & GFPGAN for face fix/upscale. Breakdown follows:1/9 #aiart #ai #aiArtist #MachineLearning #deeplearning #aiartcommunity #aivideo #aifilm
First, here's the original video in #UE5. The base model was actually from @daz3d #daz and I used Unreal's #meshtometahuman tool to make a #metahuman version. 2/9 #aiartprocess
Then I took a single still frame from that video and ran it through #img2img in a local instance of #stablediffusion WebUI. After generating a few options I ended up with this image. 3/9 #aiartprocess
イラスト制作におけるStable Diffusionの活用事例
まず二枚目みたいなざっくりしたイメージを書き起こして、AIに読み込ませる
すると三枚目が出力されます。これに加筆をしたものが四枚目です
これを更に加筆して出力…という工程を繰り返して一枚目のイラストができました
#stablediffusion #img2img
まず二枚目みたいなざっくりしたイメージを書き起こして、AIに読み込ませる
すると三枚目が出力されます。これに加筆をしたものが四枚目です
これを更に加筆して出力…という工程を繰り返して一枚目のイラストができました
#stablediffusion #img2img
ちょっと補足
最初にAIに読み込ませるイメージはラフなほうが良い出力結果が出やすいと感じました
シルエットと大まかなライティング、色合いくらいまで描き込んでStrength 0.6で出力すると、良い感じにニュアンスを汲み取ってくれる
最初にAIに読み込ませるイメージはラフなほうが良い出力結果が出やすいと感じました
シルエットと大まかなライティング、色合いくらいまで描き込んでStrength 0.6で出力すると、良い感じにニュアンスを汲み取ってくれる
参考として一枚目がシルエットと色分けのみ、二枚目がライティングまで指定した場合の生成画像です
ともに同じプロンプト、同じStrengthで生成しています
ともに同じプロンプト、同じStrengthで生成しています
