Nyghtfall's AI Experiments

[Nyghtfall3D]

Unless there's some technique I've yet to learn with A111, SD cannot handle multiple characters in one image. This is from my first Daz comic when the Reality plugin was all the rage. It's the only decent image I managed to get after about 20 attempts, including various text prompts to help it along.

It gave my male antagonist female eyes that won't look down, it merged my protagonist's right index and middle fingers, and her her tongue looks she drank acid. The text on the license plate on the background SUV is also warped.

Hi-Tech Target 3D


Hi-Tech Target AI


But it sure loves portraits. This my second attempt at Van Helsing AI, using a tweaked copy of the text prompts in your ComfyUI screenshot:

Van Helsing AI2

 

[fallenmystic]

Unless there's some technique I've yet to learn with A111, SD cannot handle multiple characters in one image. This is from my first Daz comic when the Reality plugin was all the rage. This was the only decent image I managed to get after about 20 attempts, including various text prompts to help it along.

It gave my male antagonist female eyes, it merged my protagonist's right index and middle fingers, and her her tongue looks she drank acid.

It's one of the trickier parts of using SD at the moment. There are, however, several options you can try to circumvent the issue:

• Just generate the overall scene first, then inpaint each character with a different prompt - it can work surprisingly well sometimes (like simple cases such as your example above).
• If you're using A1111, try extensions supporting "regional prompting". There are several of them, so you may need to experiment to find the one that suits your needs.
• If you're using ComfyUI, there are other options like rendering each character separately then combining them in a latent space, and so on.

I've been using such options to create my renders so far. While it gives more or less the desired result, it usually takes an unreasonable amount of effort and/or deterioration in the render quality for complex scenes.

So, I'm currently exploring more options I can use regarding this matter in ComfyUI.

 

[fallenmystic]

Unless there's some technique I've yet to learn with A111, SD cannot handle multiple characters in one image. This is from my first Daz comic when the Reality plugin was all the rage. It's the only decent image I managed to get after about 20 attempts, including various text prompts to help it along.

It gave my male antagonist female eyes that won't look down, it merged my protagonist's right index and middle fingers, and her her tongue looks she drank acid. The text on the license plate on the background SUV is also warped.

Hi-Tech Target 3D
View attachment 1373822

Compared with simple portraits, converting such a render to a photo-like image requires non-trivial effort. As such, I cannot spend sufficient time on it to make it look good. However, I made a quick setup to see how some of the methods I mentioned earlier work in ComfyUI in practice:


(I adjusted the colours using an external tool to make it look like an old horror/thriller film.)

Many details are missing or wrong, which will require at least a couple more hours for me to fix. One difference between making a 3D render and generating an AI image is how the latter often involves an iterative approach. Whereas there's not much to do once you finish a render in Daz3D or Blender, you can use the output of a stage in an AI render pipeline as the input of the next one, so you can incrementally build upon the initial image.

The above render was made that way, stacking 3-4 sampling stages. each focuses on different aspects (e.g., the background, the girl, the man, etc.)

With A1111, you can't automate such a pipeline, but you can still apply the same principle by hand. And that's why it has buttons like "Send to Img2Img".

Hope this could give you some ideas for constructing complex scenes.

P.S.: Ah... I remember those days with Reality. Back then, I didn't know what PBR was, but it didn't stop me from getting awestruck at what the plugin made possible. I don't remember exactly when it was released, but it was the time when I most actively used Daz3D.

I don't have any of the renders I made back then, but I still have good memories of the period. After all, that was how I started the whole making kinky renders business

 

[Nyghtfall3D]

However, I made a quick setup to see how some of the methods I mentioned earlier work in ComfyUI in practice:

My... GOD...

Glaring issues aside, the overall result is just... incredible... It looks exactly like a still shot from a live-action film or TV show.

The above render was made that way, stacking 3-4 sampling stages. each focuses on different aspects (e.g., the background, the girl, the man, etc.)

Haven't learned about sampling stages yet so that's all Greek to me at the moment but I think I get the idea. They're like layers?

Hope this could give you some ideas for constructing complex scenes.

Knowing it's possible definitely gives me hope for what I might be able to accomplish as I learn more.

What concerns me most right now is the "slot machine" nature of the tech and the inability to control what SD produces, even with pre-rendered scenes. I can have it generate 30 different images and it'll produce 30 slightly different variations from the same image and prompts. I understand that I can select a resulting image and make it the new root to refine the scene even more, but is there a way to retain small details while training the AI on what exactly I'm trying to improve?

:: thinks a moment ::

Is that where masking comes in?

 

[fallenmystic]

Haven't learned about sampling stages yet so that's all Greek to me at the moment but I think I get the idea. They're like layers?

It's not an established jargon but just the way I refer to a set of steps related to a single sampling ("rendering") process. For example, this was the "stage" in my setup that refined the attacker whose image got terribly mangled in the previous stage:


(Please ignore the chaotic wiring and unreasonably high sample count, by the way.)

I rearranged the nodes so that you can see how I fed the output of one sampler to another, changing the prompt, ControlNet parameters, and the region to which I want to apply the change.

But you don't have to get concerned about such technicalities too much since that's just what A1111 does under the hood when you inpaint. Just remember that it can be an iterative process in which you can refine different regions/aspects one at a time.

What concerns me most right now is the "slot machine" nature of the tech and the inability to control what SD produces, even with pre-rendered scenes. I can have it generate 30 different images and it'll produce 30 slightly different variations from the same image and prompts. I understand that I can select a resulting image and make it the new root to refine the scene even more, but is there a way to retain small details while training the AI on what exactly I'm trying to improve?

:: thinks a moment ::

Is that where masking comes in?

To control what SD produces in a way you want, you'll need to learn how to use ControlNet and inpaint feature at least. Those two are bread and butter of composing a complex scene with SD.

You can see inpainting as "photo manipulation on steroid". From that perspective, the "image slot machine" nature of SD can be a blessing since it means you have an unlimited supply of source images from which you can construct something more complex and unique.

It's not unlike how almost no Daz3D user creates one's own hairstyle or clothing props. They simply purchase them off the shelf but it doesn't mean they can't arrange and morph them in a creative process that may result in something original. It's even better with AI since you can't change the style of buttons in clothing or add a bang to hair with an existing prop in Daz3D unless the vendor included those options.

And yes, it's where masking comes in You can, for example, put a mask over the license plate of the car in your render to preserve the text, or conversely change the masked area, as I did in my screenshot above.

There are many options and ways the inpainting tool provides. Since it's the main tool along with ControlNet you'll be using to manipulate the render to match how you imagine your scene, I suggest investing sufficient time to master what it offers before moving to more advanced subjects.

 

[Nyghtfall3D]

From that perspective, the "image slot machine" nature of SD can be a blessing since it means you have an unlimited supply of source images from which you can construct something more complex and unique.

It also means Daz artists are unable to share parameter settings with each other if they want to use the same characters in their projects, because the AI never stops producing variations in their appearance.

For example, I could use the same modeler and ComfyUI settings you used on my Hi-Tech Target render and never recreate the same characters or their outfits depicted in your version of the scene. The parking garage SD created looks fantastic but I could never recreate it, or use it in another project. I really like how my second Van Helsing AI image came out, but I will never be able to re-use that look for another project.

Being able to share things like product info and skin settings is one of the cornerstones of the 3D art community that helps promote camaraderie and inspires artists to learn from each other. As impressive as the examples you and I have posted in this thread are, such conversations and learning opportunities become moot when AI is added to one's workflow because the objects and characters produced from 3D renders don't exist and can't be re-used.

Someone on VoD once asked what character I used for our logo because they wanted to try her out. I was able to point them to the character model on Daz's website, and tell them what makeup settings I used. I wouldn't have been able to do that if I had use an AI-enhanced character, and they wouldn't have been able to recreate the AI character even if I shared my A111 settings.

 

[fallenmystic]

It also means Daz artists are unable to share parameter settings with each other if they want to use the same characters in their projects, because the AI never stops producing variations in their appearance.

For example, I could use the same modeler and ComfyUI settings you used on my Hi-Tech Target render and never recreate the same characters or their outfits depicted in your version of the scene. The parking garage SD created looks fantastic but I could never recreate it. Nor would I be able to use what SD produced in another project. I really like how my second Van Helsing AI image came out, but I will never be able to re-use that look for another project.

You can, actually That's exactly what I said about seeds and how SD is largely deterministic in my previous post. Same input (parameters, control map images, etc.) + same seeds = same render.

In other words, you can import my workflow and recreate the exact same render I made provided you have the same models/nodes I used on your installation.

ComfyUI is especially good for educational purposes. Not only does it allow you to see what SD does under the hood, but you can also follow what other people did to create a specific render on a step-by-step basis and experiment with parameters.

It's one of the benefits of having a node-based workflow and deterministic process, plus the ability to import/export the whole node tree to a file.

 

[fallenmystic]

Being able to share things like product info and skin settings is one of the cornerstones of the 3D art community that helps promote camaraderie and inspires artists to learn from each other.

That's why most SD frontends embed render settings to the output so that other people can run from them. See this random image posted on Civitai, for example:



Or how people share their entire workflow using ComfyUI, like this example.

 

[Nyghtfall3D]

In other words, you can import my workflow and recreate the exact same render I made provided you have the same models/nodes I used on your installation.

Ok, that's great, if I want to re-create the same image. But I wouldn't be able to use that variation of the characters, hair, clothes, setting, or objects depicted in that image for other projects.

 

[fallenmystic]

Ok, that's great, if I want to re-create the same image. But I wouldn't be able to use that variation of the characters, hair, clothes, setting, or objects depicted in that image for other projects.

If you mean having a consistent character with the same facial features and clothes, then yes, it's something that can be done more easily by using a 3D design tool.

But it's not impossible to do with AI, either. You can always train an embedding or Lora for that, or just use a 3D character as a basis for your AI render like I often do.

Training a character Lora isn't that difficult, by the way. And that's why Civitai is flooded with all those celebrity models.

 

[Nyghtfall3D]

If you mean having a consistent character with the same facial features and clothes, then yes, it's something that can be done more easily by using a 3D design tool.

Yes, that's exactly what I mean. I wouldn't be able to take an AI character produced by SD and use it in other Daz projects.

And, even if I use the AI-enhanced character as a reference to create a 3D model from, SD will still produce its own, new interpretation of my AI-inspired 3D model when I run a 3D project through it, thus resulting in two AI-enhanced images from two separate projects featuring a character that looks nothing like itself between them.

 

[fallenmystic]

Yes, that's exactly what I mean. I wouldn't be able to take an AI character produced by SD and use it in other Daz projects.

And, even if I use the AI character as a reference to create a 3D model from, SD will still produce its own, new interpretation of my AI-inspired 3D model when I run a 3D render through it, thus resulting in two separate rendered scenes featuring a character that looks nothing like itself between them.

While it's true that character consistency is not a strong suit of AI, people reuse props or characters all the time in their AI renders (and that's why we have a common repository of sharable/reusable models like Civitai, which is also completely free unlike Daz3D Store, Renderocity, or Renderotica.)

Also, you can always control how much of the original 3D character AI is allowed to change by tweaking parameters like "denoising strength" or ControlNet weights. In a time when people use AI to generate videos, which requires frame-by-frame consistency, it's far from being something you can only achieve with a traditional 3D tool.

 

[Nyghtfall3D]

While it's true that character consistency is not a strong suit of AI

And, it seems to me, its Achilles heel where the needs of 3D artists - namely Daz and Poser users - are concerned.

people reuse props or characters all the time in their AI renders

But apps like Daz and Poser are no longer part of the workflow at that point.

 

[fallenmystic]

And, it seems to me, its Achilles heel where the needs of 3D artists - namely Daz and Poser users - are concerned. But apps like Daz and Poser are no longer part of the workflow at that point.

Things like SD, Daz3D, or Blender are tools, i.e. means to create art. Each has its own strengths and weaknesses, so what may be the best among them can be different depending on the task at hand and the intention of the user.

And many of those relative strengths and weaknesses are just trade-offs. I highly doubt most Daz3D users would choose better consistency of their characters over the possibility of making them look photorealistic. But if you value consistency above other things, there's nothing wrong with choosing Daz3D instead of SD.

It's not unlike many who still prefer Daz3D, for example, when Blender offers undeniably better capabilities as a 3D design tool. You're simply sacrificing the possibility of creating your original character to save your precious time, which can be better spent elsewhere, like in finetuning your scene composition, for example. It's just a trade-off, and nothing is inherently good or bad for every purpose.

So, in the end, it remains to be your choice. If you want to boost the realism of your renders to a new level, you may incorporate AI into your workflow to achieve that. On the other hand, if you prefer how you can lay out stuff in a 3D virtual space and get highly consistent results every time you hit the render button, it's better to stick with something like Daz3D.

The only thing that I highly oppose is such claims like how AI cannot be a legitimate tool for art, or how there's no human creativity involved in such a process, and so on.

Otherwise, it's just a matter of personal preferences. We have great artists who still draw by hand when anyone can take digital pictures. There's nothing wrong with it unless they start arguing things like how photographers are not "real artists" because they don't have hand skills, and so on.

 

[Nyghtfall3D]

The only thing that I highly oppose is such claims like how AI cannot be a legitimate tool for art, or how there's no human creativity involved in such a process, and so on.

Rest assured, I for one finally understand where you've been coming from. I completely agree, it is a fascinating tool for creating art, but I'm not willing to trade consistency for photo-realism, no matter how absolutely stunning the results.

 

[Nyghtfall3D]

I re-wrote my ToS to more accurately describe AI art as a tool but decided to keep it banned for the reason I discovered during my experiments.

And then my wife decided to weigh in after I shared the news with her (she was quite put off). She's writing her first novel (75,000 words and counting) and using Microsoft's AI art generator to visualize her characters as a sort of personal outline. She thinks I'm stifling my members' creativity by keeping them from exploring an AI-assisted workflow and sharing their experiments.

A happy wife means a happy life, so, per her suggestion, I've started a 3-month trial run for a new gallery for AI-enhanced member artwork. The rules are in the Guest Lounge for non-members interested in learning more. There will be a week-long vote afterward to determine whether the new gallery was a success and becomes permanent.

I intend to participate, so my I'm resuming my own studies and will continue sharing my experiments here as well. Since CF is open to the public, I'll be limiting anymore content I share here to SFW levels.

On that note, ControlNet is... quite impressive. I was getting all kinds of gibberish results from a bathroom strangle until I installed it. Definitely need to learn inpainting though.

 

[Nyghtfall3D]

OMFG... AI could be a game changer for postwork and artists who want to retain their artistic style in their work.

I can't draw worth shit, which is why I don't use apps like PaintShop Pro to draw ligatures. I have to MacGuyver 3D content around necks in Daz, which is a time-consuming process.

I just learned how to use AI to replace the ligature I created for this render with a thin white rope using Inpainting and the text prompt, "thin white rope". I didn't touch anything else (though I blacked out her breasts to post it here).

Strangled Socialite 3D



Strangled Socialite AI



EDIT: Posted too soon. Forgot I can't replicate the new ligature for future projects, or subsequent frames of a comic.

 

[Stungiskhan]

You can take a step further if you intend to make a photorealistic version of a Daz3D render. Here's what I got with the source image you posted. I used two different models with the same settings to see how they compare:

View attachment 1373778 View attachment 1373777

(Sorry, I didn't ask for permission to use your render. I won't claim those images as my own or use them anywhere else.)

And this is the workflow I used for the render. I attached the JSON source too, so you can load it in ComfyUI in case you want to try it yourself:

View attachment 1373776


Personally, I don't recommend using a mask/inpaint when converting a 3D render to a photo-like image. The reason is that AI tries to make the overall image consistent. So if you leave out an area of the original it will try to follow the style of it, which is that of a 3D render, in this case.

As such, it's better to just take the key features from the original using ControlNet and generate the whole image from scratch, like what I did above.


This is normally the right approach. But if all you want to achieve is converting a single comic strip, it's often enough to just render using ControlNets and the same seed. It's better to train a custom Lora or embedding, in case you're going to generate a lot of similar content, however.


I envy you have a powerful enough machine to render it in 2K But it's usually better to render in a smaller resolution and upscale it to the size you want. There are many different options you can use for upscaling.

Anyway, it's a nice start. Looks like you're enjoying yourself already.


P.S.: For those you feel interested in the technical details of my workflow above: I didn't invest too much time in that render and I suspect my use of recolouring ControlNet did more harm than good. Maybe it'd yield a better result, if you remove it and feed the latent image created from the original and render it with high denoising strength.

Also, I made a mistake of swapping the values in the “Latent from Batch” node. Better to change it if you want to try the setup.

Is it possible to preserve eyes? They are getting worse.

 

[poem21045]

Is it possible to preserve eyes? They are getting worse.

Also, see your optomologist.
(Or if your eyes are so bad you can’t see him, just feel around his office until you find him.)

 

[fallenmystic]

Is it possible to preserve eyes? They are getting worse.

It's possible, but it usually gives a better result to do a second pass for the eyes. Or you can just render at a higher resolution to minimise such a problem, in case you have sufficient VRAM.