feat: multimodal/external tool capabilities

[hpnyaggerman]

Namely, image gen, web search, MCP related things.

[OrbFrontend]

Maybe I can have some sort of plugins manager for these things. But I'm not sure how well giving the agent a bunch of random tools and expecting it to know what to do will work, even if you hand-hold and tell it when to use what.

Currently I never send tool_choice=auto in any of the passes because it would always fail, instead I always force the exact tool call.

[hpnyaggerman]

I am thinking this could be implemented by adding optional agent passes for each function which are called conditionally by the director or manually by the user. This sounds like a sensible way to do it, though given the many possible variations of such agents and their expected outputs, it might make sense to put them in a separate menu.

Since I was interested in working on #19, I would be adding a rather basic implementation of the feature of this issue.

[hpnyaggerman]

Can now be implemented as a workflow since #50 was merged.

[hpnyaggerman]

@OrbFrontend Any plans of your own on this? I might take this if you have none. Though it might take a while so if you were planning to do it yourself, you will probably be faster here, and I might send some improvements afterwards.

Image gen specifically, I mean.

[OrbFrontend]

I'm still not sure how to design a good image gen workflow. I have few approaches and concerns in mind:

1. Implement it as an MCP server. ComfyUI or Forge Neo already have MCP support, the model can discover the pipeline this way.
2. Just call backend-specific headless endpoints like /prompt
3. We probably don't want to hand over the entire pipeline to the LLM so the scope must be immaculate.
4. Should this be one-shot or a ReAct feedback loop for vision-capable models to improve the gens?
5. Cloud models like Grok API also support image gen capabilities, need to look into it as well.

Tbh I'm an image-gen novice, and have never tried driving headless image gen backends or APIs so I'm not experienced in this.

@hpnyaggerman Wdyt? You could tell me the general ideas. I can get Opus to refine and probably make a draft first.

[hpnyaggerman]

@OrbFrontend My idea was to have a pipeline which would have two LLM nodes with a number of user-side injections. Specifically, the first one would exist purely to establish the facts of the current scene in concrete terms. It would pipe the facts to the second LLM which would have history and this concrete description of the scene, and this LLM would be instructed to generate a positive prompt and a negative prompt for the image gen backend (LLM is expected to output them through tool calling). The second LLM can also be nudged in the right direction by giving it a number of items, including a compact image gen prompting guide, negative prompts it should include, etc, as needed, all user-configured with sane defaults. The request can then be sent to whatever provider this workflow would choose to implement support for. I think it would make the most sense to start with ComfyUI and allow users to use their own workflows, with a sane default.

A very rough sketch but this is how I see it.

It might be worth it to ask the anons of lmg to come and take a look and give their ideas now that such things can be implemented through workflows.

[OrbFrontend]

@hpnyaggerman Do you mean that LLM nodes will be ComfyUI-specific nodes? This is actually getting too advanced for me, should probably get some feedback first, people might not understand what the secondary workflows are for though...

[hpnyaggerman]

@OrbFrontend I meant that the two LLM nodes make the prompt for the current scene. That is, they derive from history and user-provided instruction the prompt (negative and positive) for the image gen backend. This setup is not specific to ComfyUI, really. but ComfyUI is a good universal solution for this task as it can integrate with many providers and users can configure their own pipelines to their liking, to the point where the only thing they would need to get the final image is the prompt, generating which is the purpose of the two LLM nodes.

But yes. Getting some more people involved would be good. I am not a pro in image gen by any stretch.

[OrbFrontend]

Didn't get much from the thread regarding image gen... There was a suggestion on using the adapter pattern, which is what we already have in the original tts impl anyway. Another one was to expose some of the params in the UI. I guess we can expose the generic ones like sampler, CFG, and steps (though in my experience it's normal for a ComfyUI workflow to have several nodes with their own hyperparams, some extremely elaborate, so I'm still unsure how to approach this).

Your two LLM nodes/steps idea is okay, the user may influence the second one...

I'll branch out and start exploring what's up with ComfyUI MCP server first. Need more experience on what's happening in image gen land first.

[hpnyaggerman]

@OrbFrontend I would appreciate if you didn't rush to commit and would keep it as a PR for a bit so I can try my hand at it as well. I believe I have a somewhat sound architectural solution to the problem.

[OrbFrontend]

@hpnyaggerman No problem, you can go ahead with the image gen. It's not exactly a priority on my list though I'm happy to contribute. I have other small refinements for the primary workflow (3 passes) anyway, shouldn't impact this secondary workflow.