fix(core): Infer tool `invoke` return type from provided function #8056

hntrl · 2025-04-23T18:12:37Z

Fixes #8010. Instead of just using an any alias as the return type to invoke, this adds a generic ToolOutputT type param on the tool classes that decides the return type of invoke.

A later enhancement could be to restrict the return type of the provided function if "content_and_artifact" is provided as the response format (i.e. instead of any[], the return type is forced to be extends [any, any] in the tool function)

cc @benjamincburns

vercel · 2025-04-23T18:12:41Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-docs	✅ Ready (Inspect)	Visit Preview		Apr 29, 2025 3:09am

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-api-refs	⬜️ Ignored (Inspect)			Apr 29, 2025 3:09am

benjamincburns · 2025-04-24T00:02:20Z

Hi @hntrl - looks like you might've forgotten to do a top-level yarn build? The exports tests appear to be failing due to type errors in the provider libs.

benjamincburns · 2025-04-24T00:09:44Z

Hi @hntrl - looks like you might've forgotten to do a top-level yarn build? The exports tests appear to be failing due to type errors in the provider libs.

Ah quick correction. I think the exports tests are actually just smoke testing bundled versions of the project, and they tend to catch breakages. I suspect this may be a case where the code in question is making invalid assumptions about the return type of StructuredTool, so it's maybe a false positive, but you'd need to evaluate.

Apologies for the confusion on my part - still learning the finer details about our build/test pipeline process.

hntrl · 2025-04-24T01:17:37Z

libs/langchain-aws/src/tests/chat_models.int.test.ts

-      content: await tools[0].invoke(result.tool_calls![0]),
-    })
-  );
+  messages.push(await tools[0].invoke(result.tool_calls![0]));


StructuredTool.invoke will return a tool message when a tool call is passed, so we don't have to create a new tool message here

hntrl · 2025-04-24T01:32:23Z

Hi @hntrl - looks like you might've forgotten to do a top-level yarn build? The exports tests appear to be failing due to type errors in the provider libs.

Ah quick correction. I think the exports tests are actually just smoke testing bundled versions of the project, and they tend to catch breakages. I suspect this may be a case where the code in question is making invalid assumptions about the return type of StructuredTool, so it's maybe a false positive, but you'd need to evaluate.

Apologies for the confusion on my part - still learning the finer details about our build/test pipeline process.

No worries and thanks for the callout! I think you're right in that the failing test was a false positive. The most recent commits I pushed fix the CI failures.

a0da856 fixes the failing test from the latest CI run. I added comments explaining the change.

3cbaceb fixes some other build failures I found that come from changing the interface type specifically. From what I can gather, changing that type has a (maybe unnecessary?) rippling effect through a bunch of provider packages, so I'll avoid trying to tackle that for now. At least this way the tool method works as expected when imported from the core package directly.

benjamincburns · 2025-04-24T01:39:44Z

langchain-core/src/tools/types.ts

-  ): Promise<TArg extends ToolCall ? ToolMessage : ToolOutputT>;
+  ): Promise<ToolOutputT>;


Still assessing, but I'm not sure that this change is valid.

It's a bit confusing because we don't have StructuredTool actually implementing StructuredToolInterface (I forget why that is), but the latter is meant to reflect the public interface of the former. Same goes for ToolInterface and Tool.

Additionally, extensions of StructuredTool should always be assignable to StructuredToolInterface, and extensions of Tool and DynamicTool should be assignable to ToolInterface. We should probably have a test that guards this property if we can't make it so the aforementioned classes actually implement their corresponding interfaces 🤔.

I think additionally all of those should be assignable to StructuredToolInterface (lack of type params is intentional on this and all of the above - default params should be wide enough to allow this).

Makes sense. Two paths I can see going from here (both of which I'm happy to do, more just a matter of preference):
(1) Regress the tool classes to return just ToolOutputT instead of switching based on input (which doesn't really have any impact since it was just returning an any alias before, at least this way it's an optional type param on the classes + interfaces that we can use to control the output type)
(2) Bring the output type switching back to the interface and fix the side effects in the provider packages

Option 1 gets this PR merged quicker, option 2 is more "feature complete".

Unrelated but I'd be interested to know why that direct inheritance isn't there.

I think the diff makes it confusing but this change specifically is reverting a previous change I made. On main the return type is set to Promise<ToolReturnType>

Unrelated but I'd be interested to know why that inheritance isn't there.

We've been iterating in this area recently (see #7973), so the types are a bit more well-aligned between the classes and interfaces now than when they initially came together. Looking into these issues myself, I see one bug that I introduced in that PR, where I should've aligned the constraint for SchemaT on ToolInterface with the constraint for SchemaT on Tool. I think that's all that's necessary to make Tool implement ToolInterface and StructuredTool implement StructuredToolInterface.

Two paths I can see going from here (both of which I'm happy to do, more just a matter of preference):

(1) Regress the tool classes to return just ToolOutputT instead of switching based on input (which doesn't really have any impact since it was just returning an any alias before, at least this way it's an optional type param on the classes + interfaces that we can use to control the output type)

My concern here would be that we'd drop the ToolMessage type from the output, or am I mistaken? That would trigger cases where we return something that's not expressed by the return type. Unfortunately I think it'd be kinder to consumers to regress all the way back to any as it'd be better to have no "documentation" of types here than to be misleading.

(2) Bring the output type switching back to the interface and fix the side effects in the provider packages

I think this would be the preferred approach, with a bias toward better alignment of the type hierarchy and against using casts to fix the errors outside of @langchain/core (if possible). I can probably pick this up Monday if you don't have the time for it, though.

My concern here would be that we'd drop the ToolMessage type from the output, or am I mistaken? That would trigger cases where we return something that's not expressed by the return type.

Good catch. I guess I was thinking more-so on the lines of consuming from the interface end instead of the consumer end (i.e. not creating tools using the tool method) where the output type is always implicitly any since the default for the type param is ToolReturnType. Agree this would be a bad change.

I'll have some more time to work on this tomorrow. If you don't hear from me by Monday then feel free to work off this PR

hntrl · 2025-04-25T19:34:41Z

@benjamincburns Some updates on my findings:

It turns out the error I was trying to corner has to do more with how typescript computes type inheritance rather than integration problems. When I add the conditional return types back to the interfaces (284f3e5), the error that occurs downstream is with the prebuilt agents and how it passes tools around. I haven't been able to address this yet so I imagine CI is failing atm.

The crux of the error is DynamicStructuredTool<...> is not assignable to StructuredToolInterface.

If you try to add the interface inheritance directly on the class, the error comes from the call method where it's saying that the two arg types are incompatible. <TArg extends StructuredToolCallInput<...>>(...) is not assignable to <TArg extends unknown>(...)

It's almost like when evaluating if the dynamic tool conforms to the interface, TypeScript is computing the SchemaT param on the interface (which will always fall down to unknown bc of how the tool call input type is defined), assuming that's the reference type for the generic in the call method, and complaining that StructuredToolCallInput != unknown. If that's the case, this doesn't lend itself well to comparing generic methods like we're doing here to get the conditional return type.

This is just specifically for DynamicStructuredTool. I can add inheritance from StructuredToolInterface to Tool and DynamicTool without any problems.

benjamincburns · 2025-04-28T00:57:03Z

So I confess that I got a bit nerd sniped by this last week and worked up a fix separately. I branched off of an earlier version of this PR, so unfortunately our work has diverged. My fixes are in c8d08fe and 84cf083, with the latter commit having the more substantial adjustment.

The main problem that needed fixing was that I defined the default schema type on StructuredTool and StructuredToolInterface too narrowly when I added support for JSON Schema. Since StructuredTool is essentially the base type for all LangChain style tools, its default schema type should be the union of all possible type schemas. Prior to that change, in cases where the type system couldn't infer what the schema type may be (e.g. because the tool was passed in across some external boundary), we were incorrectly assuming that the tool would've been a zod schema.

My branch includes your initial submission, so I'd be happy to merge it as-is, but if you want to review and reconcile this work against those changes I'd be happy to wait for you to do that. I suspect that you may have made some improvements to the types on internal functions that I'm missing.

…chain-aigh-8010

hntrl · 2025-04-28T17:45:07Z

No shame in that! I know I'm guilty of that sometimes too. I missed the detail about the config object also determining the output type, so I appreciate you taking a crack at it.

I merged your branch here, added some changes for readabilities sake in how the return type is defined/used, and fixed the override for the invoke method on both the interface and the class.

hntrl added 2 commits April 23, 2025 10:45

fix(core): make tool outputs a generic type (langchain-ai#8010)

174c410

fix(core): return message type when using tool calls

35b9ae9

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. auto:bug Related to a bug, vulnerability, unexpected error with an existing feature labels Apr 23, 2025

vercel bot deployed to Preview – langchainjs-docs April 23, 2025 18:25 View deployment

benjamincburns added 2 commits April 24, 2025 12:38

chore(aws): fix test typing issue

a2f37b3

fix(core): fix bad schema type constraint in ToolInterface type params

c8d08fe

hntrl commented Apr 24, 2025

View reviewed changes

vercel bot deployed to Preview – langchainjs-docs April 24, 2025 01:28 View deployment

hntrl added 4 commits April 24, 2025 13:36

fix(core): make tool outputs a generic type (langchain-ai#8010)

98226d5

fix(core): return message type when using tool calls

58dfe09

fix(aws): remove nested tool message in bedrock test

b7df050

fix(core): revert tool interface signature

6947702

benjamincburns force-pushed the fix/gh-8010 branch from 3cbaceb to 6947702 Compare April 24, 2025 01:36

benjamincburns reviewed Apr 24, 2025

View reviewed changes

vercel bot deployed to Preview – langchainjs-docs April 24, 2025 01:48 View deployment

benjamincburns and others added 3 commits April 24, 2025 17:41

chore(community): fix stagehand toolkit typing issue

d09d6c9

fix(core): fix tool typing issues, add simple tool type tests

84cf083

feat(core): add conditional return types to tool interfaces

284f3e5

vercel bot deployed to Preview – langchainjs-docs April 25, 2025 19:21 View deployment

hntrl added 3 commits April 28, 2025 10:34

Merge remote-tracking branch 'upstream/ben/fix-gh-8010' into fix/lang…

91daaf9

…chain-aigh-8010

chore(core): hoist tool return types

46e640d

fix(core): use new return type for tool invoke method

dc9a0e4

dosubot bot removed the size:L This PR changes 100-499 lines, ignoring generated files. label Apr 28, 2025

dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Apr 28, 2025

vercel bot deployed to Preview – langchainjs-docs April 28, 2025 17:49 View deployment

Merge branch 'main' into fix/langchain-aigh-8010

52aee43

vercel bot deployed to Preview – langchainjs-docs April 28, 2025 21:10 View deployment

fix(core): tool test lints

9b41066

vercel bot deployed to Preview – langchainjs-docs April 29, 2025 03:09 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(core): Infer tool `invoke` return type from provided function #8056

fix(core): Infer tool `invoke` return type from provided function #8056

hntrl commented Apr 23, 2025 •

edited

Loading

vercel bot commented Apr 23, 2025 •

edited

Loading

benjamincburns commented Apr 24, 2025 •

edited

Loading

benjamincburns commented Apr 24, 2025

hntrl Apr 24, 2025

hntrl commented Apr 24, 2025

benjamincburns Apr 24, 2025 •

edited

Loading

benjamincburns Apr 24, 2025

hntrl Apr 24, 2025 •

edited

Loading

hntrl Apr 24, 2025

benjamincburns Apr 24, 2025 •

edited

Loading

hntrl Apr 24, 2025 •

edited

Loading

hntrl commented Apr 25, 2025

benjamincburns commented Apr 28, 2025 •

edited

Loading

hntrl commented Apr 28, 2025 •

edited

Loading

		): Promise<TArg extends ToolCall ? ToolMessage : ToolOutputT>;
		): Promise<ToolOutputT>;

fix(core): Infer tool invoke return type from provided function #8056

Are you sure you want to change the base?

fix(core): Infer tool invoke return type from provided function #8056

Conversation

hntrl commented Apr 23, 2025 • edited Loading

vercel bot commented Apr 23, 2025 • edited Loading

benjamincburns commented Apr 24, 2025 • edited Loading

benjamincburns commented Apr 24, 2025

hntrl Apr 24, 2025

Choose a reason for hiding this comment

hntrl commented Apr 24, 2025

benjamincburns Apr 24, 2025 • edited Loading

Choose a reason for hiding this comment

benjamincburns Apr 24, 2025

Choose a reason for hiding this comment

hntrl Apr 24, 2025 • edited Loading

Choose a reason for hiding this comment

hntrl Apr 24, 2025

Choose a reason for hiding this comment

benjamincburns Apr 24, 2025 • edited Loading

Choose a reason for hiding this comment

hntrl Apr 24, 2025 • edited Loading

Choose a reason for hiding this comment

hntrl commented Apr 25, 2025

benjamincburns commented Apr 28, 2025 • edited Loading

hntrl commented Apr 28, 2025 • edited Loading

fix(core): Infer tool `invoke` return type from provided function #8056

fix(core): Infer tool `invoke` return type from provided function #8056

hntrl commented Apr 23, 2025 •

edited

Loading

vercel bot commented Apr 23, 2025 •

edited

Loading

benjamincburns commented Apr 24, 2025 •

edited

Loading

benjamincburns Apr 24, 2025 •

edited

Loading

hntrl Apr 24, 2025 •

edited

Loading

benjamincburns Apr 24, 2025 •

edited

Loading

hntrl Apr 24, 2025 •

edited

Loading

benjamincburns commented Apr 28, 2025 •

edited

Loading

hntrl commented Apr 28, 2025 •

edited

Loading