fix(core): Infer tool return types, with related tool type fixes #8056

hntrl · 2025-04-23T18:12:37Z

Fixes #8010. Instead of just using an any alias as the return type to invoke, this adds a generic ToolOutputT type param on the tool classes that decides the return type of invoke.

A later enhancement could be to restrict the return type of the provided function if "content_and_artifact" is provided as the response format (i.e. instead of any[], the return type is forced to be extends [any, any] in the tool function)

cc @benjamincburns

vercel · 2025-04-23T18:12:41Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-docs	✅ Ready (Inspect)	Visit Preview		May 7, 2025 1:39am

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-api-refs	⬜️ Ignored (Inspect)			May 7, 2025 1:39am

benjamincburns · 2025-04-24T00:02:20Z

Hi @hntrl - looks like you might've forgotten to do a top-level yarn build? The exports tests appear to be failing due to type errors in the provider libs.

benjamincburns · 2025-04-24T00:09:44Z

Hi @hntrl - looks like you might've forgotten to do a top-level yarn build? The exports tests appear to be failing due to type errors in the provider libs.

Ah quick correction. I think the exports tests are actually just smoke testing bundled versions of the project, and they tend to catch breakages. I suspect this may be a case where the code in question is making invalid assumptions about the return type of StructuredTool, so it's maybe a false positive, but you'd need to evaluate.

Apologies for the confusion on my part - still learning the finer details about our build/test pipeline process.

hntrl · 2025-04-24T01:17:37Z

libs/langchain-aws/src/tests/chat_models.int.test.ts

-      content: await tools[0].invoke(result.tool_calls![0]),
-    })
-  );
+  messages.push(await tools[0].invoke(result.tool_calls![0]));


StructuredTool.invoke will return a tool message when a tool call is passed, so we don't have to create a new tool message here

hntrl · 2025-04-24T01:32:23Z

Hi @hntrl - looks like you might've forgotten to do a top-level yarn build? The exports tests appear to be failing due to type errors in the provider libs.

Ah quick correction. I think the exports tests are actually just smoke testing bundled versions of the project, and they tend to catch breakages. I suspect this may be a case where the code in question is making invalid assumptions about the return type of StructuredTool, so it's maybe a false positive, but you'd need to evaluate.

Apologies for the confusion on my part - still learning the finer details about our build/test pipeline process.

No worries and thanks for the callout! I think you're right in that the failing test was a false positive. The most recent commits I pushed fix the CI failures.

a0da856 fixes the failing test from the latest CI run. I added comments explaining the change.

3cbaceb fixes some other build failures I found that come from changing the interface type specifically. From what I can gather, changing that type has a (maybe unnecessary?) rippling effect through a bunch of provider packages, so I'll avoid trying to tackle that for now. At least this way the tool method works as expected when imported from the core package directly.

benjamincburns · 2025-04-24T01:39:44Z

langchain-core/src/tools/types.ts

-  ): Promise<TArg extends ToolCall ? ToolMessage : ToolOutputT>;
+  ): Promise<ToolOutputT>;


Still assessing, but I'm not sure that this change is valid.

It's a bit confusing because we don't have StructuredTool actually implementing StructuredToolInterface (I forget why that is), but the latter is meant to reflect the public interface of the former. Same goes for ToolInterface and Tool.

Additionally, extensions of StructuredTool should always be assignable to StructuredToolInterface, and extensions of Tool and DynamicTool should be assignable to ToolInterface. We should probably have a test that guards this property if we can't make it so the aforementioned classes actually implement their corresponding interfaces 🤔.

I think additionally all of those should be assignable to StructuredToolInterface (lack of type params is intentional on this and all of the above - default params should be wide enough to allow this).

Makes sense. Two paths I can see going from here (both of which I'm happy to do, more just a matter of preference):
(1) Regress the tool classes to return just ToolOutputT instead of switching based on input (which doesn't really have any impact since it was just returning an any alias before, at least this way it's an optional type param on the classes + interfaces that we can use to control the output type)
(2) Bring the output type switching back to the interface and fix the side effects in the provider packages

Option 1 gets this PR merged quicker, option 2 is more "feature complete".

Unrelated but I'd be interested to know why that direct inheritance isn't there.

I think the diff makes it confusing but this change specifically is reverting a previous change I made. On main the return type is set to Promise<ToolReturnType>

Unrelated but I'd be interested to know why that inheritance isn't there.

We've been iterating in this area recently (see #7973), so the types are a bit more well-aligned between the classes and interfaces now than when they initially came together. Looking into these issues myself, I see one bug that I introduced in that PR, where I should've aligned the constraint for SchemaT on ToolInterface with the constraint for SchemaT on Tool. I think that's all that's necessary to make Tool implement ToolInterface and StructuredTool implement StructuredToolInterface.

Two paths I can see going from here (both of which I'm happy to do, more just a matter of preference):

(1) Regress the tool classes to return just ToolOutputT instead of switching based on input (which doesn't really have any impact since it was just returning an any alias before, at least this way it's an optional type param on the classes + interfaces that we can use to control the output type)

My concern here would be that we'd drop the ToolMessage type from the output, or am I mistaken? That would trigger cases where we return something that's not expressed by the return type. Unfortunately I think it'd be kinder to consumers to regress all the way back to any as it'd be better to have no "documentation" of types here than to be misleading.

(2) Bring the output type switching back to the interface and fix the side effects in the provider packages

I think this would be the preferred approach, with a bias toward better alignment of the type hierarchy and against using casts to fix the errors outside of @langchain/core (if possible). I can probably pick this up Monday if you don't have the time for it, though.

My concern here would be that we'd drop the ToolMessage type from the output, or am I mistaken? That would trigger cases where we return something that's not expressed by the return type.

Good catch. I guess I was thinking more-so on the lines of consuming from the interface end instead of the consumer end (i.e. not creating tools using the tool method) where the output type is always implicitly any since the default for the type param is ToolReturnType. Agree this would be a bad change.

I'll have some more time to work on this tomorrow. If you don't hear from me by Monday then feel free to work off this PR

hntrl · 2025-04-25T19:34:41Z

@benjamincburns Some updates on my findings:

It turns out the error I was trying to corner has to do more with how typescript computes type inheritance rather than integration problems. When I add the conditional return types back to the interfaces (284f3e5), the error that occurs downstream is with the prebuilt agents and how it passes tools around. I haven't been able to address this yet so I imagine CI is failing atm.

The crux of the error is DynamicStructuredTool<...> is not assignable to StructuredToolInterface.

If you try to add the interface inheritance directly on the class, the error comes from the call method where it's saying that the two arg types are incompatible. <TArg extends StructuredToolCallInput<...>>(...) is not assignable to <TArg extends unknown>(...)

It's almost like when evaluating if the dynamic tool conforms to the interface, TypeScript is computing the SchemaT param on the interface (which will always fall down to unknown bc of how the tool call input type is defined), assuming that's the reference type for the generic in the call method, and complaining that StructuredToolCallInput != unknown. If that's the case, this doesn't lend itself well to comparing generic methods like we're doing here to get the conditional return type.

This is just specifically for DynamicStructuredTool. I can add inheritance from StructuredToolInterface to Tool and DynamicTool without any problems.

benjamincburns · 2025-04-28T00:57:03Z

So I confess that I got a bit nerd sniped by this last week and worked up a fix separately. I branched off of an earlier version of this PR, so unfortunately our work has diverged. My fixes are in c8d08fe and 84cf083, with the latter commit having the more substantial adjustment.

The main problem that needed fixing was that I defined the default schema type on StructuredTool and StructuredToolInterface too narrowly when I added support for JSON Schema. Since StructuredTool is essentially the base type for all LangChain style tools, its default schema type should be the union of all possible type schemas. Prior to that change, in cases where the type system couldn't infer what the schema type may be (e.g. because the tool was passed in across some external boundary), we were incorrectly assuming that the tool would've been a zod schema.

My branch includes your initial submission, so I'd be happy to merge it as-is, but if you want to review and reconcile this work against those changes I'd be happy to wait for you to do that. I suspect that you may have made some improvements to the types on internal functions that I'm missing.

hntrl · 2025-04-28T17:45:07Z

No shame in that! I know I'm guilty of that sometimes too. I missed the detail about the config object also determining the output type, so I appreciate you taking a crack at it.

I merged your branch here, added some changes for readabilities sake in how the return type is defined/used, and fixed the override for the invoke method on both the interface and the class.

langchain-core/src/tools/index.ts

benjamincburns · 2025-05-02T07:14:55Z

Rebased on current main and linearized history, which dropped a couple of duplicated commits.

benjamincburns · 2025-05-02T07:17:07Z

Build passes locally - will merge if CI goes green 🤞.

benjamincburns · 2025-05-02T09:01:24Z

Realized that I didn't have type tests for the tool methods (namely call and invoke). Added those and found some more issues. Still need to add them for Tool, however.

hntrl · 2025-05-04T19:23:41Z

langchain-core/src/tools/tests/types.test.ts

 });

 const testDynamicStructuredToolWithZodEffects = new DynamicStructuredTool({
  name: "test",
  description: "test",
-  func: async (input: string) => `test ${input}`,
+  func: async (input: { input: string }) => `test ${input.input}`,


is this accurate? I was under the impression that the type passed in would be the input type after transformations [1]. This is the case if you take off the argument assertion here. (i.e. just leave (input) =>). Maybe it makes more sense to test the argument type inside of the function instead of washing it:

func: async (input) => { // input will be unassignable to arg if the input type changes const arg: string | undefined = input; return `test ${arg}`; }

[1] https://github.com/hntrl/langchainjs/blob/ce46fe215c57d063f31ce551d75ee2d0d8908e3b/langchain-core/src/tools/types.ts#L337

Yeah - I think I must've modified this in error. Will double check by adding a test that validates the output on invoke

Yeah you were right - good catch!

langchain-core/src/tools/index.ts

hntrl · 2025-05-04T20:44:42Z

langchain-core/src/tools/index.ts

+  override invoke<
+    TArg extends string | undefined | z.input<this["schema"]> | ToolCall,
+    TConfig extends ToolRunnableConfig | undefined
+  >(
+    input: TArg,
+    config?: TConfig
+  ): Promise<ToolReturnType<NonNullable<TArg>, TConfig, ToolOutputT>> {
+    return super.invoke(input, config) as Promise<
+      ToolReturnType<NonNullable<TArg>, TConfig, ToolOutputT>
+    >;
+  }
+


This new override is causing CI to fail. On main there is no invoke override for tool, it just assumes the signature from extending StructuredTool. Removing this gets my build to work locally

The TArg union is covered by all of the types in StructuredToolCallInput (except for undefined), but I'm lost as to why undefined is acceptable as input. Wouldn't an undefined input fail validation every time?

Edit - ignore me - this change wasn't necessary!

Original comment:
Unfortunately undefined was part of the input signature here originally. If I were writing this from scratch I definitely wouldn't allow undefined input, but we're trying to avoid making input types any narrower than they were before this PR, as otherwise this would be a breaking change.

I suppose there's a valid argument that narrowing output types is breaking here in the strictest sense, as the change can introduce new compiler errors where there weren't any previously. However I think it's reasonable to consider a change to be not breaking if the only compiler errors it introduces are ones that expose structural and/or logical errors that existed prior to the change in question.

I did some spelunking and it's been that way since at least 2 years ago.

IIRC I added the invoke override to align the invoke input type with the call input type for adherence to the interface. Maybe that wasn't necessary, though. 🤔

hntrl · 2025-05-04T20:50:49Z

Added some nit changes more than anything:

I tied the return type to DirectToolOutput instead of using an explicit object type. As far as I can tell, there's no case where lc_direct_tool_output is ever false, so I changed the type there to always assume that property is true. ce46fe2

Also hoisted the decision logic for determining if a tool call id exists in a config out to a utility just to match existing patterns. 1f10cf8

benjamincburns · 2025-05-05T23:18:53Z

call returns ToolMessage only if the toolCall property of the config is set and contains a nonempty ID.

Doesn't call also cover the case of returning a tool message if arg is a tool call? The continuation i'm following is after the validation logic (starting at let toolCallId: string | undefined;), toolCallId gets assigned if arg is a tool call, then _formatToolOutput returns a ToolMessage if toolCallId is defined.

I could very well be missing something here...

Nah you're right - I meant to go back and strike out that comment. In the end I wound up doubling back on this and making the typing a bit more explicit as you saw, based on the devX that I was seeing when writing the tests.

benjamincburns · 2025-05-07T01:16:27Z

Still need to add them for Tool, however.

This was silly of me - Tool is abstract, and DynamicTool extends it - I think this is complete now, provided CI is green.

benjamincburns · 2025-05-07T01:38:43Z

Thanks a ton for all your help on this one @hntrl! It was a pleasure working with you to get this over the finish line!

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. auto:bug Related to a bug, vulnerability, unexpected error with an existing feature labels Apr 23, 2025

vercel bot deployed to Preview – langchainjs-docs April 23, 2025 18:25 View deployment

hntrl commented Apr 24, 2025

View reviewed changes

vercel bot deployed to Preview – langchainjs-docs April 24, 2025 01:28 View deployment

benjamincburns force-pushed the fix/gh-8010 branch from 3cbaceb to 6947702 Compare April 24, 2025 01:36

benjamincburns reviewed Apr 24, 2025

View reviewed changes

vercel bot deployed to Preview – langchainjs-docs April 24, 2025 01:48 View deployment

vercel bot deployed to Preview – langchainjs-docs April 25, 2025 19:21 View deployment

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Apr 28, 2025

vercel bot deployed to Preview – langchainjs-docs April 28, 2025 17:49 View deployment

vercel bot deployed to Preview – langchainjs-docs April 28, 2025 21:10 View deployment

vercel bot deployed to Preview – langchainjs-docs April 29, 2025 03:09 View deployment

benjamincburns reviewed May 2, 2025

View reviewed changes

langchain-core/src/tools/index.ts Show resolved Hide resolved

benjamincburns force-pushed the fix/gh-8010 branch from 9b41066 to beb1c7e Compare May 2, 2025 07:13

vercel bot deployed to Preview – langchainjs-docs May 2, 2025 07:26 View deployment

vercel bot deployed to Preview – langchainjs-docs May 2, 2025 09:13 View deployment

hntrl commented May 4, 2025

View reviewed changes

vercel bot deployed to Preview – langchainjs-docs May 4, 2025 20:58 View deployment

hntrl and others added 15 commits May 6, 2025 18:14

fix(core): make tool outputs a generic type (langchain-ai#8010)

1acadf9

fix(core): return message type when using tool calls

cf0717c

fix(core): revert tool interface signature

efee5f4

feat(core): add conditional return types to tool interfaces

444dec4

fix(core): fix bad schema type constraint in ToolInterface type params

1d2c220

chore(community): fix stagehand toolkit typing issue

714e193

fix(core): fix tool typing issues, add simple tool type tests

4d3b587

chore(core): hoist tool return types

9399b8d

fix(core): use new return type for tool invoke method

2d74b76

fix(core): tool test lints

c8eaeca

fix(core): fix Tool call method return type

25a1d3d

fix(core): more tests & handle direct tool output

a6ee8d1

fix(core): use stricter DirectToolOutput type in tool return conditional

e716207

chore(core): hoist tool call config logic to a util

57ef084

fixes

37200df

benjamincburns force-pushed the fix/gh-8010 branch from cbe4989 to 37200df Compare May 7, 2025 01:14

vercel bot deployed to Preview – langchainjs-docs May 7, 2025 01:39 View deployment

benjamincburns changed the title ~~fix(core): Infer tool invoke return type from provided function~~ fix(core): Infer tool invoke return type from provided function, and related tool type fixes May 7, 2025

benjamincburns changed the title ~~fix(core): Infer tool invoke return type from provided function, and related tool type fixes~~ fix(core): Infer tool return types, with related tool type fixes May 7, 2025

benjamincburns merged commit f96352c into langchain-ai:main May 7, 2025
38 checks passed

		): Promise<TArg extends ToolCall ? ToolMessage : ToolOutputT>;
		): Promise<ToolOutputT>;

fix(core): Infer tool return types, with related tool type fixes #8056

fix(core): Infer tool return types, with related tool type fixes #8056

Uh oh!

Conversation

hntrl commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel bot commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

benjamincburns commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

benjamincburns commented Apr 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hntrl commented Apr 24, 2025

Uh oh!

benjamincburns Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hntrl Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benjamincburns Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hntrl Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hntrl commented Apr 25, 2025

Uh oh!

benjamincburns commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hntrl commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

benjamincburns commented May 2, 2025

Uh oh!

benjamincburns commented May 2, 2025

Uh oh!

benjamincburns commented May 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benjamincburns May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hntrl commented May 4, 2025

Uh oh!

benjamincburns commented May 5, 2025

Uh oh!

benjamincburns commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

benjamincburns commented May 7, 2025

Uh oh!

Uh oh!

Uh oh!

hntrl commented Apr 23, 2025 •

edited

Loading

vercel bot commented Apr 23, 2025 •

edited

Loading

benjamincburns commented Apr 24, 2025 •

edited

Loading

benjamincburns Apr 24, 2025 •

edited

Loading

hntrl Apr 24, 2025 •

edited

Loading

benjamincburns Apr 24, 2025 •

edited

Loading

hntrl Apr 24, 2025 •

edited

Loading

benjamincburns commented Apr 28, 2025 •

edited

Loading

hntrl commented Apr 28, 2025 •

edited

Loading

benjamincburns May 7, 2025 •

edited

Loading

benjamincburns commented May 7, 2025 •

edited

Loading