Google Batch prediction

[ACMCMC]

Discussed in #599

^{Originally posted by hect1c November 13, 2024}
From my understanding, please correct me if I'm wrong, when using Langchain's batch method we are essentially just running Runnables.batch which appears to just run the invoke method but in parallel using threadpools or async tasks.

So my questions are:

1. Is the invocation method for Langchain Vertex AI API is the same regardless of which method we choose (batch vs invoke)?
2. Under the hood is for Vertex is Langchain using the predict endpoint for either invocation method?
3. If above is true, are there plans to implement batch prediction API with Vertex?

I would like to you use GCPs Batch prediction with Vertex AI while leveraging the functionality and features / tools from Langchain. Is there a way to achieve this?

---

It'd be very nice to have the possibility to call the batch prediction endpoint when doing .batch(). Currently, we're calling the regular prediction endpoint individually for each element of the batch, which is suboptimal.

[lkuligin]

Afaik, Vertex batch prediction is more like a scheduled task (and it completes after quite some time, and you can get results from the GCS bucket). What use case do you have in mind by using it with LangChain?

[ACMCMC]

Exactly that - having a very large batch of inputs, scheduling a job, waiting for it to be finished, and getting the results from the bucket - this could be modeled as a synchronous or asynchronous call :)

[lkuligin]

but why do you need this as part of a chain? you can use a native SDK to schedule such a job, how does LangChain help?

[ACMCMC]

I'd appreciate having that readily available without having to write the code myself. I see LangChain as a framework to abstract and simplify interaction with language models.

I don't need LangChain to use LMs, but it can help; I don't need LangChain to use batching, but it can help.