How To Auto Update Your Knowledge Bases FlowiseAI

In this video, I will show you how to automate the
process of keeping your document stores up to

date. Document stores are a great solution for managing
all your data sources in a single container.
This way, you can easily add or remove information
from the knowledge base, without having to change the
AI chat flow itself. But one thing you guys
have been asking me in the comments is how to refresh the
document store automatically when a document
loader uses dynamic data. For instance, you might be
scraping information from a website that changes
often, or you might be pulling data from a
database, like Airtable. In those situations, you
might want the document store to be refreshed
as soon as the data is changed or like every hour, once
a day, etc. I really want to give you the best
value possible in this video, so I thought it
would be fun to build a real world application
together. Here, we have an AI powered business
directory. This is similar to the Yellow Pages, but
instead of searching businesses using
keywords, we can simply have a natural conversation
with an AI chatbot. The business listings are
actually just stored in an Airtable database, but
this could be any type of data source, like you
could be scraping data from a website or
pulling from a database. Now, the issue is if we add a
new listing to this database, for example,
let's add Oak End Barrel, which is a fictitious
restaurant, we'll just enter some address over here,
like so. Let's just enter some sort of phone
number, let's enter a description like "a
restaurant that sells steaks and sushi". Now, I've
committed the changes to this table, and if we go
back to our chatbot and we ask something like "I'm
looking for a sushi restaurant in Johannesburg",
we simply get a message saying that it can't find
that latest entry. And that brings us to the
challenge that you guys have been facing with
document stores. Whenever the data source changes,
you manually have to execute the following
steps. And in this video, we will automate this
process, but let me show you how this works. First, you
have to go to Actions, click on Preview and
Process, then we need to click on Process to load the
data and chunk it, and now we get 17 documents
back, which means that the latest entry has been
pulled in. Now we have to upsell this data into
the vector store by going to Upset Config, and
then we need to click on Upset, and now we can
finally see that that one record has been added and
16 records were skipped. So if we go back to
our chatbot now, and we ask the same question, we
should get the correct response back, and we do.
It's very possible that you might have a situation
where you want to manually refresh the data,
but in many instances, like with this business
directory, we want the data to be fresh maybe once a
day, or every few hours. So let's have a look
at automating this process. You are going to
learn a lot, because in this video, I will show you
how to set up the chat flow, the document
store, and we will use Airtable as our database in this
video. We will also use a workflow automation
tool called n8n to run this automation on our
behalf, and you will also learn how to use the
new Flow-wise API endpoints for updating the document
store. First, let's create our chat flow. I'm
using a cloud instance of Flow-wise, but you're more
than welcome to use a local instance to follow
along with this tutorial. On your local machine,
simply open up the command prompt or terminal
and enter npx flow-wise start. If this is your first
time running Flow-wise, then you will be
prompted to install Flow-wise, so simply enter y and
press enter, and afterwards, you should see a
message like this, and after that, you can access Flow-
wise by going to localhost 3000, and you
should see a dashboard like this. Now let's create the
chat flow by clicking on add new. Let's give
it a name like ai-business-directory, and let's save
this. Let's add a new node. Let's go to agents,
and let's add the tool agent node. Let's start by
adding an AI model to the canvas. So under add
nodes, let's go to chat models, and I'll be using the
chat open AI model. Now I've got plenty of
videos showing you how to use chat olama to run
models locally for free on your own machine, or
you could use grokchat which is also free, and you'll
also find the video on my channel for using
grokchat. So let's add our chat model node. Let's
attach this to our agent. Let's select our
credentials. Let's select our model as gpt40-mini, and
I'm going to lower the temperature to 0.2, as
I want our assistant to strictly follow our
prompt instructions, which we will set up in a
minute as well. Let's also add a memory node. So under
nodes let's go to memory, and let's simply add
this buffer memory node, and let's attach it to
our agent as well. Now finally let's add a
tool node. So under add nodes, scroll all the way down
to tools, and add the retriever tool, and let's
attach this tool to our agent as well. This retriever
tool will be responsible for connecting our
knowledge base to our AI agent. For the tool name,
let's call it business listings, and for the
description let's enter use this tool to find
information about businesses. Now optionally we can
also enable this toggle which will return the
source documents in the chat window. Now because this
process will be customer facing, I don't want
the user to see my data sources, so I'll leave this
disabled. Then lastly let's add a retriever. So under
vector stores let's add the document store node, and
let's attach this to our retriever tool. Great! Now we
simply need to select our document store which
we haven't created yet, but before we do that I just
want to go back to our tool agent and change the
system prompt. I'm going to clear this, but instead
I'll paste in the following prompt. Let's
expand this. You are a helpful assistant for a business
listing service called AI Business Directory.
Only answer questions related to the business. Your
role is to assist the user in finding information
related to the businesses like their location,
contact information, services, and products. Use
the provided tools to retrieve this
information. If the answer is not in the context, then
simply respond with something like "I couldn't find
the information related to that business."
Think carefully about your responses and ensure
that the information returned for each listing
does indeed belong to that listing. Feel free to
pause the video and then copy over this prompt
or enter something relevant to your business. Now
that should be everything we need for our
chatbot itself. Let's go and create our document
store. So from the dashboard let's go to
document stores, let's click on add new, let's call
this AI business directory, let's add this, then
let's click on this document store, which is
currently empty. Now we can start adding data
sources to the store. So let's click on add document
loader. Within here let's click on Airtable and
now we need to provide information about our
Airtable database. So let's go ahead and set up
Airtable. Go over to Airtable.com and sign into your
account. Within the dashboard next to workspaces
click on create a workspace. Give it a name
like AI business directory and then click on
create and let's select start from scratch. Let's
rename the space to something like AI business
directory and let's press enter. Let's also rename the
table by double clicking on this tab and let's
call this listings and let's save this as well. Now we
can define our column names as well as upload
data. To speed things up you'll find a link in
the description to a CSV file which you can
download for free. Then click on add or import, then
click on CSV file, click on browse files and
select the file that you just downloaded. Then click
on upload one file. Let's select create a new
table and let's click next. Then lastly let's click on
import and now you should see all of these columns with
some sample data. Great! Let's actually
delete the table that we created earlier by right clicking
on it. Let's delete this table and let's
rename this one. So let's call this listings and let's
save this. Great! We now have a table with data that we
can use. If we go back to Flow wise we first
have to set up our credentials. So let's click on this
drop down, let's click on create new, let's
give it a name like AI business directory and for the
access token let's go back to air table, click on your
profile, then click on builder hub, then under personal
access tokens click on create new token, call
it AI business directory. Then let's add a few scopes.
First let's add data records read, let's also add
schema basis read and let's also add a base and
let's add the base that we just created which was
AI business directory. Then let's generate a
token, let's copy the token and paste it into Flow
wise and let's click on add. Let's also select
the credential that which is created. Now for
the base id we can simply go back to our
table and the base id is this first part of the url
that starts with app and a whole bunch of
characters after that up until the first front slash
like so. Then for the table name it's this section
that starts with tbl so let's copy that and paste
it in here and then the view is this part
over here that starts with viw, it's pasted into this
field. Cool I'm also going to remove this limit
as I want this loader to read all the entries in the
database and we're not going to add a text
splitter at all as flow wise will automatically create
a document for each entry in the database and we can
see that 16 chunks were extracted and that is
because we have 16 records in our database. Let's
click on process to now store these changes. Let's
also set up our absurd config so under
embeddings because I'm using openai I'll use the
openai embeddings function I'll select my
credentials for the model name I'm going to use text
embedding three small and I'll leave all the other
values on the default values then for the vector
store I'm going to use pinecone then under
credentials let's click on create new then go over to
pinecone.io and log into your account or sign up so
the first thing we want to do is to get our api
key so go to api keys then click on create
api key I'll call this ai business directory and
let's create this then I'm going to copy this
key and paste it into flow wise and for the
credential name I'll call this ai business directory
as well let's add this let's also select the
credential now for the pinecone index let's go back
to pinecone and let's create that index under
database let's click on create index let's simply
call this flow wise then for the dimensions I'll pick
on set up by model I'm using text embedding
three small so this needs to be consistent with
whatever model name you're using over here then
I'll click on set configuration and finally we can
click on create index so I'll simply copy the
index name and add it to flow wise I like giving
namespaces to my projects so I'm going to call this
ai business directory and I'm going to change the top
k to 10 documents this simply means that when
the user asks the question the retriever tool will
retrieve 10 documents back from the vector store in
other words the 10 most relevant businesses to
the user's query now finally let's also set up the
record manager let's click on postgres then under
credentials let's click on create new let's give it a
name like ai business directory and let's
also go ahead and set up a postgres database to
simplify the process of creating a postgres
database let's go to superbase.com and sign into your
account from the dashboard let's create a new
project let's give the project a name like ai business
directory let's also set a password let's
also set the region and let's click on create new
project this will take a few minutes to
complete once the project has been created go down to
project settings then click on database and
under connection parameters we can find all the information
needed to complete the setup let's start by
copying the username and let's add it to flowwise
let's also enter the password that we created
earlier and let's click on add and let's select that
credential then for the host we can copy that from
superbase and add it to flowwise let's do the same
thing for the database name like so and let's also copy
the port and add it to flowwise as well then
the only other setting
I want to change is the cleanup method let's change it
from none to full this simply means that the
record manager will be responsible for preventing
duplicate entries in the vector store and
delete information when we delete entries from the data source
let's save this config let's test it out by
clicking on absurd and if everything was set up correctly
you should see a message sign that 16 records
were added to the vector store and if we go
to pinecon we can see that 16 records were indeed
absurd to the vector database and if we go to
superbase and if we go to database you should see this
table with 16 rows great so now that we have a
document store we can finally go back to our
chat flow let's open up our AI business directory
flow and from the document store node let's select our
newly created document store let's save this
and let's test it out let's say I'm looking for a restaurant
near the waterfront in cape town and we
instantly get a few responses back like the waterfront
seafood spot and if we go to our table and we can see
that the waterfront seafood spot is indeed a
restaurant near the waterfront in cape town now we can
expose this chatbot to our users in many
different ways by clicking on this api endpoint button
we can embed this chatbot into a website or if
we have a python or javascript application we can
easily integrate this into our custom
applications as well but for simplicity's sake i'm
simply going to go to share chatbot let's enable make
public and let's copy this url and if we open this
url in the browser we will be able to access
our AI application from outside of flow wise now we
can of course customize the look and feel of
this application from within flow wise so for
the title i'm going to add AI business
directory and i'll also link to an avatar by adding a
url to an image that is available online and if you
scroll down we can change other things as well like
the welcome message the background colors
font sizes etc i'm simply going to save these changes
and if we go back to our application and refresh
it we can see the title and avatar image over
here oh and we can also hide this power by
flow wise branding so again within flow wise all we
have to do is go to this powered by text color
and change it to white and of course remember to save
your changes and if we refresh this the power
by text is basically gone now before we move on to
the automation side of this let's simply
check if our application is working okay that's
great let's also try a complex scenario and i
really love this behavior so this is something you
can't do with the traditional business
directory let's say something like i am planning on
visiting cape town tomorrow please recommend a
museum that i can visit during the day also suggest a
restaurant and hotel near the waterfront let's say
in this and this is absolutely amazing we get a
museum back and our ai assistant recommended a
museum along with a restaurant and a hotel and
because this is an assistant with memory we can
definitely have follow-up conversations with this
assistant as well if you're enjoying this video
then please hit the like button subscribe to my channel
and share this video now let's have a look at
automating updates to our document store at
the moment when we make changes to our
database so if i add oak and barrel which is a
restaurant i'll just enter something like one oak
street in johannesburg and i'll just put in
some fictitious number and in the description i'll just
enter find dining restaurant specializing in
steaks and sushi with a rating of 4.5 great now as i
showed you in the intro of this video this
data will not be available to the chatbot automatically
and unfortunately we would and unfortunately
we would have to go through this tedious process of
manually reprocess the document loader and
then manually absurd these chunks into our vector
store using this button over here now
thankfully flow was thought about this already and what i
did was expose a lot of api endpoints that we
can consume from outside of flow wise to automate
these type of processes let me show you what i mean
when you go to flow wise's documentation and under
the api reference documentation you can find api
endpoints for pretty much every object within flow
wise including the document stores here you will
find api's for creating document stores
listing document stores fetching details of a specific
store and what we're interested in is this
process loading and chunking operation which is this
step over here as well as absurd chunks from
document store which is effectively this step over here
so to automate this process you can use any
workflow automation tool that you want like
make.com n8n xi peer etc and i already have plenty
of tutorials on my channel for setting up and
working with n8n if you do want to run n8n locally then
simply open up the command prompt or terminal
and enter npx n8n start then click on y to install all
the dependencies and after installation is
complete you will see a url that you can use to access
n8n but for this demo i will be using my
cloud instance of n8n instead so let's add a new
workflow let's give it a name like ai business
directory and let's actually delete this trigger so
what we can do in n8n is set up different types
of triggers trigger manually is simply used for testing
workflows but what we could do instead is use
something like an app event and this gives you
integration with a lot of external service
providers including air table so if i go into air table
we could trigger this workflow when something
changes within air table for example if a new entry is
added to our database table but what i'll do instead is
add this on schedule node this simply means that the
workflow will run on a set schedule for applications
like this business directory i might not want
the listing to show up immediately when it's added to
the database we could definitely do that
using the air table trigger but i personally like to
run these sort of updates once a day or once
every few hours that gives me a bit of an opportunity
to first verify the data to ensure that it's
not malicious or junk or to fix typos if i made a
mistake otherwise that
data including all its mistakes will be instantly
added to the vector store and users might actually see
that so let's say that i actually want to run
this update like every six hours great next i'm going
to add a set node we will use the set node to
store certain information about our flow wise project first
let's create a new field and let's call this
store id and for the value let's go back to flow wise and
let's actually copy this value over here as
that is the unique identifier for this document store
and let's paste it into this field i'm also
going to add one more value called the flow wise base url
now for you that might be localhost 3000 if
you are running this locally but since i'm using
a cloud instance i'm simply going to copy this url and
paste it in here without the front slash so
if i test this step i am getting this json schema out
containing our store id and our flow wise base url and
now we can use these values throughout our
project now the first thing we need to do in this
process is to fetch the details of our document
store we need to do that in order to retrieve each
and every one of the document loaders in this
document store although we only have one document
loader this document store could contain as many
document loaders as we want and what we want to do in
our automation process is to loop through
each of these document loaders and then automatically
trigger the preview and process function for each of
these in a loop so let's see how that's done to
retrieve the details of a document store we can
use this get a specific document store api endpoint
which looks something like this it's basically
the flow wise base url followed by this path and we
have to pass an id value which is that store id
so we're not going to look at the documentation for
each of these api calls but i simply wanted to
show you my process of figuring out which apis to call
and what is apis need so to call that api
within n8n let's add a new node and more specifically
the http request node for the method we will
set this as a get method again the documentation will
tell you what the method type should be for
the url i'm going to switch over to expression and
click on this button just to expand the view then
let's start with the base url and we can see a
preview on the right hand side front slash api v1
document store slash store and you might recall that we
then need to pass in the document store id as
well so let's simply grab that from this previous
node and now you should have a url that looks
something like this now if we simply test the step as
is we will get a failure and more specifically
an authorization failure and that is because
flow wise is protecting their endpoints
through api keys so under authentication let's select
generic credential type then under generic auth
type select header auth then under header auth
select create new credential and for the name
let's enter authorization and this must be
authorization with a capital a and for the value we
need to create an api key within flow wise so back
in flow wise go to api keys then click on
create key let's call this ai business directory let's
add this then let's copy this key and and
let's paste it into this field let's save these changes
let's click on test step and this time we
actually get a response back cool and if we go to the
schema view we can see the details about our document
store like its name and more importantly all
the different loaders and we only have one
loader and that's this air table loader but of course
if we add more we would see all those loaders
pull through as well so i'm just going to clean
up this workflow a little bit by renaming
this node to get document store details so what we want
to do next is simply extract all the document
loaders out of this response so that we can use them
in our workflow so let's simply add a new
node let's add the set node let's rename this to
extract document loaders let's rename this and
all we have to do to extract the document loaders into
this new structure is to simply drag this
loader's array into this fields to set we'll keep the
name as loaders and we can see that this is of
type array and if we run this all this other
information has been dropped and we now simply have a
very simple structure containing only our
loaders which is much easier to work with what we
want to do now is loop through each of these
document loaders remember we could have multiple
doc loaders and for each of these document loaders we
want to call the api to execute this preview
and process step so let's first loop through these
items by adding a new node and more specifically
the loop over items node and because the previous node
is returning an array of loaders this loop
node will automatically loop through these different
loaders so for each loader we can delete this
node and add a new node and because we're calling an
API we'll call the API request node let's
rename this to process chunks let's rename this this
method should be changed to post then for the
URL let's change this to expression again let's open
this up and again we need to start with our
base URL followed by API v1 document store
loaders slash process then for authentication
let's change this to generic credential type then for
the auth type let's select header auth and let's
select our header auth that we created earlier now for
this API we need to pass a body so it's an
able sent body and as a reminder all of this
information is available in the flowwise
documentation as well and we only have to pass two
parameters first is the store id and for the value
we can go to edit fields and grab the store id from
over there let's add another parameter and
this one is called id and this refers to this
document loaders id which we can get from this
loop and that's this id field within the loaders array
now let's go ahead and test the step now if
we go back to our application we can run this entire
flow by running test workflow and we can now
go to process chunks and from this view we can
see all the chunks that were extracted during
this process you might recall that we only had
16 chunks within flowwise but we added the
17th record in our database and if
i now refresh flowwise we can see that the amount of chunks
have changed from 16 to 17 so our API call
was indeed successful and we've just automated this
process over here now before we move on to
the absurd logic i do want to mention that the process
of processing the chunks is asynchronous and
it can take a bit of time to complete so i actually
recommend just adding a wait note into this
loop to give this process some time to complete i'll
simply make this two seconds so this means
that when we run this flow it will wait two seconds
before continuing and everything should still
be okay now let's have a look at automating the
process of observing the data into the vector store so
that's the process of clicking this button over
here when the loop is complete we'll add another note
to this done route over here and we will be
calling another HTTP request let's rename this
to absurd chunks let's change the method to
post then for the URL let's go to expression let's
open this up and of course we'll grab the URL
from the flowwise base URL over here followed by API
v1 document store vector store insert then
for authentication let's change this to
generic credential type let's change the auth
type to header auth and under header auth let's
select our header auth like so let's also enable
saint body and within the parameters we only have
one parameter that we have to pass and that's
the store id so within value let's simply go to
expression let's expand this and let's grab the
store id like so don't worry these errors will all go
away once we execute this flow let's test this
workflow and we can see that the absurd ran
successfully and we can see that 17 records were
indeed added and the old records 16 of them were
deleted so in theory if we now go back to our chatbot
so in theory if we go back to our chatbot we should
be able to find this new business listing
this oaken barrel listing over here so let's say i'm
looking for a sushi restaurant in joannesburg and
have a look at that oaken barrel came back as a
response so it seems that our workflow is indeed
working in fact to test this out i'm going to change
the interval to minutes and i'm actually
going to change this to every two minutes let's go
back to the canvas let's save this workflow
and very important change this from inactive to active
so this workflow will now run every two
minutes let's add a new entry to this table to see if
it gets added automatically so i've just added
the ai studio which is an ai services company and
for the description i'm just saying it's an
ai automation agency so let's give it a minute
or two to see if anything changes on our
document store it's been about two minutes so let's
simply refresh this page and let's see what
happens and there we have it the amount of chance is
changed to 18 and let's start a new conversation
with our agent and let's type i'm looking for an
ai automation agency and it was able to pick
up the ai studio and as you can see our knowledge
base is now automatically refreshed on a timed
interval now finally i do want to give you a pro tip
as well your document store might contain
loaders that do not change at all as an example you
might have some pdf loader or a csv file and
these document loaders never change this is static
data if you want to change the pdf document
you effectively have to delete this pdf
document loader and create a new one so that
simply means that it does not make any sense to refresh
these pdf documents in your automation process
the data is static after all so what you could do in
your workflow is add one additional node let's
move this up and let's add the filter node like
so so now what we can do is actually filter
out any document loaders that we don't want to
refresh automatically so we can grab the loader
id from the left hand side and we could say that we
only want to refresh document loaders with an
id of air table and you can add additional checks as
well so this means when we execute this
workflow all the document loaders besides air table
will be dropped out of this array this video was
a lot of work to create and if you made it this far
that i do want to thank you for your time if
you want to support my channel then please hit the
like button and subscribe and if you would
like to learn how to use flowwise or n8n then check
out these other videos and series over here
i'll see you in the next one bye bye

How To Auto Update Your Knowledge Bases FlowiseAI

Uploaded by

Copyright:

Available Formats

How To Auto Update Your Knowledge Bases FlowiseAI

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

How To Auto Update Your Knowledge Bases FlowiseAI

Uploaded by

Copyright:

Available Formats

<b>In this video, I will show you how to</b> <b>automate the

process of</b> <b>keeping your document stores</b> <b>up to

You might also like

How To Auto Update Your Knowledge Bases FlowiseAI

Uploaded by

Copyright:

Available Formats

How To Auto Update Your Knowledge Bases FlowiseAI

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

How To Auto Update Your Knowledge Bases FlowiseAI

Uploaded by

Copyright:

Available Formats

&lt;b&gt;In this video, I will show you how to&lt;/b&gt; &lt;b&gt;automate the

process of&lt;/b&gt; &lt;b&gt;keeping your document stores&lt;/b&gt; &lt;b&gt;up to

You might also like

<b>In this video, I will show you how to</b> <b>automate the

process of</b> <b>keeping your document stores</b> <b>up to