Skip to main content

TF-IDF

TF-IDF means term-frequency times inverse document-frequency.

This notebook goes over how to use a retriever that under the hood uses TF-IDF using scikit-learn package.

For more information on the details of TF-IDF see this blog post.

# !pip install scikit-learn
from langchain.retrievers import TFIDFRetriever

Create New Retriever with Texts​

retriever = TFIDFRetriever.from_texts(["foo", "bar", "world", "hello", "foo bar"])

Create a New Retriever with Documents​

You can now create a new retriever with the documents you created.

from langchain.schema import Document

retriever = TFIDFRetriever.from_documents(
    [
        Document(page_content="foo"),
        Document(page_content="bar"),
        Document(page_content="world"),
        Document(page_content="hello"),
        Document(page_content="foo bar"),
    ]
)

Use Retriever​

We can now use the retriever!

result = retriever.get_relevant_documents("foo")
result
[Document(page_content='foo', metadata={}),
 Document(page_content='foo bar', metadata={}),
 Document(page_content='hello', metadata={}),
 Document(page_content='world', metadata={})]

Save and load​

You can easily save and load this retriever, making it handy for local development!

retriever.save_local("testing.pkl")
retriever_copy = TFIDFRetriever.load_local("testing.pkl")
retriever_copy.get_relevant_documents("foo")
[Document(page_content='foo', metadata={}),
 Document(page_content='foo bar', metadata={}),
 Document(page_content='hello', metadata={}),
 Document(page_content='world', metadata={})]