Skip to main content

NVIDIA

NVIDIA AI Foundation Endpoints give users easy access to hosted endpoints for generative AI models like Llama-2, SteerLM, Mistral, etc. Using the API, you can query live endpoints available on the NVIDIA GPU Cloud (NGC) to get quick results from a DGX-hosted cloud compute environment. All models are source-accessible and can be deployed on your own compute cluster. These models are provided via the langchain-nvidia-ai-endpoints package.

Installation​

pip install -U langchain-nvidia-ai-endpoints

Setup and Authentication​

  • Create a free account at NVIDIA GPU Cloud (NGC).
  • Navigate to Catalog > AI Foundation Models > (Model with API endpoint).
  • Select API and generate the key NVIDIA_API_KEY.
export NVIDIA_API_KEY=nvapi-XXXXXXXXXXXXXXXXXXXXXXXXXX
from langchain_nvidia_ai_endpoints import ChatNVIDIA

llm = ChatNVIDIA(model="mixtral_8x7b")
result = llm.invoke("Write a ballad about LangChain.")
print(result.content)

Using NVIDIA AI Foundation Endpoints​

A selection of NVIDIA AI Foundation models are supported directly in LangChain with familiar APIs.

The active models which are supported can be found in NGC.

The following may be useful examples to help you get started: