Local LLM Integration with text-web-ui 🦙

Integrate local Large Language Models (LLMs) with oobabooga/text-generation-webui, a specialized interface that includes a custom variant of the OpenAI API for a smooth integration process.

Last updated on Dec 7, 2023


The implementation of local LLMs involves the following components:

  • text-generation-webui: A Python application with a Gradio web UI for operating Large Language Models.
    • Local Large Language Models "LLMs": Use large language models on your personal computer with consumer-grade GPUs or CPUs.
  • big-AGI: An LLM UI that offers features such as Personas, OCR, Voice Support, Code Execution, AGI functions, and more.


This guide assumes that big-AGI is already installed on your system. Note that the text-generation-webui IP address must be accessible from the server running big-AGI.

Text-web-ui Installation & Configuration:

  1. Install text-generation-webui:
    • Follow the instructions in the official page (basicall clone the repo and run a script) [~10 minutes]
    • Stop the Web UI as we need to modify the startup flags to enable the OpenAI API
  2. Enable the openai extension
    • Edit CMD_FLAGS.txt
    • Make sure that --listen --api is present and uncommented
  3. Restart text-generation-webui
    • Double-click on "start"
    • You should see something like:
      2023-12-07 21:51:21 INFO:Loading the extension "openai"...
      2023-12-07 21:51:21 INFO:OpenAI-compatible API URL:
      INFO:     Uvicorn running on (Press CTRL+C to quit)
      Running on local URL:
    • This shows that:
  4. Load your first model
    • Open the text-generation-webui at
    • Switch to the Model tab
    • Download, for instance, TheBloke/Llama-2-7B-Chat-GPTQ
    • Select the model once it's loaded

Integrating text-web-ui with big-AGI:

  1. Integrating Text-Generation-WebUI with big-AGI:
    • Go to Models > Add a model source of type: Oobabooga
    • Enter the address:
      • If running remotely, replace with the IP of the machine. Make sure to use the IP:Port format
    • Load the models
      • The active model must be selected and LOADED on the text-generation-webui as it doesn't support model switching or parallel requests.
    • Select model & Chat


Enjoy the privacy and flexibility of local LLMs with big-AGI and text-generation-webui!