This plugin allows you to setup a fully functional, multi-model AI Agent capable of having conversations with through means of voice and text input. It has a huge amount of configuration included and also has the ability to see (vision) through a video input (webcam) and answer any questions pertaining to vision capabilities (including the uploading of images and image generation).
It combines functionality from Groq, Deepgram, Cartesia, ElevenLabs, OpenAI, Gemini and Pinecone (plus a few other services) so you can have your very own voice assisted agent (or standard chat BOT) within your own site tailored to your own dataset if needed.

There are additional services for accessing real-time data and image generation features, and support for function calling is included (external and internal function calls).

Function calls can be interrupted through this plugin, meaning you can get access to the data before it gets sent onto the relevant service. This allows you to handle the data in other ways.

Custom function calls are setup through a dedicated app built to communicate with this plugin. It allows you to build out your own function calls and they will be available through the plugin when it runs. This UI is built within the tokenAuth page (see instructions).

The AI component of this plugin is driven by Groq or Gemini. It also includes Retrieval-Augmented Generation (RAG) which allows you to train your agent with your own custom data. For example, if you provided some text (or pdf files) containing relevant information about your site, then it will able to answer any questions relating to that data to further it's knowledge. You can also integrate a Pinecone database for use with RAG setups if required for much larger data sets.

A visualization element has been included that animates based on noise levels as the agent speaks (uses a THREE.JS animation) and there's also an interactive avatar element with lip-syncing abilities too.


There are a number of additional features in this plugin. Outlined are just a few.

* It can remember conversational history to form more personalized responses.

* Responses can be limited through use of tokens, so they’re not so long.

* A conversational state has been included that can be used as the data source for repeating groups to create a chat type of interface.

* Access to real-time data through Google is available through additional configuration using SerpAPI (Search Engine Results Page) or Gemini's grounding feature.

* Supports external vector database setup, to be used as a RAG application, connecting to Pinecone and able to upload data as embeddings.

* You can ask the agent to generate images based upon a textual description of how you’d like it to look. Various image settings are included around this functionality.

* Includes a voice activity detector which is capable of interrupting the agent. Various threshold settings and volume adjustments can be made when this happens.

* Microphone streaming options are provided to help reduce any background noises or other interferences that may occur.

* You can preload the agent with conversation history from a previous period in time.

* Supports Deepgram, Cartesia, ElevenLabs, OpenAI and PlayHT for streaming text to speech services (each have their own advantages).

* Can generate images from textual content (video generation is on the road map).

* You can inform the agent how you want it to behave, for example “You are a very helpful assistant who just loves to talk” or “You are calm and take on a methodical approach to questions” etc. Depending on the information you provide will determine how it responds.

* There are 12 avatars to choose from when enabling the text to speech service from Deepgram. These all have different settings, voices and colors stored in a state to help design your UI.

* ElevenLabs use more natural voices and are defined when you log into their site. They provide a much broader range of configuration when it comes to voices and also give you the ability to clone your own voice that can then be used in this plugin. Similar rules apply for both Cartesia and PlayHT.

* Support for function calls, allowing for additional requests to other providers thus expanding the agents capabilities.


Technical Info:

All keys are secured (although not entirely) through use of "tokens" which are sent to another app, further protected by privacy rules setup on the database and an additional option is provide to encrypt all the keys also. Websocket and HTTPS streaming endpoints are used for communications and access tokens/ephemeral keys are used whenever possible. This plugin does not offer complete security around any keys provided.

Simple to setup and instructions are included.