ElevenLabs Launches Customizable Conversational AI Bots, Expands Beyond Voice Cloning

ElevenLabs has rolled out a new feature that lets you create conversational AI bots. This startup, known for its AI voice cloning and text-to-speech API, launched this capability on Monday.

Now, users can build complete conversational agents on ElevenLabs’ developer platform. You can customize various aspects, like the tone of voice and response length.

Until now, ElevenLabs mainly focused on providing different voices for text-to-speech services. Sam Sklar, the head of growth at the company, mentioned that many clients were already using their tools to create conversational AI agents. However, they faced challenges with integrating knowledge bases and managing customer interruptions. That's why they decided to create a full pipeline for these bots.

To get started, just log into your ElevenLabs account. You can choose a template or start a new project. You’ll need to set the agent’s primary language, first message, and system prompt to define its persona. Also, you’ll select a large language model—like Gemini, GPT, or Claude—and adjust the response temperature to control creativity. Don't forget to set a token usage limit, too.

You can tweak other features like voice, latency, stability, authentication criteria, and the maximum length of conversations with the AI agent. It’s all about making the experience fit your needs.

Plus, you can add your own knowledge base. This could be a file, a URL, or a text block that powers your conversational bot. You can even integrate your custom large language model with it. ElevenLabs’ SDK works with Python, Javascript, React, and Swift, and they also offer a WebSocket API for extra customization.

Companies can set criteria to collect specific data, like the names and emails of customers interacting with the agent. They can also define evaluation criteria in natural language to assess how well the conversation went.

ElevenLabs is building on its existing text-to-speech capabilities. They plan to develop speech-to-text features for this new conversational AI product. Right now, they aren’t offering a standalone speech-to-text API, but that could change in the future. This would put them in competition with Google, Microsoft, Amazon, and specialized solutions like OpenAI’s Whisper and AssemblyAI.

The company is looking to raise new funding at a valuation over $3 billion. They’re also competing with other voice AI startups, such as Vapi and Retell, who are also working on conversational agents. Notably, ElevenLabs will be up against OpenAI’s real-time conversational API. However, they believe their customization options and model-switching capabilities will give them an edge.