An AI-bot and a web Admin panel: Proof-of-Concept

Type Web application
TECHNOLOGIES Java, Java Spring, JavaScript, Python, ReactJS
AREAS OF EXPERTISE External services integration (Twilio, DialogFlow, Google Cloud), Speech-to-Text and Text-to-Speech transformation, Natural language processing
TEAM 7 employees

The project is a Proof-of-Concept for the system which includes an AI-bot and a web Admin panel for a large company. The AI bot will allow Customers and Prospects of this company to get a quick and high-quality service for advertising cancellation. The admin panel will be used by the specialists of the company. The Panel will let them monitor and analyze the dialogues between the bot and Customers, Prospects in order to improve the AI bot.

The system will help the company get the following benefits:

  • reduce costs by optimizing the workload of call center operators;
  • make the company’s support focus on more complicated and important requests from Customers and Prospects rather than on routine ones (e.g. cancel advertising);
  • increase Customers and Prospects’ loyalty by improving the quality of requests handling.

The main implemented business processes are speech-to-text (STT) and text-to-speech (TTS) transformation, which looks as follows:

  1. The Customer calls the call center.
  2. The Customer listens to the voice menu options and selects the option to stop the direct advertising from the menu.
  3. The Customer is switched to the AI Bot.
  4. The Customer communicates with the bot.
  5. The bot understands the caller and puts a stop to the advertising for this Customer.
  6. The Admin views the log of this dialog via the Admin panel.
  1. The main challenge we encountered was getting a raw audio stream from Twilio. Their API doesn't provide such functionality. Only a complete record of a conversation might be obtained after it finishes.
  2. Writing a SIP client was also complicated because PJSUA* library is able to write to audio file, but not to a stream.

*PJSUA is an open source command line SIP user agent (softphone) that is used as the reference implementation for PJSIP, PJNATH, and PJMEDIA.

  1. To tackle this issue, we used SIP server provided by Twilio. In brief, we created a SIP client that accepts incoming calls and "talks" with the user using Google TTS, STT and DialogFlow.
  2. Here we wrote a custom data receiver (called "port" in PJ terms) that transfers an audio stream between PJ internals and Python side.


PoC allowed us to develop the AI Bot for calls, which correctly understands users’ requests and properly answers them and the Admin panel, where the Admin can view all the conversations between the AI bot and clients.

Besides, we developed a bot for emails and Facebook chat, which can also handle requests from clients within these channels.  

Areas of expertise

External services integration (Twilio, DialogFlow, Google Cloud) // Speech-to-Text and Text-to-Speech transformation // Natural language processing

Additional Technologies

PostgreSQL // Amazon

Data analysis app for medical garment usage prediction
Data analysis app for medical garment usage prediction
A Gas Flow Chart OCR solution
A Gas Flow Chart OCR solution