How to automate your call center quality assessment

Context

The QA (Quality Assessment) process is crucial across many industries, particularly in call centers where it is highly manual and labor-intensive. Phone calls are transcribed and analyzed against evaluation templates, a process that is both time-consuming and expensive for companies aiming to maintain high-quality communication with customers.

However, with advancements in AI, this process can now be significantly automated. Modern speech-to-text technology has advanced to a level where it can transcribe entire phone conversations with greater accuracy than human transcribers.

Large Language Models (LLMs) can then evaluate these transcriptions against specific criteria much faster and at a fraction of the cost of manual evaluations.

Specialized models like Deepgram Nova 2 Medical have emerged, outperforming benchmarks by 20% in word accuracy for medical terminology.

From retail to healthcare, companies are leveraging AI to automate the analysis of phone calls. As an example that we will use throughout this guide, a publicly-traded US-based healthcare provider collaborated with Stack AI to automate their call center QA process.

Let’s discover how they did it.

High-level Design

QA Use Case Architecture

The first step for every Stack AI project is to understand the business process and where the data sits.

In our case, a QA specialist takes an audio file from SharePoint, listens to it, takes notes, and assess it. The assessment is then uploaded into a database in a specific format.

The AI assistant that we are going to build needs a section to upload audio file, a text area to send instructions and an output box to see the assessment once processes.

Let's see step by step how to build this in Stack AI.

Building the logic in Stack AI

Audio-to-text node

Based on the above schema, let's frist drag and drop an audio-to-text model. Given the healthcare nature of the company, the Deepgram Nova 2 Medical model is certainly the most appropriate one.

Additional inputs

Let's add now an input node for the user to provide additional guidance to the model on how to perform the assessment. As we will see the bulk of the instructions come from the LLM itself.

Additionally, if requied, add a SharePoint node and select the site/folder with the information you want to index and give access to the LLM. This can be information on the company, compliance rules, policies, and anything that you think helps the model understand the transcription.

Assessment LLM

We need to select an LLM provider to perform the assessment. We select Anthropic Claude 3.5 Sonnet, given its performance (outperforming other models in most benchmarks, as of August 2024) and its HIPAA compliance.

Specifically, we use Anthropic hosted in a AWS instance. This functionality improves security and reliability as there are no dependencies on global traffic.

It’s important to provide the right instructions to the model by specifying the two sections: instructions and prompt.

The instructions section should include rules that you want the LLM to follow.

In the prompt section, include how you want the LLM to handle the data that is fed to it. In our case, we specify how to handle the transcript, the Input and Sharepoint nodes.

Be as specific as possible. Think of it as if you were to explain to a new joiner how to perform the job.

LLM for JSON format

A second LLM helps transform the assessment in the right JSON format. For this step, use OpenAI GPT-4 Mini given its speed and focus on simpler tasks. As input, it will only require the output from the first LLM. Mention it in the prompt section, and explain the task it has to perform in the instructions section.

The output of this second LLM goes into a custom API node, where you can specify the API parameters to post the information to an external database.

Finally, connect an output node to the first LLM as a way to see the assessment in the user interface that we will set up in a second.

Once tested, click Save and Publish.

NOTE: You can find the instructions and prompts of the LLMs at the end of this guide (in Appendix).

Selecting the right user interface

Available Pre-built User Interfaces

Given the nature of this business process, there is no need for a conversation with an LLM. You want it to complete the assessment for every audio file you send to it. If not satisfactory, then the user has the option of sending additional instructions via the input node.

For these reasons, the form interface is the most appropriate one. Select this UI from the list of available UIs from the export tab.

Customization

Add a name, a description, and upload your company’s logo.

Security

You can protect your application with a password or SSO.

Save and share

Click Save and see the preview. Make additional changes if required, then click on the link at the top, copy it with the button on the right, and share it with your colleagues.

The process is now ready to be automated.

Benefits

Automating the QA process for our healthcare client led to a ~70% reduction in costs and a ~10% increase in customer satisfaction due to improved call quality, which in turn boosted revenue.

Don't wait to get started! Automate your call center QA process with Stack AI.

Appendix: Instructions and Prompts of the LLMs used

For the LLM performing the assessment:

Instructions:

You are a language model designed to assess phone calls between customer service representatives and patients at a telehealth company. Your task is to evaluate these calls for quality and adherence to customer service guidelines. Please provide detailed responses based on the following criteria.

Customer Service Representative's Adherence to Guidelines:
- Identify if the customer service representative followed the standard telehealth customer service guidelines.
- Look for protocols that indicate the representative is performing necessary checks, such as verifying patient identity and confirming the purpose of the call.

Recommendation or Resolution Provided:
- Determine if the customer service representative reached a clear recommendation or resolution by the end of the call.
- Check for statements where the representative provides guidance, offers next steps, or resolves the patient's inquiry.

Patient's Description of the Issue:
- Assess whether the patient was able to clearly describe their issue.
- Identify sections where the patient explains symptoms, their duration, severity, and any prior related interactions.

Patient Satisfaction:
- Evaluate if the patient expressed satisfaction with the assistance provided.
- Look for indicators such as verbal affirmations, thankfulness, or direct feedback on the quality of the interaction.

Customer Service Representative's Adherence to Guidelines:
- Did the representative verify the patient's identity and confirm the purpose of the call?
- Did the representative ask relevant questions to fully understand the patient's issue?
- Were all necessary protocols and procedures followed during the interaction?

Recommendation or Resolution Provided:
- Did the representative provide clear guidance or a resolution to the patient's issue?
- Was there a discussion of the next steps or actions the patient should take?
- Did the representative explain the rationale behind their recommendation or resolution clearly?

Patient's Description of the Issue:
- Was the patient given adequate time to describe their symptoms or concerns?
- Did the patient mention the duration, severity, and any previous occurrences of their issue?
- Were any relevant details provided by the patient during the call?

Patient Satisfaction:
- Did the patient ask questions, and were those questions answered satisfactorily by the representative?
- Was the patient given clear instructions on what to do next?
- Did the patient express satisfaction or dissatisfaction with the interaction, either explicitly or implicitly?

Example Assessment
----
Customer Service Representative's Adherence to Guidelines:

The representative started by verifying the patient's name and date of birth.
The representative systematically asked about the patient's issue and any previous related interactions.
The representative followed standard protocols, such as advising the patient on the next steps.
Recommendation or Resolution Provided:

The representative provided guidance on scheduling a follow-up appointment with a specialist.
The representative explained how to access the patient portal for additional resources.
The representative clearly explained the rationale behind these steps.
Patient's Description of the Issue:

The patient described experiencing ongoing issues with their prescription refill.
The patient mentioned the duration of the issue and previous attempts to resolve it.
The patient felt frustrated but hopeful for a resolution.
Patient Satisfaction:

The patient asked if there were any quicker ways to resolve the issue, which the representative addressed thoroughly.
The patient thanked the representative at the end of the call and expressed relief at the clear guidance provided.

No indications of dissatisfaction were noted during the call.
Please apply this structured approach to evaluate the given phone calls, ensuring that all aspects of the interaction are thoroughly analyzed.
----

Prompt:

The user can give you additional instructions that you must follow in here:

<additional_instructions>
{in-0}
</additional_instructions>

Assess the following phone call from our customer support agent:

<call_transcription>
{audio2text-0}
</call_transcription>

Use this additional context if provided:

<additional_context>
{sharepointemb-0}
</additional_context>

For the LLM transforming the assessment into a JSON:

Instructions:

You are an AI assistant that transforms an assessment into a JSON response.

Prompt:

Assessment in here:
<assessment>
{llm-0}
</assessment>

JSON structure to follow:

---
{
  "adherence_to_guidelines": {
    "verified_identity": "yes/no",
    "asked_relevant_questions": "yes/no",
    "followed_protocols": "yes/no"
  },
  "recommendation_resolution_provided": {
    "clear_guidance": "yes/no",
    "next_steps_discussed": "yes/no",
    "explanation_provided": "yes/no"
  },
  "patient_description": {
    "time_given": "yes/no",
    "details_provided": "yes/no"
  },
  "patient_satisfaction": {
    "questions_answered": "yes/no",
    "clear_instructions": "yes/no",
    "expressed_satisfaction": "yes/no"
  },
  "additional_comments": "string"
}
---

Example of assessment:
---
1. Customer Service Representative's Adherence to Guidelines:
   - The representative started by verifying the patient's name and date of birth.
   - The representative systematically asked about the patient's issue and any previous related interactions.
   - The representative followed standard protocols, such as advising the patient on the next steps.

2. Recommendation or Resolution Provided:
   - The representative provided guidance on scheduling a follow-up appointment with a specialist.
   - The representative explained how to access the patient portal for additional resources.
   - The representative clearly explained the rationale behind these steps.

3. Patient's Description of the Issue:
   - The patient described experiencing ongoing issues with their prescription refill.
   - The patient mentioned the duration of the issue and previous attempts to resolve it.
   - The patient felt frustrated but hopeful for a resolution.

4. Patient Satisfaction:
   - The patient asked if there were any quicker ways to resolve the issue, which the representative addressed thoroughly.
   - The patient thanked the representative at the end of the call and expressed relief at the clear guidance provided.
   - No indications of dissatisfaction were noted during the call.
---

Output in JSON:
---
{
  "adherence_to_guidelines": {
    "verified_identity": "yes",
    "asked_relevant_questions": "yes",
    "followed_protocols": "yes"
  },
  "recommendation_resolution_provided": {
    "clear_guidance": "yes",
    "next_steps_discussed": "yes",
    "explanation_provided": "yes"
  },
  "patient_description": {
    "time_given": "yes",
    "details_provided": "yes"
  },
  "patient_satisfaction": {
    "questions_answered": "yes",
    "clear_instructions": "yes",
    "expressed_satisfaction": "yes"
  },
  "additional_comments": "The patient described experiencing ongoing issues with their prescription refill. The patient mentioned the duration of the issue and previous attempts to resolve it. The patient felt frustrated but hopeful for a resolution."
}
---