How MyLead achieved 65% AI-powered Automation for Customer Service with Softwise.AI

Key results

We built an OpenAI GPT-4-based, tailored solution that made it possible to leverage historical data and insights to build powerful automation for the customer service team.

65% automation rate -> 16,200 requests monthly🚀
6 people were freed for more complex tasks🥷
1200% ROI in the first year 💰

About MyLead

The MyLead affiliate network offers a convenient and accessible way for individuals to earn income from home. Since it was established, it has become a significant player in the online affiliate marketing industry. This platform provides a diverse range of affiliate programs, making it easy for users from various backgrounds to participate without needing initial investments or specialized skills. As of now, MyLead has attracted users from all around the world, each creating free accounts and leveraging the opportunity to earn by promoting products from the comfort of their own homes. This growth and user diversity highlight MyLead's appeal and effectiveness in the evolving landscape of digital marketing and remote earning opportunities.

Key challenges

Extensive number of daily inquiries. MyLead works with thousands of campaigns daily and must ensure the campaign specification matches the marketing strategy described by the publisher.
Sensitivity. MyLead works with all different kinds of customers and publishers from around the world, so it's critical to avoid publishing some kind of campaigns (e.g. with children's diapers) on certain websites (e.g. in this case - adult websites).
Seasonality. The number of campaigns varies depending on the time of day and year.
Opportunity cost. When publishers are waiting to be accepted to join campaigns, they can't earn money and participate in a larger number of campaigns.
Human errors. Humans make mistakes that generate complaints. Complaints mean less satisfied publishers, longer wait times to join campaigns, and increased workload (complaints must be resolved).
Extended new agents onboarding. Agents need weeks of training to be effective at their work which causes increased costs and constant search for a new talent.

Solution

At Softwise.AI our foremost objective is to quickly provide tangible value to our clients. This entails conducting a thorough analysis to ascertain the viability of AI automation for their specific needs. We focus on evaluating the potential outcomes and the timeframe required for delivery. This approach ensures we equip our stakeholders with comprehensive information, enabling them to make well-informed decisions about initiating a project. That's why, we usually start from the Discovery phase.

Step 1 Discovery

In this case, the Discovery phase focused on the historical data analysis. MyLead has gathered around 1,000,000 inquiries so far so we had a lot of data to work with (AWESOME!🍾). Our goal was to discover, how many inquiries have a potential for automation.

Historical Application Data Acquisition

We received labeled application data from the client's system.

Sample: 5,000 records - even though we had a much larger set of available data, we concluded that it would be enough. Sometimes it might be a challenge to extract archived data, some of the gathered parameters have not been tracked from the beginning so too broad historical data would not give us quality insights and conclusions. Often, a more carefully chosen but smaller dataset is also easier to analyze.
Continuous addition of new data points during the process - to find correlations and dependencies we needed to figure out where there are. The initial set of data was not enough and we needed to organize additional workshops to find links between parameters and flow of the customer service acceptances or declines of inquiries.

Exploratory Data Analysis

When we agreed on the final set of data with proper parameters, we continued with analytical analysis. We identified key features for Application Classification:

Application Data: user scoring, description
Campaign Data: provider sensitivity, campaign restrictions

The below chart shows our initial findings based on historically Accepted or Rejected inquiries.

Statistical Solution for Automatic Rejection

Before you go with LLM model usage, having such a great dataset it's always worth checking alternative approaches. LLMs costs are high and bring their own challenges. Often, a much simpler statistical solution is a better fit for an initial round of automation and only then, the remaining data set can be sent to the LLM for further analysis. This allows to save, very often extensive, costs, improve accuracy and predictability.

During the further analysis, we noticed the following correlations:

User scoring less than 12
Campaign sensitivity is categorized as "sensitive"

We discovered that automatic rejection is possible for 23.4% of all applications based on the identified criteria, with an anticipated accuracy of 94%. It means we are able to automate nearly 1/4 of inquiries with minimal effort, achieving accuracy comparable to human performance and without incurring additional LLMs API costs.

GPT Solution for Automatic Rejection

After exhausting the statistical approach, we moved on with choosing the LLM model that best fits our use case. Main criteria:

We needed to analyze unstructured text and a form completed by a publisher with campaign criteria (structured questions + unstructured text)
We needed to work with around 40 languages seamlessly
It needs to be cost-effective due to the high volume of daily processed inquiries

We have tested multiple LLMs but finally decided to go with GPT-3.5 and GPT-4 by OpenAI further in the process. GPT-4 had much better accuracy, but at the same time, it was 10x more expensive. GPT-3.5 would be enough if we had to work only with English, but multiple languages from Europe and Asia, needed GPT-4 for the highest effectiveness.

SMALL HINT! You can now easily calculate LLM costs HERE -> https://softwise.ai/llm-cost-calculator/

Sometimes, at certain scale or confidentiality requirements, it might be better to set up your own LLM model. More about this on our blog -> LINK

In this solution, we crafted a prompt detailing the decision-making process and furnishing all essential information for the model to make well-informed decisions regarding inquiries. We instructed the model to provide responses in a specific format, and upon processing, we assessed the accuracy of the solution.

Our final analysis results for GPT-4 usage:

Automatic rejection is expected for 23.5% of all applications
Anticipated Accuracy: 75%. In this case, this level of accuracy was acceptable by MyLead. We need to remember, that the analysis is based on historical, actual data, which includes human error and non-definitive cases (they could be either accepted or rejected depending on the customer care agent)

Final predictions

Combining statistical and LLM solutions, the final expectations looked like this:

Statistical solution

automatically rejected: 25%
accuracy: 95%

GPT solution

automatically rejected: 23,5%
accuracy: 75%

Combined

automatically rejected: 43%
accuracy: 87%

Step 2 Proof-of-concept

After the Discovery phase, we had the most important project stage completed:

MyLead learned quickly if and what is the potential of automation with minimal initial investment
MyLead's stakeholders had all the information they needed to build a business case internally and secure funding for the following project stages
MyLead learned the timeline and was able to manage dependencies and other work in their teams and plan optimal production launch

Now, there was a time for the initial solution to confirm our analysis. We iterated on prompt engineering after the Discovery phase to make it even more effective and set up a groundwork for the infrastructure.

Then, we chose a test data set (another data set on which Discovery analysis has not been performed) and ran the model on the pre-production environment. That allowed us to check if everything worked as planned and mitigate the risk before the production launch.

Step 3 MVP

The Proof-of-concept stage helped us uncover some additional areas for optimizations - both, in statistical and GPT solutions. After the production launch, and ~3 months period, the final results are better than estimated before:

65% automation rate -> 16,200 requests monthly🚀
6 people were freed for more complex tasks🥷
1200% ROI in the first year 💰

Step 4 Evaluation & Iteration

After the MVP launch, we have been monitoring the solution to optimize it further. Here are a few examples of implemented improvements:

Quality: The GPT model was overly lenient, changes in prompt were introduced that resulted in rejection when uncertainty arose
Explainability: To comprehend the model's decisions, a diagnostic mode was implemented, providing explanations for each decision
Reliability: Ensuring consistent classifications for identical data inputs, we experimented with model Temperature and Top_p sampling parameters

Summary

The MVP launch took around 2 calendar months (from the first meeting to the production), accommodating the summer holiday period, ongoing dependencies on other open projects by MyLead and standard operational work.

As you see, LLMs move AI automation capabilities to new levels. With rapid, customized solutions, companies can now achieve unprecedented operational excellence with low initial investment, rapid time-to-value, and extraordinary ROI.

Our collaboration with Softwise.ai has been instrumental in reshaping MyLead's customer service landscape. With a 65% AI-powered automation rate, efficiently managing 16,200 monthly requests, we've witnessed a significant transformation and an impressive ROI within the first year. The integration has been seamless, enhancing operational efficiency and, more importantly, customer satisfaction. The precision and speed of AI-driven responses have notably improved our service quality.

Paweł Wojcieszak, Head of Affiliate at MyLead