The Arbitrage Rebuild: How We Made Our Provider Scoring Smarter

Every OTP Prelude sends has to be handed to one of a network of SMS companies. That choice happens in milliseconds, with no human involved, and it decides which provider delivers the message. Choosing between those providers, for every single message, is what Prelude calls arbitrage.

To make that choice well, you need to know which providers actually deliver. Every SMS provider will tell you their delivery rates. That number is not comparable from one provider to the next, which makes it useless as a ranking signal.

"Each provider doesn't provide the delivery report in the same way, and some of them have their own definition of what is in transit and what is delivered," says Hugo Bauer, a software engineer on Prelude's verification infrastructure. "Because SMS is a kind of legacy technology, depending on our providers, we might have data that is inaccurate.” 

Some providers count the SMS as delivered once it reaches the cell tower. Others wait until it reaches the phone itself. That inconsistency opens the door to something worse: a provider that knows traffic is awarded based on delivery rate has an incentive to inflate that number. "They might cheat by increasing the delivery rates to get more traffic," Hugo explains, "because we split the traffic between providers.” The first challenge, then, was to overcome these discrepancies among telco providers. 

So the team at Prelude decided to build their own signal. The Verify API includes a /check endpoint that customers call when a user enters their code. A successful check means the message arrived, the user read it, and they typed the right digits back. That is what we call conversion. No ambiguity, no manipulation. The user either got the code or they didn't.

That decision gave Prelude a reliable foundation, and it's the reason many customers trust Prelude, where the focus is about users getting in, not about inflating SMS or attempt counts.

How the Original System Worked

The original arbitrage system was simple by design. For each provider in each country, it calculated a score combining conversion rate and price. A cheaper provider could beat a better-converting one if the economics justified it. It was a good starting point, and for a while, it was enough.

When Hugo joined, the team added granularity. Performance differences were appearing at the carrier and device level. "We started to receive many more signals and add granularity," Hugo says. "Before it was only statistics at the country level, but then we added carrier granularity and device, because for some reasons SMS were less delivered on some specific Android or iOS devices." The logic stayed the same, but the scoring was now far more granular: one score per provider per country became one score per provider, per carrier, per device type. 

Then a large e-commerce customer joined and a gap in that approach became impossible to ignore: a single conversion score shared across all customers on a provider cannot distinguish between customers who convert differently for reasons that have nothing to do with the provider.

A Use Case the Model Hadn't Seen Before

Most of Prelude's customers used OTPs for login or account registration. This customer placed the OTP step at the end of their checkout flow as a final confirmation before completing a purchase.

That difference showed up directly in the numbers. "The Prelude Verify product was mostly used to allow users to log in to a platform," Hugo explains. For this particular customer's end-users, it was a different use case entirely. "They were using it at the end of the shopping flow to confirm an order. So their conversion rate was much lower, because some of their users were requesting OTPs but not entering the OTP code. They changed their minds."

The user hadn't failed to receive the message. They'd simply decided not to buy. Checkout flows carry a drop-off that login never does. Some percentage of shoppers abandon a purchase between adding an item to cart and confirming it, regardless of whether the OTP arrives. That drop happens whether or not the SMS ever gets delivered.

This customer was sending enough traffic to be a large share of Prelude's volume in some smaller markets. At that weight, their structurally lower conversion rate pulled the shared average down for every provider. The arbitrage system responded by moving traffic to the next-best provider. That provider's score then dropped for the same reason. "The whole system was not broken but inaccurate," Hugo says.

What One Incident Revealed

The large customer incident turned out to be a window into three patterns that had been there all along.

Fraud. Fraud attacks generate OTP requests that will never be entered. On the rare occasion that fraudulent traffic gets through any detection layer, those unconverted attempts drag down whichever provider carries them. "When it goes through, usually the fraud doesn't convert," Hugo explains. "So we are sending attempts with providers but we will never receive checks. So the global conversion drops for those providers." The best-ranked provider gets hit first. Its score collapses. Traffic moves to the next one. The attack follows.

Check out our previous article in the engineering series if you’d like to explore how Prelude detects and responds to agent-driven fraud.

Sender ID filtering. Prelude registers sender IDs with carriers so customers can send SMS under their brand name. But some carriers block names they consider potentially deceptive, and when they do, it is invisible in the global numbers. 

One customer had a brand name that included a common retail term. Certain carriers had placed that term on their blocklist because scammers frequently use generic retail-sounding names to impersonate e-commerce businesses. The customer's messages weren't being delivered through those carriers at all, but the global statistics for the provider looked fine, because only a fraction of their total traffic ran through the affected routes.

"Only a few carriers were filtering that name, so nothing showed at any level we looked at," Hugo says. "The provider looked fine on those carriers because other customers went through, and the customer looked fine in that country because the other carriers delivered. You only see it when you look at one customer on one carrier, and then it is not a small drop, it is a wall." 

Customer baselines. Some customers simply convert at lower rates than others, not because of anything a provider does, but because of how their product works. A bank asking users to authenticate converts differently from an e-commerce app asking them to confirm a purchase. "On average, there is a difference on the same route," Hugo says. "A route is a provider in a country, and there is around 10% of spread between customers." In other words, Customer A and Customer B could both be sending through the same provider in the same country, and their conversion rates could sit 10 percentage points apart, not because one is getting better delivery than the other, but because their users behave differently. The same provider, the same country, a 10-point gap. The score was describing the customers, not the provider.

The result in all three cases: the shared average had no way to tell "this provider is underperforming" from "this provider happens to serve customers who convert differently."

Building Towards a Better Model

Attempt One: Cap Customer Contribution

The first instinct was to limit how much any single customer could influence the shared score. If one customer represents half of traffic in a country, cap their weight.

The approach had a fundamental constraint. "We are just removing a lot of data out of the equation," Hugo says. "In some countries where a couple of customers are most of the traffic, if we limit the amount of data per customer, the global statistics might be too inaccurate because we are lacking data."

The cap protected the integrity of the shared score in theory. In practice, it meant discarding most of the traffic data, and a score built on a small, unrepresentative sample is no more reliable than one built on skewed data. One problem traded for another.

Attempt Two: Separate Rankings Per Customer

The next approach gave each large customer their own dedicated model, built entirely from their own data.

This produced a real, concrete gain. "As we are using only their data, we were finding some carriers that were not delivering because of this registration issue," Hugo says. "The system was automatically using other providers." Isolating one customer's data made their sender ID problem visible for the first time.

But it created a different problem for smaller customers. "The system needs a lot of data to work well," Hugo explains, "and using only the customer's data, if the customer doesn't have enough traffic, the statistics are too small. We are not confident enough on the result. So we were picking the wrong providers."

The problem was real: per-customer data gives you signal but loses volume. Shared data gives you volume but loses signal.

The Live System: Borrow the Data, Bet on the Odds

A customer without much history borrows the crowd's data and leans more on its own as it builds up. Traffic then spreads across providers in proportion to how likely each one is to be the best, rather than going to a single winner.

"We just decided to do some formula to use global statistics," Hugo says, "and the more traffic they gain, the more data they have, and the more we use their data. But we are still using the global one. It's a mix of global data and customer data, and the more data the customer has, the more impact it has on the score."

The transition is continuous, not a threshold. There is no moment where the model switches.

Hugo uses a concrete example: "Let's say a new e-commerce customer is coming on board. The first issue is that their base conversion rates will be lower than our base conversion rates. And the second issue is that as they start, they don't have traffic at all. So we can't rely on their statistics. We will start with the global statistics, and the more traffic they will have, the more of it will be used in a mix between the global data and their data."

The weighting is built around statistical confidence rather than arbitrary tiers. "A provider that has only ten attempts a day, the data is statistically not reliable," Hugo explains. "So the confidence of the data is not strong enough. In the new system, we have built a mathematical formula that computes this confidence. Based on the confidence we will now pick providers. The more data we have, the more confident we will be on the performance of the provider." 

"For smaller customers we still use global statistics," Hugo adds. "So the system works for both of them."

Where the Difference Actually Shows Up for Customers

The most honest description of the day-to-day impact is resilience, not uplift.

"On a daily basis you will not see much improvement," Hugo says. "Maybe a small conversion rate improvement, but it's very small, like half a point maybe. The improvement shows up if there is a technical issue with one provider, a registration one, or something like this, or a fraud attack that is not caught. The system will behave and adapt accordingly."

The August provider outage makes the difference concrete. Prelude was running the old and new systems in parallel, which made the comparison direct. "We had an issue with one provider that was not delivering any messaging," Hugo says, "and the new system is adapting itself way better than the previous one. The system switches from one provider to another more rapidly. The conversion rate is more stable in case of an incident."

Before this, stability required manual work. "When the first version of arbitrage was live, we had some monitoring and we were seeing issues, but the arbitrage was not reacting correctly. So we were closing routes manually to mitigate the impact on conversion rates for our customers but now it's automatic."

Running in the Dark: The Next Piece

One problem remains open: different customers have different baseline conversion rates, independent of which provider carries their traffic.

"Each customer behaves differently," Hugo says. "A fintech app rather than an e-commerce app. So we measure the conversion rates per customer to know what their base conversion rate is, so we can know the impact of each provider independently, without looking at conversion rates only."

The next step is to measure each customer's own baseline, remove it from what that customer observed, and compare providers on what is left. "It's not about volume," Hugo explains. "It's also about how their users behave on their application, because it can affect their login flow and their conversion rates."

This work is ongoing. The architecture to support it is in place.

Hugo Bauer is a software engineer at Prelude, working on verification infrastructure and routing systems.

This is part of Prelude's engineering series, where the team shares how the product gets built. Read more on the blog or get started with Prelude.