SMS testing is a challenge regardless of location, device, or network- due to inaccurate reporting and regional regulations it’s almost impossible to consistently and accurately generate reports. DLR, or Delivery Reporting, is a feature of SMS message termination that reports back if the SMS message has been delivered. Typically, this message is sourced directly from the handset.
Telecoms providers can report whether an SMS has been delivered, but with no regulations in reporting, much of the data is inaccurate. To make matters more complex, many of the reports that are generated can only report if the transaction was a success or failure.
In this post, we demonstrate how we can leverage the HeadSpin platform for SMS testing in three high-volume countries: India, China, and Brazil, and discuss how each can be improved.
Active testing means your service sends an SMS to a phone or device that you control and then checks to see if the SMS was received and what aspects of the message changed in the process. However, the scope of active testing is larger than running unit or end-to-end testing. Test SMS messages can be sent ad hoc when setting up a new service, or for detecting delivery issues on a particular carrier or within a specific country. The classic approach to “active testing” is to call your colleague in India and tell him you’re sending a message. Understandably, this approach doesn’t scale.
Active testing is the best tool for determining if a message sent via a provider will arrive at the destination looking like it is expected to. Active testing can identify many of the following issues:
With HeadSpin, you can identify all kinds of telecom pitfalls from the user’s perspective including:
HeadSpin’s POP, or Point of Presence, phones allow any support or engineering team to have a physical presence across the globe.
Examples of failures identified with HeadSpin and “active testing” include:
Let’s see how a Leading CPaaS provider’s general (CPaaS SMS) and turnkey (CPaaS Turnkey) solutions work out of the box to deliver in three high-volume countries – India, China, and Brazil. India has a heavily regulated telecom system with many rules around when and what you can send. China, on the other hand, is constrained by the great firewall which could, at any time, suspend or delay messages. China is also quite draconian if you send messages with illicit content. Doing so may very well result in the suspension or loss of your SMS sending privileges. Brazil has historically had issues with carriers and Unicode making it another fun country to experiment with.
We’ll use the CPaaS Turnkey API’s default configuration to send OTPs. For India, this means it should deliver to recipients via a registered Alpha Sender ID. For Programmable SMS, we’ll simply register a US long code with CPaaS provider to send directly to the HeadSpin POP handsets.
Once we’ve found a phone to test with, the very first step is to simply text your own phone as a reality check. Honestly, it’s pretty amazing to initiate an SMS from the other side of the world and watch it show up to your own local phone within seconds. This first step just demonstrates that everything is working.
The intent of this testing is to see the real-world SMS experience a user has on their phone in each location. We will record each data point as below:
A Note 5 Pro device on HeadSpin’s public cloud is depicted below. This is one of 3 devices we’re using to test telephony delivery in India. Each device uses a different carrier network and supports both automated and manual QA testing via any testing framework.
Mobile Number+91XXXXX+91XXXXX+91XXXXXDeviceiPhone 6iPhone 11XIAOMI Redmi Note 5 ProLocationBangaloreBangaloreBangaloreCarrierAir TelJioVodafoneIndia to USNo lagNo lagNo lagCPaaS SMS to IndiaDNADNADNACPaas Turnkey > IndiaNo lagNo lagNo lagExpected Body?YesYesYesUnicode Issues?NoNoNoDelivered By?57575565757556 / BZ-AUTHMS503501 / 5676735LagNoNoLittle
Admittedly, this is a pretty small sample size but it gives you a general idea of what kind of experience a user can have with different carriers in each location.
The most obvious issue here is the non-delivery of SMSs to India via the CPaaS SMS API. This is expected behavior as delivery to India typically requires jumping through several technological and bureaucratic hoops.
The real winner in this small sample size is the CPaaS Turnkey. Obviously, they have nailed their implementation as everything was delivered in a timely manner. The interesting observation is that out of the six attempts, it seems only one was delivered via the Alpha Sender ID registered in India. This could be a side-effect of the intelligent retry logic or simply Indian carrier mischief.
For testing in China, we focused on two major metro areas, Shanghai and Beijing. You’ll also note that each phone used a completely different carrier. DNA denotes, did not arrive.
Mobile Number+86XXXXX+86XXXXX+86XXXXXDeviceiPhone 7XIAOMI Mi A2iPhone 7LocationBeijingShanghaiShanghaiCarrierChina MobileChina UnicomChina TelecomChina to USZero LagZero LagZero LagCPaaS SMS to ChinaDNAMinimal LagMinimal LagExpected Body?DNAYesYesUnicode Issues?DNAVery FastVery FastDelivered By?DNA1306764314513136240203LagDNANoneFew secondsMultibody?DNASupportedSupportedCPaaS Turnkey > ChinaDNADNADNA
Testing with HeadSpin in China resulted in the complete opposite experience from India. In this case, 33% of CPaas SMS failed to arrive, and NO CPaaS Turnkey messages sent to China arrive. Like most Voice or SMS API calls to phones terminating in China, misconfiguration or mischief by the Chinese firewall is surely to blame.
Interestingly enough, the message sent from China to the US arrived quicker than any other test.
Testing in Brazil was limited to a single carrier network. Overall, Brazil demonstrated the best performance across devices. The carrier behavior did not vary between phones and provided the fastest experience across all three test countries.
Mobile Number+55XXXXX+55XXXXX+55XXXXXDeviceiPhone XSiPhone 6siPhone 7LocationSão PauloSão PauloSão PauloCarrierVivoVivoVivoIndia to USNo lagNo lagNo lagCPaaS SMS to BrazilNo lagNo lagNo lagExpected Body?YesYesYesUnicode Issues?NoNoNoDelivered By?271992719927199LagNoNoNoCPaaS Turnkey > BrazilNo lagNo lagNo lagExpected Body?YesYesYesUnicode Issues?NoNoNoDelivered By?271992719927199LagNoNoNo
Testing in Brazil seems to have provided the best results out of the three countries. At the time of testing, there were limited Brazilian carriers to test with but on a positive note, the experience was exactly the same across each device.
Working with telecom is a fickle beast. When talking with customers, I frequently compare sending an SMS to sending a UDP packet. With both of these technologies, you initiate a send and the data is off! But, you never know if it has gotten to its destination. Any number of issues could affect the delivery.
Another way to think of SMS telephony is to consider hiring a container ship to deliver goods to a port of call. You can hire a cheaper company that takes the long route around South America or the more expensive company which goes through the Panama Canal. It is this latter route that has a far higher likelihood of arriving on time at the port of call. Your cheaper company’s container ship may end up at the bottom of the sea while rounding Cape Horn.
As a mobile developer, you’ll have a clear understanding of how your mobile experience will play when leveraging HeadSpin’s robust suite of testing and performance monitoring solutions. You can take immediate control of a phone anywhere in the world with our Global Device Cloud and test to your heart’s content. Looking for a more robust long term solution? Leverage your own private cloud for automated testing or setup hourly, daily, weekly testing and benchmark reporting to identify trends and issues around your KPIs.
HeadSpin can help you automate functional, performance, and load testing across applications, devices, and networks to optimize digital experiences enabled by mobile, web, IoT, and 5G. Want to give HeadSpin a try? We have a Point of Presence in more than 90+ locations, consisting of thousands of devices and endpoints. Reach out via our Contact Us form and we will coordinate a discussion and demo.
Introducing HeadSpin University
Active testing is the testing process where the tester interacts with the software and directly executes the test. It usually verifies that the critical parts of the application are working as intended. Some examples of active testing include functional testing and structural testing.
Passive testing is finding issues and bugs in the software without the tester interacting with the application, e.g., extracting and analyzing the application logs. Passive testing is practical when the regular run of the application cannot be interrupted, or the QA team does not have access to the application interface.
API testing validates an API’s functionality, security, robustness, performance, and reliability. Automated testing of APIs is recommended as automated testing can identify risks and security vulnerabilities of APIs, which manual testing can easily miss.
User acceptance testing is usually close to the final stages of SDLC and done close to deployment. The customer or the end-user typically performs it once the development team is reasonably sure that the application is bug-free.
-1280X720-Final-2.jpg)
