Tracking where a WhatsApp user came from

At Amal, we have an AI chat bot that works over WhatsApp — allowing our customers to speak to their bank account, asking questions like “What’s my balance” and “How much have I spent on Uber’s this month?”.

A problem we faced really early on when connecting to WhatsApp was that when a user started a conversation with our bot on WhatsApp, we had no idea where they came from!

This made it impossible for us to customize conversation flows based on the channel the user came from (e.g. if we know they came from an Instagram ad, we really shouldn’t need to ask users “how did you hear about us?”). We also couldn’t know which referral channels worked best for us, across our website, ads and referral partners.

The Problem

The only way users can start a conversation with a business over WhatsApp is through links like https://wa.me/18001231234?text=hi+there; these links only include a phone number, and a text parameter that serve as the default message the user can choose to send[1].

These links will typically be shared on a website, landing page or social media. However, we have no way of knowing where a user came from — since the link we’d share on e.g. Twitter is the same as the one we’d have on our website.

In most messaging platforms (including Facebook’s own messenger), you’d have some sort of ref parameter that you can set to e.g. fb-ad-v0 to track attribution, but WhatsApp did not have a ref param.

In our case, our analytics platform assigns a unique “Device ID” to every user on the website. We want to find a way to send that Device ID to WhatsApp, so we can associate it with the user conversation.

Some Possible Solutions

According to the official documentation, we can only include two pieces on information in the WhatsApp link: the phone number and the text (which is a message that the user can send).

The phone number is constant[2], so our only resort is to somehow encode the Device ID into the text parameter.

Here are some ideas for how to do that:

Option 1

Include it directly in the message and parse it on the incoming webhook (e.g. Hi, my Device ID is abdef-12810a-dac-ee-f312312) which — well, let’s just say it’s not the ideal user experience.

Option 2

Use non printable characters in the message to encode the Device ID based on the position of the non-printables (e.g. paste “hi t‌h‍e‎r‏e” in this tool to view the hidden Unicode characters):

Option 3

Encode it in the string itself. For example, “hi” vs “hello” gives you 1 bit of entropy; “there” vs “,” gives you another bit of entropy. So there are 4 possibilities for the incoming text:

Encoded Text	ID
hi there	00
hi,	01
hello there	10
hello,	11

To achieve 16 bits of entropy (to encode 2^16 = 65k possibilities) we would need a 16 word long message (at 2 variants per word) — which seems like a stretch!

The Solution: Option 2

We can achieve this using non printables!

The beauty of this option is that it can encode alot of data in a small space. Also, if we use multiple possible non-printable characters, we can encode even more results in there!

You just need to make sure that they don’t get filtered on the way to WhatsApp, and eventually to your backend application via the Webhook from your Business API service provider.

With just 4 non-printable character variations, we can encode 5^10 = 10,000,000 sessions (5 because the absence of a character can be used as a bit)

The Code

This all sounds great in theory, but let’s see some code!

# This class handles encoding and decoding of non printable characters
# into strings

class NonPrintableCoder
  # All the characters that properly make it through to WhatsApp, and back
  # The more characters, the better
  NON_PRINTABLES = %W(\u200a \u200b \u200c \u200d \u200e)
  
  class << self
    # We use the same approach commonly used for converting between integer
    # bases.
    def encode(num)
      # e.g. say we want to convert the decimal "13" to binary (base 2). 
      # The steps are as follows:
      #
      # 13 % 2 = 1, so the output is ???1
      # 13 / 2 = 6.5 ≈ 6
      #  6 % 2 = 0, so the output is ??01
      #  6 / 2 = 3
      #  3 % 2 = 1, so the output is ?101
      #  3 / 2 = 1.5 ≈ 1
      #  1 % 2 = 1, so the output is 1101
      #  1 / 2 = 0.5 ≈ 0 ==> ⏹ STOP
      # 
      # From the above, it's clear that we form the output from the right,
      # i.e. starting with the least significant bit. We follow the same
      # approach when encoding our output, with a base that is equal to the
      # length of NON_PRINTABLES
      # 
      # More generally, we form the encoded form by looping over the input,
      # picking the non printable based on the remainder (output of % op),
      # and dividing the number until the result is 0.
      base = NON_PRINTABLES.length
      output = ""

      while(num > 0) do
        output = NON_PRINTABLES[num % base] + output # MSB -> LSB
        num /= base
      end

      output
    end

    def decode(str)
      base = NON_PRINTABLES.length
      output = 0

      str.length.times do |i|
        chr = str[str.length - i - 1]
        output += NON_PRINTABLES.index(chr) * base ** i
      end

      output
    end
  end
end

Which you can then use like so:

# Generate link (encode '12345' into text string)
"https://wa.me/..?text=" + NonPrintableCoder.encode(12345) + "hi there!"

# Incoming webhook (extract '12345' from text string)
non_printables = msg.split("").filter do |char|
  char.in? NonPrintableCoder::NON_PRINTABLES
end
NonPrintableCoder.decode(non_printables) # => 12345

Works like a charm! ✨

👣 Footnotes

[1] The user has the option to edit the message being sent, which makes Solutions 1 and 3 less desirable than Solution 2. While the user could easily delete the non printable characters too, it’s less likely they will if they are prepended to the message (as they will not be visible)

[2] Technically, you could have multiple phone numbers — and encode a few bits that way. However, this would add complexity to your backend since you’d now need to manage multiple points of ingress (across webhooks). It would also become ALOT more expensive since the WhatsApp Business Partner we use charges by number.