Motivation.

There are a number of scenarios where we might want to send an SMS message to someone. And there are numerous services tailored specifically to each scenario. Some services can be extended to cover most. And not all services work globally, so what is available in your region may be very different than what is available somewhere else. Here are a few scenarios that come to mind.

• User Authentication. If a project offers users the ability to log in with a username/password, then we'd also like to offer the ability to recover a lost or forgotten password. But we need an external way to authenticate the user. E-mail notifications are good for this, browser notifications are a little less so. However, using SMS is increasingly popular as it tends to be more reliable and more secure than e-mail. It is less likely that the message will be dropped in a junk or spam folder, for example.
  
• Phone Number Validation. One way to validate a phone number is to send an SMS message to it. Some providers, including Twilio, have an extensive voice API available as well, but as mobile devices have proliferated everywhere, this has become a valid option in many cases.
• MFA. Multi-factor authentication is also a popular feature these days. Being able to automatically send an SMS to someone logging in might help streamline this process in environments calling for higher security. Some vendors offer more complete solutions in this space, like Twilio's Verify service, naturally at an additional cost.
  
• Notifications. Rather than get an e-mail when some event has transpired, an SMS can work just as well. MMS messages (text messages that include multimedia) are also typically available using the same APIs, also usually at a slightly higher cost.
• Customer Service. SMS messages can be responded to. A chat interface can be built that allows customer service representatives to collectively respond to such messages through a web-based interface, where incoming messages can be routed to specific CSRs, logged, or managed in other ways.
  

Plenty to work with here. As money is involved, typically, there are provisions for API keys, message authentication, and all sorts of callback mechanisms. Some of this is implemented with simple REST API calls. Some require a little bit more effort to sort through.

Sample Project.

For the sample project today, we're going to make some additions to the Template Project from a little while ago (GitHub repository). This is a TMS XData and TMS WEB Core project that uses the AdminLTE template for much of its client interface. We had extended it previously when we looked at ChatGPT, linking a chat interface with OpenAI's REST API for a handful of their AI models.

As with e-mail notifications, there are solid reasons to offload much of the work to XData. The most important reason, though, is to protect the API keys that we'll be using by hiding them behind our own REST API so nobody else can access them. But to start with, before we do anything with TMS XData, we first have to get set up with a Twilio account.

Twilio Programmable Messaging.

Signing up for Twilio doesn't cost anything initially, and the trial account comes with a bit of credit and access to everything we need to get our app up and running. The first thing we'll need is a Twilio Phone Number. Only one phone number is available to trial accounts, which is enough for now. After leaving the trial, multiple phone numbers can be purchased for around $1/month. These can be numbers that are specific to a region and can be linked to different messaging services or grouped together, depending on what it is you're trying to do.

Next, we'll need to create a Twilio Messaging Service. This just needs a name and some idea as to what you're using it for. In our case, we're going with "Notify Users". The service we're setting up is destined for a public demo version of the Template project, called 500 Dashboards (https://www.500dashboards.com) so that's what we'll call the service here.

Twilio Messaging Service Setup.

The next part is a little more involved. While sending an SMS doesn't require anything on our part other than a REST API call, receiving an SMS, and receiving additional information about any SMS we send, makes use of webhook callbacks.

What are those? Well, when Twilio wants to let us know about anything, they make an HTTPS POST request to a website or URL of our choosing, passing it parameters that include the details of whatever it is they want to tell us. This may be a delivery confirmation message of some kind, a notification of an incoming call or SMS message, or something else. We supply the URL that they will contact, and we have options for HttpGet or HttpPost. We're going to use HttpPost (the default for XData) as it might work better for some situations.

This is where XData comes in. We can set up an XData server to receive and handle these requests. And, conveniently, the Template project already makes extensive use of an XData server. But before we get to that, we first have to configure the callback URL in Twilio. Once we've added the new Twilio Messaging Service, we can specify the URLs for both a callback endpoint and a fallback endpoint, used if the callback endpoint doesn't respond in a timely fashion.

Twilio Integration Page.

We can look at the Twilio Programmable Messaging API to figure out what exactly is going to be sent over this webhook, but that might be quite a lot to define in terms of XData parameters. And as we'll see later, parameters can also change based on various factors entirely outside of our control. There are numerous references in the documentation relating to how the list of parameters can change at any time. Defining the list of parameters explicitly in our XData endpoint is likely not such a great idea. Let's try something a little more generic.

As a bit of a side adventure, we'd like the callback URL we're providing to point directly at our own XData endpoints. But, typically, our XData endpoints are not necessarily running on standard web ports (like 80 (gasp!) or 443). Rather, they are configured to run on their own custom port numbers, especially if running multiple XData servers (and other web servers) on the same server, virtual or otherwise. And Twilio won't let us enter a URL with a port number for whatever reason. But that's ok, we can improvise.

In my environment, XData servers run on one system (Windows), and TMS WEB Core apps are served up on a separate (Ubuntu) system. This is because I've not gotten around to building Linux XData applications (yet!) but also because I find it handy to have XData (VCL-based) running as Windows apps that I can interact with. The Linux system I use (deliberately) hasn't been configured with a desktop interface. To serve TMS WEB Core apps, I use Apache. What I've done in this situation is configure Apache to provide a proxy connection to the other system, which includes the port number and URL in the proxy. It works like this.

Ubuntu system running Apache and TMS WEB Core app (example): www.tmswebcore.com
Windows system running TMS XData app (example): www.tmsxdata.com:2468/tms/xdata

To run the TMS WEB Core app, I'd enter "https://www.tmswebcore.com" into a browser. To get access to Swagger (part of the XData app), I would enter "https://www.tmsxdata.com:2468/tms/xdata/swaggerui" into a browser. Let's add this proxy rule to the Apache configuration on the Windows system.

  SSLProxyEngine on
  ProxyPass /data/ https://www.tmsxdata.com:2468/tms/xdata/
  ProxyPassReverse /data/ https://www.tmsxdata.com:2468/tms/xdata/

After restarting Apache, Swagger can now be accessed using "https://www.tmswebcore.com/data/swaggerui". How handy is that? Pretty darn handy, I'd say. For our webhook callback URLs, we can then enter a normal-looking URL into their system and have it redirected to our XData server, no matter where it happens to live. Note that in this example, both hosts have domain names and SSL certificates, so we need the SSLProxyEngine command.

Webhooks with XData.

The idea with webhooks is that some external system (Twilio in this case) will make an HTTPS request of some kind whenever they like, and our XData server will need to process that request. Sometimes a response is required, like an acknowledgment. Sometimes an action needs to be taken, like sending an incoming SMS to a client web app used by a CSR, or logging a delivery notification against an SMS message that has been sent previously.

Normally, when we configure XData service endpoints, we set up a new "service" and then add endpoints as functions with parameters of various flavors. In this case, we don't know (or, really, we don't want to know) in advance what the parameters will be. Well, we do, but we don't want to have to specify them in the XData interface directly. This is because there may be many parameters, and they may change based on whatever it is we're being contacted about, or because Twilio decides one day to change something.

Instead, what we can do is receive whatever is coming our way into a TStream, and then parse that and figure out what to do with it. This simplifies the XData interface, but really it's just kicking the ball down the road a short way. Let's define the interface for our webhook callback endpoint like this, complete with the Swagger documentation.

    ///  <summary>
    ///    Twilio Webhook Callback
    ///  </summary>
    ///  <remarks>
    ///    After sending a message, Twilio issues a webhook callback to here
    ///    with the details of the transaction. Incoming messages arrive here
    ///    as well.
    ///  </remarks>
    ///  <param name="Incoming">
    ///    This is the body of the Twilio message which may contain a number
    ///    of URL-encoded parameters.
    ///  </param>
    [HttpPost] function Callback(Incoming: TStream):TStream;

• Figure out what kind of callback it is (new SMS, delivery confirmation, etc.).
• Validate the message (did it really come from Twilio?).
• Respond to the message (send back an acknowledgment).
• Deal with the message itself (log to a database, send to the client, ignore, etc.).

First, we need to get the incoming data out of the stream and decode it into something we can use. The parameters come in as a URL-encoded string, so we can first decode it, and then split the string based on the & delimiter. For the most part, this gets us the parameters easily enough. In some cases, the parameter value may contain JSON that we can then further process.

In order to send back a response, Twilio is expecting it to be in a specific format. Initially, at least, we'll just return an empty response in this format. An empty response can be represented as "<Response></Response>". Sending this back will allow the transaction to complete successfully. If we leave it out, Twilio will think their request has failed and will then send the same request to the fallback URL, which we would rather avoid.

Here's the start of the callback implementation.

function TMessagingService.Callback(Incoming: TStream):TStream;
var
  i: Integer;
  Request: TStringList;
  Processed: TStringList;
  Response: TStringList;
  Body: String;
  AddOns: TJSONObject;
begin

  Request := TStringList.Create;
  Request.LoadFromStream(Incoming);

  Processed := TStringList.Create;
  Processed.Delimiter := '&';
  Processed.DelimitedText :=  System.Net.URLClient.TURI.URLDecode(Request.Text);

  MainForm.LogEvent('');
  MainForm.LogEvent('Twilio Message Received:');
  i := 0;
  while i < Processed.Count do
  begin
    // Filter out any junk, like when uploading a file via Swagger
    if Pos('=',Processed[i]) > 0 then
    begin

      // Get the Body of the Message
      if Pos('Body=',Processed[i]) = 1 then
      begin
        Body := Copy(Processed[i],6,Length(Processed[i]));
        MainForm.LogEvent('* BODY: '+Body);
      end

      // Parse JSON of AddOns
      else if Pos('AddOns=',Processed[i]) = 1 then
      begin
        AddOns := TJSONObject.ParseJSONValue(Copy(Processed[i],8,Length(Processed[i]))) as TJSONObject;
        MainForm.LogEvent('* ADDONS: '+AddOns.ToString);
      end

      // Show Other Fields
      else
      begin
        MainForm.LogEvent('- '+Processed[i]);
      end;

    end;
    i := i + 1;
  end;
end;

We can test this from the Twilio website using their "Try this" feature, sending to a local phone number (an iPhone in this case). This will generate a callback request from Twilio. Note that the trial account only allows one such phone number to be used. To clarify, that means they provide one phone number on their end to send/receive messages and allow for sending/receiving from that one Twilio-supplied phone number to one other phone number, as part of the trial.

Multiple callback requests may be issued for a single SMS message. And if we reply to the message, more callback requests will likely be issued as well. For example, if we replied with "Perfect, thank you 👍" our endpoint might output something like the following.

2023-11-16 17:33:14.034  Twilio Message Received:
2023-11-16 17:33:14.056  - ToCountry=US
2023-11-16 17:33:14.063  - ToState=AL
2023-11-16 17:33:14.069  - SmsMessageSid=SM566edc5db02608fee594cc30d019XXXX
2023-11-16 17:33:14.075  - NumMedia=0
2023-11-16 17:33:14.083  - ToCity=NOTASULGA
2023-11-16 17:33:14.090  - FromZip=
2023-11-16 17:33:14.096  - SmsSid=SM566edc5db02608fee594cc30d01XXXX
2023-11-16 17:33:14.103  - FromState=BC
2023-11-16 17:33:14.109  - SmsStatus=received
2023-11-16 17:33:14.116  - FromCity=VANCOUVER
2023-11-16 17:33:14.131  * BODY: Perfect,+thank+you+👍
2023-11-16 17:33:14.251  - FromCountry=CA
2023-11-16 17:33:14.271  - To=%2B133XXXX
2023-11-16 17:33:14.283  - MessagingServiceSid=MGec8ddf1c3187241921577f73f59eXXXX
2023-11-16 17:33:14.291  - ToZip=36866
2023-11-16 17:33:14.314  * ADDONS: {"status":"successful","message":null,"code":null,"results":{"twilio_carrier_info":{"request_sid":"XRe34b599e6b4dec5ff89801a5d1c9XXXX","status":"successful","message":null,"code":null,"result":{"phone_number":"+1604505XXXX","carrier":{"mobile_country_code":"302","mobile_network_code":null,"name":null,"type":null,"error_code":60601}}}}}
2023-11-16 17:33:14.321  - NumSegments=1
2023-11-16 17:33:14.328  - MessageSid=SM566edc5db02608fee594cc30d01XXXX
2023-11-16 17:33:14.336  - AccountSid=AC5346342f844fc5b7dd41412ed8ecXXXX
2023-11-16 17:33:14.345  - From=%2B1604505XXXX
2023-11-16 17:33:14.357  - ApiVersion=2010-04-01

That's a lot of extra stuff for such a short message. But we can see that there's everything we might need to lookup either the sender or receiver of the message, as well as of course the message itself. Note also that in this example, we used an older URL-decoding function (System.Net.URLClient.TURI.URLDecode) that didn't quite work as well as it could have. In later revisions, this was updated to a newer URL-decoding function (TNetEncoding.URL.Decode) that worked much better.

In this example, there is also an "AddOns" parameter that has a value provided as JSON. There are a number of such add-ons available, usually with an extra fee associated with each transaction, that can perform other tasks. In this case, the "Twilio Carrier Information" and "Twilio Caller Name" were used, but didn't actually return much useful information. Other plugins can pull in additional information from other sources, to validate, filter, or categorize this kind of data. For example, there is an add-on that will add company information such as owner, address, social media links, and so on to incoming messages.

Logging Messages.

For this endpoint, let's just log these parameters in the database as best we can for the time being. We'll need a table for this, of course, so we'll create a "Messaging" table with the columns above, along with some others that appeared with other callback requests.

There are a couple of wrinkles here. First, we can see the names of the parameters easily enough, but they don't necessarily work as table column names - some will have to be changed. Also, we can't really make assumptions as far as whether there will be a perfect matching between parameters and columns - Twilio explicitly states that the list of parameters may change at any time. So our table doesn't really make any assumptions about this either, and just tries to store what it knows about, ignoring the rest.

Here's our table definition for SQLite, using the same conventions as all of the other tables in the Template project. In this case, though, all the fields are text fields. This is partly due to being lazy, partly due to not knowing what to expect, and partly due to anticipating that different messaging vendors are likely going to do their own thing, so text fields are used as the lowest common denominator.

// [table] messaging

TableName := 'messaging';

if (DatabaseEngine = 'sqlite') then
begin

  with Query1 do
  begin

    // Check if the table exists
    SQL.Clear;
    SQL.Add('select count(*) records from '+TableName+';');
    try
      Open;
      LogEvent('...'+TableName+' ('+IntToStr(FieldByName('records').AsInteger)+' records)');

    except on E:Exception do
      begin
        LogEvent('...'+TableName+' (CREATE)');
        SQL.Clear;
        SQL.Add('create table if not exists '+TableName+' ( '+
                '  service                    text      NOT NULL, '+
                '  created_at                 text, '+
                '  MessageStatus              text, '+
                '  ErrorMessage               text, '+
                '  ErrorCode                  text, '+
                '  Direction                  text, '+
                '  RawDlrDoneDate             text, '+
                '  SmsStatus                  text, '+
                '  SmsSid                     text, '+
                '  SmsMessageSid              text, '+
                '  AccountSid                 text, '+
                '  MessageSid                 text, '+
                '  MessagingServiceSid        text, '+
                '  Body                       text, '+
                '  ToNum                      text, '+
                '  ToCountry                  text, '+
                '  ToState                    text, '+
                '  ToCity                     text, '+
                '  ToZip                      text, '+
                '  FromNum                    text, '+
                '  FromCountry                text, '+
                '  FromState                  text, '+
                '  FromCity                   text, '+
                '  FromZip                    text, '+
                '  NumSegments                text, '+
                '  NumMedia                   text, '+
                '  Price                      text, '+
                '  PriceUnit                  text, '+
                '  AddOns                     text, '+
                '  Uri                        text, '+
                '  Resource                   text, '+
                '  ApiVersion                 text '+
                ');'
               );
        ExecSQL;

        // Try it again
        SQL.Clear;
        SQL.Add('select count(*) records from '+TableName+';');
        Open;
      end;
    end;
  end;
end;

// [query] log_messaging

if (MainForm.DatabaseEngine = 'sqlite') then
begin

  with Query1 do
  begin

    SQL.Clear;
    SQL.Add('insert into '+
            '  messaging '+
            '    (service, created_at, MessageStatus, ErrorMessage, ErrorCode, Direction, RawDlrDoneDate, SmsStatus, SmsSid, SmsMessageSid, AccountSid, MessageSid, MessagingServiceSid, Body, '+
            '     ToNum, ToCountry, ToState, ToCity, ToZip,'+
            '     FromNum, FromCountry, FromState, FromCity, FromZip, '+
            '     NumSegments, NumMedia, Price, PriceUnit, AddOns, Uri, Resource, ApiVersion )'+
            'values( '+
            '  :service, '+
            '  Datetime("now"), '+
            '  :MessageStatus, '+
            '  :ErrorMessage, '+
            '  :ErrorCode, '+
            '  :Direction, '+
            '  :RawDlrDoneDate, '+
            '  :SmsStatus, '+
            '  :SmsSid, '+
            '  :SmsMessageSid, '+
            '  :AccountSid, '+
            '  :MessageSid, '+
            '  :MessagingServiceSid, '+
            '  :Body, '+
            '  :ToNum, '+
            '  :ToCountry, '+
            '  :ToState, '+
            '  :ToCity, '+
            '  :ToZip, '+
            '  :FromNum, '+
            '  :FromCountry, '+
            '  :FromState, '+
            '  :FromCity, '+
            '  :FromZip, '+
            '  :NumSegments, '+
            '  :NumMedia, '+
            '  :Price, '+
            '  :PriceUnits, '+
            '  :AddOns, '+
            '  :Uri, '+
            '  :Resource, '+
            '  :ApiVersion '+
            ');'
           );

  end;
end;

In our endpoint, we'll have to then add the parameters we know about, and make a note of parameters we don't know about. We can then log everything we receive into the database. All we're after at this stage is to capture the data. Here's the gist of what that looks like.

  // Use this query to log messages. Set parameters to null string to start
  {$Include sql\messaging\messaging\log_message.inc}
  Query1.ParamByName('service').AsString := ServiceName;
  for i := 1 to Query1.ParamCount - 1 do
    Query1.Params[i].AsString := '';

  i := 0;
  while i < Processed.Count do
  begin
    FieldName := '';
    FieldValue := '';
    if Pos('=', Processed[i]) > 0 then
    begin
      FieldName := Copy(Processed[i], 1, Pos('=', Processed[i]) - 1);
      FieldValue := TNetEncoding.URL.Decode(Copy(Processed[i], Pos('=', Processed[i]) + 1, Length(Processed[i])));
    end;

    // Filter out any junk, like when uploading a file via Swagger
    if FieldName <> '' then
    begin
      // Parse JSON of AddOns
      if FieldName = 'AddOns' then
      begin
        AddOns := TJSONObject.ParseJSONValue(FieldValue) as TJSONObject;
        Query1.ParamByName('AddOns').AsString := Addons.toString;
      end

      // To= is assigned to ToNum field
      else if FieldName = 'To' then
      begin
        Query1.ParamByName('ToNum').AsString := FieldValue;
      end

      // From= is assigned to FromNum field
      else if FieldName = 'From' then
      begin
        Query1.ParamByName('FromNum').AsString := FieldValue;
      end

      // Process Other Fields
      else
      begin
        if Query1.Params.FindParam(FieldName) <> nil then
        begin
          Query1.ParamByName(FieldName).AsString := FieldValue;
        end
        else
        begin
          MainForm.LogEvent('WARNING: Message Received with Unexpected Field: ');
          MainForm.LogEvent('[ '+Processed[i]+ ' ]');
        end;
      end;

    end;
    i := i + 1;
  end;

  // Log the callback request
  try
    Query1.ExecSQL;
  except on E: Exception do
    begin
      MainForm.LogException('MessagingService.Callback: Log Query', E.ClassName, E.Message, Query1.SQL.Text);
    end;
  end;

  // Returning XML, so flag it as such
  TXDataOperationContext.Current.Response.Headers.SetValue('content-type', 'text/xml');

  // Return an empty, but valid, response
  Result := TMemoryStream.Create;
  Response := TStringList.Create;
  Response.Add('<Response></Response>');
  Response.SaveToStream(Result);

There's more going on in the final implementation, particularly related to logging the endpoint activity itself (something we do in all the Template endpoints) and general connection build-up and tear-down, releasing objects, and other administrative items.

Authenticating Messages.

For our next little side adventure, let's quickly look at how to authenticate these messages. Twilio has a comprehensive security document that describes how this is done. Here's a breakdown of what is involved.

• Callback messages come with a signature in the request header, "x-twilio-signature".
• To verify the signature, we need to produce a matching HMAC-SHA-1 value.
• This uses the Twilio Auth token for the account we're using as the encryption key.
  
• The value is generated first from the specific URL for the callback - the one we entered earlier.
• All the (URL decoded) parameters included with the request are appended to the URL.
• The parameters first need to be sorted, and the "=" is removed from the parameters.
• The output of HMAC-SHA-1 needs to be formatted as a Base64-encoded string.
  

It's not the most fun thing to sort out, and it would've been nice if we could just take the parameters as they were passed, but it isn't much extra work. Generating an HMAC-SHA-1 value is a bit tricky, though, as this produces a 160-bit value using the SHA-1 algorithm. Which is a bit out of date, honestly. The TMS Cryptography pack doesn't offer it, as a result. But we can get by with Indy well enough here. First, we'll need our HMAC-SHA-1 function that we can pass the URL, the encoded parameters, and the key, getting back a Base64-encoded string.

  function GenerateSignature(aURL, aParameterList, aKey: String):String;
  begin
     with TIdHMACSHA1.Create do
     try
       Key := ToBytes(aKey);
       Result := TIdEncoderMIME.EncodeBytes(HashValue(ToBytes(aURL+aParameterList)));
     finally
       Free;
     end;
  end;

The URL and the Key will ultimately come from the configuration JSON for the XData project. Something special is being cooked up for that, coming soon to a blog near you, but for now, we'll assume these are string values that we've gotten our hands on. These are both values that are available from the Twilio website and of course, the URL is one we provided to them, as we covered a bit earlier. The Twilio docs refer to being mindful of trailing "/" for these URLs that might inadvertently get added by the web server. Doesn't seem to be an issue for us here, but your mileage may vary. Here's the rest of the code.

  i := 0;
  SignaturePAR := '';
  while i < Processed.Count do
  begin
    SignaturePAR := SignaturePAR + TNetEncoding.URL.Decode(StringReplace(Processed[i],'=','',[]));
    i := i + 1;
  end;

  Signature := TXdataOperationContext.Current.Request.Headers.Get('x-twilio-signature');
  SignatureURL := 'XData URL goes here';
  SignatureTOK := 'Twilio AUTH token goes here';
  SignatureGEN := GenerateSignature(SignatureURL, SignaturePAR, SignatureTOK);

  // Not authenticated
  if (SignatureGEN <> Signature) then
  begin
    Request.Free;
    Processed.Free;
    raise EXDataHttpUnauthorized.Create('Invalid Signature Detected');
  end;

If the signature match fails, the exception is raised and the code exits at that point. Note that if this is enabled, testing with Swagger won't work very well because (1) we don't have a way to provide the "x-twilio-signature" and (2) the Swagger interface adds a couple of extra parameters that result in a signature that doesn't match anyway (the "input" and "filename" parameters). However, we can comment out that section of the code (the signature comparison) when testing everything. For example, by uploading an example of the Twilio body, we can run through the rest of the function without having to constantly send messages from Twilio directly.

In order to test the HMAC-SHA-1 calculations, that's precisely what was done. The "Processed.Text" value was copied from an actual Twilio request, along with the URL and the Twilio Auth token, and then run through an online HMAC calculator that produced a Base64-encoded string that could be compared with the "x-Twilio-signature".

https://dinochiesa.github.io/hmachash/index.html

Sending Messages.

With all that business sorted, we can now have a look at how to send messages ourselves rather than using the Twilio website. All we're doing here is making a REST API call with some of the same tokens and codes we used previously. If successful, we'll get a response that we can similarly log into the database.

This is ideally a quick process - we request that a text message be sent, and Twilio responds with the details of that request. Subsequent callbacks then report on the ultimate disposition of that request - whether the message was delivered and so on, potentially with several such callbacks (or fallbacks) being issued. So while our initial request here in the endpoint isn't crafted as an async request, it works very much as an async process overall.

Unlike the callback/fallback situation, our XData endpoints for sending or retrieving messages are most definitely going to be protected with our usual JWT mechanism. We don't want just anyone sending messages through our system, after all. To get a message on its way, we'll need five pieces of information.

1. Twilio Account Number.
2. Twilio Auth Token.
3. Twilio Messaging Service identifier.
4. Destination phone number.
5. Message to send.

The first three values are available from the Twilio website. We used the Auth Token in the callback to verify the signature, but the Account number could be passed in using the same configuration JSON. We're going to configure the Messaging Service as a parameter to our endpoint, however.

We might have several Twilio Messaging Services configured. For example, one for "Customer Service", one for "Accounts and Billing" and one for "Notifications". Think of these as having separate phone numbers, though they could have separate banks of phone numbers spread around the globe.

A CSR (working for us) might be tasked with responding to requests coming in on a particular phone number. Whereas notifications that are automated might go out on a separate number. Our configuration JSON might define any number of such Messaging Services. That JSON is starting to get more complicated!

Alternatively, we could also pass the Account number as a value, if perhaps we wanted to support having multiple different Twilio accounts (each with their own Auth Tokens as well, but we'd not want to pass those around) served by the same XData server. Lots of options. But with all the pieces available, our SendAMessage endpoint looks something like this.

function TMessagingService.SendAMessage(MessageService: String; Destination: String; AMessage: String):TStream;
var  
  ServiceName: String;
  AccountNumber: String;
  AuthToken: String;
 
  DBConn: TFDConnection;
  Query1: TFDQuery;
  DatabaseName: String;
  DatabaseEngine: String;
  ElapsedTime: TDateTime;
  User: IUserIdentity;
  JWT: String;
 
  Client: TNetHTTPClient;
  Request: TMultipartFormData;
  Response: String;
  Reply: TStringList;
 
  i: integer;
  ResultJSON: TJSONObject;
 
  procedure AddValue(ResponseValue: String; TableValue: String);
  begin
    if (ResultJSON.GetValue(ResponseValue) <> nil) and
    not(ResultJSON.GetValue(ResponseValue) is TJSONNULL) then
    begin
      if (ResultJSON.GetValue(ResponseValue) is TJSONString)
      then Query1.ParamByName(TableValue).AsString  := (ResultJSON.GetValue(ResponseValue) as TJSONString).Value
      else if (ResultJSON.GetValue(ResponseValue) is TJSONObject)
      then Query1.ParamByName(TableValue).AsString  := (ResultJSON.GetValue(ResponseValue) as TJSONObject).ToString
      else if (ResultJSON.GetValue(ResponseValue) is TJSONNUMBER)
      then Query1.ParamByName(TableValue).AsString  := FloatToStr((ResultJSON.GetValue(ResponseValue) as TJSONNumber).asDouble)
    end;
  end;
 
begin
  // Time this event
  ElapsedTime := Now;

  ServiceName := 'Twilio Messaging';
  AccountNumber := 'AC5346342f844fc5b7dd41412ed8ecab83';
  AuthToken := 'df21f3f76799992f8b52164021b87004';
 
  // Get data from the JWT
  User := TXDataOperationContext.Current.Request.User;
  JWT := TXDataOperationContext.Current.Request.Headers.Get('Authorization');
  if (User = nil) then raise EXDataHttpUnauthorized.Create('Missing authentication');

  // Setup DB connection and query
  try
    DatabaseName := User.Claims.Find('dbn').AsString;
    DatabaseEngine := User.Claims.Find('dbe').AsString;
    DBSupport.ConnectQuery(DBConn, Query1, DatabaseName, DatabaseEngine);
  except on E: Exception do
    begin
      MainForm.mmInfo.Lines.Add('['+E.Classname+'] '+E.Message);
      raise EXDataHttpUnauthorized.Create('Internal Error: M-SAM-CQ');
    end;
  end;

  // Prepare the Connection
  Client := TNetHTTPClient.Create(nil);
  client.Asynchronous := False;
  Client.ConnectionTimeout := 30000; // 30 secs
  Client.ResponseTimeout := 30000; // 30 secs
  Client.CustomHeaders['Authorization'] := 'Basic '+TIdEncoderMIME.EncodeBytes(ToBytes(AccountNumber+':'+AuthToken));
  Client.SecureProtocols := [THTTPSecureProtocol.SSL3, THTTPSecureProtocol.TLS12];
 
  // Prepare the Request
  Request := TMultipartFormData.Create();
  Request.AddField('To',Destination);
  Request.AddField('MessagingServiceSid', MessageService);
  Request.AddField('Body', AMessage);

  // Submit Request
  try
    Response := Client.Post(
        'https://api.twilio.com/2010-04-01/Accounts/'+AccountNumber+'/Messages.json',
        Request
      ).ContentAsString(TEncoding.UTF8);

  except on E: Exception do
    begin
      MainForm.LogException('Twilio Send Message',E.ClassName, E.Message, Destination);
    end;
  end;

  // Prepare Response
  ResultJSON := TJSONObject.ParseJSONValue(Response) as TJSONObject;

  // Use this query to log messages. Set parameters to null string to start
  {$Include sql\messaging\messaging\log_message.inc}
  Query1.ParamByName('service').AsString := ServiceName;
  for i := 1 to Query1.ParamCount - 1 do
    Query1.Params[i].AsString := '';
 
  // Pick out the values from the Response JSON that match our Messaging table columns as best we can - the incoming JSON can change at any time!
  try
    AddValue('status',                'SmsStatus'           );
    AddValue('error_message',         'ErrorMessage'        );
    AddValue('error_code',            'ErrorCode'           );
    AddValue('direction',             'Direction'           );
    AddValue('account_sid',           'AccountSid'          );
    AddValue('sid',                   'SmsSid'              );
    AddValue('sid',                   'MessageSid'          );
    AddValue('messaging_service_sid', 'MessagingServiceSid' );
    AddValue('body',                  'Body'                );
    AddValue('to',                    'ToNum'               );
    AddValue('from',                  'FromNum'             );
    AddValue('num_segments',          'NumSegments'         );
    AddValue('num_media',             'NumMedia'            );
    AddValue('price',                 'Price'               );
    AddValue('price_unit',            'PriceUnit'           );
    AddValue('uri',                   'Uri'                 );
    AddValue('subresource_uris',      'Resource'            );
    AddValue('api_version',           'ApiVersion'          );
  except on E: Exception do
    begin
      MainForm.LogException('MessagingService.SendAMessage: Log Query Pop', E.ClassName, E.Message, Query1.SQL.Text);
    end;
  end;
 
  // Log the send response (which includes the original message)
  try
    Query1.ExecSQL;
  except on E: Exception do
    begin
      MainForm.LogException('MessagingService.SendAMessage: Log Query', E.ClassName, E.Message, Query1.SQL.Text);
    end;
  end;

  // Send back a response
  Result := TMemoryStream.Create;
  Reply := TStringList.Create;
  Reply.Add(ResultJSON.ToString);
  Reply.SaveToStream(Result);
  
  // Cleanup
  Request.Free;
  Reply.Free;
  ResultJSON.Free;

  // All Done
  try
    DBSupport.DisconnectQuery(DBConn, Query1);
  except on E: Exception do
    begin
      MainForm.mmInfo.Lines.Add('['+E.Classname+'] '+E.Message);
      raise EXDataHttpUnauthorized.Create('Internal Error: M-SAM-DQ');
    end;
  end;
end;

But with this in place, notification messages are ready to be sent. The Twilio trial account can only send to one number, but that's enough to test that it is working. The [authorize] attribute applies to this endpoint, so anyone logged in can make use of this feature, sending notifications from a TMS WEB Core web app or any other system that has been provided with a JWT.

JSON Configuration.

Before we finish up, one more item of business. As with e-mail notifications, the Template XData server has been configured to get information about what messaging services are available to it by way of its configuration.json file.

This is where we can store the Twilio Auth token and other information that doesn't really change all that often, and that we want to be secured and completely inaccessible to the client. This is also where we can configure the individual messaging services that we can use to determine what the outbound SMS phone number to use is (or bank of phone numbers, if so configured). 

If we were to build out a fully configured SMS client app (coming soon...!) then we could use this list of messaging services in the client as well. With that in mind, we'll give those a friendly name (just as they have done on the Twilio website) and potentially make those available to certain accounts. At some point, access controls would likely be needed, controlling which user accounts can use a particular messaging service. Maybe John has access to the "accounts" messages, and Jane has access to the "customer service" messages, for example.

In any event, let's update our configuration JSON to include the following stanza. Here we're providing details for Twilio, but if we were to add support for other vendors, we could add extra stanzas with whatever options are specific to them as well.

  "Messaging Services":{
    "Twilio": {
      "Service Name":"Twilio Messaging",
      "Account": "AC5346342f844fc5b7dd41412ed8eXXXXX",
      "Auth Token": "df21f3f76799992f8b52164021bXXXXX",
      "Send URL":"https://api.twilio.com/2010-04-01/Accounts/AC5346342f844fc5b7dd41412ed8eXXXXX/Messages.json",
      "Messaging Services":[
        {"500 Dashboards Notify":"MGec8ddf1c3187241921577f73f5XXXXX"},
        {"500 Dashboards Support":"MGec8ddf1c3187241921577f73fYYYYY"},
        {"500 Dashboards Billing":"MGec8ddf1c3187241921577f73f5ZZZZZ"}
      ]
    }
  }

We can then use these values within our endpoints. If no services are configured, or some key configuration element is missing, we can exit them right away. We can pass in the friendly name for the SendAMessage function, rather than the number, to make things a little easier, potentially, where we can then do the lookup for the MessagingServiceSid in the endpoint. 

In the startup phase of the XData server, we can do a few sanity checks to see if any messaging systems have been configured. Not extensive, really, more just a matter of checking whether a few keys are defined.

  // Are Messaging Services avialable?
  if (AppConfiguration.GetValue('Messaging Services') = nil) or ((AppConfiguration.GetValue('Messaging Services') as TJSONObject) = nil)
  then LogEvent('- Messaging Services: UNAVAILABLE')
  else
  begin
    LogEvent('- Messaging Services:');
    i := 0;
    if ((AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') <> nil) and
       (((AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') as TJSONObject).GetValue('Service Name') <> nil) then
    begin
      i := i + 1;
      LogEvent('        '+(((AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') as TJSONObject).getValue('Service Name') as TJSONString).Value);
    end;
    if ((AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('RingCentral') <> nil) and
       (((AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('RingCentral') as TJSONObject).GetValue('Service Name') <> nil) then
    begin
      i := i + 1;
      LogEvent('        '+(((AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('RingCentral') as TJSONObject).getValue('Service Name') as TJSONString).Value);
    end;
    LogEvent('        Messaging Services Configured: '+IntToStr(i));
  end;

  // Get Messaging System Configuration
  ServiceName := '';
  AccountNumber := '';
  AuthToken := '';
  SendURL := '';
  MessSysID := '';
  if (MainForm.AppConfiguration.GetValue('Messaging Services') <> nil) then
  begin
    if ((MainForm.AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') <> nil) then
    begin
      if (((MainForm.AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') as TJSONObject).GetValue('Service Name') <> nil)
      then ServiceName := (((MainForm.AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') as TJSONObject).GetValue('Service Name') as TJSONString).Value
      else raise EXDataHttpUnauthorized.Create('Invalid Messaging System Configuration: Service Name');
      
      if (((MainForm.AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') as TJSONObject).GetValue('Auth Token') <> nil)
      then AuthToken := (((MainForm.AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') as TJSONObject).GetValue('Auth Token') as TJSONString).Value
      else raise EXDataHttpUnauthorized.Create('Invalid Messaging System Configuration: Auth Token');

      if (((MainForm.AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') as TJSONObject).GetValue('Account') <> nil)
      then AccountNumber := (((MainForm.AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') as TJSONObject).GetValue('Account') as TJSONString).Value
      else raise EXDataHttpUnauthorized.Create('Invalid Messaging System Configuration: Account');

      if (((MainForm.AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') as TJSONObject).GetValue('Send URL') <> nil)
      then SendURL := (((MainForm.AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') as TJSONObject).GetValue('Send URL') as TJSONString).Value
      else raise EXDataHttpUnauthorized.Create('Invalid Messaging System Configuration: Send URL');

      if (((MainForm.AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') as TJSONObject).GetValue('Messaging Services') <> nil) then
      begin
        MessSys := ((MainForm.AppConfiguration.GetValue('Messaging Services') as TJSONObject).GetValue('Twilio') as TJSONObject).GetValue('Messaging Services') as TJSONArray;
        i := 0;
        while i < MessSys.Count - 1 do
        begin
          if (MessSys.Items[i] as TJSONObject).GetValue(MessageService) <> nil
          then MessSysID := ((MessSys.Items[i] as TJSONObject).GetValue(MessageService) as TJSONString).Value;
          i := i + 1;
        end;
        if (MessSysID = '')
        then raise EXDataHttpUnauthorized.Create('Invalid Messaging System Configuration: Messaging Service Not Found');
      end;
    end;
  end;
  if (ServiceName = '') or (AccountNumber = '') or (AuthToken = '') or (SendURL = '') or (MessSysID = '')
  then raise EXDataHttpUnauthorized.Create('Invalid Messaging System Configuration');

Other Opportunities.

With that all wrapped up, we've now got the ability to send SMS messages from an XData endpoint, as well as receive messages. New incoming messages arrive via the same callback interface (at least in the case of Twilio). This means we've got everything coming and going being logged in our database, ready for subsequent work, if needed.

If we're just sending notifications (the purpose of this blog post, after all) then there's not really anything more that we need to do other than move to a non-trial account (unless we're just sending notifications to ourselves!). However, this might be a good time to look at opportunities to build out our Template project in other directions.

• Adding support for more vendors should be straightforward as we've got all the pieces in place. RingCentral is likely to be next as it is a service I've used for a long time. What other vendors should be considered? Post a comment below, or better yet, open an issue in the XData Template Demo repository.
• Building a full-fledged chat interface into the Template client, as part of the dashboard, similar to how the ChatGPT interface works. We've got all the pieces in place, but we'll have to figure out something about telling the app about new message arrivals. Might be a good candidate for an MQTT demo. What do you think?
• Building an autoresponder. We're just logging incoming messages right now. We could parse the body of the messages and do something else. For example, if we were running a booking system, we could accept appointment requests and then reply with a booking confirmation or alternate times. We could also use our ChatGPT work to connect the incoming messages to ChatGPT or some other AI function and send back a generated response, potentially augmenting that exchange in various ways.
• Build a more integrated CSR-style universal interface where we can combine e-mail, messages, and voice calls. Many SMS service providers also offer e-mail and voice services, often with similar APIs, potentially making quick work of this sort of thing.
• Build more support for MMS and images. We've not talked about this (much) but incorporating images and video in the messaging going back and forth is possible, just not something we managed to get to in this post. Maybe next time!

What do you think? Is there anything we didn't cover in this post that is relevant at this stage? Is there another opportunity that this kind of work can unlock? As always, questions, comments, and feedback of all kinds are of great interest around these parts.

Link to GitHub repository for TMS XData Template Demo Data.

Related Posts

Notifications via E-Mail with XData/Indy
Notifications via Browser Push Notifications
Notifications via SMS with Twilio

Follow Andrew on 𝕏 at @WebCoreAndMore or join our 𝕏 Web Core and More Community.