Continuous Integration for Twilio (part 2)

In the last post, we talked about the following topics.

• How our system works
• Why integration tests matter to our system
• Why writing Twilio integration test is hard

I hope by now you understand the complexity of the problem we are trying to solve. Let’s start your first step to tackle it now!

Hop! Your first step to start writing local integration test.

Create a separate environment for integration test that hits third party end point.

As a first step, let’s describe what is the purpose of this feature.

@integration Feature: Integration As a developer I want to perform end to end test without mocking so that I can be sure that the system is working as expected

In a normal testing scenario, anything that hits external services is mocked out. However, the purpose of this test is to hit the third party (Twilio) endpoint for real so that it gives confidence to developers.

This is the extract of our spec helper which achieves the full end to end test if we runs feature only when you run specs with #integration tag.

require 'my_app'
require 'webmock/rspec'
require 'capybara/rspec'
require 'turnip'

Capybara.app = MyApp
Capybara.register_driver :selenium_chrome do |app|
  Capybara::Selenium::Driver.new(app, browser: :chrome)
end

WebMock.disable_net_connect!(allow_localhost: true)
RSpec.configure do |config|
  config.filter_run_excluding integration: true
  config.before(:suite) do
    if config.inclusion_filter[:integration]
      Capybara.javascript_driver = :selenium_chrome
      WebMock.allow_net_connect!
    else
      MyApp.fake_connection!
    end
  end
end

There are several key points in this configuration.

By default,

• config.filter_run_excluding integration: true will exclude #integration tag from normal RSpec call so that it is excluded from CI environment.
• WebMock.disable_net_connect disable external connection by default.
• We enable MyApp.fake_connection! to mock the behavior of Twilio Ruby gem like follows (alternatively you can use VCR which records your HTTP request and response).

class MyApp
  def self.fake_connection!
    setup_fake_twilio_connection
  end
end

Then for #integration tag,

• We run WebMock.allow_net_connect! allowing tests to hit external service.
• Swap Capybara.javascript_driver to :selenium_chrome so that we runs the test on real browser rather than the default headless browser.

Assert events from Twilio on browser

Now that basic #integration environment is configured, let’s move on to write a simple feature that simulates that Worker receives a new Task and calls a user, then hang up, so that the worker is ready to pick up another call.

@integration Feature: Integration As a developer I want to perform end to end test without mocking so that I can be sure that the system is working as expected

Background: Given there are no existing tasks When I ask Twilio to set myself as away And I open the browser app Then my activity is “Away”

When I set myself as idle Then my activity is “Idle”

Scenario: Outbound call to a customer When I receive a call And the customer answers the call Then my activity is “Connected”

When I hang up Then my activity is “Ready”

And these are some of example steps to make these feature pass. We use Turnip, which allows you to write tests in Gherkin format and run them through your RSpec environment.

steps_for :integration do
  step 'I open the browser app' do
    visit '/'
  end

  step 'there are no existing tasks' do
    MyApp.clear_tasks
  end

  step 'I ask Twilio to set myself as :activity' do |activity|
    MyApp.update_worker_activity(MyApp.worker_sid, activity)
  end

  step 'I set myself as away' do
    select 'Gone Home', from: 'qa-activity-selector'
  end

  step 'I hang up' do
    click_button 'call_end'
    wait_for_device_status('ready')
  end

  def wait_for_device_status(status)
    attempts = 0
    until page.evaluate_script('Twilio.Device.status()') == status || attempts == 10
      attempts += 1
      sleep 1
    end
  end
end

Any methods in MyApp are server side REST API calls to Twilio server so that we can ensure the certain state of TaskRouter. MyApp.clear_tasks truncate any existing Task while MyApp.update_worker_activity ensures that the Worker is in certain Activity state.

Anything happening on Twilio TaskRouter can be easily asserted by calling Twilio REST Ruby gem but we did not do that as checking the Twilio Activity state is not the representation of what users see.

Instead, we assert front-end behaviors in two different ways.

This is one of the typical tactics to assert an event on the client side by assigning QA-specific ID into the DOM.

<select id='qa-activity-selector' value={activity} onChange={(e) => onChange(e.target.value)}>
  {optionsForSelect(UserActivities)}
</select>

Once the ID is specified, you can use a normal matcher to wait until Gone Home appears on the specific DOM you are interested in (and Capybara is clever enough to wait for the element to appear).

step 'I set myself as away' do
  select 'Gone Home', from: 'qa-activity-selector'
end

However, how do you know that your microphone device is actually connected via WebRTC or disconnected which may not be visibly obvious (or the first tactic contains some bugs so that it gives false positive result)?

To have extra piece of mind, we also evaluate Twilio.Device.status to grab the internal state of the JS object.

step 'I hang up' do
  click_button 'call_end'
  wait_for_device_status('ready')
end

def wait_for_device_status(status)
  attempts = 0
  until page.evaluate_script('Twilio.Device.status()') == status || attempts == 10
    attempts += 1
    sleep 1
  end
end

Unlike the first approach, this method does not wait for the desired status to appear. Thus, you need to write your logic to loop until the status changes to the one you expected.

Get task created callback from Twilio

You have almost achieved automation of your integration test. Before you assert all the events, you have to send a Task to Twilio.

Here is our step to create a new Task.

 steps_for :integration do
    step 'I receive a call' do
      step 'a task is created and sent to Twilio'
    end

    step 'a task is created and sent to Twilio' do
      event = RabbitFeed::Event.new(
        {
          'application' => 'my_app',
          'name' => 'task'
        },
        payload
      )
      rabbit_feed_consumer.consume_event(event)
    end
  end

Our entire infrastructure consists of multiple applications and we implement evented architecture to send one event from one system to another using RabbitMQ. RabbitFeed is a Ruby gem developed by our very own joshuafleck and it provides a nice DSL to define event consumer and publisher, though the usage is out of scope (Check out this](/about-us/tech/2014/10/centralised-event-logging-with-rabbitfeed/) and this](/about-us/tech/2014/11/hackathon-building-a-sales-leaderboard-with-rabbitfeed-and-websockets/) blog post to find out more detail). Once the event is published, our app consumes the event and runs MyApp.create_task to create a task.

Once a new Task is created, you need to configure so that Twilio can call back your local machine. You can either use ssl tunnel through an existing AWS instance (in our environment, we can easily create integration instance for each developer) or ngrok, a freemium service to provide dynamically assigned public url (eg: akvk1c.ngrok.com).

We initially choose the ssl tunnel because it gave you a static URL, but we switched to ngrok because we wanted to have dedicated URL endpoint not only per developer but also for different environment (such as dev, test, and integration).

Another thing worth noting about our custom setting is that we have subaccounts not only per environment but also per developer. If you have 6 developers on your team, it needs a total of 21 subaccounts.

• (dev, test, integration ) * number of developers = 3 env * 6 engineers = 18
• integration , staging, production for system account = 3

According to Twilio, this is a bit unusual but we decided to stick with our decision so that we can ensure dedicated Twilio space for each independent environment.

It looks quite tedious to create so many subaccounts and their associated resources (Workspace, Task, Worker, etc) but you can automate pretty much any admin tasks you do at www.twilio.com via their REST API so it’s not as tedious as it looks.

Summary

You have so far learnt the following.

• Separating test environment to normal test and #integration scenarios, and configure them differently.
• Asserting front end behaviors in different layers.
• Configuring public facing url for Twilio to call back.

Once you implement all the points we mentioned, you should be able to run your integration feature with this simple command.

bundle exec rspec –tag integration

When we achieved this stage, we were super excited and it was fun to see that a computer does all the laborious testing works.

However, there are still several pain points which prevented us from going completely “Hands free”.

In the next blog post, we will show you how to improve this to the point that you can enable this test suite on continuous integration environment.