Contract-based Testing for Faas

A big downside of designing a distributed system built on Faas it that it's hard to execute system tests or to even test that various functions are working together properly. In a microservices world, we can use consumer-driven contracts, which we talked about in the last chapter.

What is the analog in our event-sourced, functions-as-a-service, serverless world?

Since all activity is based on messages, and our functions are triggered based on messages, we could use bonafide messages as test inputs. For functions that also send messages, we'd like to capture those messages to use as test data for other functions.

In the previous section we broke our system by changing the format of the message that renderPage received.

Let's change how we test to be more message-focused. We'll start by using the documented messages and payloads as input to our tests, rather than hand-coding them.

Output of One is Input to Another

The problem with our system as it stands is that renderPage is expecting a message that won't be sent. We inadvertently invented a message that makes our tests pass, but that doesn't exist in production.

Let's create an event catalog that describes all the events we know about. We'll create a directory named js/test_messages and place one file in there for each event. The contents of the file will be an example event.

First, we'll create js/test_messages/emailSignup.json:

{
  "body": {
    "email": "pat@example.com"
  }
}

Next, js/test_messages/newEmailAddress.json:

{
  "email": {
    "data": {
      "id": 42,
      "email": "pat@example.com"
    }
  }
}

And finally, js/test_messages/pageRequested.json:

{}

We'll then write an event catalog in js/EventTestCatalog.js to read these in.

const fs   = require("fs");
const path = require("path");

const EventTestCatalog = {
  events: {}
};

const testMessagesPath = path.resolve(__dirname,"test_messages");
fs.readdirSync(testMessagesPath).forEach(file => {
  const message = JSON.parse(
                    fs.readFileSync(
                      path.resolve(testMessagesPath,file)
                    ));
  const eventName = file.replace(/\.json$/,"");
  EventTestCatalog.events[eventName] = message;
});


EventTestCatalog.events["get"] = EventTestCatalog.events["pageRequested"];
EventTestCatalog.events["post"] = EventTestCatalog.events["emailSignup"];

module.exports = EventTestCatalog;

Now that we've created some canonical message bodies for our messages outside of any given function implementation, let's rewrite js/renderPageTest.js to use these messages as input.

Because renderPage is configured to receive both pageRequested and emailSignup, we'll execute our tests twice, one for each.

const renderPage = require("./renderPage.js")
const EventTestCatalog = require("./EventTestCatalog.js");

var stringWritten = null;

const write = function(string) {
  stringWritten = string;
}

const events = ["pageRequested","emailSignup"];
events.forEach( (eventName) => {
  try {
    const testMessage = EventTestCatalog.events[eventName];
    testMessage.write = write;
    renderPage(testMessage);

    if (!stringWritten.match(/\<strong\>\d+ startups/)) {
      throw "could not find count";
    }
    if (!stringWritten.match(/As of \<strong\>/)) {
      throw "could not find date";
    }
    if (eventName === "emailSignup") {
      if (stringWritten.indexOf("Thanks pat@example.com") === -1) {
        throw "couldn't find 'Thanks pat@example.com' on the page";
      }
    }
    console.log(`✅ given ${eventName}, renderPage is good`);
  }
  catch (exception) {
    console.log(`🚫 given ${eventName}, renderPage is broken: ${exception}`);
  }
});

Now, when we run our test, we get a failure:

> node js/renderPageTest.js
✅ given pageRequested, renderPage is good
🚫 given emailSignup, renderPage is broken: couldn't find 'Thanks pat@example.com' on the page

We can see that our code works for the pageRequested event, but not for the emailSignup one. Nice! This gives us a better idea of where things are broken.

A brief investigation shows that we're expecting the key emailSignup and not the key email, so we can make the test pass by fixing the renderPage function to use the actual key that would be used in production:

const fs   = require("fs");
const path = require("path");
const index = fs.readFileSync(
      path.resolve(__dirname,"..","html","index.html")
    ).toString();

/*
 * @param params an object expected to have two keys:
 *        - write - the function passed by js/server.js
 *        - body - the parsed body of the request (optional).
 */
const renderPage = (params) => {
  var html = index.replace("##date##",(new Date()).toString()).
    replace("##count##",Math.floor(Math.random() * 1000));

/* start new code */
  if (params.body && params.body.email) {
/* end new code */
/* start new code */
    html = html.replace("##email##",`Thanks ${params.body.email}`);
/* end new code */
  }
  else {
    html = html.replace("##email##","");
  }
  params.write(html);
}
module.exports = renderPage;

Our tests should now pass:

> node js/renderPageTest.js
✅ given pageRequested, renderPage is good
✅ given emailSignup, renderPage is good

What does this mean?

Messages as Integration Test Data

This means that if our functions always take messages, and we clearly document the sorts of messages they accept, we can use actual messages as test input to drive our functions. Meaning: our test data is unlikely to be wrong, so the behavior of our functions given that data will be an accurate portrayal of how they will work in production.

It also means that we can record any messages sent by our code to use as test data for other functions. Consider sendWelcomeEmail. It accepts a somewhat complex structure as input. As we did in renderPageTest.js, let's remove the hard-coded test data and grab a message from our EventTestCatalog:

const sendWelcomeEmail = require("./sendWelcomeEmail.js")
const EventTestCatalog = require("./EventTestCatalog.js");

const data = EventTestCatalog.events["newEmailAddress"];

const emailMailed = sendWelcomeEmail(data);
if (emailMailed === "pat@example.com") {
  console.log("✅ sendWelcomeEmail is good");
}
else {
  console.log(`🚫 sendWelcomeEmail is failed.  Expected pat@example.com, got ${emailMailed}`);
}

Our test still passes:

> node js/sendWelcomeEmailTest.js
Sending welcome email to pat@example.com
✅ sendWelcomeEmail is good

OK, that was a repeat of the above, but let's suppose that we have a test of our database that actually outputs the message to our catalog?

First, we'll add a function to EventTestCatalog to allow saving new message payloads:

const fs   = require("fs");
const path = require("path");

const EventTestCatalog = {
/* start new code */
  events: {},
  saveEvent: (name,body) => {
    fs.writeFileSync(path.resolve(testMessagesPath,`${name}.json`),JSON.stringify(body,null,2))
  }
/* end new code */
};

const testMessagesPath = path.resolve(__dirname,"test_messages");
fs.readdirSync(testMessagesPath).forEach(file => {
  const message = JSON.parse(
                    fs.readFileSync(
                      path.resolve(testMessagesPath,file)
                    ));
  const eventName = file.replace(/\.json$/,"");
  EventTestCatalog.events[eventName] = message;
});


EventTestCatalog.events["get"] = EventTestCatalog.events["pageRequested"];
EventTestCatalog.events["post"] = EventTestCatalog.events["emailSignup"];

module.exports = EventTestCatalog;

Now, our test of Database.js will update the test message if its tests pass:

const Database = require("./Database.js");
const EventBus = require("./EventBus.js");
const EventTestCatalog = require("./EventTestCatalog.js");

/* store the messages we got by subscribing
   to the event we expect to be fired */
var receivedMessage = null;
const listener = (data) => {
  receivedMessage = data;
}
EventBus.register("newEmailAddress",listener)

try {
  const savedId = Database.save("pat@example.com");

  if (Database.find(savedId) !== "pat@example.com") {
    throw "save & find FAILED";
  }

  if (receivedMessage === null) {
    throw "Events did not fire";
  } else if (receivedMessage.email.data.id  !== savedId) {
    throw `Expected ${savedId}, but got ${receivedEmailId}`;
  }
  console.log("✅ Database is good");

  // Save the message we received to the shared test catalog
  EventTestCatalog.saveEvent("newEmailAddress",receivedMessage);
} catch (exception) {
  console.log(`🚫 Database failed: ${exception}`);
}

We'll run the test:

> node js/DatabaseTest.js
✅ Database is good

And we can see that the test message for newEmailAddress has changed:

> cat js/test_messages/newEmailAddress.json
{
  "email": {
    "action": "created",
    "data": {
      "id": 1,
      "email": "pat@example.com"
    }
  }
}

If we re-run js/sendWelcomeEmailTest.js, it will read this file, and should still pass:

> node js/sendWelcomeEmailTest.js
Sending welcome email to pat@example.com
✅ sendWelcomeEmail is good

So far so good. Now, let's say the database changes its payload format for the messages, by changing the field email to emailAddress:

const EventBus = require("./EventBus.js");

class Database {
  constructor() {
    this.data = {};
    this.nextId = 1;
  }
  save(email) {
    const id = this.nextId;
    this.nextId++;

    this.data[id] = email;
/* start new code */
    EventBus.fire("newEmailAddress",{ email: { action: "created", data: { id: id, emailAddress: email }}});
/* end new code */
    return id;
  }
  find(id) {
    return this.data[id];
  }
}
module.exports = (new Database());

The test for the database still passes:

> node js/DatabaseTest.js
✅ Database is good

Since it does, it outputs an updated test message:

> cat js/test_messages/newEmailAddress.json
{
  "email": {
    "action": "created",
    "data": {
      "id": 1,
      "emailAddress": "pat@example.com"
    }
  }
}

Which now breaks sendWelcomeEmail:

> node js/sendWelcomeEmailTest.js
Sending welcome email to undefined
🚫 sendWelcomeEmail is failed.  Expected pat@example.com, got undefined

Nice!

What Does it Mean?

What this demonstrates is that it's possible to create a system based on events and functions and, with sufficient investment in how we write tests, we can achieve the level of confidence we'd get from an end-to-end test by keeping track of the messages and expectations of the various functions and what they do.

You could imagine taking the above concept to the next level by having a central database of messages, and a registry of functions that send those messages along with those that expect to receive them. It would then be possible to examine this registry to see if anything broke. For example, if we'd made the change to Database above under such a system, we could include, as part of our CI process, a step that asks the central registry “Will this updated payload break any registered consumers?”.

Such a system could also monitor all messages and, armed with the knowledge of who is expecting what from whom, indicate if an expected message stopped being sent, or if no one was registered to receive a particular message.

Neat! So, we should all do serverless now?