6 min read

Model-Based Testing Applied to Shopify Development

blog post thumbnail
SHARE
Reading Time: 6 minutes

The repository is accessible here, it has been updated with the last major xstate version

Model Based Testing applied to a Shopify Liquid Theme
https://github.com/djang0dev/mbt-shopify-liquid-theme
0 forks.
1 stars.
0 open issues.

Recent commits:

Model-Based Testing (MBT) involves creating a model that describes how a system should work and then using this model to automatically generate tests. The model is often a state machine, representing the stages or states the system can reach, and the transitions between them. Ok this a bit boring, A concrete example would be to represent the different possible paths of a user on an e-commerce site (a bit like when you spy on Hotjar and see the navigation steps of a customer).

Basic Example

Here is a model representing a basic user flow on an e-commerce site.

From this model, we can derive the states and transitions:

States

  1. onHomePage: User will be redirected to the Home Page
  2. inShopAllPage: User is on the “Shop All” page.

Events

  1. HEADER_MENU_GO_TO_HOME_PAGE: Transition from any page back to the home page via the header menu.
  2. HERO_GO_TO_SHOP_ALL: Transition from the home page to the “Shop All” page via the hero button or section.

With these events and states, we can impute the playwrights' related actions.

{
  states: {
    onHomePage: async () => {
      await page.goto(`${baseShopUrl}/`);
    },
    inShopAllPage: async () => {
      await page.waitForURL(`${baseShopUrl}/collections/all`);
    },
  },
  events: {
    HERO_GO_TO_SHOP_ALL: async () => {
      const heroButton = await page.$(".banner a");
      if (!heroButton) throw new Error("Hero button not found");
      await heroButton.click();
    },
    HEADER_MENU_GO_TO_HOME_PAGE: async () => {
      const homeButton = await page.$("#HeaderMenu-home");
      if (!homeButton) throw new Error("Home button not found");
      await homeButton.click();
    },
  },
}

How many paths (and thus tests) do you think this will generate? 1? 2? even more?

Answer: Only one 😅

In MBT each state is unique, so the return path is not set up as a standard

To do so we need to create a “unique” state:

Here we go, using @xstate/test and playwright

import { test } from "@playwright/test";
import { createTestMachine } from "@xstate/test";

const machine = createTestMachine({
  id: "basicHomePage",
  initial: "onHomePage",
  states: {
    onHomePage: {
      on: {
        HERO_GO_TO_SHOP_ALL: {
          target: "#basicHomePage.inShopAllPage",
          reenter: false,
        },
      },
    },
    inShopAllPage: {
      on: {
        HEADER_MENU_GO_TO_HOME_PAGE: {
          target: "#basicHomePage.inHomePageDestination",
          reenter: false,
        },
      },
    },
    inHomePageDestination: {
      type: "final",
    },
  },
});
createTestModel(machine)
  .getSimplePaths()
  .forEach((path) => {
    test(path.description, async ({ page }) => {
      await path.test({
        states: {
          onHomePage: async () => {
            await page.goto(`${baseShopUrl}/`);
          },
          inHomePageDestination: async () => {
            await page.waitForURL(`${baseShopUrl}/`);
          },
          inShopAllPage: async () => {
            await page.waitForURL(`${baseShopUrl}/collections/all`);
          },
          inShopAllPageDestination: async () => {
            await page.waitForURL(`${baseShopUrl}/`);
          },
        },
        events: {
          HERO_GO_TO_SHOP_ALL: async () => {
            const heroButton = await page.$(".banner a");
            if (!heroButton) throw new Error("Hero button not found");
            await heroButton.click();
          },
          HEADER_MENU_GO_TO_HOME_PAGE: async () => {
            const homeButton = await page.$("#HeaderMenu-home");
            if (!homeButton) throw new Error("Home button not found");
            await homeButton.click();
          },
        },
      });
    });
  });

In this way, We go back to the first theoretical state onHomePage:

  1. The user starts on the home page (onHomePage).
  2. The user clicks on the hero button or section (HERO_GO_TO_SHOP_ALL).
  3. The user is navigated to the “Shop All” page (inShopAllPage).
  4. The user then clicks on the header menu to return to the home page (HEADER_MENU_GO_TO_HOME_PAGE).
  5. The user ends back on the home page (inHomePageDestination).

onHomePage -> HERO_GO_TO_SHOP_ALL -> inShopAllPage -> HEADER_MENU_GO_TO_HOME_PAGE -> inHomePageDestination

Round-trip result 🏎️

Declarative > Imperative

By specifying the various states and events, we give control over the generation of test steps to the model, rather than making it explicit ourselves.

Advanced Structure

We can now generate the user workflow that wants, for instance, to buy a product using different paths

import { createTestMachine } from "@xstate/test";

const advancedAddToCartMachine = createTestMachine(
  {
    initial: "onProductPage",
    id: "advancedAddToCartMachine",
    context: {},
    states: {
      onProductPage: {
        on: {
          ADD_TO_CART: "cartDrawer.visible",
          BUY_NOW: "inCheckout",
        },
      },
      cartDrawer: {
        initial: "hidden",
        states: {
          hidden: {
            on: {
              CLICK_ON_CART_ICON: "#advancedAddToCartMachine.inCartPage",
            },
          },
          visible: {
            on: {
              CHECK_PRODUCT_ADDED_TO_CART:
                "#advancedAddToCartMachine.productAddedToCart",
              CLICK_ON_HIDE_CART: "hidden",
              CLICK_ON_VIEW_CART_PAGE_BUTTON:
                "#advancedAddToCartMachine.inCartPage",
            },
          },
        },
        on: {
          CHECK_PRODUCT_ADDED_TO_CART: "productAddedToCart",
        },
      },
      productAddedToCart: {
        on: {
          GO_TO_CHECKOUT: "inCheckout",
        },
      },
      inCartPage: {},
      inCheckout: {
        type: "final",
      },
    },
  },
  {}
);

And generate a tiny factory for the related events and states to bind the playwright's logic:

const advancedAddToCartMachinePlaywrightStatesAndEventsFactory = (
  page: Page
): StateAndEvents => {
  return {
    states: {
      inCartPage: {
        decoratedHandler: async () => {},
        meta: {
          urlRegex: new RegExp(
            `^${baseShopUrl.replace(/[\/\^$*+?.()|[\]{}-]/g, "\\$&")}/cart`,
            "i"
          ),
          waitFor: true,
          expectUrl: true,
        },
      },
      onProductPage: {
        decoratedHandler: async () => {
          await page.goto(singleProductPageBaseUrl);
          await page.waitForURL(singleProductPageBaseUrl);
        },
      },
      "cartDrawer.visible": async () => {
        await page.waitForSelector("#cart-notification");
        const cartDrawer = await page.$("#cart-notification");
        if (!cartDrawer) throw new Error("Cart drawer not found");
        const isActive = cartDrawer.evaluate((node) =>
          node.classList.contains("active")
        );
        expect(isActive).toBeTruthy();
      },
      addedToCart: async () => {
        const cartItem = await page.$("#cart-notification-product");
        if (!cartItem) throw new Error("Cart item not found");
        const titleText = await cartItem.evaluate(async (node) => {
          const title = await node.querySelector("h3");
          if (!title) throw new Error("Cart item title not found");
          const titleText = title.textContent;
          return titleText;
        });
        expect(titleText).toBe("The 3P Fulfilled Snowboard");
      },
      inCheckout: {
        meta: {
          urlRegex: new RegExp(
            `^${baseShopUrl.replace(
              /[-\/\^$*+?.()|[\]{}]/g,
              "\\$&"
            )}/checkouts/.*$`,
            "i"
          ),
          waitFor: true,
          expectUrl: true,
        },
      },
      hidden: {},
    },
    events: {
      ADD_TO_CART: async () => {
        const addToCartButton = await page.$('button[name="add"]');
        if (!addToCartButton) throw new Error("Add to cart button not found");
        await addToCartButton.click();
      },
      GO_TO_CHECKOUT: async () => {
        const goToCheckoutButton = await page.$('button[name="checkout"]');
        if (!goToCheckoutButton)
          throw new Error("Go to checkout button not found");
        await goToCheckoutButton.click();
      },
      BUY_NOW: async () => {
        const buyNowButton = await page.$(
          ".shopify-payment-button__button.shopify-payment-button__button--unbranded"
        );
        if (!buyNowButton) throw new Error("Buy now button not found");
        await buyNowButton.click();
      },
      SHOW_CART: async () => {
        const cartButton = await page.$("#cart-notification-button");
        if (!cartButton) throw new Error("Cart button not found");
        await cartButton.click();
      },
      CLICK_ON_VIEW_CART_PAGE_BUTTON: async () => {
        const viewCartPageButton = await page.$("#cart-notification-button");
        if (!viewCartPageButton)
          throw new Error("View cart page button not found");
        await viewCartPageButton.click();
      },
      CLICK_ON_HIDE_CART: async () => {
        const hideCartButton = await page.$(".cart-notification__close");
        if (!hideCartButton) throw new Error("Hide cart button not found");
        await hideCartButton.click();
      },
      CLICK_ON_CART_ICON: async () => {
        const cartIcon = await page.$("#cart-icon-bubble");
        if (!cartIcon) throw new Error("Cart icon not found");
        await cartIcon.click();
      },
    },
  };
}

With some tiny tricks, we can be in pure declarative, trying to further abstract the playwright's logic and start some tests (this is a bit boring and not essential), by using some metadata.

 
import { test } from "@playwright/test";
import { createTestMachine, createTestModel } from "@xstate/test";

const isAPlaywrightStateFunction = (
  state: PlaywrightState
): state is PlaywrightFunctionState => {
  return isFunction(state);
};
const generateXstateTestStateFromPlaywrightState = (
  playwrightState: PlaywrightState,
  page: Page
): any => {
  if (isAPlaywrightStateFunction(playwrightState)) {
    return async () => {
      await playwrightState();
    };
  }
  const meta = playwrightState.meta;
  return async () => {
    if (playwrightState?.decoratedHandler) {
      await playwrightState.decoratedHandler();
    }
    if (meta?.waitFor && meta.urlRegex) {
      await page.waitForURL(meta.urlRegex);
    }
    if (meta?.expectUrl) {
      expect(page.url()).toMatch(meta.urlRegex);
    }
  };
};
const generateXstateStateAndEventsFromPlaywrightStatesAndEvents = (
  playwrightStatesAndEvents: StateAndEvents,
  page: Page
) => {
  const states = {};
  const events = {};
  Object.keys(playwrightStatesAndEvents.states).forEach((key) => {
    const playwrightState = Reflect.get(playwrightStatesAndEvents.states, key);
    Reflect.set(
      states,
      key,
      generateXstateTestStateFromPlaywrightState(playwrightState, page)
    );
  });
  Object.keys(playwrightStatesAndEvents.events).forEach((key) => {
    const playwrightEvent = Reflect.get(playwrightStatesAndEvents.events, key);
    Reflect.set(events, key, playwrightEvent);
  });
  return {
    states,
    events,
  };
};

createTestModel(advancedAddToCartMachine)
  .getSimplePaths()
  .forEach((path) => {
    test(path.description, async ({ page }) => {
      await path.test(
        generateXstateStateAndEventsFromPlaywrightStatesAndEvents(
          advancedAddToCartMachinePlaywrightStatesAndEventsFactory(page),
          page
        )
      );
    });
  });

This gives us five paths to be tested:

Shortest Paths vs Simplest Paths?

Shortest Paths

  • The shortest path between two points (or states) without passing through the same point twice.
  • Often used to identify the fastest or most efficient path.
  • Does not necessarily cover all test cases or features.

Simplest Paths

  • Any path that passes from a starting point to an end point without revisiting any intermediate point.
  • Unlike “shortest path”, it may not be the shortest path, but it ensures that no node is visited more than once.
  • Can cover more features or test cases than the “shortest path”, as it can take longer or different routes.

Translated with www.DeepL.com/Translator (free version)

Results

The model automatically generated five tests for us with a consistent workflow 🤯

Conclusion

Using a finite state machine and playwright, we were able to generate e2e tests by targeting the key steps of a user journey and creating the various associated paths.

Benefits:

  • Self-generated test (even if github copilot works fine as usual)
  • Ultra-fast maintenance: if you want to add a step along the way, the machine can be changed very quickly.
  • High Test Coverage
  • Easy to automate
  • More reliable tests
  • Explicit docs
  • Helps you to shine in society 🤠

Cons:

  • It hurts to learn

October 17, 2023 - 12:10 pm

Related posts

blog post thumbnail

Finite State Machines in E-commerce by Example

06 min read
blog post thumbnail

Why do Shopify Functions provide an unfair advantage to create exceptional stores?

06 min read

Do you like my work?

Let's make a wonderful
websiteexperienceappstrategy together!

Get in touch!