A/B Testing & Gradual Rollouts - Remark Documentation

Remark includes built-in A/B testing capabilities that allow you to gradually roll out the chat widget to your customers while maintaining a control group. This enables you to measure the impact of Remark on your key business metrics.

Test Results

Read published test results and understand report metrics.

Control Group Data

Read A/B test assignments, lead IDs, and set up external monitoring.

Testing Parameters

Open Remark automatically for testing and campaigns.

Hide Competing Tools

Automatically hide alternative chat solutions during A/B tests.

Rollout Strategy

Best practices for phased rollouts from 10% to 100%.

How A/B Testing Works

Remark manages control group assignment automatically on the backend. When a visitor arrives on your site, the backend determines whether they should be in the control group based on your configured rollout percentage.

Cohorts and Show Rate

An A/B test splits eligible shopper sessions into two cohorts:

Cohort	What shoppers see
Control	Remark is hidden, so shoppers continue through your site without the Remark chat experience.
Treatment	Remark can appear according to your widget settings.

The show rate is the percentage of eligible sessions assigned to the treatment cohort. For example, a 25% show rate means roughly 25% of eligible sessions can see Remark and the remaining eligible sessions are control. Changing the show rate changes future assignment. It does not rewrite sessions that were already assigned.

Control Group Assignment

Persistent – The same visitor will always see the same experience across sessions
Automatic – No client-side configuration required
Backend-controlled – Managed by Remark’s services to maintain consistency across all entry points

Why Backend Control?

Remark uses backend control for A/B test assignment to ensure consistent experiences across multiple entry points and devices. Visitors can engage with Remark through various channels:

Initial page load – First visit to your site
Page reloads – Navigating between pages or refreshing
Email follow-ups – Clicking links in conversation notification emails
Mobile transfers – Switching between mobile and desktop devices
Cross-device continuity – Moving from one device to another mid-conversation

If A/B test assignment were determined client-side only, a visitor could see Remark on desktop but not on mobile, or vice versa. Backend control ensures that once assigned to a group, a visitor remains in that group regardless of how they access your site.

Reviewing Test Results in the Dashboard

After shoppers have been assigned to cohorts, Remark publishes lift analyses to the Dashboard. Open the Remark dashboard and select the A/B Test Results flask icon on the home page. For brand users, the icon appears when results become available for your workspace. If you do not see the icon, results may not be available yet.

Result dates

Each analysis has a date range and an analyzed timestamp. If there are multiple published analyses, they may cover different date ranges or analysis settings. Use the date range first when comparing reports.

Metrics

The report can include summary metrics, comparison tables, and statistical test sections. Common metric types include:

Session or visitor counts
Conversion rate
Revenue or average order value
Lift or percent change between control and treatment
Confidence intervals for estimated lift

Statistical tests

Statistical test sections summarize whether the observed difference is strong enough to treat as evidence rather than normal variation.

Item	How to read it
Verdict	Plain-language result for the metric, such as whether the result is significant or inconclusive.
p-value	Lower values mean the observed difference is less likely to be random noise.
95% CI	The estimated range for the true effect. If the range crosses zero, the direction is less certain.
Power	How much ability the test had to detect the measured effect size. Low power means the test may need more data.

An inconclusive result does not mean Remark had no effect. It means the analysis did not have enough evidence to call a clear result for that metric and date range.

Report tables

Tables compare control and treatment values side by side. Rows usually represent metrics or segments, and columns show counts, rates, revenue, lift, confidence intervals, or notes. Use tables to check:

Whether both cohorts have enough sessions
Whether conversion and revenue metrics move in the same direction
Whether one metric is driven by a small number of outlier orders
Whether the reported lift matches the business question you are trying to answer

Missing or inconclusive results

If results are missing, check whether:

Remark has published an analysis for your workspace
The test has enough eligible sessions and conversions
You are looking at the intended date range
The test was active during the period you are reviewing

If results are inconclusive, the most common reasons are limited traffic, limited conversions, a short test window, or high order-value variance. Contact support@remark.ai if you need help interpreting a report or want Remark to publish a new analysis for a different date range.

Accessing A/B Test Data

SDK Initialization Event

Remark dispatches a remark:sdk-initialized event when the SDK has fully loaded and initialized. This is the ideal time to set up external monitoring or analytics.

window.addEventListener("remark:sdk-initialized", () => {
  // SDK is fully loaded and ready
  // Safe to access control group data
  const isControlGroup = localStorage.getItem("isInRemarkABControlGroup");
  console.log("Control group status:", isControlGroup);
});

Reading Control Group Status

The control group status is stored in localStorage and can be accessed synchronously:

const isInControlGroup = localStorage.getItem("isInRemarkABControlGroup");

if (isInControlGroup === "true") {
  // User is in the control group (will NOT see Remark)
  console.log("Control group - Remark hidden");
} else if (isInControlGroup === "false") {
  // User is in the treatment group (will see Remark)
  console.log("Treatment group - Remark visible");
}

The value is stored as a string, not a boolean. Always use === "true" or === "false" for comparisons.

Reading Lead ID

Remark also stores the lead ID (unique visitor identifier) in localStorage. This is useful for correlating Remark’s A/B test data with your own analytics:

const leadId = localStorage.getItem("remark_lead");

if (leadId) {
  // Use this ID to track the same visitor in your analytics
  console.log("Remark Lead ID:", leadId);
}

This lead ID remains consistent across:

Page reloads and navigation
Multiple sessions on the same device
Cross-device tracking (when visitors return via email links)

Including the lead ID in your external A/B test monitoring allows you to:

Match Remark’s user identification with your analytics
Track individual visitor journeys across the A/B test
Analyze conversion rates at the individual lead level
Correlate chat engagement with purchase behavior

Setting Up External Monitoring

If you want to run your own analytics or A/B test monitoring alongside Remark, you can use the SDK initialization event and control group data.

Google Analytics Integration

window.addEventListener("remark:sdk-initialized", () => {
  const isInControlGroup = localStorage.getItem("isInRemarkABControlGroup");
  const leadId = localStorage.getItem("remark_lead");
  
  // Send A/B test group to Google Analytics
  if (typeof gtag !== "undefined") {
    gtag("event", "remark_ab_test", {
      event_category: "ab_testing",
      event_label: isInControlGroup === "true" ? "control" : "treatment",
      remark_lead_id: leadId,  // Include lead ID for tracking
      non_interaction: true
    });
    
    // Set as custom dimensions for all subsequent events
    gtag("set", "user_properties", {
      remark_ab_group: isInControlGroup === "true" ? "control" : "treatment",
      remark_lead: leadId
    });
  }
});

Segment Integration

window.addEventListener("remark:sdk-initialized", () => {
  const isInControlGroup = localStorage.getItem("isInRemarkABControlGroup");
  const leadId = localStorage.getItem("remark_lead");
  
  if (typeof analytics !== "undefined") {
    analytics.track("Remark A/B Test Assigned", {
      group: isInControlGroup === "true" ? "control" : "treatment",
      remark_lead_id: leadId,
      timestamp: new Date().toISOString()
    });
    
    // Optionally set lead ID as user identifier for cross-session tracking
    if (leadId) {
      analytics.identify(leadId, {
        remark_ab_group: isInControlGroup === "true" ? "control" : "treatment"
      });
    }
  }
});

Custom Analytics Endpoint

window.addEventListener("remark:sdk-initialized", () => {
  const isInControlGroup = localStorage.getItem("isInRemarkABControlGroup");
  const leadId = localStorage.getItem("remark_lead");
  
  // Send to your custom analytics endpoint
  fetch("https://your-analytics.com/api/track", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      event: "remark_ab_test_assigned",
      properties: {
        controlGroup: isInControlGroup === "true",
        leadId: leadId,
        timestamp: Date.now(),
        pageUrl: window.location.href
      }
    })
  });
});

Testing with URL Parameters

For A/B testing purposes, you can use the remark_open URL parameter to open the Remark widget automatically when the page loads. This ensures visitors in the treatment group see and interact with Remark.

https://yoursite.com?remark_open=true

This parameter is useful for:

Demo and sales links – Guarantee prospects see Remark when visiting your site
Email campaigns – Direct customers to chat from notification emails
Customer support – Send customers directly into a conversation
A/B test verification – Confirm Remark appears for treatment group testing

Combining with Analytics Parameters

You can combine remark_open=true with your existing analytics parameters:

# Track email campaign performance
https://yoursite.com?remark_open=true&utm_source=email&utm_campaign=ab_test

# Track demo link clicks
https://yoursite.com?remark_open=true&utm_source=demo&utm_medium=sales_deck

Hiding Competing Chat Tools

When running an A/B test between Remark and another chat solution (e.g., HubSpot, Gorgias, Intercom), Remark will automatically hide the competing tool for users in the treatment group.

Two-Pronged Approach

Remark uses a dual strategy to ensure competing tools are hidden reliably:

JavaScript API – If the tool exposes an API, Remark will call it to hide/remove the widget
CSS Injection – Inject CSS styles to hide the tool’s DOM elements as a fallback

This redundancy ensures the competing tool is hidden even if Remark loads before the other tool’s API is available.

Example: HubSpot

const styleTag = document.createElement('style');

function hideHubSpot() {
  // Approach 1: Use HubSpot's JavaScript API
  try {
    window.HubSpotConversations.widget.remove();
  } catch (e) {
    console.debug('HubSpot API not yet available');
  }
  
  // Approach 2: Inject CSS to hide the widget
  styleTag.innerHTML = `
    div#hubspot-messages-iframe-container[role="region"] { 
      display: none !important; 
    }
  `;
  document.head.appendChild(styleTag);
}

function showHubSpot() {
  // Remove CSS
  styleTag.remove();
  
  // Reload HubSpot widget
  try {
    window.HubSpotConversations.widget.load();
  } catch (e) {
    console.debug('Error loading HubSpot chat');
  }
}

Example: Gorgias

const styleTag = document.createElement('style');

function waitForGorgias(callback) {
  let checkInterval = setInterval(() => {
    const containerDiv = document.querySelector('div#gorgias-chat-container');
    if (typeof window.GorgiasChat !== 'undefined' && containerDiv) {
      clearInterval(checkInterval);
      window.GorgiasChat.init().then(() => {
        callback();
      });
    }
  }, 50);
  
  // Timeout after 60 seconds
  setTimeout(() => {
    if (typeof window.GorgiasChat === 'undefined') {
      clearInterval(checkInterval);
      console.error('GorgiasChat not loaded after 60 seconds');
    }
  }, 60000);
}

function hideGorgias() {
  if (typeof window.GorgiasChat !== 'undefined') {
    window.GorgiasChat.hideChat(true);
  }
  
  // CSS fallback
  styleTag.innerHTML = `
    div#gorgias-chat-container { 
      display: none !important; 
    }
  `;
  if (!styleTag.isConnected) {
    document.head.appendChild(styleTag);
  }
}

function showGorgias() {
  if (typeof window.GorgiasChat !== 'undefined') {
    window.GorgiasChat.hideChat(false);
  }
  
  styleTag.remove();
  
  const widgetDiv = document.querySelector('div#gorgias-chat-container');
  if (widgetDiv) {
    widgetDiv.removeAttribute('style');
  }
}

Example: Intercom

const styleTag = document.createElement('style');

function hideIntercom() {
  // JavaScript API
  try {
    if (typeof window.Intercom !== 'undefined') {
      window.Intercom('hide');
    }
  } catch (e) {
    console.debug('Intercom API not yet available');
  }
  
  // CSS fallback
  styleTag.innerHTML = `
    .intercom-lightweight-app,
    #intercom-container { 
      display: none !important; 
    }
  `;
  document.head.appendChild(styleTag);
}

function showIntercom() {
  styleTag.remove();
  
  try {
    if (typeof window.Intercom !== 'undefined') {
      window.Intercom('show');
    }
  } catch (e) {
    console.debug('Error showing Intercom');
  }
}

Testing & Debugging

Verify A/B Test Data

// Check if localStorage is being set correctly
console.log("Control Group Status:", localStorage.getItem("isInRemarkABControlGroup"));
console.log("Lead ID:", localStorage.getItem("remark_lead"));

// Verify both values are available for external tracking
const abTestData = {
  isControlGroup: localStorage.getItem("isInRemarkABControlGroup") === "true",
  leadId: localStorage.getItem("remark_lead")
};
console.log("A/B Test Data:", abTestData);

Test SDK Initialization

let sdkInitialized = false;

window.addEventListener("remark:sdk-initialized", () => {
  sdkInitialized = true;
  console.log("✓ SDK initialized successfully");
});

// Check after a few seconds
setTimeout(() => {
  if (!sdkInitialized) {
    console.error("✗ SDK did not initialize within 5 seconds");
  }
}, 5000);

Testing Scenarios

Demo Link with Auto-Open

Create a shareable demo link that opens Remark automatically:

https://yoursite.com?remark_open=true

Combine with UTM parameters for tracking:

https://yoursite.com?remark_open=true&utm_source=demo&utm_campaign=sales

Test Control vs Treatment Side-by-Side

Open two browser profiles or incognito windows to compare experiences: Window 1 - Control Group:

localStorage.setItem("isInRemarkABControlGroup", "true");
location.reload();

Window 2 - Treatment Group:

localStorage.setItem("isInRemarkABControlGroup", "false");
location.href = location.origin + location.pathname + "?remark_open=true";

Best Practices

Wait for SDK Initialization

Always wait for the remark:sdk-initialized event before accessing control group data or setting up integrations.

// ✓ Good - wait for SDK
window.addEventListener("remark:sdk-initialized", () => {
  const isControlGroup = localStorage.getItem("isInRemarkABControlGroup");
  // Use the data...
});

// ✗ Bad - might run before SDK initializes
const isControlGroup = localStorage.getItem("isInRemarkABControlGroup");

Handle Missing Data

The control group status might not be available immediately on the first page load.

window.addEventListener("remark:sdk-initialized", () => {
  const isControlGroup = localStorage.getItem("isInRemarkABControlGroup");
  
  if (isControlGroup === null) {
    console.warn("Control group status not yet available");
    return;
  }
  
  // Proceed with your logic...
});

Non-Intrusive Monitoring

Ensure your monitoring code doesn’t interfere with the user experience:

window.addEventListener("remark:sdk-initialized", () => {
  try {
    // Your monitoring code
  } catch (e) {
    // Fail silently - don't break the user experience
    console.error("Monitoring error:", e);
  }
});

Respect Privacy

Only collect A/B test data that’s necessary for your analysis. Avoid collecting PII without proper consent.

// ✓ Good - anonymous tracking
analytics.track("Remark A/B Test", {
  group: isControlGroup === "true" ? "control" : "treatment"
});

// ✗ Bad - avoid PII unless necessary
analytics.track("Remark A/B Test", {
  group: isControlGroup === "true" ? "control" : "treatment",
  email: userEmail,  // Avoid
  name: userName     // Avoid
});

Troubleshooting

SDK Event Not Firing

If the remark:sdk-initialized event isn’t firing:

Check browser console for errors
Verify Remark script is loading correctly
Check network requests for GraphQL calls to Remark’s API
Ensure no ad blockers are interfering

Debug helper:

console.log("Checking for Remark SDK...");
console.log("window.remark exists:", typeof window.remark !== "undefined");

setTimeout(() => {
  console.log(
    "Control group status:", 
    localStorage.getItem("isInRemarkABControlGroup")
  );
}, 3000);

Control Group Status Not Set

If isInRemarkABControlGroup is always null:

Check that your Remark installation is complete
Verify backend API is reachable
Check browser console for GraphQL errors
Contact Remark support if issue persists

Competing Tool Not Hiding

If the competing chat tool isn’t being hidden:

Verify the tool’s JavaScript API is available (window.HubSpotConversations, etc.)
Check CSS selectors are correct for the tool’s DOM structure
Inspect DOM to find the tool’s container elements
Update CSS selectors if the tool has changed its markup

Use browser DevTools to inspect the competing tool’s DOM structure and adjust your hide/show functions accordingly.

Test Results

Control Group Data

Testing Parameters

Hide Competing Tools

Rollout Strategy

​How A/B Testing Works

​Cohorts and Show Rate

​Control Group Assignment

​Why Backend Control?

​Reviewing Test Results in the Dashboard

​Result dates

​Metrics

​Statistical tests

​Report tables

​Missing or inconclusive results

​Accessing A/B Test Data

​SDK Initialization Event

​Reading Control Group Status

​Reading Lead ID

​Setting Up External Monitoring

​Google Analytics Integration

​Segment Integration

​Custom Analytics Endpoint

​Testing with URL Parameters

​Open Widget on Load

​Combining with Analytics Parameters

​Hiding Competing Chat Tools

​Two-Pronged Approach

​Example: HubSpot

​Example: Gorgias

​Example: Intercom

​Testing & Debugging

​Verify A/B Test Data

​Test SDK Initialization

​Testing Scenarios

​Demo Link with Auto-Open

​Test Control vs Treatment Side-by-Side

​Best Practices

​Wait for SDK Initialization

​Handle Missing Data

​Non-Intrusive Monitoring

​Respect Privacy

​Troubleshooting

​SDK Event Not Firing

​Control Group Status Not Set

​Competing Tool Not Hiding

How A/B Testing Works

Cohorts and Show Rate

Control Group Assignment

Why Backend Control?

Reviewing Test Results in the Dashboard

Result dates

Metrics

Statistical tests

Report tables

Missing or inconclusive results

Accessing A/B Test Data

SDK Initialization Event

Reading Control Group Status

Reading Lead ID

Setting Up External Monitoring

Google Analytics Integration

Segment Integration

Custom Analytics Endpoint

Testing with URL Parameters

Open Widget on Load

Combining with Analytics Parameters

Hiding Competing Chat Tools

Two-Pronged Approach

Example: HubSpot

Example: Gorgias

Example: Intercom

Testing & Debugging

Verify A/B Test Data

Test SDK Initialization

Testing Scenarios

Demo Link with Auto-Open

Test Control vs Treatment Side-by-Side

Best Practices

Wait for SDK Initialization

Handle Missing Data

Non-Intrusive Monitoring

Respect Privacy

Troubleshooting

SDK Event Not Firing

Control Group Status Not Set

Competing Tool Not Hiding