Skip to main content
Remark includes built-in A/B testing capabilities that allow you to gradually roll out the chat widget to your customers while maintaining a control group. This enables you to measure the impact of Remark on your key business metrics.

Test Results

Read published test results and understand report metrics.

Control Group Data

Read A/B test assignments, lead IDs, and set up external monitoring.

Testing Parameters

Open Remark automatically for testing and campaigns.

Hide Competing Tools

Hide another chat tool whenever Remark appears.

How A/B Testing Works

Remark manages control group assignment automatically on the backend. When a visitor arrives on your site, the backend determines whether they should be in the control group based on the show rate Remark sets for your test. To start a test, change its show rate, or end it, contact support@remark.ai.

Cohorts and Show Rate

An A/B test splits eligible shopper sessions into two cohorts: The show rate is the percentage of eligible sessions assigned to the treatment cohort. For example, a 25% show rate means roughly 25% of eligible sessions can see Remark and the remaining eligible sessions are control. Changing the show rate changes future assignment. It does not rewrite sessions that were already assigned. A visitor whose Remark session starts while Remark can’t appear stays in the control group for the rest of the test. That happens when chat is turned off, the brand is unpublished, the IP allowlist excludes the visitor, or no expert is available to pair. Page allow and block lists don’t affect assignment. When the test ends, control visitors start seeing Remark.

Control Group Assignment

  • Persistent – Remark assigns the cohort once, when it creates the visitor’s Remark session, and keeps it as long as that browser keeps its remark_lead. Visitors whose session began before the test aren’t assigned and keep seeing Remark.
  • Automatic – No client-side configuration required
  • Backend-controlled – Managed by Remark’s services to maintain consistency across all entry points

Why Backend Control?

Remark stores the assignment on the visitor’s Remark session, so it holds across every way that visitor returns to Remark:
  • Initial page load – First visit to your site
  • Page reloads – Navigating between pages or refreshing
  • Email follow-ups – Clicking links in conversation notification emails
Links in Remark’s conversation emails carry the visitor’s remark_lead, so a visitor who opens one on another device keeps their cohort. A visitor who switches devices any other way gets a new assignment on the new device.

Reviewing Test Results in the Dashboard

After shoppers have been assigned to cohorts, Remark publishes lift analyses to the Dashboard. Open the Remark dashboard and select the A/B Test Results flask icon on the home page. For brand users, the icon appears when results become available for your workspace. If you do not see the icon, results may not be available yet.

Result dates

Each analysis has a date range and an analyzed timestamp. If there are multiple published analyses, they may cover different date ranges or analysis settings. Use the date range first when comparing reports.

Metrics

The report can include summary metrics, comparison tables, and statistical test sections. Common metric types include:
  • Session or visitor counts
  • Conversion rate
  • Revenue or average order value
  • Lift or percent change between control and treatment
  • Confidence intervals for estimated lift

Statistical tests

Statistical test sections summarize whether the observed difference is strong enough to treat as evidence rather than normal variation. An inconclusive result does not mean Remark had no effect. It means the analysis did not have enough evidence to call a clear result for that metric and date range.

Report tables

Tables compare control and treatment values side by side. Rows usually represent metrics or segments, and columns show counts, rates, revenue, lift, confidence intervals, or notes. Use tables to check:
  • Whether both cohorts have enough sessions
  • Whether conversion and revenue metrics move in the same direction
  • Whether one metric is driven by a small number of outlier orders
  • Whether the reported lift matches the business question you are trying to answer

Missing or inconclusive results

If results are missing, check whether:
  • Remark has published an analysis for your workspace
  • The test has enough eligible sessions and conversions
  • You are looking at the intended date range
  • The test was active during the period you are reviewing
If results are inconclusive, the most common reasons are limited traffic, limited conversions, a short test window, or high order-value variance. Contact support@remark.ai if you need help interpreting a report or want Remark to publish a new analysis for a different date range.

Accessing A/B Test Data

Session Initialization Event

Remark dispatches a remark:session-initialized event when the client session has initialized. This is the ideal time to set up external monitoring or analytics.

Reading Control Group Status

The control group status is stored in localStorage and can be accessed synchronously:
The value is stored as a string, not a boolean. Always use === "true" or === "false" for comparisons.

Reading Lead ID

Remark also stores the lead ID (unique visitor identifier) in localStorage. This is useful for correlating Remark’s A/B test data with your own analytics:
This lead ID remains consistent across:
  • Page reloads and navigation
  • Multiple sessions on the same device
  • Cross-device tracking (when visitors return via email links)
Including the lead ID in your external A/B test monitoring allows you to:
  • Match Remark’s user identification with your analytics
  • Track individual visitor journeys across the A/B test
  • Analyze conversion rates at the individual lead level
  • Correlate chat engagement with purchase behavior

Setting Up External Monitoring

If you want to run your own analytics or A/B test monitoring alongside Remark, you can use the session initialization event and control group data.

Google Analytics Integration

Segment Integration

Custom Analytics Endpoint

Testing with URL Parameters

You can use the remark_open URL parameter to open the Remark widget automatically when the page loads. It opens the widget only for visitors who can already see Remark, so it doesn’t override A/B assignment. To make sure Remark appears, add remark_force_widget=true.

Open Widget on Load

This parameter is useful for:
  • Demo and sales links – Combine it with remark_force_widget=true so prospects see Remark
  • Email campaigns – Direct customers to chat from notification emails
  • Customer support – Send customers directly into a conversation
  • A/B test verification – Confirm Remark appears for treatment group testing
Each visit opened with remark_force_widget=true permanently moves that browser’s visitor into the treatment group. Keep it out of campaign and email links while a test is running.

Combining with Analytics Parameters

You can combine remark_open=true with your existing analytics parameters:

Hiding Competing Chat Tools

Remark can hide another chat tool (for example Gorgias, Zendesk, HubSpot, or Intercom) whenever Remark appears, and show it again whenever Remark doesn’t, such as for visitors in the control group. Remark configures this for your storefront. To set it up, contact support@remark.ai with the name of the tool you use.

Two-Pronged Approach

Remark uses a dual strategy to ensure competing tools are hidden reliably:
  1. JavaScript API – If the tool exposes an API, Remark will call it to hide/remove the widget
  2. CSS Injection – Inject CSS styles to hide the tool’s DOM elements as a fallback
This redundancy ensures the competing tool is hidden even if Remark loads before the other tool’s API is available.

Testing & Debugging

Verify A/B Test Data

Test Session Initialization

Testing Scenarios

Create a shareable demo link that shows and opens Remark automatically:
Combine with UTM parameters for tracking:

Test Control vs Treatment

To see the treatment experience, open any page with ?remark_force_widget=true. To start over as a new visitor with a fresh assignment, open a page with ?remark_reset_chat=true or use a new private window. After remark:session-initialized fires, read isInRemarkABControlGroup to see which group you landed in. Setting isInRemarkABControlGroup yourself has no effect. Remark rewrites it from the backend on every page load.

Best Practices

Wait for Session Initialization

Always wait for the remark:session-initialized event before accessing control group data or setting up integrations.

Handle Missing Data

The control group status might not be available immediately on the first page load.

Non-Intrusive Monitoring

Ensure your monitoring code doesn’t interfere with the user experience:

Respect Privacy

Only collect A/B test data that’s necessary for your analysis. Avoid collecting PII without proper consent.

Troubleshooting

Session Event Not Firing

If the remark:session-initialized event isn’t firing:
  1. Check browser console for errors
  2. Verify Remark script is loading correctly
  3. Check network requests for GraphQL calls to Remark’s API
  4. Ensure no ad blockers are interfering
Debug helper:

Control Group Status Not Set

If isInRemarkABControlGroup is always null:
  1. Check that your Remark installation is complete
  2. Verify backend API is reachable
  3. Check browser console for GraphQL errors
  4. Contact Remark support if issue persists

Competing Tool Not Hiding

If the competing chat tool still shows while Remark is visible, contact support@remark.ai. The tool may have changed its markup, and Remark can update the hiding rules for your storefront.