Test Results
Read published test results and understand report metrics.
Control Group Data
Read A/B test assignments, lead IDs, and set up external monitoring.
Testing Parameters
Open Remark automatically for testing and campaigns.
Hide Competing Tools
Hide another chat tool whenever Remark appears.
How A/B Testing Works
Remark manages control group assignment automatically on the backend. When a visitor arrives on your site, the backend determines whether they should be in the control group based on the show rate Remark sets for your test. To start a test, change its show rate, or end it, contactsupport@remark.ai.
Cohorts and Show Rate
An A/B test splits eligible shopper sessions into two cohorts:
The show rate is the percentage of eligible sessions assigned to the treatment cohort. For example, a 25% show rate means roughly 25% of eligible sessions can see Remark and the remaining eligible sessions are control.
Changing the show rate changes future assignment. It does not rewrite sessions that were already assigned.
A visitor whose Remark session starts while Remark can’t appear stays in the control group for the rest of the test. That happens when chat is turned off, the brand is unpublished, the IP allowlist excludes the visitor, or no expert is available to pair. Page allow and block lists don’t affect assignment. When the test ends, control visitors start seeing Remark.
Control Group Assignment
- Persistent – Remark assigns the cohort once, when it creates the visitor’s Remark session, and keeps it as long as that browser keeps its
remark_lead. Visitors whose session began before the test aren’t assigned and keep seeing Remark. - Automatic – No client-side configuration required
- Backend-controlled – Managed by Remark’s services to maintain consistency across all entry points
Why Backend Control?
Remark stores the assignment on the visitor’s Remark session, so it holds across every way that visitor returns to Remark:- Initial page load – First visit to your site
- Page reloads – Navigating between pages or refreshing
- Email follow-ups – Clicking links in conversation notification emails
remark_lead, so a visitor who opens one on another device keeps their cohort. A visitor who switches devices any other way gets a new assignment on the new device.
Reviewing Test Results in the Dashboard
After shoppers have been assigned to cohorts, Remark publishes lift analyses to the Dashboard. Open the Remark dashboard and select the A/B Test Results flask icon on the home page. For brand users, the icon appears when results become available for your workspace. If you do not see the icon, results may not be available yet.Result dates
Each analysis has a date range and an analyzed timestamp. If there are multiple published analyses, they may cover different date ranges or analysis settings. Use the date range first when comparing reports.Metrics
The report can include summary metrics, comparison tables, and statistical test sections. Common metric types include:- Session or visitor counts
- Conversion rate
- Revenue or average order value
- Lift or percent change between control and treatment
- Confidence intervals for estimated lift
Statistical tests
Statistical test sections summarize whether the observed difference is strong enough to treat as evidence rather than normal variation.
An inconclusive result does not mean Remark had no effect. It means the analysis did not have enough evidence to call a clear result for that metric and date range.
Report tables
Tables compare control and treatment values side by side. Rows usually represent metrics or segments, and columns show counts, rates, revenue, lift, confidence intervals, or notes. Use tables to check:- Whether both cohorts have enough sessions
- Whether conversion and revenue metrics move in the same direction
- Whether one metric is driven by a small number of outlier orders
- Whether the reported lift matches the business question you are trying to answer
Missing or inconclusive results
If results are missing, check whether:- Remark has published an analysis for your workspace
- The test has enough eligible sessions and conversions
- You are looking at the intended date range
- The test was active during the period you are reviewing
support@remark.ai if you need help interpreting a report or want Remark to publish a new analysis for a different date range.
Accessing A/B Test Data
Session Initialization Event
Remark dispatches aremark:session-initialized event when the client session has initialized. This is the ideal time to set up external monitoring or analytics.
Reading Control Group Status
The control group status is stored inlocalStorage and can be accessed synchronously:
Reading Lead ID
Remark also stores the lead ID (unique visitor identifier) inlocalStorage. This is useful for correlating Remark’s A/B test data with your own analytics:
- Page reloads and navigation
- Multiple sessions on the same device
- Cross-device tracking (when visitors return via email links)
- Match Remark’s user identification with your analytics
- Track individual visitor journeys across the A/B test
- Analyze conversion rates at the individual lead level
- Correlate chat engagement with purchase behavior
Setting Up External Monitoring
If you want to run your own analytics or A/B test monitoring alongside Remark, you can use the session initialization event and control group data.Google Analytics Integration
Segment Integration
Custom Analytics Endpoint
Testing with URL Parameters
You can use theremark_open URL parameter to open the Remark widget automatically when the page loads. It opens the widget only for visitors who can already see Remark, so it doesn’t override A/B assignment. To make sure Remark appears, add remark_force_widget=true.
Open Widget on Load
- Demo and sales links – Combine it with
remark_force_widget=trueso prospects see Remark - Email campaigns – Direct customers to chat from notification emails
- Customer support – Send customers directly into a conversation
- A/B test verification – Confirm Remark appears for treatment group testing
Combining with Analytics Parameters
You can combineremark_open=true with your existing analytics parameters:
Hiding Competing Chat Tools
Remark can hide another chat tool (for example Gorgias, Zendesk, HubSpot, or Intercom) whenever Remark appears, and show it again whenever Remark doesn’t, such as for visitors in the control group. Remark configures this for your storefront. To set it up, contactsupport@remark.ai with the name of the tool you use.
Two-Pronged Approach
Remark uses a dual strategy to ensure competing tools are hidden reliably:- JavaScript API – If the tool exposes an API, Remark will call it to hide/remove the widget
- CSS Injection – Inject CSS styles to hide the tool’s DOM elements as a fallback
Testing & Debugging
Verify A/B Test Data
Test Session Initialization
Testing Scenarios
Demo Link with Auto-Open
Create a shareable demo link that shows and opens Remark automatically:Test Control vs Treatment
To see the treatment experience, open any page with?remark_force_widget=true.
To start over as a new visitor with a fresh assignment, open a page with ?remark_reset_chat=true or use a new private window. After remark:session-initialized fires, read isInRemarkABControlGroup to see which group you landed in.
Setting isInRemarkABControlGroup yourself has no effect. Remark rewrites it from the backend on every page load.
Best Practices
Wait for Session Initialization
Always wait for theremark:session-initialized event before accessing control group data or setting up integrations.
Handle Missing Data
The control group status might not be available immediately on the first page load.Non-Intrusive Monitoring
Ensure your monitoring code doesn’t interfere with the user experience:Respect Privacy
Only collect A/B test data that’s necessary for your analysis. Avoid collecting PII without proper consent.Troubleshooting
Session Event Not Firing
If theremark:session-initialized event isn’t firing:
- Check browser console for errors
- Verify Remark script is loading correctly
- Check network requests for GraphQL calls to Remark’s API
- Ensure no ad blockers are interfering
Control Group Status Not Set
IfisInRemarkABControlGroup is always null:
- Check that your Remark installation is complete
- Verify backend API is reachable
- Check browser console for GraphQL errors
- Contact Remark support if issue persists
Competing Tool Not Hiding
If the competing chat tool still shows while Remark is visible, contactsupport@remark.ai. The tool may have changed its markup, and Remark can update the hiding rules for your storefront.