> ## Documentation Index > Fetch the complete documentation index at: https://docs.remark.ai/llms.txt > Use this file to discover all available pages before exploring further. # A/B Testing & Gradual Rollouts > Learn how to run A/B tests with Remark and set up external monitoring Remark includes built-in A/B testing capabilities that allow you to gradually roll out the chat widget to your customers while maintaining a control group. This enables you to measure the impact of Remark on your key business metrics. Read published test results and understand report metrics. Read A/B test assignments, lead IDs, and set up external monitoring. Open Remark automatically for testing and campaigns. Automatically hide alternative chat solutions during A/B tests. Best practices for phased rollouts from 10% to 100%. ## How A/B Testing Works Remark manages control group assignment automatically on the backend. When a visitor arrives on your site, the backend determines whether they should be in the control group based on your configured rollout percentage. ### Cohorts and Show Rate An A/B test splits eligible shopper sessions into two cohorts: | Cohort | What shoppers see | | --------- | -------------------------------------------------------------------------------------------- | | Control | Remark is hidden, so shoppers continue through your site without the Remark chat experience. | | Treatment | Remark can appear according to your widget settings. | The **show rate** is the percentage of eligible sessions assigned to the treatment cohort. For example, a 25% show rate means roughly 25% of eligible sessions can see Remark and the remaining eligible sessions are control. Changing the show rate changes future assignment. It does not rewrite sessions that were already assigned. ### Control Group Assignment * **Persistent** – The same visitor will always see the same experience across sessions * **Automatic** – No client-side configuration required * **Backend-controlled** – Managed by Remark's services to maintain consistency across all entry points ### Why Backend Control? Remark uses backend control for A/B test assignment to ensure consistent experiences across multiple entry points and devices. Visitors can engage with Remark through various channels: * **Initial page load** – First visit to your site * **Page reloads** – Navigating between pages or refreshing * **Email follow-ups** – Clicking links in conversation notification emails * **Mobile transfers** – Switching between mobile and desktop devices * **Cross-device continuity** – Moving from one device to another mid-conversation If A/B test assignment were determined client-side only, a visitor could see Remark on desktop but not on mobile, or vice versa. Backend control ensures that once assigned to a group, a visitor remains in that group regardless of how they access your site. ## Reviewing Test Results in the Dashboard After shoppers have been assigned to cohorts, Remark publishes lift analyses to the Dashboard. Open the Remark dashboard and select the **A/B Test Results** flask icon on the home page. For brand users, the icon appears when results become available for your workspace. If you do not see the icon, results may not be available yet. ### Result dates Each analysis has a date range and an analyzed timestamp. If there are multiple published analyses, they may cover different date ranges or analysis settings. Use the date range first when comparing reports. ### Metrics The report can include summary metrics, comparison tables, and statistical test sections. Common metric types include: * Session or visitor counts * Conversion rate * Revenue or average order value * Lift or percent change between control and treatment * Confidence intervals for estimated lift ### Statistical tests Statistical test sections summarize whether the observed difference is strong enough to treat as evidence rather than normal variation. | Item | How to read it | | ------- | -------------------------------------------------------------------------------------------------------------- | | Verdict | Plain-language result for the metric, such as whether the result is significant or inconclusive. | | p-value | Lower values mean the observed difference is less likely to be random noise. | | 95% CI | The estimated range for the true effect. If the range crosses zero, the direction is less certain. | | Power | How much ability the test had to detect the measured effect size. Low power means the test may need more data. | An inconclusive result does not mean Remark had no effect. It means the analysis did not have enough evidence to call a clear result for that metric and date range. ### Report tables Tables compare control and treatment values side by side. Rows usually represent metrics or segments, and columns show counts, rates, revenue, lift, confidence intervals, or notes. Use tables to check: * Whether both cohorts have enough sessions * Whether conversion and revenue metrics move in the same direction * Whether one metric is driven by a small number of outlier orders * Whether the reported lift matches the business question you are trying to answer ### Missing or inconclusive results If results are missing, check whether: * Remark has published an analysis for your workspace * The test has enough eligible sessions and conversions * You are looking at the intended date range * The test was active during the period you are reviewing If results are inconclusive, the most common reasons are limited traffic, limited conversions, a short test window, or high order-value variance. Contact `support@remark.ai` if you need help interpreting a report or want Remark to publish a new analysis for a different date range. ## Accessing A/B Test Data ### SDK Initialization Event Remark dispatches a `remark:sdk-initialized` event when the SDK has fully loaded and initialized. This is the ideal time to set up external monitoring or analytics. ```javascript theme={null} window.addEventListener("remark:sdk-initialized", () => { // SDK is fully loaded and ready // Safe to access control group data const isControlGroup = localStorage.getItem("isInRemarkABControlGroup"); console.log("Control group status:", isControlGroup); }); ``` ### Reading Control Group Status The control group status is stored in `localStorage` and can be accessed synchronously: ```javascript theme={null} const isInControlGroup = localStorage.getItem("isInRemarkABControlGroup"); if (isInControlGroup === "true") { // User is in the control group (will NOT see Remark) console.log("Control group - Remark hidden"); } else if (isInControlGroup === "false") { // User is in the treatment group (will see Remark) console.log("Treatment group - Remark visible"); } ``` The value is stored as a **string**, not a boolean. Always use `=== "true"` or `=== "false"` for comparisons. ### Reading Lead ID Remark also stores the lead ID (unique visitor identifier) in `localStorage`. This is useful for correlating Remark's A/B test data with your own analytics: ```javascript theme={null} const leadId = localStorage.getItem("remark_lead"); if (leadId) { // Use this ID to track the same visitor in your analytics console.log("Remark Lead ID:", leadId); } ``` This lead ID remains consistent across: * Page reloads and navigation * Multiple sessions on the same device * Cross-device tracking (when visitors return via email links) Including the lead ID in your external A/B test monitoring allows you to: * Match Remark's user identification with your analytics * Track individual visitor journeys across the A/B test * Analyze conversion rates at the individual lead level * Correlate chat engagement with purchase behavior ## Setting Up External Monitoring If you want to run your own analytics or A/B test monitoring alongside Remark, you can use the SDK initialization event and control group data. ### Google Analytics Integration ```javascript theme={null} window.addEventListener("remark:sdk-initialized", () => { const isInControlGroup = localStorage.getItem("isInRemarkABControlGroup"); const leadId = localStorage.getItem("remark_lead"); // Send A/B test group to Google Analytics if (typeof gtag !== "undefined") { gtag("event", "remark_ab_test", { event_category: "ab_testing", event_label: isInControlGroup === "true" ? "control" : "treatment", remark_lead_id: leadId, // Include lead ID for tracking non_interaction: true }); // Set as custom dimensions for all subsequent events gtag("set", "user_properties", { remark_ab_group: isInControlGroup === "true" ? "control" : "treatment", remark_lead: leadId }); } }); ``` ### Segment Integration ```javascript theme={null} window.addEventListener("remark:sdk-initialized", () => { const isInControlGroup = localStorage.getItem("isInRemarkABControlGroup"); const leadId = localStorage.getItem("remark_lead"); if (typeof analytics !== "undefined") { analytics.track("Remark A/B Test Assigned", { group: isInControlGroup === "true" ? "control" : "treatment", remark_lead_id: leadId, timestamp: new Date().toISOString() }); // Optionally set lead ID as user identifier for cross-session tracking if (leadId) { analytics.identify(leadId, { remark_ab_group: isInControlGroup === "true" ? "control" : "treatment" }); } } }); ``` ### Custom Analytics Endpoint ```javascript theme={null} window.addEventListener("remark:sdk-initialized", () => { const isInControlGroup = localStorage.getItem("isInRemarkABControlGroup"); const leadId = localStorage.getItem("remark_lead"); // Send to your custom analytics endpoint fetch("https://your-analytics.com/api/track", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ event: "remark_ab_test_assigned", properties: { controlGroup: isInControlGroup === "true", leadId: leadId, timestamp: Date.now(), pageUrl: window.location.href } }) }); }); ``` ## Testing with URL Parameters For A/B testing purposes, you can use the `remark_open` URL parameter to open the Remark widget automatically when the page loads. This ensures visitors in the treatment group see and interact with Remark. ### Open Widget on Load ``` https://yoursite.com?remark_open=true ``` This parameter is useful for: * **Demo and sales links** – Guarantee prospects see Remark when visiting your site * **Email campaigns** – Direct customers to chat from notification emails * **Customer support** – Send customers directly into a conversation * **A/B test verification** – Confirm Remark appears for treatment group testing ### Combining with Analytics Parameters You can combine `remark_open=true` with your existing analytics parameters: ``` # Track email campaign performance https://yoursite.com?remark_open=true&utm_source=email&utm_campaign=ab_test # Track demo link clicks https://yoursite.com?remark_open=true&utm_source=demo&utm_medium=sales_deck ``` ## Hiding Competing Chat Tools When running an A/B test between Remark and another chat solution (e.g., HubSpot, Gorgias, Intercom), Remark will automatically hide the competing tool for users in the treatment group. ### Two-Pronged Approach Remark uses a dual strategy to ensure competing tools are hidden reliably: 1. **JavaScript API** – If the tool exposes an API, Remark will call it to hide/remove the widget 2. **CSS Injection** – Inject CSS styles to hide the tool's DOM elements as a fallback This redundancy ensures the competing tool is hidden even if Remark loads before the other tool's API is available. ### Example: HubSpot ```javascript theme={null} const styleTag = document.createElement('style'); function hideHubSpot() { // Approach 1: Use HubSpot's JavaScript API try { window.HubSpotConversations.widget.remove(); } catch (e) { console.debug('HubSpot API not yet available'); } // Approach 2: Inject CSS to hide the widget styleTag.innerHTML = ` div#hubspot-messages-iframe-container[role="region"] { display: none !important; } `; document.head.appendChild(styleTag); } function showHubSpot() { // Remove CSS styleTag.remove(); // Reload HubSpot widget try { window.HubSpotConversations.widget.load(); } catch (e) { console.debug('Error loading HubSpot chat'); } } ``` ### Example: Gorgias ```javascript theme={null} const styleTag = document.createElement('style'); function waitForGorgias(callback) { let checkInterval = setInterval(() => { const containerDiv = document.querySelector('div#gorgias-chat-container'); if (typeof window.GorgiasChat !== 'undefined' && containerDiv) { clearInterval(checkInterval); window.GorgiasChat.init().then(() => { callback(); }); } }, 50); // Timeout after 60 seconds setTimeout(() => { if (typeof window.GorgiasChat === 'undefined') { clearInterval(checkInterval); console.error('GorgiasChat not loaded after 60 seconds'); } }, 60000); } function hideGorgias() { if (typeof window.GorgiasChat !== 'undefined') { window.GorgiasChat.hideChat(true); } // CSS fallback styleTag.innerHTML = ` div#gorgias-chat-container { display: none !important; } `; if (!styleTag.isConnected) { document.head.appendChild(styleTag); } } function showGorgias() { if (typeof window.GorgiasChat !== 'undefined') { window.GorgiasChat.hideChat(false); } styleTag.remove(); const widgetDiv = document.querySelector('div#gorgias-chat-container'); if (widgetDiv) { widgetDiv.removeAttribute('style'); } } ``` ### Example: Intercom ```javascript theme={null} const styleTag = document.createElement('style'); function hideIntercom() { // JavaScript API try { if (typeof window.Intercom !== 'undefined') { window.Intercom('hide'); } } catch (e) { console.debug('Intercom API not yet available'); } // CSS fallback styleTag.innerHTML = ` .intercom-lightweight-app, #intercom-container { display: none !important; } `; document.head.appendChild(styleTag); } function showIntercom() { styleTag.remove(); try { if (typeof window.Intercom !== 'undefined') { window.Intercom('show'); } } catch (e) { console.debug('Error showing Intercom'); } } ``` ## Testing & Debugging ### Verify A/B Test Data ```javascript theme={null} // Check if localStorage is being set correctly console.log("Control Group Status:", localStorage.getItem("isInRemarkABControlGroup")); console.log("Lead ID:", localStorage.getItem("remark_lead")); // Verify both values are available for external tracking const abTestData = { isControlGroup: localStorage.getItem("isInRemarkABControlGroup") === "true", leadId: localStorage.getItem("remark_lead") }; console.log("A/B Test Data:", abTestData); ``` ### Test SDK Initialization ```javascript theme={null} let sdkInitialized = false; window.addEventListener("remark:sdk-initialized", () => { sdkInitialized = true; console.log("✓ SDK initialized successfully"); }); // Check after a few seconds setTimeout(() => { if (!sdkInitialized) { console.error("✗ SDK did not initialize within 5 seconds"); } }, 5000); ``` ### Testing Scenarios #### Demo Link with Auto-Open Create a shareable demo link that opens Remark automatically: ``` https://yoursite.com?remark_open=true ``` Combine with UTM parameters for tracking: ``` https://yoursite.com?remark_open=true&utm_source=demo&utm_campaign=sales ``` #### Test Control vs Treatment Side-by-Side Open two browser profiles or incognito windows to compare experiences: **Window 1 - Control Group:** ```javascript theme={null} localStorage.setItem("isInRemarkABControlGroup", "true"); location.reload(); ``` **Window 2 - Treatment Group:** ```javascript theme={null} localStorage.setItem("isInRemarkABControlGroup", "false"); location.href = location.origin + location.pathname + "?remark_open=true"; ``` ## Best Practices ### Wait for SDK Initialization Always wait for the `remark:sdk-initialized` event before accessing control group data or setting up integrations. ```javascript theme={null} // ✓ Good - wait for SDK window.addEventListener("remark:sdk-initialized", () => { const isControlGroup = localStorage.getItem("isInRemarkABControlGroup"); // Use the data... }); // ✗ Bad - might run before SDK initializes const isControlGroup = localStorage.getItem("isInRemarkABControlGroup"); ``` ### Handle Missing Data The control group status might not be available immediately on the first page load. ```javascript theme={null} window.addEventListener("remark:sdk-initialized", () => { const isControlGroup = localStorage.getItem("isInRemarkABControlGroup"); if (isControlGroup === null) { console.warn("Control group status not yet available"); return; } // Proceed with your logic... }); ``` ### Non-Intrusive Monitoring Ensure your monitoring code doesn't interfere with the user experience: ```javascript theme={null} window.addEventListener("remark:sdk-initialized", () => { try { // Your monitoring code } catch (e) { // Fail silently - don't break the user experience console.error("Monitoring error:", e); } }); ``` ### Respect Privacy Only collect A/B test data that's necessary for your analysis. Avoid collecting PII without proper consent. ```javascript theme={null} // ✓ Good - anonymous tracking analytics.track("Remark A/B Test", { group: isControlGroup === "true" ? "control" : "treatment" }); // ✗ Bad - avoid PII unless necessary analytics.track("Remark A/B Test", { group: isControlGroup === "true" ? "control" : "treatment", email: userEmail, // Avoid name: userName // Avoid }); ``` ## Troubleshooting ### SDK Event Not Firing If the `remark:sdk-initialized` event isn't firing: 1. Check browser console for errors 2. Verify Remark script is loading correctly 3. Check network requests for GraphQL calls to Remark's API 4. Ensure no ad blockers are interfering Debug helper: ```javascript theme={null} console.log("Checking for Remark SDK..."); console.log("window.remark exists:", typeof window.remark !== "undefined"); setTimeout(() => { console.log( "Control group status:", localStorage.getItem("isInRemarkABControlGroup") ); }, 3000); ``` ### Control Group Status Not Set If `isInRemarkABControlGroup` is always `null`: 1. Check that your Remark installation is complete 2. Verify backend API is reachable 3. Check browser console for GraphQL errors 4. Contact Remark support if issue persists ### Competing Tool Not Hiding If the competing chat tool isn't being hidden: 1. Verify the tool's JavaScript API is available (`window.HubSpotConversations`, etc.) 2. Check CSS selectors are correct for the tool's DOM structure 3. Inspect DOM to find the tool's container elements 4. Update CSS selectors if the tool has changed its markup Use browser DevTools to inspect the competing tool's DOM structure and adjust your hide/show functions accordingly.