StopShootingintheDark HOW TESTANDLEARNANALYTICSCANHELPORGANIZATIONSACE EVERYDAYDECISIONMAKING AnImpactAnalyticsPerspective
Ataglance Intoday sfast-movingonlineworldofe-commerce,traditionalretailersarefacing majorhardships.theyneed to innovateand testnew strategiesfastto remain competitive.testandlearnanalyticsisvitalfortoday'sretailerstobeefectiveinfast experimentation. TestandLearnanalytics,however,isacomplexanalyticalproblem whichcomprises chalenges.theanswerstothesechalengeslieinbeingabletoapplymethodswhich amalgamatebusinessunderstandingandstatisticalmethods.thekeychalengesare: 1.Appropriatetestandcontrolgroupidentification:Identificationoftestandcontrol groupsiscriticaltorolingouttheexperimentstootherstoresofthecompany 2.Measuringtheefectivenessoftheexperiment:Itisimperativetoidentifythe rightmetricstoevaluateforcomparingtestandcontrolstores. Solution to these chalenges requires an end-to-end application which enables retailerstoexecuteinnovativeandprofitablebusinessstrategies.thekeysteps neededforthisapplicationare:createhypothesis,design,execute,analyzeand evaluate results,and roloutsolutions based on outcomes.testand Learn implementationischalengingbutwithrightguidanceandtools,favorableresultscan beachieved.
1 BRICK-AND-MORTARSTORES:THEGOINGISTOUGH Brick-and-mortarstoresareundoubtedlyfacingaspateoftoughchalenges.Afterrunninga successfulbusinessformorethan4decades,aretailbookstoregiant(whichhadmorethan 500physicalstores)hadtodownitsshuters.Anestimated7000storesclosedin2017,which is17%higherthanthenumberduringthefinancialmeltdownperiod,i.e.,2008.theusmedia termedthisclosureofstoresas RetailApocalypse (refertoexhibit1).today,retailersacross industriesarefacingmultiplechalengesinoperatingtheirstores,beit Mom &Pop storesor even retailgiants.accordingtoindustryexperts,herearesomeofthemajorchalenges: Brick-andmortarstoresare facingmultiple chalengesin operatingtheir stores Onlinethreat:E-Commercehaspampereditscustomerswithamyriadofserviceslikevast productoptions,latestproductdesigns,freereturns/exchange,personalizedmarketing strategies,greaterdiscounts,etc.therangeofserviceskeepexpandingwhichisamajor threattoretailersastheydonothavethebandwidthtoprovidetheseservices. Erodingmargins:Retailstoresdonothavemuchleewaytochangethepricesoftheir productsasfrequentlyasonlineretailers.inadditiontothat,onlinecompetitorscaneasily providediferentialandspecializedpricesbasedonthedatacolectedandothermethods suchascustomersegmentation.pricechangerestrictionsandhigherinventorycostseat intothebotom-linemarginsofbrick-and-mortarstores. Personalizedofering:Onlinestorescanproviderelevantoferingswhereasretailstores struggletodothat.increaseddemandforthepersonalizedshoppingexperienceandashift inbuyingpaternshavebecomemajorissuesforretailstorestocontinuemakingaprofit. StoreclosuresinUS 17% Exhibit1:Numberof storeclosuresover theyearsinus 6134 7000 2008 Financial meltdownperiod Source:htps:/www.cbinsights.com 2017 2018
Brick-and-mortarretailersneedtoinnovatetofindsuitablestrategiestoengagecustomersand testthem efficiently.therearemanyretailerswhousetestandlearnapproachregularlyto achievebusinessgoals.onesuchretailerusedtestandlearntounderstandifthepricesoffew productscouldbeincreasedwithoutthelossofcustomers.anotherfoodretailerwantedtotest ifdecreasingsandwichpricescouldatractmorecustomers.someretailersemploytestand Learnapproachtoquantifytheefectivenessofpromotions. Whiletherearemanychalengesthatretailstoresface,theyhaveonemajoradvantageoverthe onlinechannels,i.e.,physicalspace.physicalspacesalow customerstohaveanengaging experiencewhilebrowsingformoreoptionsinperson.retailstoreshelpalthebrandstostand outandimprovetheirmarketvisibilitytoo.thebrandspresentintheseretailstoresalso encouragecustomerstospreadthewordaboutthem andenablethem toefficientlyruntheir marketingstrategies. Physicalspacecomeswithitsownsetofchalenges.Forexample, Whatistheideallocation toplacesnacksinaretailstore?shouldthejuicesbeplacednexttochips?shouldthestore investinanatractiveentertainmentzone? Providingagreatin-storeexperiencecanresultin greattractioninfootfalsviapositivewordofmouth.the WOW efectisoneofthemost sought-afterresponseswhiledecidingacustomerexperiencestrategy,however,onemustalso balancethiswithprofitabilityandfinancialsustainability. BRICK-AND-MORTARSTORESCANINNOVATEANDWIN THROUGHTESTANDLEARN SourceforExhibit2images:htps:/pixabay.com/ Retailersneedto takeadvantage oftheirbiggest asset,i.e., physicalspace Exhibit2:Physical spacesalow customerstohavean engagingexperience 2 Testingmultiplestorelayoutsandevaluatingtheresponsetoeachstrategyatasmalernumber ofunitscanhelparetailerfine-tunechangesanddeveloptheperfectcustomerengagement strategy. Testingmultiple storelayoutsand evaluatingthe responsetoeach strategyata smalernumber ofunitscanhelp ofunitscanhelp aretailerfine-tune changesand developthe perfectcustomer engagement strategy.
3 TESTANDLEARNISNOTEASYFORBRICK-AND-MORTAR RETAILERS TestandLearn strategies cannotbedevised overnight IdealTestandLearnstrategiescannotbedevisedovernight.Mostoftheexperimentsgo throughapainfulprocessoftrialanderorwithextensivetargetmeasurements.online companieshaveanaturaladvantageintestingouttheirinnovativeideas.theirtestandlearn costislowerincomparisontothebrick-and-mortarretailers.theexecutiontimefortestand Learncanbesignificantlylow ononlineplatformsthanonoflinechannels. TestandLearnisquitechalengingforalretailers.Evenwithin-depthinsightsandadvanced data analytics solutions,retailers face the majorchalenge ofdesigning the testand interpretingtheresults.however,themostsignificantchalengeistoconvinceastoreto completelyaccepttheresultsofanalyticswithouthavingtheassuranceofprofitabilityor scalabilityofresults. Theconceptwasadoptedintheretailindustryafteritwaschampionedbyotherindustries, especialybye-commercegiants.testandlearnwasinitialychampionedbyabankforsome ofitssmalprojectsbutlatertheprocesswasencouragedacrossmultipledomains.testand Learncanbecomequitechalenginginthemathematicalcomplexityofinferingresults,and someofthesechalengesinclude: CHALLENGES1:APPROPRIATETESTANDCONTROL GROUPIDENTIFICATION 1.Testgroupidentification Aretailerwithmanystoresneededtotestoutastrategythatwouldachieveitsbusiness requirementssuchasincreasedmargin.themajordrawbackwasthattheretailercouldnot employthestrategyinalstores.itwasimportanttocaryouttheexperimentinasmalnumber ofstoresthatrepresentthebrand.thisimpliesthatthecostatachedtoexperimentationalso increaseswithincreasingnumberofstores.thus,identificationofteststoresbecomes critical.thechalengeassociatedwithidentifyingagroupofteststoresishow toensurethe grouprepresentsalthestores.thebetertheteststoresrepresentation,thebeteristhe accuracyoftests in otherstores.generaly,teststores are identified afterthe store segmentationisfinishedandthenselectedfrom variousclusters.numerouschalengescan bepresentinclusteringbutalofthem canbesolvedwiththerightalgorithms.
4 2.Controlgroupidentification InaTestandLearnexperiment,theTestGroup(TG)isagroupofstoreswhereafreshstrategy action is piloted,and the ControlGroup (CG)is a group ofstores againstwhich the performanceofthetgismeasured.thecg,therefore,representsthestatusquo(ascenario ofnottakinganynew actionornew experimentation).thechalenge,therefore,in Testand Learnanalysisistoidentifytestandcontrolgroupsthatareasidenticaltoeachotheras possible.forexample,theperformanceofa10% cashbackdealatalargestoremustbe evaluatedagainstnotrunninganydealorstatusquoatanotherlargestore.comparingsales ofalargestorewiththatofasmaleronewouldbemisleading,evenifbothstoresareinthe samecountyandpossiblyhavesimilarconsumerprofileswiththem. Exhibit3 shows how diferentstores can be grouped orclustered togetherbased on similarities,andteststoresandtheircorespondingcontrolstorescanbeidentifiedfrom a group.here,theclusteringisonlybasedontwometricsforeaseofvisualilustration.however, inrealworldsituations,theclusteringisoftenbasedonmultiplefactors. Theright algorithmsand solutionscan helpovercomethe biggestchalenges oftestandlearn TestandControlstoresselection GrowthPercentage(%) 8.5 8.0 7.5 7.0 6.5 6.0 StoreClusters Exhibit3:Clusters ofstoresrequired forbestcontrol storematches 5.5 5.0 0M 2M 4M 6M 8M 10M 12M 14M 16M 18M 20M 22M 24M 26M 28M 30M Sales(USD) 3.Multipletestsorparaleltests:Howtoseparate individualimpact? Manyretailershavealimitednumberofstoresandwanttoutilizethesestoresfornumerous tests.thereisalwaysapossibilitywhereasinglestoremayhavemultipletestsrunning.insuch scenarios,the chalenge is to accurately alocate contribution from each testto the sales/marginliftobservedinthestores.ifthereexistsaconditioninwhichtestsarecompletely independentofeachother,onecanassumethatthereisnooverlapinresultsfrom thetests.to isolatethetestefectsistrickyandiftestsarenotdoneproperly,efortandmoneyarelostfor good.
Performancemetricsaretoonoisy Asthetestsarerunforacertaindurationandtherecouldbeseveralotherfactorsthatdrive salesduringthetestperiod,example,seasonality,promotions,new productlaunch,etc.when thetestdataisavailableondailybasis,thesefactorswilappearasarandom variationinsales. Therearethreepossiblewaystohandlethesenoisemeasurementissues: Aggregatedataovertesttimeframe Aggregatedsalesoraggregatedconversionratesmaybebetermatchedtothoseofcontrol stores.thisworksbestwhenthetestiscariedoutinatleast20ormorestores. 2. Multiplemeasuresformatchingtestandcontrolstores Matching Matchingthetestandcontrolunitsonlyonrevenues(asbeterperformanceintermsof revenueistheintendedoutcome)canbeinsufficient.taketheexampleoftwofoodoutlets, oneeachinlasvegasandmiami.thetwostoresmayhavesimilarannualtotalsales,yet theseasonalityofdemandcandominatetheimpactofthetestifitiscariedoutduringpeak travelseason.matchingtheoutletsbasedonseasonalsales,categorysales,demographics, etc.maygivetheextraconfidenceneededincomparing orangeswithoranges. Liftasaperformancemeasure Lift Liftinsales/marginintheteststoreorY-o-Y (YearoverYear)changeaccountsfor seasonalityandisalwaysabetercomparisonthanmeasuringtheabsolutechangein revenue. Severalcomparisonsinsteadofonepairwisecomparison Thetestandcontrolgroupscouldbeone-to-oneorone-to-many.Iftheresultsarerobust acrossalone-to-oneandone-to-manycomparisons,thentheanalysispresentsresultswith moreconfidencefortherol-out. 5 CHALLENGE2:MEASURINGTHEEFFECTIVENESSOFA TESTANDLEARNEXPERIMENT Lackofcorectbasereference Typicaly,severalsimultaneousmarketingactionsandseasonalityconfoundthecalculationof basemetrics.thebasemetricshavetworeferences:performanceoftestunitsvis-à-vistheir ownpastandperformanceoftestunitsvis-à-visthecontrolunits.thiscanbeaddressedin3 specificsteps: 1. IncreasingthesamplesizeoftheTGreducesthevarianceinthedata.Furthermore,itisan accuraterepresentationofalstoresoroveralcustomersandgivesbeterconfidencein possibilitieswitharolout. Increaseunitsinthetest/controlgroup Therecouldbe severalfactors thatdrivesales duringthetest period,which canappearas random
6 Includeadditionalvariablesthatinfluenceperformancemetrics Themeasurementoftestimpactiscompletewhileconsideringadditionalsalesdriver variablessuchasbrandpreference,categorypreference,etc.inancova (analysisof covariance) regression.ancova accounts for other drivers of performance that systematicalyinfluenceperformanceinadditiontothetestthatispiloted. TESTANDLEARNEXPERIMENTSTEPS EfficientTestandLearnneedsaluserstouseoneapplicationwhichcanenableautomationof TestandLearnandalow stackholderstocolaborate.thekeyfunctionalitiesneededinsuch anapplicationare: 2 1 Create Hypothesis Design No Execute 3 Exhibit4:Stepsto folowfortestand Learnexperiment 5 Evaluate Results RolOuts Yes Analyze 4 CreateHypothesis:ThefirststepinanyTestandLearnexperimentistoidentifytheoveral objectiveoftheexperiment.theobjectivecanberepresentedasahypothesiswhichneeds tobeproved/disproved.forexample,afast-foodchainwantedtoreducecostsbyremoving mid-sizedbeveragecupsfrom theirmenuastheybelieveditwilnotafectsales.inthiscase, thenulhypothesis(h 0 )wilstatethatmid-sizedcupsdonothavehighsalesvolumeandcan beeasilysubstitutedwithothersizedcups. Design:Metricsarethenidentifiedwhichcanhelpquantifythesuccessoftheobjective.In thiscase,numberofmid-sizedbeveragecupssold(quantity),marginsgeneratedandtheir associationwithotheritemsbecomekeymetricstoapprove/rejecth 0.Thenextstepisto selectafew storesfortheexperimentbeforetheresultscanberoledouttoalthestores.
7 Testandcontrolstoresmustbeidentifiedthroughstoreclusteringandsimilaritiesamong storesbasedonautomatedanalytics.theapplicationshouldsuggestthebesttestand controlgroups. Execute:Oncethetestandcontrolstoresareidentified,theexperimentneedstoberoled outinaphasedmannertoprovideopportunitiestoironoutthewrinklesexperiencedinthe firstfew stores.thesestoresalsoneedtobetaggedastestandcontrolstoresforaccurate trackingofresults. Analyze:Theapplicationthenrecordsandanalyzesthedataafterthetestandtracksthe performanceofthekeymetrics.theseresultscanalsobevisualizedusingasimple dashboardthatrecordsthemetricsatastorelevel. Evaluate Results:Bycomparing the keymetrics between TG and CG stores using techniquessuchast-tests,anovaandancova,thehypothesiscanbeapproved/rejected withcertainstatisticalconfidence.exhibit5below isanilustrationthatshowsa 2%positive growthintgstores. RolOuts:Thefinalstepistoestimatetheimpactiftheexperimentistoberoledouttoal stores.usersshouldbeabletovisualizetheimpactthroughin-builtsimulationsinthe application. InExhibit5,itisshownhow testandcontrolstoreswereidentifiedandtheirperformance duringtestperiodwascomparedwithsameperiod,thepreviousyear.controlstoresshowed anaturalgrowthof3%,whileteststoresshowedagrowthof5%,whichincludesthenatural growthof3%.thus,2%isthenetimpactofthetestandlearnexperimentconducted.these resultscanbegeneratedbyanovamethodwhichcanprovideconfidencevaluesonthe results. 1.03M 1M s 3% Sales Exhibit5:ANOVA analyticsshowing theimpactofthetest whencomparedtothe performanceof controlactions (USD) 1.05M 1M Lastyear Presentyear s 5% 3% Growthdueto experiment 2% Sales (USD) Naturalgrowth Lastyear Presentyear
8 CASESTUDIES ImpactAnalyticshasdoneseveralTestandLearnprojectsformultipleclientsintheretailsector andimplementedthetestandlearnapplication.hereareafew casestudiesshowingour approachtotestandlearnformajorusretailers: CASESTUDYI:INVENTORYMANAGEMENTFORASPECIALTYRETAILER IAhasworkedwitharetailerstrugglingwithdecreasingtransactionsize,decliningrevenuesand oldstockinventory.a strategytoreducetheoldinventoriesbyputingthem inprominent positionswasdesigned.thebusinessobjectivewasanalyzedbasedonseveralparameterssuch asanincreaseinrevenue,increaseinaffinityrelated categories,cannibalizationinother categories,adecreaseinrevenueofreplacedcategories,etc.someoftheinsightswere: Storeswereclusteredbasedonbusinessparametersandclusteringalgorithms.Idealtest andcontrolstoreswereidentified,andthepilotprojectwasroledoutin60stores. Theresultsweretrackedfor6weeksanditwasdiscoveredthattherewasanetincreaseof 9.75%inrevenuepertransaction.Theexperimentalsoledtoincreasedinventoryspacefor highmovingitems,whichshouldincreasebasketsizefurther. Thepilottestwasthenroledouttoasetofnext100stores. Revenuepertransaction $21.95 $20 9.75% Exhibit6:Revenueper transactionsizeduring testperiod
9 CASESTUDY I:PRICEEFFECTIVENESSFORABEVERAGERETAILER Anotherfamousretailerwantedtoincreasethemarginsthroughraisingthepricesofbeverages acrosscategoriesanddecidedtotestthisoutbyrunninganexperiment.ia developeda specializedstoresegmentationandmultiplepricingmodelswiththehelpofmachinelearning algorithms.thepricingmodelssuggestedmanyproductstobepriceinelastic,i.e.,thedemand wouldnotchangesignificantlyevenifthepriceswereincreased.thefolowingapproachwas used: Intheteststores,pricesofcertainitemswereincreasedby3% onaverageafterthe productswerecarefulyselected.productswithcannibalizationcapabilitywereremoved andaffinityrelateditemswerenotintroducedeither. Thepilotranfor3monthsandresultsshowedanincreaseof5.2%inmargin.Thenumberof unitssolddecreasedmarginalywhichwasexpectedbutoveraltherevenueincreased leadingtoanincreaseinthemargin. Averagepriceperproduct Unitssold (alunitsin000 s) $3.15 $3.26 3.49% 142 141-0.7% Exhibit7:Efectsof TestandLearnon diferentbusiness metricsfora beverageretailer Incrementinrevenue 460k 448k 2.6% Incrementinmargin 303k 5.2% 288k
10 CASESTUDY I:TARGETINGCAMPAIGNFORASPECIALTYRETAILER Customersegmentationwasdoneforapetsuppliesretailer.12distinctclusterswerecreated aftercomputingcustomerbehaviormetricsinan8-dimensionalhyperplaneandbymeasuring Euclideandistancesbetweenthem. Apromotionstrategywasdevisedforasegmentofcustomerswhogeneralytransactwith abasketsizeof$46,generating$13inmargin.theobjectiveofthepromotionstrategywas toincreasethemarginandrevenuefrom thecustomers. A testwasdesignedtointroducespecificcouponsforasegmentofcustomers.the customerswereprovidedwithcouponsthroughemailwhichcouldberedeemedduring weekendsonly.asimilarsetofcustomerswereidentifiedasthecgwhowouldnotreceive thecoupons. Afterthetestwasexecuted,itwasfoundthatthebasketsizeforthetargetedcustomers increasedfrom $46to$80,withmarginsincreasingsubstantialy.TheCGdidnotshow any significantincreaseinbasketsizeormargin. Thetestwasdeclaredsuccessfulwith73%increaseinbasketsizeandanincreaseof23.5% inmarginforthecustomersegment. Discountpercentage Incrementinvisitfrequency (000 s) 20% 17% 3% 100 105 5% Incrementinbasketsize $80 Incrementinprofitpertransaction $21 Exhibit8:Efectsof TestandLearnon diferentbusiness metricsfora specialtyretailer $46 73.9% $17 23.5%
11 CONCLUSION Innovationisamustforanybusinesstosurviveinthelongrun.Onlineplatformshaveanatural advantagewhenimplementinginnovativestrategies,butbrick-and-mortarretailerscanalsowin. RetailerscanleverageinnovativestrategiesusingtheTestandLearnprinciples.TestandLearn implementationischalengingbutwiththerightmethodologyandtools,favorableresultscanbe achieved.impactanalyticshasprovideditstestandlearnservicestomanyretailersinthepast andimplementedanapplicationwhichcanbecustomizedaspertheclients requirements.
AtImpactAnalytics,weconceptualize,developanddeploy360- degreedatasciencesolutionsfordatadrivendecisionmaking. Ouroferingsspanacrossforecasting,pricingandpromotions management,consumerandmarketinganalytics,visualisation andreporting,etc.,alrightsreserved Learnmoreat www.impactanalytics.co Formoreinformationcontactusat info@impactanalytics.co Address 780ElkridgeLandingRoad Linthicum MD.21090 Folow Us