Setting up a proper A/B split test to measure the effects of a new search engine can be difficult for some eCommerce retailers. Especially for those that haven't worked with split testing in the past.
Analysing the effect of a change in functionality without running a proper A/B test can be tricky since it involves comparing the performance of a website today with how it performed during a previous period. Moreover, there is no way of controlling all noise in the data, such as seasonality, changes in marketing efforts and so on.
To mitigate this problem, an assumption can be made that the performance change of non-searching visitors between two periods of time acts as a reference as to how one would expect the performance of searching visitors to change during the same period.