Category

Engineering research

Problem pattern

Less studied problem / Well studied problems

Idea pattern

empirical research

Motivation

 Background

What is the research problem? Why it is an important problem? Why these solutions cannot address the problem satisfactorily?

The arms race nature of abuse has spawned a contentious battle in the realm of web cloaking. Here, miscreants seeking to short-circuit the challenge of acquiring user traffic turn to search engines and advertisement networks as a vehicle for delivering scams, unwanted software, and malware to browsing clients. Although crawlers attempt to vet content and expunge harmful URLs, there is a fundamental limitation to browsing the web: not every client observes the same content. While split views occur naturally due to personalization, geo optimization, and responsive web design, miscreants employ similar targeting techniques in a blackhat capacity to serve enticing and policy-abiding content exclusively to crawlers while simultaneously exposing victims to attacks.

Literature Review

What are the existing solutions? What is their methodology?

Indeed, earlier studies predicated their analysis on a limited set of known cloaking techniques. These include redirect cloaking in search engine results or search visitor profiling based on theUser-Agent and Referer of HTTP requests. An open question remains as to what companies and crawlers blackhat cloaking software targets, the capabilities necessary for security practitioners to bypass state of the art cloaking, and ultimately whether blackhat techniques generalize across traffic sources including search results and advertisements.

Research Niche

What motivates this work? e.g. motivated by the limitations of existing work or the lack of study for a specific issue.

Work

Research Objectives

What does the authors want to achieve in this work? What are the scope of this work? What the questions that this paper want to address?

The contentious battle between web services and miscreants involved in blackhat search engine optimization and malicious advertisements has driven the underground to develop increasingly sophisticated techniques that hide the true nature of malicious sites. These web cloaking techniques hinder the effectiveness of security crawlers and potentially expose Internet users to harmful content. In this work, we study the spectrum of blackhat cloaking techniques that target browser, network, or contextual cues to detect organic visitors.

Challenge

Why achieving the research objectives is difficult?

First, dynamic content remains a concern. If miscreants can limit the deviations introduced by cloaking to within typical norms (e.g., including only a small new button or URL), the system may fail to detect the attack. That said, this also constrains an attacker in a way that reduces click through from users. Additionally, there is a risk with news sites and other frequently updated pages that a crawler will serve incoming visitors a stale digest due to an outdated crawl, thus burdening users with alerts that are in fact false positives.

Insight

What is the main inspiration that lead to the new solution? 

Research summary

What is the proposed approach/framework/technique(s)?

In this work, we explored the cloaking arms race playing out between security crawlers and miscreants seeking to monetize search engines and ad networks via counterfeit storefronts and malicious advertisements. While a wealth of prior work exists in the area of understanding the prevalence of content hidden from prying eyes with specific cloaking techniques or the underlying monetization strategies, none marries both an underground and empirical perspective that arrives at precisely how cloaking operates in the wild today. We addressed this gap, developing an anti-cloaking system that covers a spec- trum of browser, network, and contextual blackhat targeting techniques that we used to determine the minimum crawling capabilities required to contend with cloaking today.

We informed our system’s design by directly engaging with blackmarket specialists selling cloaking software and services to obtain ten of the most sophisticated offerings. The built- in capabilities of these packages included blacklisting clients based on their IP addresses, reverse DNS, User-Agent, HTTP headers, and the order of actions a client takes upon visiting a miscreant’s webpage. We overcame each of these techniques by fetching suspected cloaking URLs from multiple crawlers that each emulated increasingly sophisticated legitimate user behavior. We compared and classified the content returned for 94,946 labeled URLs, arriving at a system that accurately detected cloaking 95.5% of the time with a false positive rate of 0.9%.

Evaluation

Evaluation summary

How do the authors compare the new solution with existing ones? What are the comparison metrics? What is the scale of the experiment? What interesting observations do the author make from their experiment results?

We present the overall accuracy of our system in Table VIII. We correctly detect 99.1% of Alexa URLs as non-cloaked with a false positive rate of 0.9%. To achieve this degree of accuracy, we overlook 18.0% of potentially cloaked counterfeit storefronts. If we examine the trade-off our system achieves between true positives and false positives, presented in Fig- ure 2, we find no inflection point to serve as a clear optimum. As such, operators of de-cloaking pipelines must determine an acceptable level of false positives. For the remainder of our study we rely on a false positive rate of 0.9%.

Implications

What are the implications of the evaluation results? What does it mean to the practitioners?

Novelty

Contributions

What are the contributions of the work? 

We frame our contributions as follows:

• We provide the first broad study of blackhat cloaking techniques and the companies affected.
• We build a distributed crawler and classifier that detects and bypasses mobile, search, and ads cloaking, with 95.5% accuracy and a false positive rate of 0.9%.
• We measure the most prominent search and ad cloaking techniques in the wild; we find 4.9% of ads and 11.7% of search results cloak against Google’s generic crawler.
• We determine the minimum set of capabilities required of security crawlers to contend with cloaking today.

Limitations

Are there any unrealistic assumptions in the approach? Are there any case where the approach does not work?

In the process, we exposed a gap between cur- rent blackhat practices and the broader set of fingerprinting techniques known within the research community which may yet be deployed. As such, we discussed future directions for breaking the cloaking arms race that included clients reporting browsing perspective to crawler operators, hindering the ability of miscreants to show benign content exclusively to search engines and ad networks.