Download Report

Safe Browsing
Quick setup
8.10.2014
Safe Browsing. Quick setup. Version 1.1
Document build date: 8.10.2014.
This volume is a part of Yandex technical documentation.
Yandex helpdesk site: http://help.yandex.ru
© 2008—2014 Yandex LLC. All rights reserved.
Copyright Disclaimer
Yandex (and its applicable licensor) has exclusive rights for all results of intellectual activity and equated to them means of individualization, used for development, support,
and usage of the service Safe Browsing. It may include, but not limited to, computer programs (software), databases, images, texts, other works and inventions, utility models,
trademarks, service marks, and commercial denominations. The copyright is protected under provision of Part 4 of the Russian Civil Code and international laws.
You may use Safe Browsing or its components only within credentials granted by the Terms of Use of Safe Browsing or within an appropriate Agreement.
Any infringements of exclusive rights of the copyright owner are punishable under civil, administrative or criminal Russian laws.
Contact information
Yandex LLC
http://www.yandex.com
Phone: +7 495 739 7000
Email: pr@yandex-team.ru
Headquarters: 16 L'va Tolstogo St., Moscow, Russia 119021
Contents
Introduction .......................................................................................................................................................................................... 4
Overview .............................................................................................................................................................................................. 4
Restrictions .......................................................................................................................................................................................... 4
Integration .......................................................................................................................................................................................... 5
SafeBrowsing API PHP SDK ...................................................................................................................................................... 6
SafeBrowsing API Python Client Version 0.3 ............................................................................................................................. 6
phpGSB v0.2.3 ............................................................................................................................................................................. 6
Google Safe Browsing API for .NET (April 25, 2011) ............................................................................................................... 7
General recommendations for changing parameters .................................................................................................................... 8
Processing user complaints .......................................................................................................................................................... 9
Safe Browsing
Quick setup
Quick setup 4
Introduction
The Safe Browsing API is a programming interface that lets you check whether a URL is in the lists of suspected
malicious pages. The lists are compiled and maintained by Yandex.
This document summarizes and classifies widespread threats and covers restrictions to using the programming
interface. It also describes how to quickly integrate the API using freely distributed libraries. The quick setup
is available for projects implemented in the following programming languages:
•
Python
•
PHP
•
C#.Net
Overview
As the internet expands, there are a growing number of documents that are potential threats to users' technical
devices (computers and mobile devices).
Potentially dangerous documents can be divided into two groups:
•
“Malware” causes malicious code to be executed. Running it may cause data to be leaked or lost, and can
also harm user devices. It may be authorized (for example, when downloading and running an executable
file) or unauthorized (for example, a spyware attack).
•
“Phishing” requests confidential user data for further unauthorized use. These pages are copies of sites where
the user probably has an account. When the user authenticates, confidential data is intercepted without
authorization.
Yandex maintains a database of pages that are considered dangerous for user devices. The database is updated
regularly.
External applications have access to the database via the Safe Browsing API (SB API). If the address of a
requested document matches a URL from the database, it indicates a high risk of threat posed by this document.
Access to the potentially dangerous page is blocked, and the appropriate warning message is returned to the
application user.
Restrictions
Access to the SB API requires an API key. Each API key is subject to the following daily limits (depending
on which comes first):
•
Verifying 100,000 URLs.
•
Requests from 10,000 users.
If you expect to exceed these limits when using the SB API, send email to safesearch@yandex.ru and negotiate
terms for using the SB API under a high load.
Safe Browsing
Quick setup
5
Integration
To quickly integrate the SB API with a project implemented in Python, PHP or C#.Net, complete the following
steps:
1. Get an API key.
2. Download library source files.
Libraries are available for projects that are implemented using one of the following programming languages:
Programming language
Libraries available
Python
SafeBrowsing API Python Client Version 0.3 .
PHP
C#.Net
•
SafeBrowsing API PHP SDK.
•
phpGSB 0.2.3.
C#.Net
Note:
For C++ projects, you can extract and re-use open code from code bases for the Mozilla Firefox
and Chromium browsers.
3. Make corrections to the source code of the selected library.
The list of changes depends on the library and version you are using:
•
SafeBrowsing API PHP SDK.
Tip:
The library is ready for use. You do not need to make any changes.
•
SafeBrowsing API Python Client Version 0.3.
•
phpGSB v0.2.3.
•
Google Safe Browsing API for .NET (April 25, 2011).
Attention!
Libraries are updated regularly. The names of files that contain the required parameters might not match
the ones described here if you are using a different version of a library. In this case, you need to follow
the general recommendations for changing parameters.
4. Integrate a library function call in your project.
5. Optional. Configure an output page containing detailed information about the suspected document.
The page is called using an HTTP GET request in the following format:
http://yandex.ru/infected?l10n=<language code for returning the information
page (ISO 639-1 format)>&url=<address of the infected page>
http://yandex.ru/infected?l10n=ru&url=http://www.wmconvirus.narod.ru/
After completing these steps, initialize downloading the SB database using the selected library. Download speed
depends on data transfer capacity.
Safe Browsing
Quick setup
6
SafeBrowsing API PHP SDK
To download the library, use this link.
The library is ready for use. You do not need to make any changes.
SafeBrowsing API Python Client Version 0.3
To download the library, use this link.
To make changes to a library, follow these steps:
1. Save the diff file in the library directory that contains the client.py file.
2. Execute this command:
patch client.py sb-python.diff
In some cases (for example, if you are using a different version of the library), it may be necessary to do this
manually.
The table below shows a list of source files in the library that require changes.
File
client.py
Changes
1.
Find the following section:
def
__init__(self,
ds,
apikey,
hp=('safebrowsing.clients.google.com', 80),
ssl_hp=('sb-ssl.google.com', 443), base_path='/
safebrowsing',
2.
Substitute the following:
def __init__(self, ds, apikey, hp=('sba.yandex.net', 80),
ssl_hp=('sba.yandex.net', 443), base_path='',
server.py
1.
Find the following section:
sbls.append(sblist.List(line.strip()))
2.
Substitute the following:
if line.strip().endswith('-shavar'):
sbls.append(sblist.List(line.strip()))
phpGSB v0.2.3
To download the library, use this link.
The table below shows a list of source files in the library that require changes.
File
Changes
phpgsb.class.ph
1.
p
Find the following line:
var $apiversion = "2.2";
2.
Safe Browsing
Substitute the following:
Quick setup
7
File
Changes
var $apiversion = "2.3";
3.
Find the following section:
function googleDownloader($url,$options,$followbackoff=false)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
4.
Substitute the following:
function googleDownloader($url,$options,$followbackoff=false)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
5.
Find the following line:
$result
=
$this->googleDownloader("http://
safebrowsing.clients.google.com/safebrowsing/downloads?
client=api&apikey=".$this->apikey."&appver=".$this>version."&pver=".$this->apiversion,$buildopts,"data");
6.
Substitute the following:
$result = $this->googleDownloader("http://sba.yandex.net/downloads?
client=api&apikey=<API key value>"."&appver=2.3"."&pver=".$this>apiversion,$buildopts,"data");
Attention!
Substitute with the API key value provided to you.
7.
Find the following line:
$result
=
$this->googleDownloader("http://
safebrowsing.clients.google.com/safebrowsing/gethash?
client=api&apikey=".$this->apikey."&appver=".$this>version."&pver=".$this->apiversion,$buildopts,"lookup");
8.
Substitute the following:
$result = $this->googleDownloader("http://sba.yandex.net/gethash?
client=api&apikey=<API key value>”."&appver=2.3"."&pver=".$this>apiversion,$buildopts,"lookup");
Attention!
Substitute with the API key value provided to you.
Google Safe Browsing API for .NET (April 25, 2011)
To download the library, use this link.
The table below shows a list of source files in the library that require changes.
File
Changes
Google.SafeBrow
1.
sing/API.cs
Safe Browsing
Find the following line:
Quick setup
8
File
Changes
return
@"http://safebrowsing.clients.google.com/safebrowsing/{0}?
client=api&apikey={1}&appver={2}&pver=2.2";
2.
Substitute the following:
return
@"http://sba.yandex.net/{0}?client=api&apikey=<API
value>&appver=2.3&pver=2.3”;
key
Attention!
Substitute with the API key value provided to you.
General recommendations for changing parameters
Parameters can be stored in configuration files, or they may be hardcoded in the program code.
The way that parameters are stored can differ depending on the version of the library you are using. For this
reason, when using a different library version than the ones provided, you must first determine where
the parameters to be changed are located.
The list of parameters that need to be changed differs depending on the programming language that the program
is written in.
In general, you must find and replace the following lines in the library files, using the data you received when
generating an API key:
Safe Browsing
Found string
Modified string
Example
http://
safebrowsing.clients.google
.com/safebrowsing/
downloads?
client=api&apikey=APIKEY&ap
pver=CLIENTVER&pver=PVER&wr
key=MACKEY
http://sba.yandex.net/
downloads?
client=api&apikey=<API key
value>&appver=2.3&pver=2.3
http://sba.yandex.net/
downloads?
client=api&apikey=12345&appv
er=2.3&pver=2.3
http://
safebrowsing.clients.google
.com/safebrowsing/gethash?
client=api&apikey=APIKEY&ap
pver=CLIENTVER&pver=PVER&wr
key=MACKEY
http://sba.yandex.net/
gethash?
client=api&apikey=<API key
value>&appver=2.3&pver=2.3
http://sba.yandex.net/
gethash?
client=api&apikey=12345&appv
er=2.3&pver=2.3
https://sb-ssl.google.com/
safebrowsing/newkey?
client=api&apikey=APIKEY&ap
pver=CLIENTVER&pver=PVER
https://sba.yandex.net/
newkey?
client=api&apikey=<API key
value>&appver=2.3&pver=2.3
http://sba.yandex.net/
newkey?
client=api&apikey=12345&appv
er=2.3&pver=2.3
http://
safebrowsing.clients.google
.com/safebrowsing/list?
client=api&apikey=APIKEY&ap
pver=CLIENTVER&pver=PVER
http://sba.yandex.net/list?
client=api&apikey=<API key
value>&appver=2.3&pver=2.3
http://sba.yandex.net/list?
client=api&apikey=12345&appv
er=2.3&pver=2.3
Quick setup
9
Attention!
Substitute with the API key value provided to you.
Processing user complaints
The SB API includes the ability to update the database of potentially harmful documents based on user
complaints. When a complaint is received, it is reviewed by Yandex employees. If the threat is confirmed,
the URL is added to the list of malicious pages.
User complaints are made using the feedback form for the Yandex.Webmaster service. To embed links in the
feedback form, you need to find and modify the following lines in the source files of the library you are using:
Found string
Modified string
Depending on the library you are using, you need to find one or more http://webmaster.yandex.ru/
of the following lines:
delspam.xml?l10n=<code for the
Safe Browsing
•
http://www.google.com/safebrowsing/
report_phish/?continue=http%3A%2F
%2Fwww.google.com%2Ftools%2Ffirefox%2Ftoolbar
%2FFT2%2Fintl%2F%3Clang%3E
%2Fsubmit_success.html&hl=en
•
http://www.google.com/safebrowsing/
report_error/?continue=http%3A%2F
%2Fwww.google.com%2Ftools%2Ffirefox%2Ftoolbar
%2FFT2%2Fintl%2Fen%2Fsubmit_success.html&hl=en
•
http://www.google.com/safebrowsing/
report_badware/
language used on the feedback page
(in ISO 639-1
format)>&url=<address of the page
the user reported>
Quick setup
Safe Browsing
Quick setup
8.10.2014