ATRG - URL Filtering

5/6/22, 1:26 PM ATRG: URL Filtering
Free Demo Contact Us Support Center Blog Welcome: Girum Mesfin
TRY OUR QUOTING OFFERINGS / ASSETS / SUPPORT /

PRODUCTS TOOLS UPSELLS INFO SERVICES
Support Center > Search Results > SecureKnowledge Details
Search Support Center
ATRG: URL Filtering Technical Level
Rate This My Favorites Email Print
Solution ID sk92743
Technical Level
Product URL Filtering

Version R77.30 (EOL), R80.10 (EOL), R80.20, R80.30, R80.40, R81, R81.10
Platform / Model All
Date Created 01-May-2013
Last Modified 19-Apr-2022
Solution
Table of Contents
Introduction
URL Filtering Categorization Flow
Features
HTTPS Filtering
Safe Search
Cached and Translated pages categorization
URL Filtering Agent
Related documentation
Related solutions
Introduction
Check Point's latest firewall innovation brings the industry's strongest URL Filtering, application and identity control to organizations of all sizes. You can easily create
policies which detect or block thousands of applications and internet sites.
Use the Application Control and URL Filtering blades to:
Learn about the applications
Use Check Point's comprehensive AppWiki to understand what applications are used for and what their risk levels are.
Create a Granular Policy
Make rules to allow or block applications or Internet sites, by individual application, application or URL categories, or risk levels. When you use Identity Awareness,
you can easily make rules for individuals or different groups of users. You can also create an HTTPS policy that enables the gateway to inspect HTTPS traffic to
prevent security risks related to the SSL protocol.
Learn What Your Employees are Doing
Use SmartView Tracker and SmartEvent to understand the application and site traffic that really occurs in your environment. Then change the policy to make it even
more effective. Only administrators that have been assigned with relevant permissions can see all the fields in a log. Using these permissions makes sure that
restricted data is kept private in logs and cannot be seen by all administrators.
Keep Your Policies Updated
Application and URL Filtering Database is updated regularly with applications and site categories to help you keep your policy current. The gateway connects to the
Check Point Online Web Service to identify social networking widgets and website categories for URLs that it does not recognize. Results are stored on a local cache
on each Security Gateway. Subsequent uncategorized URLs are first checked against the local cache before querying the Check Point Online Web Service.
https://supportcenter.checkpoint.com/supportcenter/portal?action=portlets.SearchResultMainAction&eventSubmit_doGoviewsolutiondetails=&solutioni… 1/8
Custom Applications, Sites, Categories and Groups
You can create applications, websites, categories, and groups that are not in the Application and URL Filtering Database for use in the policy. Use these custom
objects to create a Rule Base that meets your organization's requirements. It is also possible to contact Check Point to create customized application signatures that
can be imported into the database. This file can contain, for example, a database with an organization's internal applications that are not necessarily web-based.
URL Filtering Categorization Flow
URL Categorization occurs mostly in the kernel:
The kernel cache is queried for the URL.

The cache lookup first checks for the host, and if there are members, it looks in the members array. If the URL is found in the cache, its categories are returned.
If there is no URL in the cache, then a-sync request is issued, asking for online categorization.
The user space daemon sends the URL to its user-space module that communicates with Check Point's online detection service.
Once the online service responds with categories, the kernel cache is updated.
Features
Feature - HTTPS Filtering
Categorization of HTTPS sites without HTTPS inspection (passive HTTPS). Supports URL Filtering on HTTPS traffic without HTTPS inspection.
To enable it, enable the URL Filtering blade:
In SmartDashboard, go to Application & URL Filtering tab -> Advanced -> Engine Settings -> Enable "Categorize HTTPS sites", and install Security Policy.
How it works
A cache table cptls_server_cn_cache is used here. The cache saves a mapping between IP+Port to CN (certificate's Canonical Name), and a flag if the CN is valid.
When SSL traffic is coming, the request (client hello) is recognized by PM (Pattern Matcher) matching on one of SSL request signatures internal procedure. Then the IP+Po
will be extracted from the connection and will be searched in the CPRLS cache and a CN (certificate's Canonical Name) might be found.
If a CN is found and valid, it will be sent to RAD (Resource ADvisor - internal engine responsible for URL Filtering) and the traffic will be categorized according to it.
Otherwise, the traffic is continued to be inspected.
When a response (server "Hello") arrives, it is recognized by PM (Pattern Matcher) matching on one of SSL response signatures internal procedure, the certificate will be
extracted and sent to WSTLS process in User mode (in one or more traps). The process will try to resolve the certificate and find its CN.
It will return an asynchronous ioctl with response to the kernel with the IP+Port and the CN (if found).
If a CN is returned, it will be sent to RAD and will be categorized according to it. Otherwise, the IP address will be sent to RAD and will be categorized according to it.
Example: if the certificate's CN is google.com, the categories will be "search engine/portal"
Debugging & Troubleshooting
Run the following debug on the Security Gateway:
CLI command Description
Debug wstlsd in User space - refer to sk105559 (output is saved in

for PROC in $(pidof wstlsd) ; do fw debug $PROC on TDERROR_ALL_ALL=6 ; done
$FWDIR/log/wstlsd.elg)
fw ctl debug 0
fw ctl debug -buf 32000
Debug cptls in kernel (output is saved in /var/log/kernel_debug.txt)

fw ctl debug -m fw + cptls
fw ctl kdebug -T -f > /var/log/kernel_debug.txt
fw ctl debug 0
Debug APPI in kernel (output is saved in /var/log/appi_debug.txt)

fw ctl debug -m APPI all
fw ctl kdebug -T -f > /var/log/appi_debug.txt
fw ctl debug 0
Debug RAD in kernel (output is saved in /var/log/rad_debug.txt)

fw ctl debug -m RAD_KERNEL all
fw ctl kdebug -T -f > /var/log/rad_debug.txt
Debug RAD in User space (output is saved in $FWDIR/log/rad.elg)
rad_admin rad debug on all To stop, run:
rad_admin rad debug off ALL
Problem: HTTPS Filtering feature is not working.
Troubleshooting:
1. Make sure you are surfing via port 443 (and not through proxy).
2. Check that the website you are surfing to is certificated by one of the trusted CAs configured in SmartDashboard.
In SmartDashboard, go to Application & URL Filtering tab -> Advanced -> HTTPS Inspection -> Trusted CAs.
If not, add its certificate.
3. Check that wstlsd process is up.
Run:
ps -ax | grep wstlsd
4. Check that the 'cptls_server_cn_cache' cache table is loaded.
Run:
fw tab -t cptls_server_cn_cache -s
5. Try to find your connection in the table.
Run:
fw tab -t cptls_server_cn_cache
and search your IP+Port (in hex) in the 2 left columns.
6. Analyze the debug output file:
a. Check that the client side was matched by the PM.
Search for:
appi_rad_uf_cmi_handler_match_signature: group_data->group_id 10002002
b. Check if the IP+Port was found in the CPTLS table.
Search for:
fwtls_get_server_cn_from_cache_ex
c. Check if the server side was matched by the PM.
Search for:
d. Check if the certificate was extracted from the traffic.
Search for:
"CERTIFICATE"
e. Check if the traps were sent to WSTLSD process.
Search for:
appi_rad_uf_cmi_handler_ssl_sn_trap
when you see "_last 1", it means it is the last trap.
f. Check if you got ioctl from WSTLDS process.
Search for:
appi_rad_uf_ssl_cn_cmi_ioctl
g. Check if anything was sent to RAD.
Search for:
rad_kernel_urlf_request_set_url
7. Make sure you have connectivity to RAD and get response.
Feature - Safe Search
Safe Search feature in web browsers is designed to screen sites that contain sexually explicit content and remove them from the search results. While no filter is 100%
accurate, Safe Search helps to avoid content you may prefer not to see or would rather your children did not stumble across.
User can change the settings of Safe Search from the browser for each specific search engine.
This feature supports Safe Search in search engines (currently Google, Bing, and Yahoo).
To enable it, enable the URL Filtering blade. In SmartDashboard, go to Application & URL Filtering tab -> Advanced -> Engine Settings -> enable "Enforce Safe Search in
search engines", and install Security Policy.
How it works
Each search engine has its saved "&meter=value" that is added to the URL when 'Safe search' feature is enabled or disabled. For example, when enabling Safe Search
engine in Google, it adds "&safe=active" to the URL. When disabling it, it adds "&safe=off" to the URL.
fw ctl debug 0
Debug Application Control in kernel (output is saved in

/var/log/appi_debug.txt)
Replicate the issue, or wait for the issue to occur.
Press CTRL+C to stop the kernel debug.
fw ctl debug 0
Problem: Safe Search feature is not working.
Troubleshooting:
1. Make sure that you use one of the supported search engines (Google, Bing or Yahoo).
2. Capture the traffic using WireShark and see if you get a redirect from the Security Gateway.
3. Verify that your URL is matched on one of the signatures from $FWDIR/appi/update/appi_db.C:
In the debug output file, search for:
appi_rad_uf_cmi_handler_match_signature: group_data->group_id #Group Number of Relevant Section of Signatures from urlf_group#
4. If one of the signatures was matched on your URL, analyze the debug output and see how the redirect URL was built.
appi_rad_uf_cmi_handler_safe_search_handler: redirect_url: #Editted URL#
5. If no signature was matched on your URL, send your URL to app_request@checkpoint.com with a request to add a new signature to our list.
Feature - Cached and Translated pages categorization
With this feature disabled, when a user accesses the cached pages or Google translated pages, the matched category is ‘Search Engine’ since we drop the query string and
send to RAD (Resource ADvisor) the Search Engine domain only.
For example, when accessing http://www.ynet.co.il cached page from Google, the URL looks like: http://webcache.googleusercontent.com/search?
ix=seb&sourceid=chrome&ie=UTF-8&q=cache%3Awww.ynet.co.il.
The category will be the category of "webcache.googleusercontent.com" (Search Engine).
This feature extracts the correct domain from the cached / Google translated URL buffer, sends it to RAD and then gets the exact category of the page.
To enable it, enable the URL Filtering blade. In SmartDashboard, go to Application & URL Filtering tab -> Advanced -> Engine Settings -> Enable "Enforce Safe Search in
search engines", and install Security Policy.
How it works
The signatures are compiled to one PM (Pattern Matcher) that recognizes URLs that try to access the cached/translated pages.
When a URL is coming, the PM matching (built from the relevant signatures of this feature) is executed against it.
If there is a match on one of the signatures, it means that it is a cached or translated page.
According to the signature match offset, the host is extracted and sent to RAD and the client will get the correct category of the page.
fw ctl debug 0
fw ctl kdebug -T -f > /var/log/rad_debug.txt

Debug RAD in kernel (output is saved in /var/log/rad_debug.txt)
fw ctl debug 0
rad_admin rad debug on all
Debug RAD in User space (output is saved in $FWDIR/log/rad.elg)

fw ctl debug 0

Debug APPI in kernel (output is saved in /var/log/appi_debug.txt)
fw ctl debug 0
Problem: Cached/Translated pages feature is not working.
Troubleshooting:
1. Make sure that you are using one of the supported search engines on cached pages (Google, Bing or Yahoo). And Google for translated pages.
2. Verify that your URL is matched with one of the signatures from $FWDIR/appi/update/appi_db.C file.
3. If one of the signatures was matched on your URL, run debug and see how the cached/translated domain is extracted from the URL.
appi_rad_uf_cmi_handler_extract_url_after_pm_match
4. See that the extracted domain is sent to RAD.
In the debug file, search for:
rad_kernel_urlf_request_set_url
5. Make sure you have connectivity to RAD and get response.
6. If no signature was matched on your URL, send your URL to app_request@checkpoint.com with a request to add a new signature to our list.
URL Filtering Agent

General Description
The new URL Filtering software blade uses a hosted categorization service (cloud-based categorization) that is being constantly updated to cope with the dynamic nature of
the web. The Security Gateway is not required to download updates, the categorization database is not limited by the storage space on the Security Gateway, and customers
receive the latest categorization to keep policies current.
As requests are being sent to the cloud, responses are being cached locally on the Security Gateway so that subsequent requests for the same sites will not query the cloud
but will be answered by the local cache. In deployments of R75.20 URL Filtering, we have seen that more than 99% of requests are being met by the cache, providing
customers with the most up-to-date categorization (with the cloud) on the one hand and minimal latency (as most requests are being met by the cache), on the other.
When there are critical updates, for example, a previously benign site has been hacked and is now used to spread malware, the cache is invalidated and forced to retrieve
updates from the cloud.
Processes
The URL Filtering Software Blade (R75.20 and higher) uses a single user mode process called RAD ($FWDIR/bin/rad) to communicate with the online URL Filtering service.
The process is initiated once Application Control, or URL Filtering Software Blades, are active on the Security Gateway. Whenever RAD successfully categorizes a URL, it
caches it in its kernel cache.
The URL Filtering kernel cache holds the entire URL Family (all the URLs within the same host). Whenever there is a request for a new URL categorization, RAD sends to th
online categorization service only the host part of the URL and receives the entire URL family. The response is then cached for future use.
Important Files
RAD holds its configuration in these files:
$FWDIR/conf/rad_conf.C - RAD settings.

$FWDIR/conf/rad_settings.C - RAD proxy settings.
$FWDIR/conf/rad_scheme.C - A file describing where and how to connect to the online services; reloaded on each install policy event.
$FWDIR/conf/rad_services.C - A database file that holds the kernel cache setting, such as cache table sizes and max allowed memory usage.
URL Filtering Updates and local database (LDB) files
The RAD kernel cache holds the entire URL Family. Whenever there is a request for a new URL categorization, RAD sends to the online categorization service only the host
part of the URL and receives the entire URL family. To keep the response size short, the online service answers only if the URL family has 100 or fewer members. 99% of the
URL database are part of families with 100 or fewer members and can be retrieved using the online service. The remaining 1% of the database is aggregated in a special
file: $FWDIR/appi/update/urlf_db.bin
The file is reloaded on each URL Filtering update. Before sending a new online categorization request to the service, RAD looks in the local LDB file for a match. Only if the
URL does not exist in the file, a request to the service is made.
To reload the LDB file, run:
rad_admin urlf update $FWDIR/appi/update/urlf_db.bin
Website categorization mode
You can select the mode that is used for website categorization:
Background: requests are allowed until categorization is complete. When a request cannot be categorized with a cached response, an uncategorized response is
received. Access to the site is allowed. In the background, the Check Point Online Web Service continues the categorization procedure. The response is then cached
locally for future requests (default). This option reduces latency in the categorization procedure.
Hold: requests are blocked until categorization is complete. When a request cannot be categorized with the cached responses, it remains blocked until the Check
Point Online Web Service completes categorization.
RAD CLI Commands

Topic Command Explanation Example

Activation rad_admin start Start RAD process
Activation rad_admin stop Stop RAD process
Activation rad_admin install Install new RAD policy. RAD will reparse its rad_setting.C file.
Debug rad_admin rad debug on <topic> Start debug (output is saved in $FWDIR/log/rad.elg)
rad_admin rad debug on all
Note: "all" must be in small letters
Debug rad_admin rad debug off Stop debug

Note: "ALL" must be in capital letters
Statistics rad_admin stats on <service> Start collecting statistics rad_admin stats on urlf
Statistics rad_admin stats print <service> Print current statistics to rad_admin stats print urlf
$FWDIR/log/<service>_rad_stats.csv
Statistics rad_admin stats reset <service> Reset all counters rad_admin stats reset urlf
Statistics rad_admin stats print_reset <service> Print current statistics to rad_admin stats print_reset urlf
$FWDIR/log/<service>_rad_stats.csv and reset the counters
Statistics rad_admin stats off <service> Stop collecting statistics rad_admin stats off urlf
Update rad_admin urlf update <full path> Reload URL Filtering local database file
Fast Path rad_admin fastpath <service> Issue a request and update security categories in Security rad_admin fastpath urlf
Gateway's cache
Main Debug Flags

GW/
Kernel/
CLI Command Description

Mgmt User Space
fw ctl debug 0
Security Kernel Start debug of RAD kernel module
Gateway
fw ctl kdebug -T -f > /var/log/debug_kernel.txt
fw ctl debug 0
rad_admin rad debug on

Security User Space Start debug of RAD process

Gateway (debug is saved in $FWDIR/log/rad.elg)
rad_admin rad debug off
Getting the list of URLs Security Gateway sends to RAD (Resource ADvisor)
Important - This procedure makes the Security Gateway log every RAD request. This increases the size of log files and takes disk space. Make sure to restore the default
setting as soon as possible.
1. Connect to the command line on the Security Gateway / each Cluster Member.
2. Log in to the Expert mode.
3. Back up the current $FWDIR/conf/rad_conf.C file:
cp -v $FWDIR/conf/rad_conf.C{,_BKP}
4. Edit the current $FWDIR/conf/rad_conf.C file:
vi $FWDIR/conf/rad_conf.C
5. Change the value of the parameter "normal_flow_events":
from
:normal_flow_events (false)
to
:normal_flow_events (true)
6. Save the changes in the file and exit Vi editor.
7. Restart the RAD process:
rad_admin restart
8. Wait at least 30 minutes.
9. Get the list of URLs and save it in a file:
grep "Resource=" $FWDIR/log/rad_events/Normal/* -A 1 | awk -F "=" '{print $2}' | sort | uniq -c > /var/log/RAD_URLs.txt
10. Make sure the file contains URLs:
cat /var/log/RAD_URLs.txt
11. Restore the default $FWDIR/conf/rad_conf.C file:
cp -v -f $FWDIR/conf/rad_conf.C{_BKP,}
12. Restart the RAD process:
rad_admin restart
Related Documentation
Security Management Administration Guide
Security Gateway Administration Guide
Managing Pre-R80 Application Control and URL Filtering with R80 Administration Guide
Related Solutions
sk163793 - How to scale up requests/responses RAD handling rates
sk65124 - URL Filtering Blade FAQ
sk115652 - Users are still able to post and chat on various web sites, although Application & URL Filtering policy is configured to block those web sites
sk169920 - Sporadic timeouts in the Anti-Malware/URL Filtering service (RAD)
Give us Feedback Please rate this document [1=Worst,5=Best]
Enter your comment here

Comment
SECURE YOUR EVERYTHING ™ Follow Us    

©1994-2022 Check Point Software Technologies Ltd. All rights reserved.
Copyright | Privacy Policy

ATRG - URL Filtering

Uploaded by

Copyright:

Available Formats

ATRG - URL Filtering

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ATRG - URL Filtering

Uploaded by

Copyright:

Available Formats

5/6/22, 1:26 PM ATRG: URL Filtering

Free Demo Contact Us Support Center Blog Welcome: Girum Mesfin

TRY OUR QUOTING OFFERINGS / ASSETS / SUPPORT /

Search Support Center

ATRG: URL Filtering Technical Level

Rate This My Favorites Email Print

Product URL Filtering

Use the Application Control and URL Filtering blades to:

Learn about the applications

Create a Granular Policy

Learn What Your Employees are Doing

Keep Your Policies Updated

URL Filtering Categorization Flow

URL Categorization occurs mostly in the kernel:

The kernel cache is queried for the URL.

To enable it, enable the URL Filtering blade:

Example: if the certificate's CN is google.com, the categories will be "search engine/portal"

Debugging & Troubleshooting

Run the following debug on the Security Gateway:

CLI command Description

Debug wstlsd in User space - refer to sk105559 (output is saved in

fw ctl debug -buf 32000

Debug cptls in kernel (output is saved in /var/log/kernel_debug.txt)

fw ctl kdebug -T -f > /var/log/kernel_debug.txt

fw ctl debug -buf 32000

Debug APPI in kernel (output is saved in /var/log/appi_debug.txt)

fw ctl kdebug -T -f > /var/log/appi_debug.txt

fw ctl debug -buf 32000

Debug RAD in kernel (output is saved in /var/log/rad_debug.txt)

fw ctl kdebug -T -f > /var/log/rad_debug.txt

Debug RAD in User space (output is saved in $FWDIR/log/rad.elg)

rad_admin rad debug on all To stop, run:

rad_admin rad debug off ALL

Problem: HTTPS Filtering feature is not working.

If not, add its certificate.

3. Check that wstlsd process is up.

ps -ax | grep wstlsd

4. Check that the 'cptls_server_cn_cache' cache table is loaded.

5. Try to find your connection in the table.

and search your IP+Port (in hex) in the 2 left columns.

a. Check that the client side was matched by the PM.

b. Check if the IP+Port was found in the CPTLS table.

c. Check if the server side was matched by the PM.

d. Check if the certificate was extracted from the traffic.

e. Check if the traps were sent to WSTLSD process.

when you see "_last 1", it means it is the last trap.

f. Check if you got ioctl from WSTLDS process.

g. Check if anything was sent to RAD.

7. Make sure you have connectivity to RAD and get response.

Feature - Safe Search

Debugging & Troubleshooting

Run the following debug on the Security Gateway:

CLI command Description

fw ctl debug -buf 32000

fw ctl debug -m APPI all

Debug Application Control in kernel (output is saved in

Press CTRL+C to stop the kernel debug.

Problem: Safe Search feature is not working.

In the debug output file, search for: