SRX UTM: Web Filtering (Enhanced Web Filtering)

In this post I am going to look at Enhanced Web Filtering as a follow up to the Web Filtering (Local) post.

Enhanced Web Filtering is known as the Integrated option and you should know that it has replaced the Surfcontol option. Juniper has advised me that Surfcontrol is no longer orderable.

GET AND INSTALL THE EWF LICENSE 

a) Download the trial license.
a. Login to this site with your Juniper ID.. https://www.juniper.net/lcrs/mylic.do?methodToCall=setUpTrial&family_id=1
b. Select your version of Junos, enter your serial number and click “Get Available Trials”
c. Select Websense Enhanced Web Filtering (Trial Period : 30 Days ) and click the "Generate" button to get your license key

Here is what it should look like
... 

blogger@LEFTY> show system license
License usage:
                                 Licenses     Licenses    Licenses    Expiry
  Feature name                       used    installed      needed
  idp-sig                               1            1           0    2012-09-18 10:00:00 EST
  dynamic-vpn                           0            2           0    permanent
  ax411-wlan-ap                         0            2           0    permanent
  mem-upg                               0            1           0    permanent
  wf_key_websense_ewf                   0            1           0    2012-10-09 11:00:00 EST

Licenses installed:
  License identifier: JUNOS125706
  License version: 2
  Valid for device: AU2811AF0436
  Features:
    idp-sig          - IDP Signature
      date-based, 2012-08-19 10:00:00 EST - 2012-09-18 10:00:00 EST

  License identifier: JUNOS134119
  License version: 2
  Valid for device: AU2811AF0436
  Features:
    wf_key_websense_ewf - Web Filtering EWF

      date-based, 2012-09-09 10:00:00 EST - 2012-10-09 11:00:00 EST

  License identifier: JUNOS322061
  License version: 2
  Valid for device: AU2811AF0436
  Features:
    mem-upg          - Memory Upgrade
      permanent
  
CONFIGURATION

a) Configure UTM URL Pattern Custom Objects (Optional)

We have this already configured from Web Filtering (Local) post so we will use again.
 

blogger@LEFTY> show configuration security utm custom-objects and categories.
url-pattern {
    good-list {
        value http://www.juniper.net;
    }
    bad-list {
        value http://www.cisco.com;
    }
}
custom-url-category {
    good-category {
        value good-list;
    }
    bad-category {
        value bad-list;
    }
}

b) Configure the Web Filtering Feature Profile

blogger@LEFTY# set security utm feature-profile web-filtering type ?
Possible completions:
  juniper-enhanced    
  juniper-local       
  surf-control-integrated 
  websense-redirect   
[edit]
blogger@LEFTY# set security utm feature-profile web-filtering type juniper-enhanced

blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced ?             
Possible completions:
+ apply-groups         Groups from which to inherit configuration data
+ apply-groups-except  Don't inherit configuration data from these groups
> cache               
> profile              Juniper enhanced profile
> server               Juniper enhanced server

--> Set the cache size and timeout

blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced cache size 500

[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced cache timeout 1800

---> Set the Threatseeker cloud hostname and the port to communicate to it on

set security utm feature-profile web-filtering juniper-enhanced server host rp.cloud.threatseeker.com
set security utm feature-profile web-filtering juniper-enhanced server port 80

Note that this URL is also used in some some Juniper doco...cluster-k.cloud.threatseeker.com however the one in the set commands above is working.

---> Create a profile to be used as the UTM Web Filtering policy

blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile ?        
Possible completions:
  <name>               Juniper enhanced profile name
[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile ?  
Possible completions:
+ apply-groups         Groups from which to inherit configuration data
+ apply-groups-except  Don't inherit configuration data from these groups
> block-message        Juniper enhanced block message settings
> category             Juniper enhanced category
  custom-block-message  Juniper enhanced custom block message sent to HTTP client
  default              Juniper enhanced profile default
> fallback-settings    Juniper enhanced fallback settings
  no-safe-search       Do not perform safe-search for Juniper enhanced protocol
> site-reputation-action  Juniper enhanced site reputation action
  timeout              Juniper enhanced timeout (1..1800 seconds)


---> Define categories and actions

blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile category ?
Possible completions:
  <name>               Name of Juniper enhanced category
  Enhanced_Abortion   
  Enhanced_Abused_Drugs 
  Enhanced_Adult_Content 
  Enhanced_Adult_Material 
  Enhanced_Advertisements 
  Enhanced_Advocacy_Groups 
  Enhanced_Alcohol_and_Tobacco 
  Enhanced_Alternative_Journals 
  Enhanced_Bandwidth  
  Enhanced_Blogs_and_Personal_Sites 
  Enhanced_Bot_Networks 
  Enhanced_Business_and_Economy 
  Enhanced_Computer_Security 
  etc etc etc


There is 100+ categories to choose from.

Lets say we want to block access to porn and permit but log access to social networking sites such as facebook.

set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile category Enhanced_Adult_Content action block
set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile category Enhanced_Adult_Material action block
set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile category Enhanced_Sex action block
set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile category Enhanced_Social_Networking_and_Personal_Sites action log-and-permit

---> Set site reputation values. This Enhanced Web Filtering service returns reputation values for sites as well as category.

blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile site-reputation-action ?                                
     
Possible completions:
  <[Enter]>            Execute this command
+ apply-groups         Groups from which to inherit configuration data
+ apply-groups-except  Don't inherit configuration data from these groups
  fairly-safe          Action when site reputation is fairly safe
  harmful              Action when site reputation is harmful
  moderately-safe      Action when site reputation is moderately safe
  suspicious           Action when site reputation is suspicious
  very-safe            Action when site reputation is very safe
  |                    Pipe through a command

[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile site-reputation-action very-safe ?
Possible completions:
  block               
  log-and-permit      
  permit  
            
[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile site-reputation-action very-safe permit 

[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile site-reputation-action fairly-safe permit 

[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile site-reputation-action moderately-safe log-and-permit   
 
[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile site-reputation-action suspicious block                 

[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile site-reputation-action harmful block


---> Set a block message for the profile

blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile custom-block-message "BLOCKED BY ENHANCED!"

---> Set the default actions for the profile  

blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile default ?                                    
Possible completions:
  block                Block action
  log-and-permit       Log and permit action
  permit               Permit action
[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile default log-and-permit


---> Set the fallback setting for the profile

blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile fallback-settings ?           
Possible completions:
  <[Enter]>            Execute this command
+ apply-groups         Groups from which to inherit configuration data
+ apply-groups-except  Don't inherit configuration data from these groups
  default              Fallback default settings
  server-connectivity  Fallback action when device cannot connect to server
  timeout              Fallback action when connection to server timeout
  too-many-requests    Fallback action when requests exceed the limit of engine
  |                    Pipe through a command
[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile fallback-settings default ?
Possible completions:
  block               
  log-and-permit      
[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile fallback-settings default log-and-permit

[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile fallback-settings default server-connectivity log-and-permit                                         

[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile fallback-settings timeout log-and-permit               

[edit]
blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile fallback-settings too-many-requests block 

---> Set the timeout to trigger fallback settings


blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile timeout 10  


---> Disable safe search

About this, Juniper says.."search requests have safe-search strings attached to them, and redirect response is sent to ensure that all search requests are safe or strict"

Most Juniper examples seem to show this feature disabled.

blogger@LEFTY# set security utm feature-profile web-filtering juniper-enhanced profile wf-e-profile no-safe-search

Ok lets see what it all looks like...



blogger@LEFTY> show configuration security utm feature-profile web-filtering
url-blacklist bad-category;
type juniper-enhanced;
juniper-local {
    profile wf-profile {
        default permit;
        custom-block-message ">>NOT PERMITTED<<";
        fallback-settings {
            default block;
            too-many-requests block;
        }
    }
}
juniper-enhanced {
    cache {
        timeout 1800;
        size 500;
    }
    server {
        host rp.cloud.threatseeker.com;
        port 80;
    }
    profile wf-e-profile {
        category {
            Enhanced_Adult_Content {
                action block;
            }
            Enhanced_Adult_Material {
                action block;
            }
            Enhanced_Sex {
                action block;
            }
            Enhanced_Social_Networking_and_Personal_Sites {
                action log-and-permit;
            }
        }
        site-reputation-action {
            very-safe permit;
            moderately-safe log-and-permit;
            fairly-safe permit;
            suspicious block;
            harmful block;
        }
        default log-and-permit;
        custom-block-message "BLOCKED BY ENHANCED!";
        fallback-settings {            
            default log-and-permit;    
            server-connectivity log-and-permit;
            timeout log-and-permit;    
            too-many-requests block;   
        }                              
        timeout 10;                    
        no-safe-search;
    }
}
 
Note the old juniper-local setting are still there from my SRX UTM: Web Filtering (Local) post

c) Apply the Web Filtering profile to the UTM Policy

blogger@LEFTY# set security utm utm-policy utm-protect web-filtering http-profile wf-e-profile

TESTING



First check the status..

blogger@LEFTY# run show security utm web-filtering status         

 UTM web-filtering status:
    Server status: Juniper Enhanced using Websense server UP


Ok, now lets see what happens when we try Playboy and Penthouse...




Worth noting that the categories for those 2 sites are different if you wish to block that type of thing in the corporate environment.

Here is the log messages...

Sep 10 17:13:13  LEFTY RT_UTM: WEBFILTER_URL_BLOCKED: WebFilter: ACTION="URL Blocked" 10.10.10.50(2093)->66.254.102.216(80) CATEGORY="Enhanced_Adult_Content" REASON="BY_PRE_DEFINED" PROFILE="wf-e-profile" URL=www.playboy.com OBJ=/ USERNAME=N/A ROLES=N/A

Sep 10 17:15:02  LEFTY RT_UTM: WEBFILTER_URL_BLOCKED: WebFilter: ACTION="URL Blocked" 10.10.10.50(2105)->208.88.180.54(80) CATEGORY="Enhanced_Sex"
REASON="BY_PRE_DEFINED" PROFILE="wf-e-profile" URL=www.penthouse.com OBJ=/ USERNAME=N/A ROLES=N/A

So again, as per the Local web filtering post -  Block means to log by default

Now lets see that when we access facebook that it records a log hit.

Sep 10 16:54:14  LEFTY RT_UTM: WEBFILTER_URL_BLOCKED: WebFilter: ACTION="URL Blocked" 10.10.10.50(1816)->184.31.207.139(80) CATEGORY="Enhanced_Social_Networking_and_Personal_Sites" REASON="BY_PRE_DEFINED" PROFILE="wf-e-profile" URL=connect.facebook.net OBJ=/en_us/all.js USERNAME=N/A ROLES=N/A


So that worked great - logged and permitted as per policy.


Finally lets try Cisco which isn't in any Enhanced category in the config but is still in the Blacklist...



So that shows that black/white lists and Enhanced Web Filtering can work together and it also shows the lists take precedence over the pre-defined categories as we have blocked Cisco by blacklist but not via any pre-definedcategory.

The full search order as per Juniper info is..

blacklist
whitelist
user-defined category
pre-defined category
safe-search
site reputation
default action

This also means that any specific site or sites in a blocked category could be specifically allowed by adding them to and using a whitelist

You can see that stats with this command

blogger@LEFTY> show security utm web-filtering statistics   
 UTM web-filtering statistics:
    Total requests:                  449
    white list hit:                  0
    Black list hit:                  4
    Queries to server:               436
    Server reply permit:             396
    Server reply block:              14
    Custom category permit:          0
    Custom category block:           0
    Site reputation permit:          26
    Site reputation block:           0
    Cache hit permit:                5
    Cache hit block:                 4
    Safe-search redirect:            0
    Web-filtering sessions in total: 4000
    Web-filtering sessions in use:   0
    Fallback:                       log-and-permit           block
          Default                                 0               0
          Timeout                                 0               0
     Connectivity                                 0               0
Too-many-requests                                 0               0

Finally lets remove the old black list (As we are now blocking by Threatseeker Cloud category) and add a custom category for sites we wish to permit access to but which are being blocked by category - I.e an overrides user-defined category. I like this way of doing things as it keeps the config consistent with only the use of categories rather than categories and black/white lists.

For example ..let say we wanted to block all social networking sites except facebook (Look its just an example ok!)

We block the predefined category "Enhanced_Social_Networking_and_Personal_Sites" and create a custom user-defined category using a custom-url-category that has facebook in it. Then we permit that user-defined category. We will call the custom category "overides".  


An important note - when doing these kinds of overrides, always use the leading wildcard "*" otherwise other elements of the web page may be blocked. Sometimes that isnt even enough as you will see below.

Here is a guide on using wildcards in the url-patterns

https://www.juniper.net/techpubs/software/junos-security/junos-security10.1/junos-security-swconfig-security/index.html?topic-44593.html

Here is the config.

blogger@LEFTY> show configuration security utm   
custom-objects {
    url-pattern {
        good-list {
            value http://*.facebook.com;
        }
    }
    custom-url-category {
        overides {
            value good-list;
        }
    }
}
feature-profile {
    web-filtering {
        type juniper-enhanced;
        juniper-enhanced {
            cache {
                timeout 1800;
                size 500;
            }
            server {
                host rp.cloud.threatseeker.com;
                port 80;
            }
            profile wf-e-profile {
                category {
                    Enhanced_Adult_Content {
                        action block;
                    }
                    Enhanced_Adult_Material {
                        action block;
                    }
                    Enhanced_Sex {
                        action block;
                    }
                    Enhanced_Social_Networking_and_Personal_Sites {
                        action block;
                    }
                    overides {
                        action permit
;
                    }
                }
                site-reputation-action {
                    very-safe permit;
                    moderately-safe log-and-permit;
                    fairly-safe permit;
                    suspicious block;
                    harmful block;
                }
                default log-and-permit;
                custom-block-message "BLOCKED BY ENHANCED!";
                fallback-settings {
                               

And this is what we see when we access facebook...


Well that doesn't look quite right.

A look in the log shows us that facebook is being permitted by our user-defined overides catergory...

Sep 15 21:15:28 LEFTY clear-log[10234]: logfile cleared
Sep 15 21:15:39  LEFTY utmd[1264]: WEBFILTER_SERVER_CONNECTED: Successfully connected to webfilter server rp.cloud.threatseeker.com
Sep 15 21:15:39  LEFTY last message repeated 5 times
Sep 15 21:15:49  LEFTY RT_UTM: WEBFILTER_URL_PERMITTED: WebFilter: ACTION="URL Permitted" 10.10.10.50(2239)->69.171.228.70(80) CATEGORY="overides" REASON="BY_USER_DEFINED" PROFILE="wf-e-profile" URL=www.facebook.com OBJ=/ USERNAME=N/A ROLES=N/A


However we do see a lot of URLs being blocked at the same time by the pre-defined social networking catergory even though we have permitted facebook in the overides category which should take precedence and allow it to be permitted. 


What we see though is those blocked URLs do not match *.facebook.com which is why they are being blocked and causing the page not to load properly.

Sep 15 21:15:50  LEFTY RT_UTM: WEBFILTER_URL_BLOCKED: WebFilter: ACTION="URL Blocked" 10.10.10.50(2244)->203.206.129.90(80)CATEGORY="Enhanced_Social_Networking_and_Personal_Sites" REASON="BY_PRE_DEFINED" PROFILE="wf-e-profile" URL=static.ak.fbcdn.net OBJ=/rsrc.php/v2/yu/r/qjwkundl5sb.css USERNAME=N/A ROLES=N/A
 

Sep 15 21:15:50  LEFTY RT_UTM: WEBFILTER_URL_BLOCKED: WebFilter: ACTION="URL Blocked" 10.10.10.50(2245)->203.206.129.67(80)CATEGORY="Enhanced_Social_Networking_and_Personal_Sites" REASON="BY_PRE_DEFINED" PROFILE="wf-e-profile" URL=photos-d.ak.fbcdn.net OBJ=/photos-ak-snc7/v85005/171/451311101547435/app_105_451311101547435_1883532661.jpg USERNAME=N/A ROLES=N/A

So turns out Facebook is being served up by Akamai. I will guess that ak.fbcdn means Akamai Facebook Content Delivery Network.

Lets add that to the url-pattern good-list which is what the overides use.

blogger@LEFTY# set security utm custom-objects url-pattern good-list value http://*.ak.fbcdn.net

Tested and working now. Take my work for it ...I am not showing the facebook frontpage here...

So the moral of the story is you may need to do a bit of troubleshooting to get an overide to work. The log is your friend..as always.

To end ..here is the command to clear the web filtering cache

blogger@LEFTY> request security utm web-filtering juniper-enhanced cache flush   
Web-Filtering cache flush request result: Flush cache OK



Model: srx100h
JUNOS Software Release [12.1R3.5]

2 comments:

  1. Nice post.very usefull information you have posted here.please carry on with posting such informative posts

    Web filtering software

    ReplyDelete
  2. Thanks Japos!
    Appreciate the feedback.

    Rgs

    Junosblogg

    ReplyDelete