Logging Policy and TCPDUMP (SRX)

How can we know if our policies are being hit or if traffic is getting to the firewall?

* SETUP

The policy in question for all the below work is just the default trust to untrust policy on the SRX. The source IP on trust is 192.168.56.50 and the destination on IP on untrust is 172.20.123.2

blogger@LEFTY> show configuration security policies from-zone trust to-zone untrust
policy trust-to-untrust {
    match {
        source-address any;
        destination-address any;
        application any;
    }
    then {
        permit;
    }
}

* LOGGING

First lets look at logging. Lets add logging to the policy. If your going to have logging you must choose session-init or session-close or both. From my perspective session-init is way more useful as if you have very long lasting sessions you may never know that you got a policy hit. Sure you might want to know when a session ends but at the very least you would want to know when and if your policy even got hit so make sure you at least use session-init.

blogger@LEFTY> show configuration security policies from-zone trust to-zone untrust   
policy trust-to-untrust {
    match {
        source-address any;
        destination-address any;
        application any;
    }
    then {
        permit;
        log {
            session-init;
        }

    }
}

With this in place you have a couple of options to check for hits. You can examine the log to see if it got hit. The log file in question to see these hits, by default, is rtlogd.

RT_FLOW_SESSION_CREATE tells us a security session was created by policy - I.e a "hit" on the permit policy.

blogger@LEFTY> show log rtlogd | match RT_FLOW                                        
Feb 25 07:58:55 rtlogd: receives log RT_FLOW_SESSION_CREATE from RT_FLOW at severity 6, miscellaneous string=session created 192.168.56.50/1066->172.20.123.2/22 junos-ssh 172.20.123.1/56800->172.20.123.2/22 source-nat-rule None 6 trust-to-untrust trust untrust 1866, attribute-list=source-address 13 192.168.56.50 source-port 4 1066 destination-address 12 172.20.123.2 destination-port 2 22 service-name 9 junos-ssh nat-source-address 12 172.20.123.1 nat-source-port 5 56800 nat-destination-address 12 172.20.1

You could also monitor the log in real time to see the hit. Be as specific as you need to be with the match filters to get just what your looking for. You could filter on any of the strings in the message including the name of the actual policy.

blogger@LEFTY> monitor start rtlogd | match RT_FLOW | match 192.168.56.50   
blogger@LEFTY>
*** rtlogd ***
Feb 25 08:08:37 rtlogd: receives log RT_FLOW_SESSION_CREATE from RT_FLOW at severity 6, miscellaneous string=session created 192.168.56.50/1068->172.20.123.2/22 junos-ssh 172.20.123.1/18384->172.20.123.2/22 source-nat-rule None 6 trust-to-untrust trust untrust 1876, attribute-list=source-address 13 192.168.56.50 source-port 4 1068 destination-address 12 172.20.123.2 destination-port 2 22 service-name 9 junos-ssh nat-source-address 12 172.20.123.1 nat-source-port 5 18384 nat-destination-address 12 172.20.1

blogger@LEFTY> monitor stop rtlogd

Another way to see hits is with the "count" option. If you set the count option like so..

blogger@LEFTY> show configuration security policies from-zone trust to-zone untrust   
policy trust-to-untrust {
    match {
        source-address any;
        destination-address any;
        application any;
    }
    then {
        permit;
        log {
            session-init;
        }
        count;
    }
}

Then you can examine quite a lot of stats on the policy. The whole section in bold below will not show up unless you have "count" on the policy.

blogger@LEFTY> show security policies policy-name trust-to-untrust detail   
Policy: trust-to-untrust, action-type: permit, State: enabled, Index: 4, Scope Policy: 0
  Policy Type: Configured
  Sequence number: 1
  From zone: trust, To zone: untrust
  Source addresses:
    any-ipv4: 0.0.0.0/0
    any-ipv6: ::/0
  Destination addresses:
    any-ipv4: 0.0.0.0/0
    any-ipv6: ::/0
  Application: any
    IP protocol: 0, ALG: 0, Inactivity timeout: 0
      Source port range: [0-0]
      Destination port range: [0-0]
  Per policy TCP Options: SYN check: No, SEQ check: No
  Session log: at-create
  Policy statistics:
    Input  bytes     :                10368                  301 bps
    Output bytes     :                10368                  301 bps
    Input  packets   :                   92                    2 pps
    Output packets   :                   92                    2 pps
    Session rate     :                    2                    0 sps
    Active sessions  :                    1
    Session deletions:                    1
    Policy lookups   :                    1


You could simply count hits this way..

blogger@LEFTY> show security policies policy-name trust-to-untrust detail | match "policy lookups"
    Policy lookups   :                    2

* MONITOR

Now knowing that a policy got a hit may not be enough depending on your situation. Sometimes you may want to know did the traffic in question even get to the firewall, let alone match a policy especially when your expecting to get policy hits but are not seeing any.

Let see if we can use the monitor command to see the raw traffic as passes from trust to untrust still using the default policy. Specifically we will monitor traffic on the untrust interface.

blogger@LEFTY> monitor traffic no-resolve interface fe-0/0/0.0   
verbose output suppressed, use <detail> or <extensive> for full protocol decode
Address resolution is OFF.
Listening on fe-0/0/0.0, capture size 96 bytes

10:24:50.052166  In STP 802.1d, Config, Flags [none], bridge-id 8000.2c:21:72:20:9e:08.8201, length 43
10:24:51.865712  In STP 802.1d, Config, Flags [none], bridge-id 8000.2c:21:72:20:9e:08.8201, length 43
10:24:52.975799 Out IP truncated-ip - 4 bytes missing! 172.20.123.1 > 224.0.0.5: OSPFv2, Hello, length 44
10:24:53.802649  In STP 802.1d, Config, Flags [none], bridge-id 8000.2c:21:72:20:9e:08.8201, length 43
^C
4 packets received by filter
0 packets dropped by kernel

blogger@LEFTY>

We don't see the traffic we were looking for (192.168.56.50/1068->172.20.123.2/22)even though we were looking at all traffic that went in and out the interface while the connection was successfully initiated. All we see is spanning tree and OSPF traffic. I.e traffic initiating or terminating on the interface.

The reason is the monitor command cannot be used like you would TCPDUMP on a Checkpoint or (Sadly) Snoop on SceenOS. Monitor only sees traffic going to or from the device, not through it. I.e Its only looking at traffic to do with the Control Plane.

At this point, I'll link to the monitor command reference which is worth going through
http://www.juniper.net/techpubs/en_US/junos10.4/information-products/topic-collections/swcmdref-basics-services/index.html?topic-19319.html

Lets test this by telnetting to the device rather than through the device.

blogger@LEFTY> monitor traffic no-resolve interface fe-0/0/0.0 matching "tcp"   
verbose output suppressed, use <detail> or <extensive> for full protocol decode
Address resolution is OFF.
Listening on fe-0/0/0.0, capture size 96 bytes

10:41:53.644129  In IP 172.20.123.2.63886 > 172.20.123.1.23: S 641953623:641953623(0) win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 3103949 0,sackOK,eol>
10:41:53.644304 Out IP truncated-ip - 4 bytes missing! 172.20.123.1.23 > 172.20.123.2.63886: S 1960505699:1960505699(0) ack 641953624 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 10657977 3103949,[|tcp]>
10:41:53.654023  In IP 172.20.123.2.63886 > 172.20.123.1.23: . ack 1 win 33304 <nop,nop,timestamp 3103959 10657977>
10:41:53.660026  In IP 172.20.123.2.63886 > 172.20.123.1.23: P 1:25(24) ack 1 win 33304 <nop,nop,timestamp 3103965 10657977>

^C
20 packets received by filter
0 packets dropped by kernel

blogger@LEFTY>

That's cool. We see the 3 way handshake. Notice the default capture size of 96 bytes. To get rid of the truncated IP messages increase the capture size...Saying that the default capture size is still obviously big enough to see the handshake which can be important for trouble shooting.

blogger@LEFTY> monitor traffic no-resolve interface fe-0/0/0.0 matching "tcp" size 130
verbose output suppressed, use <detail> or <extensive> for full protocol decode
Address resolution is OFF.
Listening on fe-0/0/0.0, capture size 130 bytes

10:48:23.848685  In IP 172.20.123.2.62464 > 172.20.123.1.23: S 1137534827:1137534827(0) win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 3489552 0,sackOK,eol>
10:48:23.848833 Out IP 172.20.123.1.23 > 172.20.123.2.62464: S 3300985708:3300985708(0) ack 1137534828 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 11042634 3489552,sackOK,eol>
10:48:23.852947  In IP 172.20.123.2.62464 > 172.20.123.1.23: . ack 1 win 33304 <nop,nop,timestamp 3489557 11042634>
10:48:23.861831  In IP 172.20.123.2.62464 > 172.20.123.1.23: P 1:25(24) ack 1 win 33304 <nop,nop,timestamp 3489561 11042634>

etc....

Lets look at how much more detail we can get with the monitor.
The same again with detail output..

blogger@LEFTY> monitor traffic no-resolve interface fe-0/0/0.0 matching "tcp" size 130 detail
Address resolution is OFF.
Listening on fe-0/0/0.0, capture size 130 bytes

10:52:18.252222  In IP (tos 0x10, ttl  64, id 13842, offset 0, flags [DF], proto: TCP (6), length: 64) 172.20.123.2.63603 > 172.20.123.1.23: S 114195160:114195160(0) win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 3720719 0,sackOK,eol>
10:52:18.252373 Out IP (tos 0x0, ttl  64, id 4336, offset 0, flags [DF], proto: TCP (6), length: 64) 172.20.123.1.23 > 172.20.123.2.63603: S 867324599:867324599(0) ack 114195161 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 11273226 3720719,sackOK,eol>
10:52:18.256913  In IP (tos 0x10, ttl  64, id 13844, offset 0, flags [DF], proto: TCP (6), length: 52) 172.20.123.2.63603 > 172.20.123.1.23: . ack 1 win 33304 <nop,nop,timestamp 3720723 11273226>
10:52:18.265279  In IP (tos 0x10, ttl  64, id 13847, offset 0, flags [DF], proto: TCP (6), length: 76) 172.20.123.2.63603 > 172.20.123.1.23: P 1:25(24) ack 1 win 33304 <nop,nop,timestamp 3720728 11273226> [telnet DO SUPPRESS GO AHEAD, WILL TERMINAL TYPE, WILL NAWS, WILL TSPEED, WILL LFLOW, WILL LINEMODE, WILL NEW-ENVIRON, DO STATUS]


The same again with extensive output..

blogger@LEFTY> monitor traffic no-resolve interface fe-0/0/0.0 matching "tcp" size 130 extensive
Address resolution is OFF.
Listening on fe-0/0/0.0, capture size 130 bytes

10:53:48.304132  In
        Juniper PCAP Flags [Ext, no-L2, In], PCAP Extension(s) total length 16
          Device Media Type Extension TLV #3, length 1, value: Ethernet (1)
          Logical Interface Encapsulation Extension TLV #6, length 1, value: Ethernet (14)
          Device Interface Index Extension TLV #1, length 2, value: 34048
          Logical Interface Index Extension TLV #4, length 4, value: 70
        -----original packet-----
        PFE proto 2 (ipv4): (tos 0x10, ttl  64, id 13899, offset 0, flags [DF], proto: TCP (6), length: 64) 172.20.123.2.50036 > 172.20.123.1.23: S 2946283342:2946283342(0) win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 3809643 0,sackOK,eol>
10:53:48.304278 Out
        Juniper PCAP Flags [Ext], PCAP Extension(s) total length 16
          Device Media Type Extension TLV #3, length 1, value: Ethernet (1)
          Logical Interface Encapsulation Extension TLV #6, length 1, value: Ethernet (14)
          Device Interface Index Extension TLV #1, length 2, value: 34048
          Logical Interface Index Extension TLV #4, length 4, value: 70
        -----original packet-----
        2c:21:72:26:da:c0 > 2c:21:72:20:9e:08, ethertype IPv4
(0x0800), length 78: (tos 0x0, ttl  64, id 4419, offset 0, flags [DF], proto: TCP (6), length: 64) 172.20.123.1.23 > 172.20.123.2.50036: S 3295572203:3295572203(0) ack 2946283343 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 11361960 3809643,sackOK,eol>
10:53:48.308186  In
        Juniper PCAP Flags [Ext, no-L2, In], PCAP Extension(s) total length 16
          Device Media Type Extension TLV #3, length 1, value: Ethernet (1)
          Logical Interface Encapsulation Extension TLV #6, length 1, value: Ethernet (14)
          Device Interface Index Extension TLV #1, length 2, value: 34048
          Logical Interface Index Extension TLV #4, length 4, value: 70
        -----original packet-----
        PFE proto 2 (ipv4): (tos 0x10, ttl  64, id 13901, offset 0, flags [DF], proto: TCP (6), length: 52) 172.20.123.2.50036 > 172.20.123.1.23: . 1:1(0) ack 1 win 33304 <nop,nop,timestamp 3809647 11361960>
10:53:48.322772  In
        Juniper PCAP Flags [Ext, no-L2, In], PCAP Extension(s) total length 16
          Device Media Type Extension TLV #3, length 1, value: Ethernet (1)
          Logical Interface Encapsulation Extension TLV #6, length 1, value: Ethernet (14)
          Device Interface Index Extension TLV #1, length 2, value: 34048
          Logical Interface Index Extension TLV #4, length 4, value: 70
        -----original packet-----
        PFE proto 2 (ipv4): (tos 0x10, ttl  64, id 13904, offset 0, flags [DF], proto: TCP (6), length: 76) 172.20.123.2.50036 > 172.20.123.1.23: P 1:25(24) ack 1 win 33304 <nop,nop,timestamp 3809653 11361960> [telnet DO SUPPRESS GO AHEAD, WILL TERMINAL TYPE, WILL NAWS, WILL TSPEED, WILL LFLOW, WILL LINEMODE, WILL NEW-ENVIRON, DO STATUS]
10:53:48.360171 Out
        Juniper PCAP Flags [Ext], PCAP Extension(s) total length 16
          Device Media Type Extension TLV #3, length 1, value: Ethernet (1)
          Logical Interface Encapsulation Extension TLV #6, length 1, value: Ethernet (14)
          Device Interface Index Extension TLV #1, length 2, value: 34048
          Logical Interface Index Extension TLV #4, length 4, value: 70
        -----original packet-----


One final example show how to filter on an IP with no timestamps.

blogger@LEFTY> monitor traffic interface fe-0/0/0.0 no-resolve no-timestamp matching "host 172.20.123.2" size 130
verbose output suppressed, use <detail> or <extensive> for full protocol decode
Address resolution is OFF.
Listening on fe-0/0/0.0, capture size 130 bytes

 In IP 172.20.123.2.55852 > 172.20.123.1.23: S 1912765779:1912765779(0) win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 6783486 0,sackOK,eol>
Out IP 172.20.123.1.23 > 172.20.123.2.55852: S 2704698022:2704698022(0) ack 1912765780 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 14328581 6783486,sackOK,eol>
 In IP 172.20.123.2.55852 > 172.20.123.1.23: . ack 1 win 33304 <nop,nop,timestamp 6783496 14328581>
 In IP 172.20.123.2.55852 > 172.20.123.1.23: P 1:25(24) ack 1 win 33304 <nop,nop,timestamp 6783527 14328581>


Thats a quick look at monitor with a few output options showing what we can do to see traffic coming to or from the device. Also try "monitor interface" to see realtime interface stat counters.

So what about seeing traffic the goes through the device....Tcpdump.

* TCPDUMP

There are a few comments in the Juniper forums that mention that you can't use Tcpdump for transit traffic on an SRX or you can only use it for traffic that terminates on or initiates from the device. Now that's certainly true if you just go to the shell and run Tcpdump. However you CAN use it to look at transit traffic on the SRX with a couple of methods. Please note I only make that statement with reference to Branch SRXs not high end SRXs.

Method 1  - no filtering

With this method you just fire up sampling on the interface you want to capture the traffic on. You can't control what you capture though.

Step 1) Set up a packet capture file. The 10k is the maximum file size.

forwarding-options {
    packet-capture {
        file filename dumptest files 2 size 10k world-readable;
    }
}


Step 2) Enable sampling on the interface.

vlan {
        unit 0 {
            family inet {
                sampling {
                    input;
                    output;
                }
                address 10.10.10.5/24;
            }
        }
    }


Step 3) See the result - examine the capture

root@LEFTY% tcpdump -r /var/tmp/dumptest.vlan
Reverse lookup for 172.20.123.2 failed (check DNS reachability).
Other reverse lookup failures will not be reported.
Use <no-resolve> to avoid reverse lookups on IP addresses.

20:54:49.077362  In IP truncated-ip - 18 bytes missing! 10.10.10.100 > 224.0.0.5: OSPFv2, Hello, length 48
20:54:51.066998  In IP truncated-ip - 42 bytes missing! 192.168.56.50.1057 > 10.10.10.5.ssh: P 578217943:578217995(52) ack 2029920434 win 63928
20:54:51.067134 Out IP truncated-ip - 42 bytes missing! 10.10.10.5.ssh > 192.168.56.50.1057: P 1:53(52) ack 52 win 65535

Looking at the above 3 capture lines first I see an incoming OSPF hello to the SRX, then part of my ssh management session to the SRX and finally an ssh session through the device. So with this method we are capturing everything to/from and through the SRX

Method 2  -  filtering

With this method we can control exactly what we want to capture  by using firewall filters.

If you were previously using the above method and want to filter at capture time then you must remove the sampling from the interface as that will override the input/output filter applied to the interface.

Step 1) Same as above

Step 2) Create filters to sample what you want to capture and permit everything else. Assuming you want to see both directions of a flow you will need 2 filters. Here we want to capture an ssh session from 192.168.56.50 to 172.20.123.2. The IPs need to be the real IPs as the interface you intend to apply the filters on would see them coming or going so do take NAT into account when setting up the filters. "Port" here means src or dst port.

[edit firewall]
blogger@LEFTY# show
filter TCPDUMP-INGRESS {
    term CAPTURE-TERM {
        from {
            source-address {
                192.168.56.50/32;
            }
            destination-address {
                172.20.123.2/32;
            }
            port ssh;
        }
        then {
            sample;
            accept;
        }
    }
    term PERMIT-ALL {
        then accept;
    }
}
filter TCPDUMP-EGRESS {
    term CAPTURE-TERM {
        from {
            source-address {
                172.20.123.2/32;
            }
            destination-address {
                192.168.56.50/32;
            }
            port ssh;
        }
        then {
            sample;
            accept;
        }
    }
    term PERMIT-ALL {
        then accept;
    }
}

Step 3) Apply the filters to the interface where you want the capture to happen.

vlan {
    unit 0 {
        family inet {
            filter {
                input TCPDUMP-INGRESS;
                output TCPDUMP-EGRESS;
            }
            address 10.10.10.5/24;
        }
    }
}

Step 4) See the result - examine the capture

root@LEFTY% tcpdump -r /var/tmp/dumptest.vlan
Reverse lookup for 172.20.123.2 failed (check DNS reachability).
Other reverse lookup failures will not be reported.
Use <no-resolve> to avoid reverse lookups on IP addresses.

21:53:19.314903  In IP 192.168.56.50.1100 > 172.20.123.2.ssh: S 1489299913:1489299913(0) win 64240 <mss 1460,nop,nop,sackOK>
21:53:19.315036 Out IP 172.20.123.2.ssh > 192.168.56.50.1100: S 1231992630:1231992630(0) ack 1489299914 win 65535 <mss 1460,sackOK,eol>
21:53:19.315123  In IP 192.168.56.50.1100 > 172.20.123.2.ssh: . ack 1 win 64240
21:53:19.315189  In IP truncated-ip - 37 bytes missing! 192.168.56.50.1100 > 172.20.123.2.ssh: P 1:48(47) ack 1 win 64240
21:53:19.315289 Out IP 172.20.123.2.ssh > 192.168.56.50.1100: . ack 48 win 65535

So there it is ... the 3 way handshake of a flow going THROUGH the SRX with no uninteresting packets. That's a lot more of a hassle than running up snoop in a minute on ScreenOS eh! But just shows it can be done. And it needs to be done ...sometimes you need to just know if a packet even got to your firewall before the firewall had it chance to do whatever it had to do to that packet. You could leave the filters there permanently and just change parameters as needed and apply where/when needed to speed things up for subsequent captures.

Finally, Lets apply a security policy to block the traffic we are trying to capture. Here is the security policy from trust to untrust now...

blogger@LEFTY> show configuration security policies from-zone trust to-zone untrust
policy blocker {
    match {
        source-address any;
        destination-address 172.20.123.2/32;
        application any;
    }
    then {
        deny;
        log {
            session-init;
        }
    }
}
policy trust-to-untrust {
    match {
        source-address any;
        destination-address any;
        application any;
    }
    then {
        permit;
        log {
            session-init;
        }
        count;
    }
}

Lets see what happens..

Policy blocking the flow...

Feb 28 22:56:40 rtlogd: receives log RT_FLOW_SESSION_DENY from RT_FLOW at severity 6, miscellaneous string=session denied 192.168.56.50/1114->172.20.123.2/22 junos-ssh 6(0) blocker trust untrust, attribute-list=source-address 13 192.168.56.50 source-port 4 1114 destination-address 12 172.20.123.2 destination-port 2 22 service-name 9 junos-ssh protocol-id 1 6 icmp-type 1 0 policy-name 7 blocker source-zone-name 5 trust destination-zone-name 7 untrust
Feb 28 22:56:40 rtlogd: forwarding 452 bytes

and the filter capturing the repeated SYN attempts...

root@LEFTY% tcpdump -r /var/tmp/dumptest.vlan
Reverse lookup for 192.168.56.50 failed (check DNS reachability).
Other reverse lookup failures will not be reported.
Use <no-resolve> to avoid reverse lookups on IP addresses.

22:56:40.573021  In IP 192.168.56.50.1114 > 172.20.123.2.ssh: S 2869939349:2869939349(0) win 64240 <mss 1460,nop,nop,sackOK>
22:56:46.572293  In IP 192.168.56.50.1114 > 172.20.123.2.ssh: S 2869939349:2869939349(0) win 64240 <mss 1460,nop,nop,sackOK>

As we expected. Note the identical timestamp for the first SYN and the source port (1114) proving it is the same packet in question and showing that packet based input firewall filter is applied before the flow based filters (Security policy)

Model: srx100h
JUNOS Software Release [10.4R7.5]

No comments: