I love doing CTF challenges, they usually involve binary exploitation or reverse engineering but this challenge is a whole different story... I was asked to give input into a networking challenge that was anything but straight forward.
I like keeping things simple... The more complex the solution, the more likely it will go wrong. I do not advocate for complexity like in the solution outlined in this gist. My goal is to show that with some networking knowledge, some well placed rules, anything is possible. I would love to have a simpler solution, and if you have ideas I'm all ears. I do not advocate for putting something like this into production without trying to exhaust other avenues to simplify the overall design.
Internet
|
|
+---------------------------+ +--------------------------------------------------------------+ | +-----------------------------------+
| Client Device | | Router | | | Remote Server |
| | | | | | |
| MAC: 00:00:00:AA:BB:CC | | | | | |
| | | Internal Network External Network | | | External Network |
| ------------ IF: lan0 IF: wan0 | | | IF: wan0 |
| | | MAC: 00:00:00:11:22:43 MAC: 00:00:00:22:44:44| | | MAC: 00:00:00:22:44:66 |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | EoGRE Tunnel |------------------------------------------------|EoGRE Tunnel |
| | | IF: eogre |------------------------------------------------|IF: eogre |
| | | MAC: 00:00:00:44:44:44 | | | MAC: 00:00:00:44:44:66 |
| | | | | | |
| | | Internal Network | | | |
| | | IF: lan1 | | | |
| | | MAC: 00:00:00:11:22:44 | | | |
+---------------------------+ +------------|-------------------------------------------------+ | +-----------------------------------+
| |
| |
+------------|-------------------+
| Captive Portal |
| |
| Internal Network |
| IF: lan0 |
| MAC: 00:00:00:99:99:10 |
| |
| |
| |
| |
| |
| |
+--------------------------------+
Diagram created using textik.com
Here are some of the constraints / requirements. I'd like to declare that this is a judgement free zone, but this is going to get convoluted, so judge away. This is what I was given to work with.
Client_Deviceis a "bring your own device",- We have no control over the software/hardware.
- Assume phone/tablet
Client_Deviceneeds to be "on" the remote network provided by the tunnel- It needs to get a IP from the DHCP server on the remote end of the tunnel
- Remote server side provides a Gateway out to the internet
- Remote server does accounting/billing per device MAC Address
- This means that all packets from Client Device need the correct Source MAC when transmitted over the tunnel
Still with me? Seems fairly simple thus far. I hear you saying, "just create a bridge and be done with it!" But wait, there is more!
Client_Deviceneeds to be able to establish a VoWi-Fi connection (ipsec tunnel) over the eogre tunnel to known servers.Client_Devicemust not be allowed to 'browse' to the internet without first purchasing a planClient_Devicemust be redircted to the Captive Portal if no purchase has been madeClient_Devicemust choose and purchase a plan for general browsing:- Pay now: Internet Traffic must be routed out over wan0 (direct internet)
- Pay remote server: Internet Traffic must be routed via eogre (Tunnel)
OK, so what's the problem? Still fairly simple, still a bridge, with some IPTables rules based on plan selection. But wait, there is more!
- Captive Portal server can't have an IP address on the remote server network
- Router server can't have an IP address on the remote server network
And then the kicker!
- Router interfaces lan0 and lan1 don't support promiscous mode and therefore can not be bridged to the tunnel interface!
OUCH... Maybe an explaination of that last point is needed. It's 2020, what linux network interface driver doesn't support promiscous mode? Well, The router is in a VM, the interfaces are both Virtual Functions from a SR-IOV enabled card on the host PC, the NICs are passed to the VM via PCI-Passthrough. The kernel driver on the hypervisor doesn't support "trust mode" for Virtual Functions which is needed to promiscously receive packets.
Just as an aside, the original network architecture included bridging the traffic from the Client_Device to the tunnel, and even though that is simple when all traffic goes over the tunnel, it gets complicated when trying to meet the other requirements of redirecting to a captive Portal etc... I won't go into the details of this, but it involves some ebtable broute drop rules, 2 different SNAT rules depending on traffic destination etc... It's not pretty, and definitely just as convoluted.
- Put
Client_Deviceon a local network subnet where the router is the DNS/DHCP/Gateway etc - Put the Captive portal on the same network as the Client_Device
- Create an application that will create a custom DHCP request with the
Client_DeviceMAC address in the DHCP Request that is sent over the tunnel. - Use the resulting DHCP response information to set up SNAT rules
- Do SNAT (Source Network Address Translation) on both Layer 2 (MAC) and Layer 3 (IP) over the tunnel
- Since we'll need to respond to ARP requests for our NAT'd IP over the tunnel, we'll need to utilize ebtables
- Since Layer 2 SNAT is required, we'll need to utilize ebtables
- Since Layer 3 SNAT is required, we'll need to utilize iptables
- And if that wasn't complicated enough, the router can't have an IP on the remote network, so it needs to 'borrow' a
Client_DeviceIP in order to receive ARP responses on the EoGRE tunnel - Once a plan has been purchased, traffic will need to be routed either over the wan0 connection or the eogre tunnel, so advanced routing tables will be utilized along with firewall markings (fwmark in iptables)
Pretty simple right? What could possibly go wrong?
I won't go into details about the different things I tried and failed at (like digging into nftables kernel source to get a better understanding of why a packet was being dropped unexpectedly in the mangle:PREROUTING table), but rather will just outline the final working solution. If you're working on something similar and are getting stuck, don't give up! I'm documenting this approach so that I can look back on it next week, by which time I'll probably have forgotten it if I didn't write it down.
Let's start by adding some IP addresses to reference.
Internet
|
+---------------------------+ +--------------------------------------------------------------+ | +-----------------------------------+
| Client Device | | Router | | | Remote Server |
| | | | | | |
| MAC: 00:00:00:AA:BB:CC | | | | | |
| IP: 192.168.10.50/24 ------------ Internal Network External Network | | | External Network |
| | | | IF: lan0 IF: wan0 | | | IF: wan0 |
| | | | MAC: 00:00:00:11:22:43 MAC: 00:00:00:22:44:44| | | MAC: 00:00:00:22:44:66 |
| | | | IP: 192.168.10.1/24 IP: 68.183.x.x | | | IP: 20.13.x.x |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | EoGRE Tunnel |------------------------------------------------|EoGRE Tunnel |
| | | | Bridge Device IF: eogre |------------------------------------------------|IF: eogre |
| | | | IF: brtun Master: brtun | | | MAC: 00:00:00:44:44:66 |
| | | | MAC: 00:00:00:44:44:44 MAC: 00:00:00:44:44:44 | | | IP: 10.10.0.1/24 |
| | | | IP: NONE IP: NONE | | | |
| | | | | | | |
+---------------------------+ | +--------------------------------------------------------------+ | | |
| | +-----------------------------------+
+----------------|---------------+ |
| Captive Portal | |
| |
| Internal Network |
| IF: lan0 |
| MAC: 00:00:00:99:99:10 |
| IP: 192.168.10.10/24 |
| |
+--------------------------------+
-
Create a bridge and add eogre as the only interface
- Note that we need a 'bridge' interface so that we can use ebtables rules, we're not actually bridging between two interfaces.
ip link add name brtun type bridgeip link set eogre master brtun
- Note that we need a 'bridge' interface so that we can use ebtables rules, we're not actually bridging between two interfaces.
-
Modify
/etc/iproute2/rt_tablesand add a new table. For this example I'll pick 4, but it's irrelevant as long as it doesn't conflict with any other table IDs:4 EOGRE
-
Set up rule to send all packets that are marked with
0x4/0x4to the EOGRE routing table. This rule is for the kernel routing decision making process. More on this later.ip rule add from all fwmark 0x4 lookup EOGRE- NOTE: I'm picking fwmark of 0x4 to match the table ID, but they are completely separate and don't have to be the same.
-
Configure bridge to pass frames to iptables
sysctl -w net.bridge.bridge-nf-call-iptables = 1sysctl -w net.bridge.bridge-nf-filter-vlan-tagged = 1
-
Configure interfaces to turn off reverse path filtering.
sysctl -w net.ipv4.conf.all.rp_filter=0sysctl -w net.ipv4.conf.brtun.rp_filter=0
When the Client_Device connects to our network we need to also do a request over the Tunnel interface to get a DHCP IP address from the remote server. This can be triggered by capturing the DHCP request/response on our server, this can be done with the use of a nfqueue:
iptables -t raw -A PREROUTING -i lan0 -p udp -m udp --dport 67 -j NFQUEUE --queue-num 100
Note: you may some funky behavior if using dhcpd. It uses a raw socket, and therefore bypasses iptables.
And then create an application that will issue a DHCP request out over the tunnel interface. This is beyond the scope of this exercise, it has been done already and not too interesting.
+---------------+ +---------+ +---------------+
| Client_Device | | Router | | Remote_Server |
+---------------+ +---------+ +---------------+
| | |
| DHCP Discover | |
|------------------------------------->| |
| | |
| DHCP Offer (192.168.10.50/24) | |
|<-------------------------------------| |
| | |
| DHCP Request (192.168.10.50/24) | |
|------------------------------------->| |
| | |
| DHCP ACK | |
|<-------------------------------------| |
| | |
| | DHCP Discover (Client MAC: 00:00:00:AA:BB:CC) |
| |------------------------------------------------------>|
| | |
| | DHCP Offer (IP: 10.10.0.50/24) |
| |<------------------------------------------------------|
| | |
| | DHCP Request (IP: 10.10.0.50/24) |
| |------------------------------------------------------>|
| | |
| | DHCP ACK |
| |<------------------------------------------------------|
| -----------------\ | |
| | Parse response |-| |
| |----------------| | |
| | |
| | Setup SNAT Rules |
| |----------------- |
| | | |
| |<---------------- |
| | |
At this point we should have the following information in the DHCP response:
Client IP address on Remote server network(Option 1) Subnetmask of the remote network(Option 3) Gateway(remote router) address(Option 6) DNS server IPs on remote network(Option 51) Lease Time
So what rules do we need to set up in order to make this work.
First let's make user the router can represent the Client_Device to the remote server:
ebtables -t nat -A PREROUTING -i eogre -p ARP --arp-op Request --arp-ip-dst 10.10.0.50 -j arpreply --arpreply-mac 00:00:00:AA:BB:CC
This takes care of the case where the remote server is requesting who has 10.10.0.50:
+---------------+ +---------+ +---------------+
| Client_Device | | Router | | Remote_Server |
+---------------+ +---------+ +---------------+
-------\ | | |
| Idle |-| | |
|------| | | |
| | |
| | ARP - Who Has 10.10.0.50? |
| |<-------------------------------------------------|
| | |
| | ARP - 10.10.0.50 is at 00:00:00:AA:BB:CC |
| |------------------------------------------------->|
| | |
The router must have an IP in order to receive the MAC address of the GW on the remote server. But the requirements state that we can't request one from the DHCP server (Yes it sucks, but it is what it is...)
+---------------+ +---------+ +---------------+
| Client_Device | | Router | | Remote_Server |
+---------------+ +---------+ +---------------+
-------\ | | |
| Idle |-| | |
|------| | | |
| | |
| | ARP - Who Has 10.10.0.1, Tell 10.10.0.50 |
| |------------------------------------------------->|
| | |
| | ARP - 10.10.0.1 is at 00:00:00:44:44:66 |
| |<-------------------------------------------------|
| | |
So since we know we'll receive packets destined for 10.10.0.50, we can use that IP in our ARP request.
ip addr add 10.10.0.50/24 dev brtun
This will result in the default routing table to get the following entry:
10.10.0.0/24 dev brtun proto kernel scope link src 10.10.0.50
But we also need to set up the default route for the advanced EOGRE routing table
ip route add default via 10.10.0.1 dev brtun table EOGRE
Now the router needs to make sure that the traffic is correctly SNAT'd:
iptables -t nat -A POSTROUTING -o brtun -s 192.168.10.50 -j SNAT --to 10.10.0.50
ebtables -t nat -A POSTROUTING -p IPv4 -o eogre --ip-source 10.10.0.50 -j snat --to-source 00:00:00:44:44:66
OK, Client_Device traffic needs to be routed over the right interfaces, since there is also a wan0 interface that is the default route of the router, we need to explicitly state what we want in the kernel routing decisions
iptables -A PREROUTING -t mangle -i lan0 -s 192.168.10.50/32 -j MARK --set-mark 4
This rule sets a firewall mark on packets from the Client_Device so that when the routing decision is made, it will be directed to the correct routing table.
This can be modified as needed so that only certain traffic goes over the tunnel.
EG: only http traffic:
iptables -A PREROUTING -t mangle -i lan0 -s 192.168.10.50/32 -p tcp -m tcp --dport 80 -j MARK --set-mark 4
EG: only pings:
iptables -A PREROUTING -t mangle -i lan0 -s 192.168.10.50/32 -p icmp -j MARK --set-mark 4
Let's look at how an ICMP packet sent from the Client_Device will traverse this route.
+---------------+ +---------+ +---------------+
| Client_Device | | Router | | Remote_Server |
+---------------+ +---------+ +---------------+
| | |
| ARP - Who Has 192.168.10.1, Tell 192.168.10.50 | |
|------------------------------------------------------------------------------->| |
| | |
| ARP - 192.168.10.1 is at 00:00:00:11:22:43 | |
|<-------------------------------------------------------------------------------| |
| | |
| ICMP Req: SrcIP: 192.168.10.50, DstIP 8.8.8.8, SrcMAC: 00:00:00:AA:BB:CC | |
|------------------------------------------------------------------------------->| |
| | |
| | ARP - Who Has 10.10.0.1, Tell 10.10.0.50 |
| |------------------------------------------------------------------------------->|
| | |
| | ARP - 10.10.0.1 is at 00:00:00:44:44:66 |
| |<-------------------------------------------------------------------------------|
| | |
| | ICMP Req: SrcIP: 10.10.0.50, DstIP 8.8.8.8, SrcMAC: 00:00:00:AA:BB:CC |
| |------------------------------------------------------------------------------->|
| | |
| | ICMP Resp: SrcIP: 8.8.8.8, DstIP 10.10.0.50, DstMAC: 00:00:00:AA:BB:CC |
| |<-------------------------------------------------------------------------------|
| | |
| ICMP Resp: SrcIP: 8.8.8.8, DstIP 192.168.10.50, DstMAC: 00:00:00:AA:BB:CC | |
|<-------------------------------------------------------------------------------| |
| | |
Or we can enable some debugging and trace the packets with IPTables and EBTables:
modprobe nfnetlink_log
iptables -t raw -A PREROUTING -i lan0 -p icmp -j TRACE
iptables -t raw -A PREROUTING -i brtun -p icmp -j TRACE
ebtables -t broute -A BROUTING -p IPv4 --log-level info --log-prefix "EBFW-BROUTE" --log-ip -j CONTINUE
ebtables -t filter -A INPUT -p IPv4 --log-level info --log-prefix "EBFW-INPUT" --log-ip -j CONTINUE
ebtables -t filter -A FORWARD -p IPv4 --log-level info --log-prefix "EBFW-FORWARD" --log-ip -j CONTINUE
ebtables -t nat -A PREROUTING -p IPv4 --log-level info --log-prefix "EBFW-NAT" --log-ip -j CONTINUE
ebtables -t nat -A POSTROUTING -p IPv4 --log-level info --log-prefix "EBFW-NATPOST" --log-ip -j CONTINUE
Now lets monitor the log and send a single ICMP request from the client device:
Feb 21 19:29:03 router kernel: TRACE: raw:PREROUTING:policy:4 IN=lan0 OUT= MAC=00:00:00:11:22:43:00:00:00:AA:BB:CC:08:00 SRC=192.168.10.50 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=62569 DF PROTO=ICMP TYPE=8 CODE=0 ID=19831 SEQ=1
Feb 21 19:29:03 router kernel: TRACE: mangle:PREROUTING:rule:1 IN=lan0 OUT= MAC=00:00:00:11:22:43:00:00:00:AA:BB:CC:08:00 SRC=192.168.10.50 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=62569 DF PROTO=ICMP TYPE=8 CODE=0 ID=19831 SEQ=1
Feb 21 19:29:03 router kernel: TRACE: mangle:PREROUTING:policy:2 IN=lan0 OUT= MAC=00:00:00:11:22:43:00:00:00:AA:BB:CC:08:00 SRC=192.168.10.50 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=62569 DF PROTO=ICMP TYPE=8 CODE=0 ID=19831 SEQ=1 MARK=0x4
Feb 21 19:29:03 router kernel: TRACE: nat:PREROUTING:policy:1 IN=lan0 OUT= MAC=00:00:00:11:22:43:00:00:00:AA:BB:CC:08:00 SRC=192.168.10.50 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=62569 DF PROTO=ICMP TYPE=8 CODE=0 ID=19831 SEQ=1 MARK=0x4
Feb 21 19:29:03 router kernel: TRACE: mangle:FORWARD:policy:1 IN=lan0 OUT=brtun MAC=00:00:00:11:22:43:00:00:00:AA:BB:CC:08:00 SRC=192.168.10.50 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=62569 DF PROTO=ICMP TYPE=8 CODE=0 ID=19831 SEQ=1 MARK=0x4
Feb 21 19:29:03 router kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=brtun SRC=192.168.10.50 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=62569 DF PROTO=ICMP TYPE=8 CODE=0 ID=19831 SEQ=1 MARK=0x4
Feb 21 19:29:03 router kernel: TRACE: nat:POSTROUTING:rule:1 IN= OUT=brtun SRC=192.168.10.50 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=62569 DF PROTO=ICMP TYPE=8 CODE=0 ID=19831 SEQ=1 MARK=0x4
The last rule hit is the actual SNAT, and unfortunately it doesn't hit any other rules to show that the Source IP did actually change.
Now when the ICMP Response is received again
Feb 21 19:29:03 router kernel: EBFW-BROUTE IN=eogre OUT= MAC source = 00:00:00:44:44:66 MAC dest = 00:00:00:AA:BB:CC proto = 0x0800 IP SRC=8.8.8.8 IP DST=10.10.0.50, IP tos=0x00, IP proto=1
Feb 21 19:29:03 router kernel: EBFW-NAT IN=eogre OUT= MAC source = 00:00:00:44:44:66 MAC dest = 00:00:00:AA:BB:CC proto = 0x0800 IP SRC=8.8.8.8 IP DST=10.10.0.50, IP tos=0x00, IP proto=1
Feb 21 19:29:03 router kernel: TRACE: raw:PREROUTING:policy:4 IN=brtun OUT= PHYSIN=eogre MAC=00:00:00:AA:BB:CC:00:00:00:44:44:66:08:00 SRC=8.8.8.8 DST=10.10.0.50 LEN=84 TOS=0x00 PREC=0x00 TTL=55 ID=0 PROTO=ICMP TYPE=0 CODE=0 ID=19831 SEQ=1
Feb 21 19:29:03 router kernel: TRACE: mangle:PREROUTING:policy:2 IN=brtun OUT= PHYSIN=eogre MAC=00:00:00:AA:BB:CC:00:00:00:44:44:66:08:00 SRC=8.8.8.8 DST=10.10.0.50 LEN=84 TOS=0x00 PREC=0x00 TTL=55 ID=0 PROTO=ICMP TYPE=0 CODE=0 ID=19831 SEQ=1
Feb 21 19:29:03 router kernel: EBFW-INPUT IN=eogre OUT= MAC source = 00:00:00:44:44:66 MAC dest = 00:00:00:44:44:44 proto = 0x0800 IP SRC=8.8.8.8 IP DST=192.168.10.50, IP tos=0x00, IP proto=1
Feb 21 19:29:03 router kernel: TRACE: mangle:FORWARD:policy:1 IN=brtun OUT=lan0 PHYSIN=eogre MAC=00:00:00:44:44:44:00:00:00:44:44:66:08:00 SRC=8.8.8.8 DST=192.168.10.50 LEN=84 TOS=0x00 PREC=0x00 TTL=54 ID=0 PROTO=ICMP TYPE=0 CODE=0 ID=19831 SEQ=1
Feb 21 19:29:03 router kernel: TRACE: mangle:POSTROUTING:policy:1 IN= OUT=lan0 PHYSIN=eogre SRC=8.8.8.8 DST=192.168.10.50 LEN=84 TOS=0x00 PREC=0x00 TTL=54 ID=0 PROTO=ICMP TYPE=0 CODE=0 ID=19831 SEQ=1
The tunnel/bridge combo seemed to cause return packets to be seen as martian packets and therefore dropped in mangle:PREROUTING for the longest time. My first attempt to disable reverse_path filtering on the bridge didn't seem to have any effect and had me scratching me head for a while.
| brtun.rp_filter | all.rp_filter | Result |
|---|---|---|
| 1 | 1 | Dropped |
| 0 | 1 | Dropped |
| 1 | 0 | Dropped |
| 0 | 0 | Passed through |
Kernel message when sysctl -w net.ipv4.conf.brtun.log_martians=1 is set:
Feb 21 19:56:01 router kernel: IPv4: martian source 192.168.10.50 from 8.8.8.8, on dev brtun
With a bit of ingenuity and a lot of debuging/packet capturing etc, we have the end result we wanted. A router that does a Layer 2 and Layer 3 source NAT while not having its own IP on the network it's routing to and without having any of the virtual function interfaces in promiscous mode.
I'm hoping that this will help someone in the future. Feel free to leave comments.
Under SNAT Rules, should the ebtables SNAT use the router's EOGRE MAC address 00:00:00:44:44:44 as the --to-source parameter instead of the MAC address of the remote gateway's EOGRE interface?