Paradise ponders: routers, sprinklers, and ARP proxies edition...

Paradise ponders: routers, sprinklers, and ARP proxies edition...  It was a long, long day for me yesterday.  The end result of yesterday's work (and some more this morning) is at right: the Internet speed test on my office laptop.  That's what I get when I send a gigabit stream of Internet data through two routers and a radio link to my laptop.  It actually varies from around 500 to 800 mbps, and 800 mbps is the limit of what my laptop's network interface can do.  Woo hoo!  I got data now!

Yesterday I ripped out our two older MikroTik routers and replaced them with newer, (much) faster models (RB/1100AHX2).  I brought up the one in my barn office first.  That was a tedious, but fairly straightforward affair, and by 1 pm I had that up and running.  After returning from a very pleasant lunch with Debbie and our friend Michelle H., I started on the house side.  I had it all installed by 6pm, and it was talking between the house and the barn just fine (over my new radio link).  But it wouldn't connect to the Internet at all.

I troubleshot it for three hours, with (to me!) very puzzling symptoms.  The new MikroTik router could talk to the cable modem just fine, but nothing connected to the router could do so.  Sounds like a routing problem, right?  I inspected and re-inspected all the address and routing configuration, and found no problems at all.  No reason for it not to work!  So then I did some packet sniffing, using a constantly-running ping session on Debbie's workstation as a source of known data.  The outbound ping got routed to the cable modem just fine, but then the cable modem never responded.  Tried the same thing with a ping from the router, and the cable modem responded just fine.  My tired brain couldn't process that information, so I went to bed and hoped that with fresh, caffeinated neurons in the morning I could figure it out.

Round about 2 am I woke up, visions of router configurations dancing in my head.  After thinking about it some more, it occurred to me that it might be a problem with ARP (the Address Resolution Protocol).  Unless you're a networking geek, you probably have no idea what that is.  In technical terms, it gives networked devices a way to translate an IP address into an Ethernet destination (a MAC address).  This isn't a great analogy, but it's a bit like a service that translates ZIP+4 codes into a street address.  In terms of my problem, if the router was trying to send a packet to an IP address (in this case, Debbie's terminal) that it didn't know the corresponding MAC address for, it would broadcast an ARP request to all the devices directly connected to the modem, and then my new router should reply (because it already knows how to send something to that IP address).  If that ARP request was never sent, or if my new router never replied to it, the symptoms would match what I was seeing.

So I got up around 3 am and started working on the problem.  With a bit more packet sniffing, I was able to determine that the cable modem was sending an ARP request – but my new router never responded.  At last, something concrete to track down!  With a bit of poring through documentation and configuration screens, I found the place where ARP behavior is configured.  That ARP setting on the screen at right is ordinarily set to “enabled”, but on this particular interface (the one connected to the cable modem) it needs to be set for “proxy-arp”.  Why?  My router's ether1 interface is connected to the cable modem, and it's address is 10.0.0.2/8.  All the other router interfaces are subnets of that one.  For instance, the client machines are on 10.1.4.x/24.  That matters because all the IP addresses that the router issued ARP requests for are within 10.0.0.0/8 – so my new router didn't know that it had to reply; for all it knew some other device would reply.  By changing that setting to proxy-arp, the router sent the ARP request to the interface with a matching subnet, and that interface replied to the ARP request.  With one simple little tweak, all of a sudden the entire network on the house side started working correctly.

Amazing what a little sleep will do for your troubleshooting capability!

We had another little milestone yesterday: Mark T., the fellow installing our new lawn sprinklers, arrived and started digging the trenches.  Progress!  His trenching machine is working much better now that our soil has dried out a bit. :)  He's starting on the section of our yard that we call the “driving range” (because that's what the previous owner did with it).  It's about half our total grass area.  He thinks he'll be finished trenching that part by Friday...