6

(This may be better answered on serverfault, but it's not technically about a server, so suggestions welcome...)

We just moved offices at work. We're in Cambridge, MA, and have a Comcast business-class cable modem. Once every few days, for most of the day, we have trouble reaching some, but not all, web sites - for example, Slashdot. As it happens, I live three miles from the office, and I also have a Comcast business-class cable modem at home. From work, I can ssh into my server at home, and though I go through some of the same routers - and all of the same general POPs - I don't have these problems from home.

15 years ago, I knew how to troubleshoot this and call NOCs and figure it out. Nowadays, with load balancers and virtual IPs, I'm stumped. I tried contacting Savvis with the traceroutes below, and they said "It's not us". I sent them to Slashdot, and of course no response - but it's not just a Savvis problem and not just a Slashdot problem anyway.

We have also occasionally seen 10-30% packet loss when pinging Google's 8.8.8.8; I don't know if the problem occurs at the same time, and I don't have any failed traceroutes for that at the moment, but a successful traceroute leaves 111eighthave.ny.ibone.comcast.net and goes straight to Google without hitting Savvis.

Failing traceroute from the office:

~% traceroute slashdot.org
traceroute to slashdot.org (216.34.181.45), 64 hops max, 52 byte packets
 1  * * *
 2  te-7-1-ur01.cambridge.ma.boston.comcast.net (68.87.36.241)  10.628 ms  7.029 ms  14.147 ms
 3  be-51-ar01.needham.ma.boston.comcast.net (68.85.162.157)  10.648 ms  13.714 ms  13.754 ms
 4  pos-2-1-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.95.29)  20.171 ms  18.774 ms  17.866 ms
 5  pos-1-6-0-0-pe01.111eighthave.ny.ibone.comcast.net (68.86.87.110)  20.177 ms  18.549 ms  18.130 ms
 6  er2-tengig3-3.newyork.savvis.net (208.173.138.13)  20.854 ms  19.490 ms  16.720 ms
 7  cr1-tengig-0-8-3-0.newyork.savvis.net (204.70.198.13)  15.856 ms  20.863 ms  16.717 ms
 8  cr2-tengig-0-0-2-0.chicago.savvis.net (204.70.196.242)  59.632 ms  47.147 ms  52.665 ms
 9  hr2-tengigabitethernet-12-1.elkgrovech3.savvis.net (204.70.195.122)  40.771 ms  55.918 ms  39.418 ms
10  das4-v3044.ch3.savvis.net (64.37.207.206)  45.907 ms  45.159 ms  46.643 ms
11  64.27.160.198 (64.27.160.198)  42.509 ms  39.425 ms  67.412 ms
12  * * *
13  * * *
14  * * *
15  * * *

Successful traceroute from home:

~% traceroute slashdot.org
traceroute to slashdot.org (216.34.181.45), 64 hops max, 52 byte packets
 1  73.164.80.1 (73.164.80.1)  10.194 ms  13.718 ms  9.876 ms
 2  te-7-4-ur01.cambridge.ma.boston.comcast.net (68.85.160.17)  9.680 ms  6.937 ms  9.150 ms
 3  be-51-ar01.needham.ma.boston.comcast.net (68.85.162.157)  8.392 ms  7.986 ms  8.621 ms
 4  pos-2-2-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.93.185)  16.350 ms  18.983 ms  19.961 ms
 5  pos-1-4-0-0-pe01.111eighthave.ny.ibone.comcast.net (68.86.86.194)  17.208 ms  16.946 ms  20.909 ms
 6  er2-tengig3-3.newyork.savvis.net (208.173.138.13)  16.934 ms  18.493 ms  23.790 ms
 7  cr2-tengig-0-15-4-0.newyork.savvis.net (204.70.198.17)  26.530 ms  16.009 ms  14.924 ms
 8  cr2-pos-0-7-3-0.chicago.savvis.net (204.70.192.109)  40.031 ms  39.496 ms  39.807 ms
 9  hr2-tengigabitethernet-12-1.elkgrovech3.savvis.net (204.70.195.122)  41.065 ms  45.294 ms  41.091 ms
10  das3-v3039.ch3.savvis.net (64.37.207.186)  47.867 ms  40.606 ms  40.157 ms
11  64.27.160.194 (64.27.160.194)  50.774 ms  56.097 ms  51.147 ms
12  slashdot.org (216.34.181.45)  39.788 ms  41.741 ms  39.871 ms
Jay Levitt
  • 549
  • 6
  • 15

1 Answers1

2
  1. Just to note: icmp-tests (traceroute|ping) aren't always accurate and correct - you may have successful TCP-connection with endpoint, but filtered ICMP-responses from some hops (including destination) and you haven't ability to detect (easy)- is it timeout or suppressed echo-reply
  2. Same physical source location for you doesn't mean same networks (I can't see office IP, but I suppose it have to be in 68.87.3? network somewhere, but home-net is 73.164.80.) and same AS (Autonomous Systems), which are base of routing (if I write it in simplest form and drop NOC-details)
  3. In order to troubleshot you can check icmp-tcp connectivity (as you done before, but for 2 types is more preferable), know (it's better) AS of target, AS of good source and AS of bad source and place ticket to support@ smth. near "Detected connectivity problem to AS X from AS Y of your area of responsibility, while your AS Z doesn't show same type of problem". In case of same AS for good and bad source only networks are enough.
  4. "It's not us" isn't answer to NOC!!! You can read SLA in order to have law tools, or just require (if you can it) "escalation of problem" to management or to the neighbors along the route

HTH

Lazy Badger
  • 3,676
  • 13
  • 12
  • BTW, sources ARE in different AS *68.80.0.0/15* origin: **AS33287** *73.164.0.0/15* origin: **AS7015** – Lazy Badger Oct 17 '11 at 00:42
  • Yep, I tried tcptraceroute too, but can't get it to build on Mac, and it looks like abandonware. Is there a modern equivalent? And you're right, I'm on a different AS, and it may be dynamic at work - today my external work IP is 24.63.250.125. I'll keep this open and update when I know more. As for SLA, I'm not Savvis's customer so they have no responsibility to me; what they actually said was "It's on Slashdot's side", but I'm not convinced they're right - else we wouldn't have widespread problems. – Jay Levitt Oct 17 '11 at 15:45
  • You haven't say to Savvis, to have to ask Comcast - **their** address-space (part of) have connectivity troubles – Lazy Badger Oct 18 '11 at 09:33