http server response time measuring tool (free or cheap?)

[masraum]

Tech dudes!

I'd like to know if you guys know of a (hopefully, free or cheap) tool to measure http server response times. I figure someone somewhere must have written a small freeware tool, or maybe there's some sort of script that I could run.

I'll explain.

At work, we got a bargain (100Mb Eth) Internet connection (2 of them actually). Since we migrated to them, occasionally a user will find a page that loads very slowly over one of them. Today, someone noticed that it could take over a minute to see any response from http://www.wapa.gov. I then checked from my house and over our secondary connection and the page seemed to load fine. To test, I've been telnetting to the server on port 80 and then issuing something like "GET http://www.wapa.gov HTTP/1.1" Issuing that command from several places got me a response in less than a second. Issuing that same command from our router just outside of our firewall meant that I'd have to wait for >1min for the webpage to come back.

It sure would be nice, if there was a tool or script that I could run that would give me times and output on a PC, that I could use for evidence.

I found a perl script named "torture.pl" that will do it, but there's not a Windows version, and I don't have a UNIX box handy.

[stomachmonkey]

Not sure what the question is.

Do you believe the issue is with one of the 2 connections?

Is it consistently the same connection or bounce between the 2?

Sounds like a lookup issue. What DNS do you have set?

If the issue really is one of the connections running trace routes will get you more info than an http request.

For server response times a simple ping is easier.

[ChkbookMechanic]

tracert at the windows prompt will tell you the times and the hops it takes to ping a server.. you could use that. Or do you want something more graphical?

[Paul_Heery]

I see that you are using a Blue Coat SG 400. I was a little surprised to see it responding to inbound traffic. You should check your settings on that devices as they can be set to limit and throttle bandwidth. Just a thought.

[masraum]

Originally Posted by stomachmonkey

Not sure what the question is.

Do you believe the issue is with one of the 2 connections?

Is it consistently the same connection or bounce between the 2?

Sounds like a lookup issue. What DNS do you have set?

If the issue really is one of the connections running trace routes will get you more info than an http request.

This is the third complaint we've had in 6-8 months. The complaint is always only a single website, and is generally an issue only over one of the links.

The problem isn't DNS. DNS resolves quickly.

Unfortunately ping and/or traceroute isn't indicating any sort of problem.

The website in question this time, doesn't allow icmp, so a traceroute only gets so far, but that's not that unusual these days.


ge-3-1-0.425.ar2.MIN1.gblx.net (67.17.168.165) 8 msec 8 msec 9 msec
pos7-0-0-10G.ar2.CHI2.gblx.net (67.17.105.110) 25 msec 17 msec 17 msec
te-3-3.car3.chicago1.level3.net (4.68.110.193) 25 msec 17 msec 25 msec
ae-31-55.ebr1.Chicago1.Level3.net (4.68.101.158) 25 msec 16 msec 25 msec
ae-6-6.ebr1.chicago2.level3.net (4.69.140.190) 16 msec 17 msec
*
ae-3-3.ebr2.denver1.level3.net (4.69.132.61) 42 msec 50 msec *
- ae-21-54.car1.Denver1.Level3.net (4.68.107.102) 42 msec
- ae-22-54.car2.Denver1.Level3.net (4.68.107.103) 51 msec
- ae-21-54.car1.Denver1.Level3.net (4.68.107.102) 50 msec
ae-1-13.edge1.Denver1.Level3.net (4.68.106.154) 42 msec 50 msec
ae-2-23.edge1.Denver1.Level3.net (4.68.106.158) 42 msec

For server response times a simple ping is easier.

Except pings aren't allowed by the website...

Traceroutes or HTTP GETs reply from most sites in a normal time, or from this site over a different connection in a normal time. I've got a packet capture from our firewall that shows my send the "get" at 15 seconds and the response come back at 75 seconds. 60 seconds isn't normal.

[masraum]

Originally Posted by Paul_Heery

I see that you are using a Blue Coat SG 400. I was a little surprised to see it responding to inbound traffic. You should check your settings on that devices as they can be set to limit and throttle bandwidth. Just a thought.

Where are you seeing that? In front of http://www.wapa.gov Maybe that's part of the problem.

[mikester]

Worst case scenario you could simply use wireshark and span the port. In the packet details you will see the times - all a tool does is evaluate the delta between the packets to determine server response time.

If you are able to recreate the problem then you can easily do it in wireshark.

[Paul_Heery]

Originally Posted by masraum

Where are you seeing that? In front of http://www.wapa.gov Maybe that's part of the problem.

I ran an nmap scan against the server. Inbound is definitely hitting the proxy.

nmap scan report - scan @ May 26, 2010 - 05:47scan summary | scan info | 192.208.27.6 / www.wapa.gov | runstats scan summarynmap was initiated at May 26, 2010 - 05:47 with these arguments:
nmap -T4 -A -v -PE -PA21,23,80,3389 www.wapa.gov
The process stopped at . Debugging was disabled, the verbosity level was 1. 
192.208.27.6 / www.wapa.gov(online)ping results
address192.208.27.6 (ipv4) 
hostnameswww.wapa.gov (PTR) 
portsThe 998 ports scanned but not shown below are in state: filtered
Port State Service Reason Product Version Extra info 
80 tcp open http-proxy  syn-ack BlueCoat SG-400 http proxy      
443 tcp open http-proxy  syn-ack BlueCoat SG-400 http proxy      
remote operating system guessused port 80/tcp (open) 
os match: Blue Coat SG200 proxy server (SGOS 4.2.2.8 - 4.2.6.1) 
accuracy: 87% 
reference fingerprint line number: 3912 
os match: Blue Coat SG810 proxy server (SGOS 4.2.3.26) 
accuracy: 86% 
reference fingerprint line number: 3929 
system uptimeuptime: 402275 sec 
last reboot: (null) 
tcpsequenceindex: 255 
difficulty: Good luck! 
values: 58EE6935,2E21BDD,1A24EBB5,62D9428F,2468D4B6,4AF9CF25 
ipidsequenceclass: Incremental 
values: EC5B,EC5C,EC5D,EC5E,EC5F,EC60 
tcptssequenceclass: 2HZ 
values: C46C2,C46C2,C46C2,C46C2,C46C3,C46C3 
tracerouteport: 21 
proto: 
Hop Rtt IP Host 
1    
runstats32 sec. scanned 
1 host(s) scanned 
1 host(s) online 
0 host(s) offline

[cstreit]

Monitus has a nice online tool... Assuming the page is extranet and no intra-....

[masraum]

Been busy today. Thanks for all of the ideas guys.

Mikester, it's too bad we weren't able to hook up in Titusville/Cocoa. I ran a capture on my firewall while I did a get from a telnet on a router. That seemed to work.

Now I'm beginning to think that it's something either in our firewall or IPS SSM cards in the firewalls. I could swear that yesterday it was fast through one firewall and slow through the other, but now it looks slow if it's going through the firewalls only. Crud!

[mikester]

What kind of firewalls are these?

It is ashame but with the family 'issues' I had - it would have been difficult.

[masraum]

Asa-5540s

[mikester]

code version? If you're running one of the 8 versions memory utilization can get up there and start impacting performance.

We had this happen regularly - stuff would start getting slow and then finally just stop working. I would reboot them about every 60 days.

8.3 requires double the memory by the way. =-)

[masraum]

Originally Posted by mikester

code version? If you're running one of the 8 versions memory utilization can get up there and start impacting performance.

We had this happen regularly - stuff would start getting slow and then finally just stop working. I would reboot them about every 60 days.

8.3 requires double the memory by the way. =-)

Horrible version of code, 8.0(4). For about 5 months (maybe more) we haven't been able to "write mem" (or copy run start if you aren't old school). I'm sure a reboot would fix it, but getting the OK to reboot these things is next to impossible. Actually, we have ammunition coming to allow us to not just reboot, but upgrade. I'm not sure if we're going to 8.2(1) or 8.3(1). We have the memory upgrades on order to run 8.3 if we decide to go that route.

Fortunately, we won't have to do the upgrade remotely. We've got a spare pair. Once we get the memory, we're going to prestage the spares and ship them up and swap the cables.

[stealthn]

How are you going through both VRRP?

[masraum]

Originally Posted by stealthn

How are you going through both VRRP?

Both Internet connections?

We have sites all over the country on an MPLS cloud. All of the sites are talking BGP to the cloud. Both Internet POPs advertise a route to the Internet. Depending upon where you are, routing decides which Internet circuit you use. To test both, I just access a remote site that I think should use the other link, do a trace to confirm, and then test.

Nothing as fancy as VRRP or GLBP or anything like that. Besides, I think those are both expecting a more local network for the gateways, like 2 gateways at a particular location. Our two connections are Tempe and Omaha.

[mikester]

This isn't a pair of ASAs? If it were you could easily reboot the secondary, once it was backup fail over to it and then reboot the primary. Once it's back up - fail back to the primary.

We're on 8.2.1 now I think (I'll check tomorrow) and it's been pretty stable. I just received my memory upgrades for a bunch of 5510s and we'll be trying 8.3 here soon enough. I've been testing it for a bit now and I haven't seen any issues.

I also applied for the AnyConnect 3.0 beta - not that I have time for it with my studies...

[masraum]

Yes, it's a pair. One of the SSM modules failed and the pair failed over to the ASA with the good SSM. Because of the traffic that flows through them 24 hours a day, it's difficult to get approval to do anything through them. We're unlikely to get approval to failover and reboot without something more concrete in the way of a fix involved (not so much for the slow traffic, but for a handful of issues).

Yes, we should have our memory upgrades by the end of next week at the latest. We're also getting ready to deploy SSL VPN to replace our VPN 3000 concentrators. It's kind of a shame. I'm not a huge fan of the GUI in the concentrators, but it's been my experience that they just always work.

[mikester]

I effing hate the concentrators.

I've been doing SSL VPN for the better part of 3 years now so if you have any questions let me know.

I like the product - I'm not sure I liked what I saw with the new version of AnyConnect though.

Your failover pair is configured for stateful failover right? If it is - the hit from a failover would be less latency than the hit you're experiencing from the problem that started this thread.

[mikester]

And we're using 8.2(2) at the moment.

Will you be doing posture assessment or do you know?