Web Server Performance...

[Wayne 962]

Hi all. Bad news! Our ISP appears to be going out of business. The unfortunate thing about it is that it seems that we might have contributed to their failure (well slightly).

When I first moved servers over to them about 2 years ago, we were dishing out about 18 gigs of traffic each day. Now (and I had to check 3-4 times to make sure I was right), we're delivering more than 6 gigs per day (about 180 gigs a month). More than tenfold of where we were two years ago. Needless to say, I don't think they increased my rates enough...

Anyways, we're looking to move the server in-house and separate them into a BBS machine, and a website machine. This will make the access to the BBS a lot faster. I think lately, some of you might have noticed a hesitancy on the server when dishing out pages. This was caused by server procesor bottleneck and not connection bandwidth. I've increased the time between searches to 30 seconds (I'm sure you've noticed that) to decrease the bam-bam-bam server load associated with multiple searches, but there is still a problem.

The overall goal is to move the server(s) in-house to our HQ location in El Segundo. However, I'm not sure if we need 1 or 2 T1 lines run to our location. So, over the next day or so, I'm going to be running some simulation tests on the server, basically limited it to T1 speeds, to see if there is a slowdown.

Since some of you use the board and the site even more than I do, please let me know if all of a sudden the site seems much slower. I've limited the output of the server to 150K/sec. Granted, this should only show up in high-load times, and might even be caused by processor overload.

Well, I guess that at least these are good problems to have...

-Wayne

[widebody911]

Well, other than your 'math' issues.

I think the mission-critical nature of the site requires at least 3 clustered systems in geographicaly diverse locations with redundant OC-54 lines and ...

[Wayne 962]

Sorry if I wasn't clear:

PLEASE LET ME KNOW WHAT YOUR IMPRESSIONS OF BBS AND SITE SPEED ARE FROM YOUR END.

Thanks,

Wayne

[targa4me]

Wayne,

Site speed is always good and Im beaming in from Oz using SLOW OLD 56k.

BTW: 18 Gigs to 6 Gigs a day #!&^%?

[RJ Gilliam]

Wayne, I work for SUN Microsystem and maybe I can be of some help if your gonna get some hardware.

Can always get you a little more bang for the buck.

RJ Gilliam
1986 930 Turbo
"Julie"

[JeffW]

I'm sitting on a 1 megabit dsl line and I always have fast responce time from your site...

[nostatic]

I access the site from a variety of locations either dedicated T3 w/10/100 switches or at home via 768 DSL. Much of the time I find the site to load *abysmallyl* slow, with graphics often hanging for long periods of time. I don't know what you're running for a server, but I suspect that PHP isn't the bottlneck. You're not stuck on a wintel box, are you?

Depending on your needs you could move it inhouse. I also use an external ISP for some of my sites (www.westhost.com). They are reasonable and robust in my experience, and support MySQL.

[jriera]

Wayne, no difference with my Cable Modem ~2Mb loading pages.. seems a little bit slower while 'constructing' the indexes for each one of the different forums...

Let me know if you need any help.. web/network design is one of my 'abilities'

regards

[Wayne 962]

Sorry, 18GB a month to 6 GB a day.

Looks like we're going to go with redundant T1 lines from XO. Anyone heard anything bad about them? Also keeping the DSL line as a backup.

With the bandwidth throttled, it seems pretty good, but at HQ, we'd be sharing connections with www.maxstudio.com. They use about half the traffic of Pelican though.

PHP without a doubt is indeed the big bottleneck. I can watch as the site hangs and CPU % usage pegs at 100% from IIS and Mysql. Yes, wintel is annoying at times, however, I have been pretty happy with Win2k server on this box.

I need to move the BBS to it's own server. It will go soooo much faster then, and it won't slow down the site at all...

-Wayne

[nostatic]

Originally posted by Wayne at Pelican Parts


PHP without a doubt is indeed the big bottleneck. I can watch as the site hangs and CPU % usage pegs at 100% from IIS and Mysql. Yes, wintel is annoying at times, however, I have been pretty happy with Win2k server on this box.

I have to disagree...the problem is with IIS, not PHP. If you run Apache with PHP as a module, it is plenty fast (we've done it on ours and a number of collaborator's sites)...your pipe will be the bottleneck, not the application.

Use the same hardware, run RedHat with the Apache/MySQL/PHP setup and life will be good. Once it is configured you won't have to touch it. In our experience anything with IIS requires much more attention.

Or chuck it and get a G4

[jriera]

I agree with nostatic.. Linux+Apache or Linux+TUX is the way to go if you want a hands-off/lights-out operations..

I have been running Windows and many flavors of Unix in commercial/hosted applications for a long while and we even had to schedule weekly 'downtimes' to reboot the Windows machines to clear memory and caches, never on Unix..

As nostatic says, IIS is not bad, but requieres more attention.

[Wayne 962]

While I appreciate the feedback from Unix fans, I must insist that I know that the bottleneck is with the processor. I can watch MYSQL sop up the CPU cycles. A similar thing happened with the old BBS sofware (UBB), and from VB boards and UBB boards, it doesn't quite matter which OPSYS you're running.

IIS has done a marvelous job of dishing out the pages to our visitors. We are averaging about 300+ megabytes of traffic an hour in our peak times. When the server stops for a second or two, it's when I'm doing something (or someone else) in VBulletin or Mysql. Like deleting a message. For some reason, the software does some type of complex removal query that takes several seconds to access the database.

Don't forget that we've got 12,000 daily visitors, and about 47,000 hits an hour in our peak usage times. For one singular server (PIII 850mhz 1 gig RAM IDE mirrored), that's not too shabby (in my opinion)...

Trust me, if you could see all the statistics on my end, you would be convinced too that it's PHP and MYSQL. I don't really think that PHP is too much of the culprit, but I can see MYSQL actually eating up the processor cycles left and right...

-Wayne

[jriera]

REALLY, not bad for a PIII....

MySQL can be a little bit of a memory/CPU hog in Windows installs (after all it was not developed for Windows) specially with the rebuild of indexes in memory.. are you running MySQL 3.23??

Not a Unix fan it just happens that I run (manage the people that operate will be a better definition) 100's of servers in several OS's.. I'm just a VM fan.. but has not made it yet to PC's.. (aside from the IBM PC/VM in the mid 80's).

Bottom line if ain't broken don't fixed!!! this is not a Porsche!!

[Wayne 962]

I just made a change to the software that should speed up the reply process! (I had to remove the VB code section that usually is above the reply box).

-Wayne

[campbellcj]

If you split the tiers (database, bbs code, web server etc) across multiple boxes, be sure you use a big pipe between them - otherwise you're just creating another bottleneck.

Sounds like it also couldn't hurt to go to a multiprocessor box if you're confident that CPU throughput is an issue. While you're at it, you could put in an array of 10K or 15K RPM Ultra160 SCSI disks and 2-4GB of RAM too. The parts are all pretty cheap these days...and gosh-darnit this BBS is a mission-critical app if I've ever seen one!!!

Oh yeah...don't forget a big line-conditioning UPS and a nice cool A/C unit! Don't want any of those power grid hiccups taking out the site this summer!

[Wayne 962]

Yada, yada, yada. The box is running pretty darn well considering!

The BBS is very processor intensive. The webserver is much, much less. I'm just going to split tasks, hardware is cheap!

-Wayne

[DJB]

My 2c and Wayne you probably will not like it.

Many may suggest that the BBS is mission critical, at times it is.

Your e-commerce web site IMHO is one of the best on the internet and a testament to your technical skills and from a business (revenue) point of view, is mission critical.

IMHO running these two apps internally will cost you big money. The cost is the 'opportunity cost' of spending time away from driving your business by having to worry and administer a web hosting environment.

I would suggest that maybe you should outsource the hosting task to professionals and spend your time either driving your business or driving your car.

Wintel hardware may be cheap, but keeping it running is time consuming.

[rstoll]

Ah, "growing pains". They hurt so good!

[si2t3m]

Hi Wayne,

I'd consider myself as one spending too much time on this board browsing and collecting info!

I have i dialup connection @ 56k usually connected @ 46.6Kbps.

As for searching or browsing the forums, everything is fast enough considering my connection.

HTH

[Wayne 962]

Hah, DJB you give good advice for most people, but there is way too much custom software running on these boxes to give it away to someone else.

Part of what has made Pelican a success was our embrace and utilization of technology. Without that edge, we would be just another vendor. I personally don't know anyone qualified (nor would I trust any company) to adequately service our machines.

We've been doing it this way now for four years, and it seems to be working well, so if it ain't broke, don't fix it!



-Wayne