I think I found the problem (500 - Error)

[Wayne 962]

So, here's the deal. The software used to run the forums (MySQL) is open-source, but inherently defective in that some queries take forever to process and then the same query (to the database) will process in a few seconds only a few minutes later. I'm not sure what causes this. I have been trying to solve this problem for years now. I've been on all of the bulletin boards, and numerous other people with very large forums often have the same problem. It's caused by the search engine reading too much from the database or something like that.

So, I wrote a program that constantly monitors rogue database inquiries and kills them off (these lock the whole site up for a period of time) after 60 seconds. Trouble is, there was also another setting that set the maximum execution time for scripts to be 30 seconds. So, if everyone else was waiting, and then I killed off that rogue process, everyone else would then get killed too (and get that 500: Error). So, I changed that variable, and that should fix most of the problems with that error for now. I figured it out tonight when I happened to have two forums pages open at the same time in two separate windows, and both gave 500:Internal Server Errors, which in my opinion was impossible. That led me to investigate further and figure out that the processes were timing out - not getting killed erroneously by my watchdog program.

I also believe that the slowdown is caused by disk access - the server itself runs at about 5-10% CPU utlization all of the time - certainly not taxing. Network bandwidth is also not a problem. The only other metric is IO speed. So, I have now set the database to defragment itself every night on the most used files. Each time I have done this, the slowdown problem has gone away temporarily until the files get defragmented, and then the intermittant slowdown comes back.

Conclusion, with these two changes, I think we'll see a remarkable improvement. Less slowdowns, and we should have almost no 500: Internal Server errors now. Sorry if this also doesn't make any sense to the non-computer people...

thx,

Wayne

[Fishcop]

WTF

My brain hurts

[rnln]

Not exactly the 500 error problem, but I encoutered similar problems with several work places. I am not sure if my little experience can help but maybe a reference. Once it was dish space available. We were using SQL server and had only 2 or 3 G left. Rebooting server would fix it for 1 or 2 days. Another time was the same cause, disk space, but it behaved as application problem. Watching the numbers of loging in, it kept increasing and won't decrease after users logged out (leaving the dead process behind). It acted as you forgot to close the connection to your DB in your application. Temp fix for us was running a connection kill at night, and made sure to include the log file when you do your backup. This log file can get pretty big if you allow. Anyway, after we upgrade disk space, problem was gone.

Yes, tonight I don't see the usual slowliness problem.

Also, there is another problem related that I forgot to mention. That is the data is not saved. User would think it's saved since it acted normal. And I think you might have the same problem, since there were several replies I posted but never see it again.

[johndglynn]

Nice one Wayne, I am using the search function all the time now as I try to figure out some trick jobs on my C3, the 500 errors were making it slow going. Glad you understand this stuff! Thanks as always.

[JohnJL]

I've never actually gotten a 500 error, just 404 timeouts. I'm using IE 7 and IE6.

[Zeke]

Does this do anything for the double post problem? (which seems to be getting better)

[Wickd89]

Its fun to use your engineering degree and experience every now and then!

I just got back from a training course in the Northeast, where I hung out with fellow electrical engineers reviewing WCDMA, QPSK and 16QAM modulation techniques amongst other quantization and channelization techniques. Sometimes funner than being a manager...

Now back to the office and assume my role of time sheet approver!

[tcar]

Originally Posted by milt

Does this do anything for the double post problem? (which seems to be getting better)

Same question, plus timeouts.

[equality72521]

The error I get sometimes when I try to post says I've created identical posts in the thread even though I've never posted in but it still creates the post.

[Wayne 962]

Originally Posted by rnln

Also, there is another problem related that I forgot to mention. That is the data is not saved. User would think it's saved since it acted normal. And I think you might have the same problem, since there were several replies I posted but never see it again.

The software does a trick where it puts your post information on the page, and then writes to the database after that (speeds the process for the end user). The trouble is, the processes to post beyond that were accidentally getting killed. I think this change fixes that...

-Wayne