View Full Version : The downtime ...
JoShea
03-14-2004, 08:33 PM
For those who missed the notice today, we had to do an emergency upgrade on this server due to a security issue. The databases, being as large as they are, took much longer to restore than we had anticipated.
We appreciate your patience during this time.
Due to technical difficulties some posts made late Saturday evening or Sunday morning are not showing up, they are in the database however. If you really need one of these missing messages recovered, please email me with your username and title or topic of the post.
joshea @ amitymama.com
We can't tell if the same problem exists for PM's. Again, please ask if you suspect you are missing a PM from late last night or this morning.
RocketScientist
03-14-2004, 08:53 PM
Jo, does that go for pictures uploaded here, and polls too? Amity had posted a poll here: http://www.amitymama.com/vb/showthread.php?s=&threadid=154199 and while her post is there, the poll has disappeared. Also danica's post here: http://www.amitymama.com/vb/showthread.php?s=&threadid=154193 has a very different picture than it used to have uploaded in it. :)
Thank you for your hard work!
JoShea
03-14-2004, 09:23 PM
However, I cannot recuperate the picture. I have removed the link from her post, she'll have to re-attach it.
nuttymudder
03-14-2004, 09:34 PM
Are you free to explain what security issues it involved? "Security issues" and "emergency upgrade" doesn't sound like a good thing.
Daisy Duck
03-14-2004, 09:59 PM
Yeah...what nuttymudder said...
and being told it would be back "shortly"... then actually having it back after 8 1/2 hours is a big suspicious (sp) difference too.
Soggy Granola
03-14-2004, 10:08 PM
Thanks for your hard work Jo. :)
norasmama
03-14-2004, 10:38 PM
Originally posted by Daisy Duck
...
and being told it would be back "shortly"... then actually having it back after 8 1/2 hours is a big suspicious (sp) difference too.
It doesn't sound suspicious to me. When I worked in an office, it always seemed to take a day to reboot the system when we had to do upgrades of any size. Although our IT department could usually do them on Saturdays when no one was at work. We at Amity's are here 24-7!
Thanks for the hard work, and the warning, Jo!
I still can't get into the email
JoShea
03-14-2004, 11:10 PM
Originally posted by Daisy Duck
Yeah...what nuttymudder said...
and being told it would be back "shortly"... then actually having it back after 8 1/2 hours is a big suspicious (sp) difference too.
Our post said it was going offline shortly.
*I* had estimated a couple of hours downtime, but I had grossly underestimated the size of the databases (way over 1.5 GB!) This maintenance was necessitated to fix a software exploit... our first experience at having a server 'hacked'. Unfortunately, many servers using the same software, this one included, were hit before the exploit became known and the patch was made available. Someone was able to gain access and place a trojan and rootkit on the server. The easiest and securest way to reverse the damage done by a rootkit is to completely reinstall the OS. That was the easy part since additional spare servers are on standby. The big job was moving the content and db only from the old to the new server.
And we had planned to do it overnight ... it just didn't work out that way ... the techs were working as fast as they could, as I said, others were hit as well ... stand in line and wait your turn, KWIM?
I hope that helps you understand!
JoShea
03-14-2004, 11:29 PM
Originally posted by Ladywithafan
I still can't get into the email
LMK if you can get in now.
arasmama
03-14-2004, 11:49 PM
Thanks for your work Jo!
One other thing - search feature doesn't seem to be working right. I keep getting the database error page. Also can't sort by username.
jessica_momof7
03-14-2004, 11:56 PM
Originally posted by JoShea
Our post said it was going offline shortly.
*I* had estimated a couple of hours downtime, but I had grossly underestimated the size of the databases (way over 1.5 GB!) This maintenance was necessitated to fix a software exploit... our first experience at having a server 'hacked'. Unfortunately, many servers using the same software, this one included, were hit before the exploit became known and the patch was made available. Someone was able to gain access and place a trojan and rootkit on the server.
so does this affect us the users in anyway?
For example, one mama I'm e-mailing with can't get in. (huntersmommy) She keeps getting this message:
"continuing maintenance. we expect to be fully back online within the hour. posted 1:45 pm est."
???
JoShea
03-15-2004, 12:03 AM
Originally posted by Stacy
For example, one mama I'm e-mailing with can't get in. (huntersmommy) She keeps getting this message:
"continuing maintenance. we expect to be fully back online within the hour. posted 1:45 pm est."
???
That page is no longer on the server!. Make sure she tries refresh/reload. Or to come in directly by http://amitymama.com/vb since THAT page was only for the index.
Thanks, Jo! I let her know. Hopefully it'll work. Thanks for your hard work!
Originally posted by JoShea
That page is no longer on the server!. Make sure she tries refresh/reload. Or to come in directly by http://amitymama.com/vb since THAT page was only for the index.
JoShea
03-15-2004, 12:12 AM
Originally posted by jessica_momof5
so does this affect us the users in anyway?
The object with trojans is that they turn the server into a 'bot'. They will drop trojans onto a bunch of servers then get them to simultaneously launch a ddos attack against their chosen target The rootkit changes system files that deal with logging and/or intrusion detection - in order to cover their tracks and hide the illicit files. :(
JoShea
03-15-2004, 12:16 AM
Originally posted by arasmama
Thanks for your work Jo!
One other thing - search feature doesn't seem to be working right. I keep getting the database error page. Also can't sort by username.
Please elaborate - I am able to do searches. Does the empty search form page come up at all? What particular terms are you searching for and what settings re time,etc? *Where* are you not able to sort by username - the posts, the user list, what ?
arasmama
03-15-2004, 01:44 AM
First, I tried searching for a particular term on the sewing board. Didn't come up even though I knew it was there (and found it later a different way). It gave me the "not found" page.
Then, I tried sorting by username and it gave me the database error page (still sewing forum).
Then, I went toe the "search" page, entered the username I was searching for and "sewing forum" and again got the database error page.
This was all a couple hours ago, maybe 8:30 pm PST
Now, I just went to the search page again and tried. Username search gave me database error. Term search (that I know is there) gave me not found. Then, I searched "fabric" (which is probably listed a million times on the sewing board) and it gave me database error.
All these searches were for the sewing forum, I didn't try any of the others. HTH
nuttymudder
03-15-2004, 07:48 AM
Thanks for the details and your work, Jo. Although all the technical stuff is way over my head! I wasn't accusing at all, just curious if we had been hit randomly, targeted, in danger, if pm's had become open to someone, or what! Glad you got the patch in to help make this place a little safer from hackers.
vBulletin® v3.6.5, Copyright ©2000-2008, Jelsoft Enterprises Ltd.
SEO by
vBSEO 3.0.0 RC8