|
|
|
|
{Please reload this page often during times of trouble} HERE are announcements of interest. Note that we hide none of our problems!
Our backbone connection via our DS3 line failed this morning. This report is being entered at 4:06am, and its still offline. Many many people are looking at the problem, and its affecting many internet users in the Sacramento and points North area. 5:28am, all is fine. SBC's backbone engineers finally confessed that they took the system down for "maintenance" and didn't bother to announce it to anyone. Hope that not too many of you were inconvenienced.
We are experiencing problems with our dialup lines in Yuba City, the problem is being "fixed" then it breaks again. Hopefully this will settle down soon. It is also affecting our regular phone lines. (25 Nov) Dialup lines are working again, but the incoming voice lines are toast for at least 4 more days. 11/29/05 5:00pm. Our lines were shut down by SBC intentionally "due to an error" and it took this long for someone to figure that out. Everything is normal again.
Yuba City this morning experienced a power outage of previously unknown length. PGE worked hard to get things running again, but 4 1/2 hours was just too long for our batteries to run...
Marysville PRI down. At approx. 09:00 hrs a group of in-bound lines for our dialup and ISDN customers failed. Those affected are customers dialing the "530-634-0108" number and our ISDN centrex customers in the Marysville area. (Brownsville and Loma Rica area is unaffected) This apparently is an issue with SBC. We currently have a trouble ticket open and SBC is working on the problem. No estimated time to repair.
Telephone Systems failture. Around 5:00 this morning the voice mail / telephone system crashed. Leaving busy signals on our voice number. (755-1751) Problem has been resolved. Cause is under investigation.
Colusa phone line is fouled up today, apparently Frontier was doing some maintenance... You know the rest of the story. Ticket opened with Frontier, ESTIMATED REPAIR IS TUESDAY! Call the Meridian number at 696-0668 if you are local to Meridian (in Colusa) - 6 Apr 3:00pm all is fixed...
Power failed
again, this time a bird apparently blew a breaker on the incoming cable
a block or so away from the office.
Power failed this morning, due to excessive
(VERY excessive) storming in the Yuba Sutter area. Most servers continued
running, but a few things failed, until finally around 8:30am the backbone
server failed. Can't ever seem to have enough batteries.
Our connection to the qwest backbone is up and running, but the Qwest backbone is offline. Qwest fixed the problem at 1:30am.
Our Tiara 1400 Router hung at about 3AM in the morning an wasn't fixed till just after 7AM same day. Have changed the way we get notified so hopefully it wont happen again.
Gad, I hate power outages. A wall jack that one of our Batterys was plugged into died. Really. Probably a result of Mondays storm. Major badness followed when the server attached to that battery failed. 9:53 all better now.
Power is out, some machines are happy (battery backups are working), others are not (batterys are dying). 5:36am, power is restored, but one (of many) servers is not happy. Appears to have failed rather spectacularly. A few of our commercial hosted websites are not available until we get it back. Fully operational by 5:00pm, minor fixes until Tuesday, then its DONE.
All connections to our NNTP services are offline, no ETA, but it appears to be a physical cut line between here and San Jose. Could be other parts of the internet are not available at the same time, but this is the only KNOWN point of failure that affects all of us using NNTP. -- 6:15 PM all is right in the world again!
I have taken the mail server offline, so all smtp and pop3 services are gone for a bit. Please be patient, there is a new better server coming online.
A single Power Backup, one week old, decided to give it up and shut down the main Ethernet Switch here. When this happens, all the internal servers can't talk to each other, so things really get stupid. A replacement is ordered, and a temporary solution is in place now.
We are seeing failures with the DSL circuits, and some people are seeing DNS failures. The DSL circuits are being repaired now, the DNS failures we are working on.
Major Upgrades performed, now SYIX is operating at T3 speeds, currently setup as 6 MB, with expansion available to 12M as needed! Cool stuff this technology! (for those keeping track, thats 4 times the bandwidth we had before)
This evening one of our 2 T1 circuits failed for a short time, due to a failure on the Qwest network. Internet connectivity was not lost, but things definitely slowed down! At 10:29 the second circuit finally came back online completely and things are working as they should once again.
I have had it with viruses. I have installed blocking software that will cause our mail server to groan under the strain, but I have got to get a handle on the "W32.Hybris.gen" Virus, we have it running all over our system. If you get a rejected message from the server, and it says you have a virus in the message you are sending, CLEAN YOUR SYSTEM!!!. Follow this link to www.symantec.com, they have complete instructions on how to get the silly thing out of there. And delete any outgoing mail you that has attachments on it, and resend it again after things are cleaned up.
Today our 822-7868 number has failed, its
returning either dead air or a busy signal. We are NOT overloaded, the line
is just fouled up (Trouble ticket into PacBell today). To get the same place
in the system change your dialup number to 751-8580!
Update: Dec 3 1400
PacBell assigned our number to a residence, nice of them huh? Poor people
must have been going nuts with all the calls. All fixed now.
New Edge will be upgrading the Yuba City DSLAM for our Yuba City DSL customers, it should last a very short time (less than one hour) sometime during the time allotted.
Qwest is replacing a card in San Jose we are connected to, to finally fix the problems that have occurred the last month or so. It will result in a quick outage (5 to 10 minutes) this morning, during the maintenance window.
At this time, our Backbone Connection is
San Jose is repeatedly going up and down, but only 1/2 of it at at time.
Strange results will occur, but be patient, the page will open!
Chapter II - 1:25 PM - Still working with Qwest to get things straight,
the problem is ongoing, and we are working on it.
Fixed at 4:40 PM! Yeah
Something wrong with the backbone in Northern California. Most sties are up now, parts of Yahoo still experiencing some difficulties.
QWEST's routers in San Jose were failing. Things were up and down sporatically for about 1 1/2 hours. Everything seems to be fine now and hopefully no more problems.
QWEST's lost all of it's routes to the
Internet. Rebooting their routers took about 20 minutes and Internet traffic
was interrupted for nearly 1/2 hour. Things seem to be up and working
fine now and their routes will continue to rebuild themselves throughtout
the day.
Lost QWEST's DNS Server upstream from us and would not allow Internet connections till the early AM. Also had internal problems with our Mail Server and PortMasters, leaving us with no mail access and only about 150 incoming lines. Everything was resolved about 9:45 AM. We are sorry for any inconveniences this might have caused.
What ever PacBell lost yesterday in the power outage continues today. It appears that our dialups lines are OK but some customers can't call our office line (751-1751) as all they get are fast busy signals.
We have lost approx 1/2 of our power, a PGE error we think. Things are working but you can't get a phone call in, and we are not sure how long the power backups will run in this condition...
We have lost one of our 2 data T1 circuits
to the backbone, and this is causing strange performance problems, sorry
for the trouble! Please ignore anything that is happening today!
A Harddrive crash has eliminated all data on this page from July 29, 1999 to June 5, 2000, so you can't see any outage reports for that time period, sorry.
Today from 11:36 to 11:50 am our dialup lines, and some of our internal servers were moved and re-arranged. The total outage time was 15 minutes or so, not bad considering that all the equipment had to be moved from one room to another, very short outage, and now all our dialup lines are running on Fiber!
We are scheduled to get our dialup lines moved to the new "LightWave" Fiber Optic lines today. There will be intermittent outages all day!!!
Today was a banner day at SYIX! DOUBLED our available bandwidth to the Internet Backbone! Now running an available 3Mb of data out to the world! The speed increase is noticeable!
4 Minute hic-up. All better before I could even get this typed. Might happen again, there is a major power outage in San Jose that is affecting a LOT of internet circuits. There may be places that you can't get to until power is completely restored.
NEW WHEATLAND Phone Number installed. Please make the changes to your dialup networking to reflect the new number if you are calling the WHEATLAND number only! New number is 633-2017
Our internet connection to the world is offline, working on it now!!! More info... 7:45. Qwest lost a OC192 circuit (a LOT of lines) and ours happened to be connected to it. More later. 8:15 - No change... Please be patient! 8:30 - Talked to Qwest again. Major circuit between here and Salt Lake City, Qwest and USWest (telco) working on it. 8:49 - All circuits up and running again. Enjoy! (I'm going home!!!)
After installing our newest PRI line (23 more dialup lines) PacBell broke 2 of our older ones. Of course, these events are not connected... Anyway, we are now down 23 lines from yesterday, until they get them fixed, its hoped that repairs will go into place around 8pm or so, so it might be okay for most of prime time.
Yep, Busy signals are haunting us again. During prime time, from 7 to 9 in the evening, we are experiencing busy signals. Your support has allowed us to grow faster than PacBell can get the wires installed. Thanks for the support, and we apologize for the problems. Please be sure your system is setup to automatically redial!
Today, PRI Number 12 was ordered. (23 incoming lines on each of these).
Today PRI Number 11, working fine!!!
During the early morning hours, all our dialup ports were upgraded to the latest "V90" code and a few fixes dealing with latency over modem connections was addressed. Things should look better for modem game players today and this weekend! Additional PRI lines will be installed on Jan 6 or 7th, even though we have not had any busy signals. Just wanted to get back ahead of the game!
Our 10th PRI line was installed today, giving you all 230 individual phone lines to call into! Cool huh? And even better, 23 more are ordered already to make SURE that busy signals don't happen again!
Due to increasing demand on the system, caused by an incredible rush in new members, we are experiencing intermittent busy signals. They seem to last just a few minutes, then things are back to normal. New digital lines are being added, they are due to be installed December 9
I've changed some things on the website that you may or may NOT notice, contact me if you see any problems that I may have missed or if a page is extremely slow... webmaster@syix.com , James
There is a scheduled outage set for 1:00
am local time Sunday morning, to enable a bit of maintenance on the Qwest
network, it will last less than one hour.
Today we setup a new "Usenet"
server, but now your "new article" pointers are going to be
fouled up. It will be fine after you 1. Refresh the list of
groups, and 2., read articles the first day.
There will be some scheduled upgrades occurring
in Denver to replace a "CORE" router that is failing.
A brief outage is projected during this time, probably very close to 7:00.
This will affect our "outbound" connectivity for a bit.
Additional Note, no impact on the network, nothing changed, or failed.
Cool!
We are going to replace some of the components in the News Server tonight. It should be offline for about 30 minutes and maybe even less. We are also going to be doing some much needed maintnance on our Web Server and should speed up some things on the webpages. Hopefully.
There is going to be a router replace in Denver that we are connected to on Saturday night at 10:00 pm. The maintenance should be completed by 10:15 or so, just wanted to let you all know.
This morning, PacBell, in a scheduled upgrade that they failed to tell us about, disconnected all our PRI lines. A call to Priority Repair says at first, and I am not kidding, "Whats a PRI line?". Lines were back up after 30 minutes of phone calls to different people in PacBell.
Our 2nd Portmaster (lines 47 to 92) decided to allow connections, but not allow anything to happen after you were connected. This made all of our monitoring systems think all was fine, but it wasn't. We are looking at improvements to our monitoring system to keep this from happening again. Anyway, it took a hard reset (power switch) to get it running. All is fine now,and I am monitoring it closely for any other problems.
We now support ALL the V.90 modems, all
of them. It appears that the manufacturers finally got their act
together and agreed to actually all support the same standard, and Lucent
released the code for our Terminal Servers. For information on your
modem, check with its manufacturer, or your computers manufacturer.
We just installed a new Point to Point T1 into QWEST, and now seem to have bandwidth to burn! Point-Point T1 is great stuff, but we have to pay by the MILE to operate it! CRL circuit is disconnected now, and we say good riddance. Frame Relay T1 is disconnected now, and to that we also say good riddance. Things are FAST again!
See THIS
PAGE. We are moving bytes again.
There is a problem existing between Sacramento and San Francisco that is causing a 30% (m/l) packet loss in our signal. Its being looked at by engineering right now, should have it back and running fine SOON! Connectivity is not lost, but things are VERY slow, and telnet and constant type connections are liable to be very unstable. UPDATE: July 18, noon. Still continuing to experience this problem, still rattling cages to get it fixed, bear with us. Alternative T1 circuits are being installed to eliminate this happening again. UPDATE: July 19, am. Still Broken. More information is available now, PacBell has limited our bandwidth on our T1 circuit to 384K for a reason that I can't seem to get an answer to yet. All T1 engineers don't work on weekends, and they don't consider this a failure so won't drag them to the office to fix it. Take a look at our internal monitor and see what it looks like on this end. Especially look at the second picture, the weekly graph. You can see this slowdown and the Wednesday outage also! UPDATE:`` July 20, pm. Still Broken. PacBell has finally at 4pm this afternoon admitted that they might have something broken. Its still not fixed at 7:30 while I type this, but there is hope. I am learning to hate PacBell. Really. Hate them a LOT. Anyone want to help me start a new local phone company? Additional Note: news.syix.com is running, but no new articles will be forthcoming until after this crisis is over. News uses about 40K of bandwidth by itself, and we don't have that much to spare right now. No posting will be allowed either, sorry about that! UPDATE: July
21, all day. Still Broken. Today I have received
calls from CRL and PacBell, both stating that they have found the problem
and it will be fixed "soon" Both so far have been wrong.
Repeated calls by me has resulted in "uh, that wasn't it, but we
have people on site trying to find the problem". I am still
frustrated, as you are also. It is BETTER, we have a bit more bandwidth,
and the packet loss is down to around 15%, so its a bit more usable today,
but still not good at all. Today our T1 circuit was shutoff and
back on about 10 times, during different tests being ran to find the source
of this problem. "news.syix.com" is running after 11pm
until 9am, during times of light load so we are getting a few articles
daily (150,000 yesterday) but I shut it off during the day to allow you
to use the web easier.
Our T1 to the world is offline again, this time its not routing packets, so of course, we are isolated from the rest of the internet. Telco has been contacted, and repairs are promised "real soon" Repairs completed 10:24 pm. Hope you all enjoyed your time with your families :-) UGH, of all the times for it to break huh?
Our T1 to the world is offline, as are most of the long distance circuits out of town. PacBell is working on it, and we hopefully will see them get the repair in SOON! Repairs were done at 1:10 pm. Cause was PacBell "power" error in Modesto. Really. They let the batteries run down in Modesto, causing all T1 lines in (530) to fail. Including the 911 lines!
News decided to go nuts last evening, and your system administrator failed to catch it. Sorry everyone, all should be fine again now.
For a while, until the software upgrade is complete, news.syix.com will be offline. New INN software with new features is being added. Sorry for the inconvenience. (finished at 7:00 am, time for bed :-))
Tonight around this time, one of our Portmaster Terminal Servers went into a "reboot" cycle due to the failure of one of its modem cards. About every 20 minutes it would reboot trying to "fix" the card, and it didn't. The card was replaced and all is well again. This was a machine at the end of our hunt group, so very few people were actually impacted.
Pacific Bell broke our Hunt Group again, we are getting about 1/3 of our modems usable, before it drops to never never land. Call 751-5032 if you can't get 751-8580 to work!!! Fixed at 6:00 or so. Did you note, that there is a theme to the PacBell/Broken stuff here?
Pacific Bell broke our Hunt Group again, we are getting exactly 1/2 of our modems usable, before it drops to never never land. Call 751-5032 if you can't get 751-8580 to work!!! But of course, you can't see this unless it works huh? Fixed at 9:30 pm (or so)
The main authentication server just went down, at 6:04pm, manually rebooted it at 6:42pm. All is normal again. Sorry about that!
Nothing is broke, and I am bored stiff...
Our First Portmaster (the machine that answers the modems) decided, after running for 36+ months, that it didn't have enough RAM in it, and decided to start randomly rebooting. This would have disconnected your connection repeatedly between 12:30am and 1:40 am today. I patched it temporarily, will fix it permanently later. Extra info... Added memory to 2 different machines, all should be fine as of 3:35pm Jan 2.
Tonight we are suffering from phone company problems. At 4:00 this afternoon we started getting "we're sorry, this line has been disconnected" due to an error by PacBell on the programming of the hunt group, and at 9:30 we are getting busy signals because when they fixed the first problem, they broke it a different way. Its been called in, and should be fixed very soon.
|
SYIX.COM OUTAGES
|
|
Tonight we are suffering from phone company problems. At 4:00 this afternoon we started getting "we're sorry, this line has been disconnected" due to an error by PacBell on the programming of the hunt group, and at 9:30 we are getting busy signals because when they fixed the first problem, they broke it a different way. Its been called in, and should be fixed very soon.
|
|