Stack Network Status Updates

Status.stack.nl is a site used to list impactful maintenance and problems with the Stack systems or services or related issues. It is hosted outside the TU/e so it should be reachable when e.g. network or power problems prevent service to Stack and/or TU/e.

Planned maintenance to critical systems will always be announced in many places: on the Stack website, this status.stack.nl page, in the stack.announce newsgroup on news.stack.nl, in the /topic of #stack on the StackIL IRC network, in the MOTD of the login servers and often by email communication.

Status updates

2012-03-02 16:30 - 2012-03-02 17:00 - Network problems

A major switch in the Stack network started racing following a powercycle to an media converter in a different building, making the network highly irresponsive. The converter has been disconnected and we are investiging the exact cause.

2012-02-24 19:30 - 2012-02-24 20:00 - Storage migration

Between 19:30 and 19:45, primary userdata (including websites and mail) will be moved to the new storage system. The member login servers and most websites will be offline during this 15 minute window.

2011-12-02 - Beetle unavailable

Loginserver Beetle is temporarily unavailable due to hardware failure. The hardware failure may be a result of the power failure or vice versa. We hope to have replacement hardware up and running next week.

2011-12-02 21:30 - 2011-12-02 23:00 - Power interrupted

Power in part of the Stack room has been down on Friday evening. This affected the network uplink in the Bunker and several of the servers there; but not those in the server room. Login systems that went down include snail and turtle and member machines; but toad, www.stack.nl and www.stud.tue.nl were not affected.

2011-11-26 18:30 - 2011-10-28 - Turtle down

Turtle is currently unavailable after a crash that cannot be fixed remotely. Turtle should be available again from Monday (time unknown). In the meantime, servers Snail, Hammer, Beetle and Toad can still be used.

2011-10-19 13:30 - 2011-10-19 18:15 - Turtle and Snail down

Apparently the Bunker building had a short power outage. Servers Snail and Turtle require maintenance before being online again. This will happen later today.
Update 2011-10-19 18:15. Snail and Turtle are available again.

2011-10-09 15:00 - 2011-10-10 21:00 - Login server move / Network issue

The login servers Turtle, Snail and Hammer have been moved to the Bunker. Unfortunately, there are some issues with network stability at the new location. These servers currently remain unavailable, while we deal with the network issue. We hope to fix this later on Monday. Login servers Toad and Beetle remain fully available.
Update 2011-10-10 16:00. The network connection between Stack in the Bunker and the Stack network core in the Laplace building is functioning correctly again.
Update 2011-10-10 21:00. Loginservers Snail, Turtle and Hammer are available again.

2011-10-08 12:00 - 2011-10-09 - Login server moves

As part of the move of Stack to the new location in the Bunker, login servers Turtle, Snail and Hammer will be moved this weekend. The systems will be unavailable from Saturday, 12:00 to Sunday afternoon. Also, Jabber, XFS and XDCMP will not be available during this window. Login servers Toad and Beetle will remain fully available.

2011-08-28 12:00 - 2011-08-28 18:00 - Server moves

In preparation for the relocation of Stack to the Bunker in September, a number of servers will be moved to different locations. The login server Beetle will be unavailable for about 30 minutes. Also, mail-related services and authentication services may be briefly (for a maximum of 10 minutes) unavailable.

2011-08-14 13:30 - 2011-08-14 16:00 - Moving servers

Several servers are moved from to a different server rack. First, fileserver Mammoth will be moved (13:30-14:30). As a result, the News and Ubackup services will be unavailable for about 30 minutes. Then, the servers Edge, Volta and Meestal will be moved. As a result, (most) associations' websites and the Webmail service will be unavailable for about 20 minutes.

2011-07-09 10:00 - 2011-07-09 10:30 - University network maintenance

The university IT department will upgrade the software of several routers. As a result, the Stack network will be inaccessible using IPv4 for about five minutes during this window.
Update 2011-07-09 10:55. In reality the network upgrade took almost an hour, and Stack was unreachable over IPv4 for about 40 minutes.

2011-06-05 18:00 - 2011-06-05 22:00 - Network maintenance

Stack and the university IT department will perform network maintenance on the connection between the Stack and TUE networks. This will result in some interruptions lasting at most a few minutes at a time.

2011-03-10 - Network maintenance

Due to network maintenance in the TUE server room, the Stack network will be unreachable for about 10 minutes in the evening, most likely between 18:00 and 20:00.

2011-03-03 15:40 - 2011-03 17:10 - Association sites down

Because of a failing RAID controller Volta went down. As a result, most association websites are currently unavailable. The websites are currently being moved to different hardware so that the RAID controller in Volta can be fixed or replaced.
Update 2011-03-03 17:20. The RAID controller memory has been replaced. This appears to have fixed the problem. We'll keep monitoring the situation.

2011-02-13 23:40 - 2011-02-14 07:30 - Fileserver down; services unavailable

The Stack fileserver, "sabretooth", crashed after a low-level IO error. As this disabled the serial console, the system could not be revived until early morning. As the fcileserver was down, websites, shell service en most other services were unavailable during the night. The cause is being investigated.

2010-12-14 10:40 - Talbot issues

This moring our server Talbot crashed. When it came back up, there was no connectivity on its primary network interface and it failed to switch automatically to the backup link. We rerouted the traffic and it mostly works again now (although network is somewhat slow). Later the server crashed again. We continue to investigate and hopefully a permantent fix will be in place later today. A number of services are responding slower than usual and may be occassionally unavailable until these issues are resolved: www.stack.nl, member websites, some association sites, databases DB7 and DB9, POP3/IMAP/Webmail for members, mailing lists (website lists.stack.nl), loginserver Toad, anonymous FTP (ftp.stack.nl) and Jabber.

2010-11-05 13:30 - 2010-11-05 14:30 - Server Move

Server Talbot will be moved from a Stack server room in DH to the TUE server room in LG. As a result, a number of services will be unavailable during this window: www.stack.nl, member websites, some association sites, databases DB7 and DB9, POP3/IMAP/Webmail for members, mailing lists (website lists.stack.nl), loginserver Toad, anonymous FTP (ftp.stack.nl) and Jabber.

2010-10-31 18:00 - 2010-10-30 21:00 - IPv6 Network Maintenance

Due to a restructuring of Stack's IPv6 routers, the IPv6 uplink will be down for part of this maintenance window, as well als IPv6 end-user tunnels.

2010-10-01 07:30 - 2010-10-01 08:30 - TUE Emergency Power Test

The university will perform a test of the emergency power changeover system (switch to diesel aggregates). As a result, some Stack servers may be temporarily unavailable for a few minutes around 8:00am.

2010-09-18 - 2010-09-20 - StackFiler1 unavailable

The StackFiler1 (virtual) fileserver lost it relationship with the TUE domain. This can only be rectified by the TUE Dienst ICT. As a result, StackFiler1 is unavailable until Monday. StackFiler1 is used by several study associations for Windows file storage.
Update 2010-09-20 11:45. The domain relationship has been restored. StackFiler1 is available again.

2010-08-16 14:25 - 2010-08-16 15:10 - Network outage

Due to a faulty UPS unit, the network connections between the computer room in De Hal and the Stack association room were unavailable. As a result, Stack was also unreachable from the outside world. The equipment connected to the faulty UPS has been reconnected to a different supply, fixing the immediate problem.

2010-07-19 07:20 - 2010-07-19 08:00 - Talbot/toad/DB7 update

Physical server talbot, which runs various "virtual" systems, such as toad (webserver and loginserver) and database server DB7, received a scheduled reboot this morning. However, an issue with the remote console prevents the correct booting of the system. The system is expected to be available again at approximately 8:00 am.

2010-05-05 - 2010-05-05 - Toad software updates

The application software on Toad (webserver for all members and some associations) is being upgraded. In the process, some webpages may be temporarily unavailable. If a problem persists, please contact unix AT stack DOT nl.
Update 2010-05-06 10:00. The PHP upgrade from 5.2 to 5.3 caused some issues with installed PHP applications. Therefore, PHP is being reverted to version 5.2. This can cause some PHP errors this morning.
Update 2010-05-06 12:00. Due to issues with an important PHP 5.2 module, version 5.3 has been retained.

2010-04-30 05:30 - 2010-04-30 09:45 - Slow network response talbot

Server talbot (host of web/loginserver toad and database DB7 among others) was largely unresponsive due to a problem with its network card (or the driver for it). The problem could not be resolved by reinitialization of the link. Eventually server talbot was rebooted to fix the problem.

2010-04-27 06:30 - 2010-04-28 08:30 - Electrical systems maintenance

Due to maintenance to the building's electrical systems, Stack services will be largely unavailable between 6:30am and 8:30am on Tuesday, April 27. Most systems will be shut off between 6:30am and 6:45am and be powered on again after 8am.
Update 2010-04-27 09:00. The power was not actually turned off, so systems didn't automatically boot. All important servers were running again by 9:00. The non-essential systems (stockholm and some member-owned systems) that aren't available yet, will become so shortly after 10am.

2010-03-24 12:00 - Websites down

A peculiar network issue is causing websites/www.stack.nl to not be fully available. We are investigating the problem; in the meantime some workarounds have been implemented. We hope to have the problem fully fixed by about 14:00.

2010-03-22 16:00 - 2010-03-22 16:45 - Talbot/toad down

One of the main servers talbot has crashed and cannot be reset remotely. This server hosts toad.stack.nl: a loginserver and also the main webserver and local mail delivery server; and some databases. We are now on our way to fix it locally.
Update 2010-03-22 16:45. The problem turned out not to be with Talbot, but the switch it is connected to. The switch was reset, as well as talbot, and all hosts and services are available again.

2010-03-08 20:00 - 2010-03-09 11:45 - Authentication issues

Since yesterday's move of toad there are some issues with internal network connections to and from this server. This causes IMAP authentication to time-out and fail (but not in all cases) - and may cause other problems as well. We hope to resolve the issue soon.
Update 2010-03-09 11:45. Now 'fixed' by rebooting toad.
Update 2010-03-09 12:00. Cause found and permanently fixed.

2010-03-08 18:00 - 2010-03-08 21:00 - Toad migration

Login/web server toad will be moved to new hardware (talbot). Since toad is a virtual server, the system move will only take about 10 minutes, which is to occur somewhere between 18:00 and 21:00.
Update 2010-03-08 20:00. The move will start by 20:10.

2010-03-04 18:00 - 2010-03-04 21:00 - New snail installation

Thursday night a new operating system install (FreeBSD 8) will be installed on snail (like turtle before). As a result, snail will not be available for some time in this window (approximately 30 minutes).

2010-02-27 16:00 - 2010-02-27 20:00 - Toad recovery

A failed battery test on the UPS toad is connected to, caused to lose power. Unfortunately there was significant filesystem corruption. We are currently trying to fix this.
Update 2010-02-27 16:38. The filesystem corruptions are worsened due to bad RAID synchronisation; the filesystem is broken beyond repair. Therefore toad will be provided with a new installation. This will probably take a few hours.
Update 2010-02-27 18:30. The webservice is fully operational again.
Update 2010-02-27 19:55. All services on toad are fully available again, including logins. However, performance is decreased in this temporary configuration. Reboots will happen to move the system to faster hardware again, so screens and long term sessions are best run on other login servers.

2010-02-06 08:30 - 2010-02-06 14:00 - TUE network maintenance

The IT department of the university will perform major maintenance on the university network backbone. Unfortunately, Stack (and several other affected parties) were not informed of this operation. As a result, Stack was unavailable for some time during this window. The network operators have informed us that the operation will be over by 14:00.

2010-02-04 18:00 - 2010-02-04 22:00 - New turtle installation

Thursday night a new operating system install (FreeBSD 8) will be installed on turtle. As a result, turtle will not be available for some time in this window. Also, crab may be unavailable for a short time to assist in the installation of turtle (which is essentially a copy of crab).
Update 2010-02-04 22:00. The upgrade has been cancelled due to a partitioning problem with turtle's disk. The installation will now happen in a different way; next Thursday turtle and crab will need a reboot to activate the new installation but long downtimes will be avoided this way.

2010-02-02 14:10 - 2010-02-02 15:10 - Authentication server unavailable

As a result of an expired SSL certificate, Stack password authentication was temporarily unavailable. The problem has been corrected by installing a new certificate.

2010-01-27 08:00 - 2010-01-27 14:00 - Newsserver maintenance

The Stack Usenet service news.stack.nl will be offline for a couple of hours on 27 January for maintenance. During this time we will rebuild the overview and history databases of this service. As a result, older articles should once again be visible after this update. The service might be unavailable longer if recovery doesn't go as planned.

2010-01-26 07:00 - 2010-01-26 17:00 - Turtle NFS issues

Turtle is non-responsive due to problems with its connection to the fileserver. A reboot attempt caused the machine to hang completely. The server will return this afternoon when somebody has physical access to reset it.

2009-12-16 - 2009-12-16 14:00 - VPOP access problem

Logins to "virtual" mailboxes (VPOP) for assocations using IMAP and Webmail (but not POP3) all currently fail. We are investigating the problem and hope to have a solution soon.
Update 2009-12-16 14:00. The system clock of the VPOP server was lagging just over 30s with regard to the fileserver. As a safety feature, the IMAP server (affecting also webmail and authentication email submission, but not the POP3 server), did not allow logins for this reason. Unfortunately this was not logged or returned to users, so it took a while to track down.

2009-11-23 12:20 - 2009-11-23 14:00 - Stack outage

Stack has been unavailable for approximately 90 minutes due to an electrical problem. A short occurred in the primary supply cable, causing a fuse to break. For reasons that are currently unknown, the network did not failover to the secondary power source on loss of the primary supply. The power was available again a little before 14:00 using a temporary hookup. A new supply cable will be installed later today.
Update 2009-11-23 14:20. The systems toad (web/login), trivia (SQL databases) and turtle (login) are available again; all systems are now up and running.
Update 2009-11-23 21:00. The Stack network is currently a little slow as the failure of core switch ziggurat has reduced some of the backbone connections to 100Mbit, occasionally overloading the network. This will be fixed Tuesday.

2009-11-11 15:00 - 2009-11-12 08:00 - Snail NFS issues

Snail is again experiencing problems with NFS. The system needs to be rebooted to fix the immediate problem. This will be performed on Thursday at approximately 8am. Since the cause of the problem is currently unknown, the issue may recur. You may wish to relocate to a different login server: turtle, hammer, toad or crab.

2009-11-10 07:45 - 2009-11-10 09:45 - Power dip; hammer down

At approximately 7:45am, the university experienced a power dip. As a result, most Stack systems were power cycled. Unfortunately, hammer.stack.nl did not recover automatically. As a result, hammer and the SVN and mailing list service are not yet available. We expect to have these running again at approximately noon.
Update 2009-11-10 09:45. Hammer, along with the SVN and mailing list services, is available again.

2009-11-10 00:00 - 2009-11-10 08:00 - Snail NFS issues

Snail has had some problems with NFS (communication with the fileserver) today. The system is currently unavailable as a reboot failed as a result. The system will be available again Tuesday, hopefully without further problems. In the meantime, the shell servers turtle, hammer, crab and toad are available as alternatives.
Update 2009-11-10 08:00. The power dip caused snail to complete its reboot procedure, after which snail was available again.

2009-11-14 00:00 - 2009-11-14 02:00 - MySQL database upgrade

MySQL databases DB2 and DB5 (both running MySQL 5.0) will be upgraded to MySQL 5.1. As a result, the databases will be unavailable for some time during this window.

2009-11-07 09:00 - 2009-11-07 12:00 - TUE Network maintenance

The TUE IT department will perform network maintenance on Saturday between 9:00 and 12:00. As a result, there will be several network interruptions on the university network that will affect Stack.

2009-11-01 23:00 - 2009-11-01 23:15 - Toad reboot

Toad will reboot for a software update. As a result, loginserver toad, a number of websites and some mail services will be unavailable for a few minutes.
Update 2009-11-01 21:15. The update has been rescheduled to 23:00 (from 22:00).

2009-10-02 08:00 - 2009-10-02 08:45 - Generator test

Friday October 2nd, the annual test of the university's backup generators will be held. This will cause an interruption of power on the emergency power source to which a number of Stack systems are also connected. Stack will connect these systems to UPSes and other sources, but limited capacity may cause a short server downtime at the beginning and end of these tests.

2009-08-20 17:40 - Hammer unavaible

Some of hammer's local filesystems were damaged. We are currently repairing the damage. The system will probably be online again later tonight. In the meantime, hammer and the SVN service are down.
Update 2009-08-20 19:00. The damage has been repaired; hammer and SVN are available again. The cause of the filesystem corruption is still unknown.

2009-08-14 18:00 - 2009-08-14 19:00 - Hammer upgrade

Loginserver hammer will be unavailable for about 30 minutes, during which hammer will receive a CPU upgrade.

2009-07-28 22:00 - 2009-07-29 10:40 - Turtle unavailable

Turtle is currently unavailable, probably due to problems with NFS. The issue should be fixed later this morning when somebody has console access to the machine. Please use the other login servers for now.
Update 2009-07-29 10:40. Turtle is available again. The problem was indeed NFS related.

2009-07-24 12:30 - 2009-07-24 13:30 - Toad move (2)

Toad will be moved from the server room to the Stack main room. The move will take about 15 minutes, during which toad and most websites at Stack will be temporarily unavailable. This move was originally planned for 23-7

2009-07-23 12:30 - 2009-07-23 13:30 - Toad move

Toad will be moved from the server room to the Stack main room. The move will take about 15 minutes, during which toad and most websites at Stack will be temporarily unavailable.
Update 2009-07-23 11:15. Due to weather circumstances, the move is rescheduled for Friday, July 24th.

2009-07-21 12:30 - 2009-07-21 13:30 - Database server move

Between 12:30 and 13:30, database server trivia will be moved to a different location. As a result, all MySQL databases as well as PostgreSQL database DB4 will be unavailable for about 15 minutes during this window.

2009-07-11 11:00 - 2009-07-13 09:00 - Snail down

Snail is currently unavailable. The exact cause is not known. Unfortunately the problem cannot be fixed remotely, so snail will be unavailable until Monday 9:00. In the meantime, please use the other login servers: hammer, turtle or toad.

2009-07-01 00:00 - 2009-07-01 16:15 - Power interruption consequences

As a result of the power maintenance yesterday, some critical hardware seems to have died. This includes harddisks of vaak (dns, mail, ldap), trivia (databases), gauss and the gigabit switch trion. As a result several services are still down. More news later.
Update 2009-07-01 12:45. E-mail functionality has been (mostly) restored this morning. Databases (our priority now) and IPv6 are still down.
Update 2009-07-01 13:20. Database server trivia is now up and running again (without RAID and still finishing database repairs)
Update 2009-07-01 16:15. Finally: all standard services are now functional again - including everything on vaak.

2009-06-30 17:00 - 2009-07-01 09:00 - Major power maintenance

Due to major maintenance to the university's electrical systems, the building where Stack and its servers are housed (De Hal) will be without (main) power between 18:00 and (at least) midnight. Emergency power will remain available, so Stack's critical servers will remain available. The following services will remain available: all websites, databases, webmail and email delivery and some supporting services. Toad will be available as a login server but other login servers will be offline. Services that will be unavailable include the IRC server (irc.stack.nl), the Usenet news server (news.stack.nl), mailing lists (lists.stack.nl), Subversion (svn.stack.nl) and all IPv6 tunnels. Systems will be shut down around 17:00. Most systems will be available again after midnight. The remaining systems will be brought up by Wednesday 9:00.
Update 2009-06-30 11:00. The database server (trivia) and LDAP server (vaak) will be moved from the Stack room to to the server room so they can use the emergency power. As a result, these services will be unavailable for short periods between 11:00 and 14:00.
Update 2009-06-30 14:00. Vaak and trivia have been successfully moved to the computer room.
Update 2009-06-30 18:00. All systems have been shut down.
Update 2009-06-30 21:20. One of the remaining servers (vaak) has crashed. Under the special circumstances of tonight this has a major impact. Zen (nameserver) has been migrated to a different machine for now. Currently password logins due not work anywhere until the LDAP server is operational again. We are working on this.

2009-05-17 00:00 - 2009-05-18 18:30 - Turtle unavailable

Loginserver turtle is currently unavailable, probably due to a network filesystem problem. The system cannot currently be reset remotely. Turtle is not running critical services. The system will be reset either May 17 or May 18.
Update 2009-05-18 18:30. Turtle is available again.

2009-05-05 00:15 - 2009-05-05 03:20 - Database server problem

Database server trivia (host of DB2, DB4, DB5, DB6 and DB7) did not shutdown cleanly for a scheduled reboot. After the boot, a lengthy filesystem check was started which has yet to finish.
Update 2009-05-05 03:20. The server and all databases are available again.

2009-04-27 16:25 - Trivia (databases) unavailable

The machine Trivia, which hosts all databases at Stack, has crashed. Unfortunately, the machine cannot be restarted remotely; this will be fixed as soon as possible. In the meanwhile, any services that use MySQL or PostgreSQL database services (i.e. most websites) will be unavailable.
Update 17:10. The system did not boot correctly after a restart. This has been fixed; the server and all databases are available again.
Update 18:10. The server has been connected to a networked power switch and console concentrator for improved remote management.
Update 19:30. The databases have been inaccessible for approximately one hour due to a configuration error. Fixed.

2009-04-18 00:00 - 2009-04-18 03:00 - MySQL upgrades

MySQL server db2 is to be upgraded from MySQL 4.0 to 5.0 (via 4.1). The database upgrade only affects associational accounts using "db2" or "localhost" as the MySQL host. All www.stud.tue.nl web services will be unavailable. After the upgrade, the MySQL server will also be operating on a different machine, but this change will be transparent. Performance of both the database server and the associational webserver (www.stud.tue.nl) will be improved by this upgrade.
Update 02:30. Due to data import complications after installing MySQL5, the MySQL 5.0 upgrade will be rescheduled for another date. Database DB2 will run MySQL 4.1 until then.
Update 03:00. The database and webserver are available again.
Update 2009-04-19 02:30. MySQL 4.1 was moved to a faster server. This move is transparent for all supported configurations. Performance of databases and webserver should be improved.

2009-04-17 19:50 - 2009-04-17 20:30 - Power outage; meestal-mk5 down

A power outage has occured as a failed server power supply tripped a breaker, draining the UPS batteries. When the outage was detected, the breaker was reset. The server with the failed power supply is still down. The mailhost.stack.nl mail submission server will be moved to a different server tonight; the server itself will be physically replaced next Monday or Tuesday.

2009-03-31 06:30 - 2009-03-31 08:15 - Power maintenance

The university will perform maintenance on the power distribution of De Hal where Stack is situated. As a result, the systems outside of the server room will have to be turned off during this maintenance. Systems will be turned off by Stack Systems Administration between 06:30 and 06:45 and back on after 8:00. Some services that will not be available during this window: POP3/IMAP (popserver.stack.nl), mailing lists (lists.stack.nl), webmail (webmail.stack.nl), the news server (news.stack.nl), and shell servers hammer, snail and turtle. The web and database services should not be impacted and login servers toad and vwww will remain available for logins.
Update 2009-03-31 06:30. IPv6 networking, both internal routing and IPv6 tunnels, will also be unavailable, as will Subversion (svn.stack.nl), IRC and Bitlbee.
Update 2009-03-31 08:45. All systems were up again by 08:15. IPv6 was fully operational again by 08:45.

2009-02-27 18:00 - 2009-02-27 19:00 - Server move vaak

Server "vaak" will be moved from server room DHS01 back to the main Stack room. The move itself will take about 10 minutes. During this time, a number of services at Stack will show delays because vaak acts as the primary DNS and LDAP server. Especially noticeable will be delays (but apart from that, normal operation) in login attempts (password verification is done through LDAP).

2009-02-15 - Telnetd disabled

Telnet logins have been disabled on all of the standard FreeBSD login servers at Stack, due to an exploitable bug in telnetd. At the moment there is no fix available for this problem. We advise everybody to use SSH logins instead. SSH is much more secure by default. Unlike telnet which sends unencrypted passwords over the wire, SSH encrypts not only your passwords, but your entire session.

2009-01-22 - 2009-01-23 - Server moves

For the second phase of the building cleanup maintenance, the basement has to be cleared. Systems currently housed in the basement will therefore return (in case of snail, hammer, mud, tinkywinky, maxwell) or temporarily move (in case of helsinki/lobster) to different locations. The moves will be performed Thursday and/or Friday (January 22-23), resulting in up to 20 minutes of downtime for each of the mentioned systems. Other services running on these systems are: news.stack.nl (mud), keyserver.stack.nl (mud), irc.stack.nl (maxwell), bitlbee.stack.nl (maxwell), webmail.stack.nl (snail), fontserver.stack.nl (snail), hts.stack.nl (snail), xdmcp.stack.nl (snail), finance.stack.nl, (hammer), lists.stack.nl (hammer), svn.stack.nl (hammer).
Update 2009-01-22 20:30. All systems have been moved already and are available again.

2009-01-18 - 2009-01-20 - Mud (news, keyserver) unavailable

The server mud has suddenly died this afternoon and won't boot anymore (failure to detect SCSI interface). This server also also runs the Stack newsserver, news.stack.nl, the PGP keyserver and OuterSpace mud. We expect to have this machine up and running again within a few days with replacement hardware.
Update 2009-01-20 19:30. Mud is available again, using a different machine.

2008-12-19 - 2009-01-09 - Turtle and lobster unavailable

During the holiday break, maintenance will be performed in De Hal. Therefore a number of systems will be moved to different locations in the weeks before December 20th. Due to limitations to the number of systems that can be housed, login servers turtle and lobster (on helsinki) will not be available during the break.

2008-12-15 - 2008-12-19 - Systems down in preparation for maintenance

From Monday December 15th through Friday December 19th, all equipment in the Stack main room will be turned off. Many servers will be moved to rooms in the basement to provide service during the holidays. Other systems, such as turtle, shiroi and cyaan, will be shut off completely and put in storage. Also, network functionality will be moved to different equipment or the equipment itself will be moved. As a result of all this, systems and services may be unavailable for some time in order to move the systems. This can take up to an hour per system. Network interruptions are also possible.

2008-12-11 15:00 - 2008-12-11 17:00 - Core network problems

Due to a hardware problem, the Stack core switch and router (sequoia) stopped working. This caused a part of the Stack network to be unreachable. Workarounds were made by 17:00; we are currently investigating the problem. These problems were due to hardware failure and are not related in any way to the scheduled network maintenance.

2008-12-11 15:00 - 2008-12-11 19:00 - Short downtime due to server cabinet maintenance

On Thursday December 11th, between 15:00 and 19:00, the Stack equipment rack (rack 5) in the computer room of De Hal will be reorganized to make place for temporary equipment during the holidays break. Database server oslo will be moved to a different position, resulting in about 10 minutes of downtime of most databases (db0,db3,db4,db5,db6). Also, the rack's main gigabit switch will be replaced, resulting in approximately 5 minutes of downtime for a number of systems, including the fileserver, resulting in a 5 minute downtime of most services.
Update 2008-12-11 17:00. Oslo has been replaced. This look about 30 minutes due to mechanical problems with the mounting rails.
Update 2008-12-11 20:00. With a little delay due to the core network problem, the network maintenance in rack 5 is completed.

2008-11-06 23:00 - Meestal-mk4 and IPv6 tunnels down

Meestal-mk4 has crashed for the second time today. Unfortunately the system cannot be remotely booted again. Internal IPv6 routing has been moved to a different system and is operational; IPv6 tunnels and uplink connectivity are currently unavailable. This will be fixed tomorrow (Friday November 7th).
Update 2008-11-07 11:10. IPv6 tunnels and DHCP are operational again.

2008-11-05 15:15 - Toad ssh key changed

Somehow the ssh host key of hammer had been copied to toad a while ago. So both machines were using an identical server key. The ssh key of toad.stack.nl has now been replaced. The DSA key fingerprint for the new key is: be:75:a3:be:08:90:c9:74:aa:7c:c0:4c:1d:f0:3a:6b.

2008-10-23 15:15 - 15:45 - Uplink connections

The secondary uplink will be connected. Because the core switch sequoia has to be taken offline anyway for a reconnection to a UPS, we will also test failover of the uplink connection. During this time, there will be two or three network interruptions of about one minute each.
Update 2008-10-23 21:00. Introduction of the second network uplink is postponed until further notice. The core switch was however reconnected to a different UPS.

2008-10-22 18:30 - 2008-10-21 18:50 - UPS problem

A currently undiagnosed problem caused one of our UPS units to fail. As the core routing switch (sequoia) is connected through this UPS, Stack was unreachable for about 20 minutes.
Update 2008-10-22 19:00. The UPS may need to be disconnected tomorrow to investigate the problem. This will probably result in a 2-minute network downtime.
Update 2008-10-22 20:35. The UPS again indicated a failure and was about to fail, so it was disconnected. This resulted in a short downtime.

2008-10-20 07:00 - 2008-10-20 12:45 - Vwww network problem

A problem with a switch port that vwww (www.stud.tue.nl) was connected to, caused bad network performance (and hence slow website response). The problem has been solved by moving the connection between vwww and the fileserver to a different network connection.

2008-10-06 23:00 - 2008-10-06 23:15 - Network maintenance

The university's network operators will update the backbone switches in De Hal. These switches provide Stack's uplink connection, so Stack's network will be unavailable for about 10 minutes.
Update 23:10. Maintenance over.

2008-09-27 12:00 - 2008-09-27 17:15 - Toad upgrade

On Thursday, September 27th, toad.stack.nl will be upgraded to FreeBSD 7 and some hardware modifications will be made as well. Webservices such as www.stack.nl and virtual hosts on websites.stack.nl will be unavailable during this time. Email services will not be impacted. The hardware upgrades are done: new network, fibre channel card and disks are installed, and the system was remounetd in the rack at a different location. The operating system is also complete. We are however noticing some problems with the new network card, so we will replace it first. As a result, the upgrade will take somewhat longer.
Update 13:15. The hardware upgrades are done: new network, fibre channel card and disks are installed, and the system was remounetd in the rack at a different location.
Update 15:45. The operating system is also complete. We are however noticing some problems with the new network card, so we will replace it first. As a result, the upgrade will take somewhat longer.
Update 17:15. The network card problem appears to be with the main board (PCI slot) instead. Finally an alternative network configuration was selected. The system is now available again. However, application rebuilds are still going on, and toad may need to be rebooted later today or tomorrow (Sunday).
Update 0:00. PHP has been broken since about 20:00 due to a version conflict in a deep dependency library. PHP should work again somewhere overnight.

2008-09-16 06:30 - 2008-09-16 13:15 - Meestal-mk5 (mailhost) unavailable

Meestal-mk5 seems to have crashed. No external services are impacted by this as they are all implemented redundantly, except for mailhost.stack.nl services (authenticated mail submission, viewing of virtual host mail aliases). The system should be available again by about 13:00.
Update 13:00. Meestal-mk5 and its services are available again.

2008-08-25 15:00 - 2008-08-26 - Vwww performance issues

Vwww (www.stud.tue.nl) is currently experiencing severe performance problems. We are investigating the problem. Vwww may be occassionally be unavailable for a few minutes due to testing of new configurations (i.e., reboots).

2008-08-20 17:00 - 2008-08-22 20:00 - Network upgrade and issues

Last Wednesday through Friday, a new switch was installed in the network, to serve as the new core switch and router (multilayer switch). The installation of the switch itself was not a problem, but triggered some problems with other equipment on the network, including the VLAN table corruption on ziggurat and a final failure of old core switch ziggurat. It also triggered more issues with the old core switch (arreat), which eventually gave up completely on Thursday. Arreat has been completely disconnected and all functionality moved to the core switch. On Friday, a few final modifications were made, completing the core switch migration.

2008-08-20 20:50 - 23:20 - Switch issues

There is a problem with a server switch (ziggurat) at Stack. As a result, most of the 141 machines are unavailable. The core services however remain available.
Update 2008-09-20 23:20. Problem solved. The problem stems from a corruption of the switch's VLAN table when another switch was connected by the system. The table has been restored.

2008-08-19 15:45 - 16:30 - Webmail and snail unavailable

System snail crashed and needed a manual checkup. As snail runs webmail.stack.nl, the Stack webmail was temporarily unavailable as well.
Update 2008-06-19 16:30. Snail and webmail are available again.

2008-08-06 18:30 - 19:45 - Brief network outage

Due to a technical problem with the Stack core switch (arreat), the Stack network was effectively down for approximately 15 minutes. The issue resulted from a failed hardware initialisation, requiring manual intervention. The core switch is to be replaced in two weeks.

2008-08-06 13:00 - 16:00 - Dbadmin temporarily unavailable

The dbadmin.stack.nl site, hosting MySQL and PostgreSQL administrative interfaces, is unavailable for approximately two hours to due a migration to another server.
Update 2008-08-06 16:00. The migration was completed (although with some delay).
Update 2008-08-07 15:00. A workaround has been implemented to prevent DNS caching from redirecting you to the old (inactive) server. If the workaround is active for you, you will be redirected to dbadmindev.stack.nl, causing an certificate warning. This can be safely ignored.

2008-07-13 09:00 - Vwww problems

There is a communication problem between the webserver vwww (www.stud.tue.nl) and the fileserver (sabretooth) resulting in incredible slowness. We are investigating the problem.
Update 2008-07-13 11:15. The problem appears to be with one fiber switchport. The fileservice link has been bypassed using another port; the remaining network connectivity will be moved to a different fiber switchport later today. The acute problem appears to be resolved.
Update 2008-07-13 13:00. Vwww has been connected to a different fiber port, fixing all connectivity problems.

2008-07-10 14:15 - 15:05 - Meestal-mk4 unavailable

Meestal-mk4 is being moved to a new case. IPv6 tunnels and connectivity with the .141 machines will be temporarily unavailable.
Update 2008-07-10 15:05. Downtime was a bit longer than expected due to software trouble. Everything works again now.

2008-06-06 16:30 - 21:00 - Turtle unavailable

Due to a disk crash, turtle is unavailable and needs to be reinstalled on a new disk.
Update 2008-06-06 21:00. A new disk was installed into turtle and the turtle install was copied from snail, quickly making turtle fully available again.

2008-05-16 18:30 - 22:30 - IPv6 and 141 net interruptions

The hardware and software release of meestal-mk4 are upgraded. The system will be replaced by a faster one, and the operating system will be upgraded from FreeBSD 5 to FreeBSD 6. During these upgrades, the IPv6 routing, uplink and tunnels will be unavailable, as well as IPv4 connectivity to 141 systems. Official Stack services will not be interrupted by this (not on the 141 network). Exceptions are the news service (news.stack.nl) and PGP keyserver (keyserver.stack.nl).

2008-04-24 14:30 - 23:30 - Listservice unavailable

The listservice is being migrated from maxwell to sj2 (a jail on hammer) to improve performance.
Update 2008-04-24 23:30. The migration and processing of the original queue was taking longer than expected. The listservice is fully operational again on the new server, and the delay problems of the last few weeks are fully over.

2008-03-20 15:30 - Turtle unavailable for logins

Application software on turtle is currently being upgraded to make sure all software is using the new FreeBSD 7 libraries. Because some problems occurred, both with application availability and system stability, user logins will temporarily be unavailable until the upgrade has been completed.
Update 2008-03-31 13:30. All software rebuilds have completed. Turtle is available for logins again.

2008-03-18 13:10 - 14:10 - Fileserver crash

The main Stack fileserver (sabretooth) crashed with battery problems. We are trying to get it back online a.s.a.p.
Update 14:10. The fileserver has been completely replaced. This is because the problem is suspected to be with the internal charging system, not the battery itself."

2008-03-16 - Webmail/IMAP service unavailable for association accounts

The new IMAP mail service for normal association accounts is not working correctly due to problems within the authentication subsystem. We are currently investigating possible solutions. The service should be operational again later today. The IMAP service is also used by the Horde/IMP and SquirrelMail webmail systems. Note only association accounts are affected, not member accounts or VPOP accounts.
Update 16:15. A workaround is in place for the authentication problem. The IMAP and Webmail services for association accounts are fully operational again.

2008-03-14 17:00 - 19:00 - Upgrade vwww (www.stud.tue.nl)

The webserver for association accounts (vwww.stack.nl a.k.a. www.stud.tue.nl) will be upgraded from FreeBSD 6 to FreeBSD 7. In the meantime, websites and email for assocations will not be available.
Update 19:00. The upgrade was completed successfully. Over the next few days, all software will be rebuilt to use the new libraries. This may at times cause short interruptions of certain services (e.g., PHP may not fully work during PHP rebuilds).
Update 2008-03-15 23:00. All applications have been rebuilt, there should be no further interruptions.

2008-03-11 17:30 - 19:00 - Upgrade turtle (turtle.stack.nl)

One of the standard login servers for Stack members, turtle, will be upgraded from FreeBSD 6 to FreeBSD 7. The other servers continue to be available for logins and other services.

2008-03-04 07:00 - 07:30 - Building network maintenance

The university ICT Deparment will perform maintenance on the switch equipment in De Hal causing Stack to be unavailable for a few minutes during this window.
Update 07:30. Maintenance done, fixing a number of network connectivity problems in De Hal.

2008-02-23 03:00 - 2008-02-24 07:00 - Network outage

The Stack network in the server room in De Hal is down, most likely due to a crash of the switch connecting the Stack equipment there to the Stack main room. We will investigate onsite at 06:00. The network will probably be running again by 08:00.
Update 07:00. The switch has been reset and provided with the latest firmware. All is well again.

2008-02-07 06:30 - 08:00 - Power system maintenance (part III)

On Monday February 7th, the Building Services Department will perform maintenance on the emergency (i.e., aggregate-backed) power systems of De Hal. While some critical Stack systems are using this grid, no Stack systems will have to be shut down. However, as the building network equipment is connected to this grid, Stack will not be reachable from the outside during this window. The status.stack.nl website will of course remain available.
Update 07:45. Unfortunately our fileserver (sabretooth) and database server (oslo) were also unavailable as well due to a problematic power configuration. These servers and the building network were available again by 7:45. Other systems were not affected.

2008-01-28 09:50 - Turtle unavailable

Turtle's power supply is defective and turtle is therefore unavailable at this time. A replacement will be available within the next few days.
Update 16:00. The power supply has been quickly replaced (with some rewiring of connectors) and turtle is available again.

2008-01-28 06:30 - 08:00 - Power system maintenance (part II)

On Monday, January 28th, the Building Services Department will perform maintenance on the building power system for De Hal. This will cause most Stack services to be unavailable during this window. Exceptions are www.stud.tue.nl, www.stack.nl and of course this status website.
Update 11:00. All servers are available again with the exception of turtle. The Macintoshes shiroi and cyaan are also available again.

2008-01-15 06:30 - 08:00 - Power system maintenance

On Tuesday, January 15th, the Building Services Department will perform maintenance on the building power system for De Hal. This will cause most Stack services to be unavailable during this window. Exceptions are www.stud.tue.nl and of course this status website.
Update 08:00. Systems are available again.

2008-01-11 07:15 - 2008-01-11 08:00 - Storage maintenance vwww

The local storage in vwww (www.stud.tue.nl) will be extended to provide for more room for SVN repositories and other data. The services on vwww (www.stud.tue.nl, association user logins, POP/IMAP/SMTP for associations and SVN) will be unavailable between 7:15am and 8:00am.
Update 08:00. Storage expansion completed successfully.

2007-12-23 - 2008-01-17 - Shiroi not available

Shiroi, the Stack Mac Mini, is unavailable due to a failed disk. The system will hopefully be available again somewhere in January 2008, depending on the arrival time of the new disk.
Update 12:00. Shiroi is available again. A new disk was placed and the system reinstalled.