Two big events happened recently for the I.T. team. We moved the PLoS production servers to a new colo facility and upgraded the journal websites to the latest Ambra release.
Josh, Russ and I recently moved all of our production servers from UnitedLayer to Internet Systems Consortium (ISC) in Redwood City. Russ implemented a failover stack for the journal websites so there was no site out during the move. Unfortunately, once the servers were moved, things did not go as smoothly as planned due to server malfunctions and network configuration issues. We were able to switch the sites from the failover stack on Sunday, May 23 but needed a few more days and some quick site outages to deal with cleanup. Josh was almost given his own office at ISC because he was onsite for a number of days. The dust has settled but we still have a few things to do at the new colo and don’t expect any further site outages.
The journal websites were updated last night to the release of Ambra 1.0 (“Babbage”). This release contains the new and improved search UI for advanced search and the search results. Liz has a nice writeup on the new features. If you haven’t seen the new UI yet, head over to your favorite PLoS journal and do a simple and advanced search.
The sites were slow for a few hours after the upgrade because a Yahoo crawler bombarded our sites, completely ignoring the robots exclusion standard (robots.txt) that we have in place. Upgrades require clearing a cache that holds ~1 million objects (mostly images) and it has to slowly refill over a period of ~24 hours. The sites can normally handle this traffic when the cache is filled, but the combination of the cache just starting to refill and the Yahoo crawler slamming the sites at the same time caused the sites to slow down. We have blocked the Yahoo crawler but will re-enable it once the cache fills and/or Yahoo responds to our complaints about their crawler.
The journal websites are stable now and barring more voodoo doll shenanigans, we shouldn’t have any other unplanned site outages or slowdowns.