Blog from May, 2011

One of our commodity internet providers, NTT/Verio, successfully completed their maintenance this morning between 0200 - 0300.

The maintenance on the RNAS storage system has been successfully completed and tested.

IT is replacing hardware for "rnas" tonight, May 27, at 11:45 PM. The rnas drives and shared storage spaces are used for internal IT services and pre-OWLspace course accounts. Updates will be posted to the IT web site: http://it.rice.edu/

IT is replacing hardware for "rnas" on Friday night, May 27, at 11:45 PM. The drives and shared storage spaces hosted on rnas are primarily internal IT services and pre-OWLspace course accounts. If you use rnas, you may notice a brief connectivity interruption late Friday evening.

We have resolved the storage issue(s) and completed the maintenance on $SHARED_SCRATCH. Fortunately, we were able to preserve the existing user data on the failed file system.

Log a ticket if you see any issues.

IT monitoring has identified failing hardware in the DSPAM system. Although no customers are currently affected we will be replacing the failing components at 5:00pm.

Kenneth Marshall, PhD

Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

Webcal.rice.edu is back after a brief outage. Please report any further problems to the Helpdesk (xHELP).

Kenneth Marshall, PhD

Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

The scheduled network maintenance for today has been completed. If you have any internet related outages or issues, please contact the Service Help Desk (x4357, helpdesk@rice.edu) or Danny Eaton (x5233, dannyeaton@rice.edu).

We have worked with the Panasas engineers and have identified the root cause of the failure. The issue has required us to entirely rebuild the fast scratch file system. This is turning out to be a very slow process. We anticipate the rebuild to continue throughout the night and hope to finish tomorrow.

The ruf systems are back online. Users now have access to ruf web pages and user accounts.

The ruf.rice.edu systems are currently offline. IT is investigating. This outage affects web pages on ruf (www.ruf.rice.edu/) and ruf personal user accounts.

The problem with the CAAM SAMBA service has been identified and corrected. At this time, IT believes that SAMBA is working properly again.

Please report any lingering issues to the IT Help Desk.

We are currently experiencing issues with samba.caam.rice.edu. This services Windows users using storage space on the server.

The problem with the campus email system was resolved and tested. All email service are currently back on line.

The DSPAM system is online. Please report any problems to the Helpdesk (xHELP).

Regards,
Kenneth Marshall, PhD

Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

One of the 3 DSPAM database backends, which provides DSPAM anti-spam service for roughly 1/3 of campus (Cyrus2 users), is down and is being bypassed. It will most likely be replaced tomorrow afternoon. Until then, mail to the affected users will not be processed by DSPAM.

The cluster STIC is not accepting job submissions due to failure of the $SHARED_SCRATCH file system. Users may login as usual and $HOMES and $PROJECTS was unaffected. The fast scratch storage subsystem is under full maintenance and teh vendor has been notified. Our best estimate is that the issue will be resolved as early as Monday.

Storage systems online

The storage systems are now back online and available for use. Should you encounter any problems, staff will be onsite performing cleanup work until noon, please contact Operations.

The Rice networking department has completed the scheduled maintenance at the Primary Data Center (PDC). All network services should be up and functioning normally. If you experience any issues, please open a ticket with Operations (x4989).

All network-hosted services have been returned to standard operating levels after a 22-minute power outage in a small part of the on-campus data center. If you notice any remaining service interruptions, please contact the Help Desk: 713.348.HELP(4357) or helpdesk.rice.edu

The Ticketmaster system has been returned to service. If you notice any remaining issues with the system, please contact the Help Desk: 713.348.HELP(4357) or helpdesk@rice.edu.

www.ruf.rice.edu and kennel.ruf.rice.edu are back online. Please report any problem to the Helpdesk (xHELP).

Kenneth Marshall, PhD

Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

The Ticketmaster system is off-line due to the current power outage that has affected some network-hosted services. IT is continuing to investigate.

A power outage has affected some network-hosted services such as web sites on ruf.rice.edu and kennel.rice.edu. IT is investigating and will post updates to the IT web site: http://it.rice.edu/

Due to a power problem, www.ruf.rice.edu and kennel.ruf.rice.edu are offline. IT is investigating.

Kenneth Marshall, PhD

Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

The old ruf.rice.edu systems lost power. Power has been restored and services should be up very shortly. This includes
access to ruf web pages (www.ruf.rice.edu/) and ruf personal accounts.

There will be a partial network outage 5-8 AM, Saturday, May 21, 2011.

Rice.edu email and the home page for rice.edu will still work, but all other services hosted on the Rice network will be unavailable.

Details can be found in the IT web site at: http://it.rice.edu/Announcement1/

Power has been restored in Greenbriar. All network services are back online

Possible power outage affecting Greenbriar network service. Network Management is investigating.

At approximately 8:40, an interruption of campus internet access occurred. Networking is investigating the issue.

Rice Networking has completed the border maintenance. If you experience loss of internet connectivity, please contact the Rice Service Desk (xHELP, x4357) or Dylan Jacob (dtj1@rice.edu).

The central mail system is back online.

Kenneth Marshall

Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

The scheduled mail maintenance has started.

Rice Faculty, Staff, and Students,

IT will be performing annual maintenance on the Rice email system on Sunday morning, May 15, 2011 from 6:00 AM until 7:00 AM.
Email will be unavailable during this time, but messages will be queued for delivery at the end of the maintenance window.

Details on IT's scheduled outages can be found in the IT web site: http://it.rice.edu/Announcement1/

Main ISP link has been brought online. Please contact Helpdesk if you encounter further Internet issues.

Rice Networking has determined that one of our commercial internet providers is having technical issues at this time in the Houston market. They are aware, and working on the issue. IT Networking is switching all internet connectivity to our other peer. If you have any questions, please contact the IT Service Desk (x4357) or Danny Eaton (x5233, dannyeaton@rice.edu).

The storage maintenance has been completed and all services are available.

Maintenance will be done on the storage system at 12:01am Tuesday morning.

Storage.rice.edu and rnas.rice.edu clients will be affected by the maintenance.

Short outages will occur during the one hour maintenance window as we transfer services between redundant hardware.

There were reports today of new login attempts failing against storage.rice.edu. Waiting and retrying resulted in successfully gaining access.

Users who were already authenticated were not affected.