Blog from January, 2015

The problem related to the central authentication services has been resolved. 

A report of a problem with CAS the central authentication server has been verified.  IT staff have responded and are evaluating.

IT received a customer complaint that some email messages were bouncing to internal Rice addresses. Internal testing could not confirm the issue, however we did confirm the issue for the 2 reports and work around was quickly implemented.  No additional reports -of problems have been seen or reported at this time.  If you see any issues please report them to 713 348 4989.

The update to the campus DNS service infrastructure is complete. Please report any issues to the Helpdesk (help@rice.edu, xHELP).

Regards,
Kenneth Marshall
--
Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

The upgrade will begin at 6:00am.

Regards,

Kenneth Marshall
--
Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

Early tomorrow morning (6:00 AM on Saturday, January 31), each of Rice's four Domain Name System or DNS* servers will be upgraded, one at a time. The four DNS servers are redundant, so we anticipate no core service interruptions for users on the Rice network. 

Status updates will be posted online:http://it.rice.edu

*DNS servers route user-friendly host names like "rice.edu" OR "it.rice.edu" to their correct Internet Protocol (IP) addresses.  IP addresses are used for all Internet traffic, from email and calendaring tasks to browsing web sites and transferring files and data.  Because it acts as a traffic controller, the DNS service is a critical component in the process used by computers to access internal and external systems, services and web sites. 

IT Networking has completed its network maintenance in the Recreation Center.  Please report any connectivity issues to the Help Desk (x4357).

IT Networking will be performing maintenance on the network equipment in the Recreation Center.  An update will be sent when the work has been completed.

Database Migration Complete

The database migration was successful and all dependent services have been restarted.

The only functionality that is currently having trouble is the KeyVault under https://old.apply.rice.edu.  This will be fixed tomorrow.

IT is upgrading a database server upon which several IT services depend.  During the upgrade, the following services will be unavailable:

Services are expected to be restored by midnight.

Status updates will be posted online: http://it.rice.edu.

Early Friday morning, January 30, at 6:00 AM, one of the three network switches in Recreation and Wellness Center will be upgraded. The network connectivity for approximately third of the building will be affected during the change over, but services will be returned to full functionality after an hour. Updates will be posted to the IT web site: http://it.rice.edu

A critical software vulnerability has been identified. IT will be updating affected systems. Most systems are redundant and there will be no impact on services. For others, there may be brief outages when the updates are applied. Please report any issues to the Helpdesk (help@rice.edu, xHELP).

Regards,
Kenneth Marshall
--
Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

Machines may need to reboot to restore proper connectivity.

The planned network maintenance for Herring Hall - Center of Teaching Excellence will begin at 0600 am

Early Saturday morning (6:00 AM on January 31), each of Rice's four Domain Name System or DNS* servers will be upgraded, one at a time. The four DNS servers are redundant, so we anticipate no core service interruptions for users on the Rice network.

Status updates will be posted online:http://it.rice.edu.

*DNS servers route user-friendly host names like "rice.edu" OR "it.rice.edu" to their correct Internet Protocol (IP) addresses.  IP addresses are used for all Internet traffic, from email and calendaring tasks to browsing web sites and transferring files and data.  Because it acts as a traffic controller, the DNS service is a critical component in the process used by computers to access internal and external systems, services and web sites. 

Early Wednesday morning, January 28, at 6:00 AM, one of the five network switches in Herring Hall will be upgraded. The network connectivity for Center of Teaching Excellence will be affected during the change over, but services will be returned to full functionality before classes begin that morning. Updates will be posted to the IT web site: http://it.rice.edu

Network connectivity has been restored.  Please contact IT Operations (x4989) if you are still having problems accessing the network.

There is a partial network outage in Anderson Clark. IT Networking is investigating.

There is a partial network outage in Anderson Clark. IT Networking is investigating.

The CAS Web Authentication service experienced a short unplanned outage from 12:50PM to 1:00 PM. The CAS service should be up and available again.

Network management have isolated the root cause. 

Network management is investigating intermittent network connection on multiple buildings. Will provide update when available.

The scheduled maintenance for the Mir3 Emergency Communication service has been completed.  No service disruption was required and all services are online.

Machines may need a reboot to restore proper connectivity.

The planned network maintenance for Mech Lab will begin at 0600

The Confluence software used on the two wiki servers, docs.rice.edu and wiki.rice.edu, was patched to correct a security flaw.

Both are back up and running normally now.

 

Due to a security issue with the software, Confluence will be patched Thursday Jan 22 at 9pm on the two servers running the software, docs.rice.edu and wiki.rice.edu.

During the patch, wiki services will be shutdown and unavailable.  

The outage is expected to last 15 minutes. Status updates will be posted online: http://it.rice.edu.

IT Networking has completed maintenance on the Internet service.  Please contact IT Operations at x4989 if you are experiencing Internet connectivity issues. 

IT Networking will be performing maintenance on one of the Internet connections.  No impact is expected.  Communication will be sent when the work has been completed.

At 6:00 PM on Wednesday, January 28, IT will upgrade a database server upon which several IT services depend.  During the upgrade, the following services will be unavailable:

Status updates will be posted online: http://it.rice.edu.

Early tomorrow morning, IT will migrate the existing Internet2 connection to new hardware located within the Southeast Texas GigaPop network infrastructure.  During this maintenance, web traffic will be managed by the remaining three ISPs (Internet Service Providers) and no impact is anticipated. The work begins at 6:00 AM on Thursday, January 22. Status updates will be posted online: http://it.rice.edu.

Early Friday morning, January 23, at 6:00 AM, the second of two network switches in Mech Lab (Mechanical Engineering Laboratory) will be upgraded. The network connectivity for approximately half of the building will be affected during the change over, but services will be returned to full functionality before classes begin that morning. This work completes the upgrade begun on January 16. Status updates will be posted online: http://it.rice.edu

The scheduled maintenance has been completed.

Please report any issues to the helpdesk, helpdesk@rice.edu

Tonight between 8:15 PM and 10:00 PM, IT will be working with a vendor to complete maintenance on the infrastructure that supports VMs (Virtual Machines).  This work completes an upgrade begun on January 6. No impact on services that depend on the VMs is anticipated.  Updates will be posted online: http://it.rice.edu

The planned network maintenance for Mech lab is complete.  - Danny Eaton (x5233, dannyeaton@rice.edu)

The planned network maintenance for Mech Lab will begin at 0600.   - Danny Eaton (x5233, dannyeaton@rice.edu)

Early Friday morning, January 16, at 6:00 AM, one of the two network switches in Mech Lab (Mechanical Engineering Laboratory) will be upgraded. The network connectivity for approximately half of the building will be affected during the change over, but services will be returned to full functionality before classes begin that morning. Updates will be posted to the IT web site: http://it.rice.edu

One of the two network switches in Mech Lab (Mechanical Engineering Laboratory) will be upgraded at 6:00AM on Friday, January 16. The network connectivity for approximately half of the building will be affected during the change over, but services will be returned to full functionality before classes begin that morning. Updates will be posted to the IT web site: http://it.rice.edu

SPICE (Shared Pool of Integrated Computing Environments) will be offline for battery replacement at 10AM on Tuesday, January 13. The research groups that utilize this high performance computing (HPC) cluster were previously notified of the scheduled maintenance, and the cluster should be returned to service shortly after lunch.  Updates will be posted online: http://it.rice.edu.

The software issue with the web server has been resolved.

An older web server experienced a software issue this evening that is affecting 17 sites.  IT staff have been notified and are working the issue.

The planned maintenance on the crate and rnas storage systems has been completed.

One of the high performance computing (HPC) clusters will be offline for maintenance to replace a battery at 10AM on Tuesday, January 13. The research groups and customers who utilize SPICE (Shared Pool of Integrated Computing Environments) have already been notified. The cluster should be returned to service shortly after lunch.  Updates will be posted online: http://it.rice.edu.

Rooms.rice.edu maintenance was completed and the system was fully operational again at 5:15pm this afternoon, July 8, 2015.

The problems that resulted in intermittent connection interruptions for Banner, Esther, Rice Marketplace and related systems have been contained and the network traffic is stable.  If you notice further issues, please contact us at admsys@rice.edu.

Today at 5:00 PM, Rooms.rice.edu will be offline for approximately 10 minutes for maintenance.  This room reservations system is used for spaces in the Student Center, the Jones School and Duncan Hall.  Updates will be posted to the IT web site: http://it.rice.edu.

IT and Administrative Systems staff continue working on the issues that resulted in intermittent dropped connections for customers using Banner, Esther, Rice Marketplace and related systems. If your session is dropped, you should be able to relaunch the application with a fresh login, Updates will be posted online: http://it.rice.edu

The battery replacement scheduled for servers hosting RNAS* and Crate** storage solutions has been postponed until after business hours on Friday, January 9, 2015.  See previous IT-Alert for details.  Updates will be posted to the IT web site: http://it.rice.edu

 

*RNAS ("r-naz") is a long-term file storage solution, purchased with department funds and shared between of all members in group or department. 

**Crate is a storage solution for Rice researchers.  

The external payment processing system, Touchnet, is still experiencing issues related to a coordinated denial of service attack which began January 7 (see IT-Alert).  As their web sites (IP addresses) are flooded with irrelevant hits by an attacker, the hosting servers become overwhelmed with activity and have difficulty responding to legitimate payment processing actions.  Although the service provider is external to Rice, several departments depend on the payment processing service and IT will continue providing updates online at http://it.rice.edu.

Banner, Esther, Rice Marketplace and related systems are experiencing intermittent connection interruptions.  IT and Administrative Systems staff are troubleshooting the issues now.  Updates will be posted online: it.rice.edu.

On Thursday, January 8, between 6:00 AM and 6:00 PM, IT will complete the battery replacement maintenance for equipment supporting RNAS* and Crate.**  These systems were already failed over (migrated from one server to another) on January 7.  Customers using these storage systems during the maintenance may notice a brief connectivity interruption.  Personal and department files saved in the storage.rice.edu system will not affected by this maintenance. Updates will be posted to the IT web site: it.rice.edu.


*RNAS ("r-naz") is a long-term file storage solution, purchased with department funds and shared between of all members in group or department. 

**Crate is a storage solution for Rice researchers.  

Touchnet, an external payment processing system used by several departments at Rice University, is experiencing issues related to a cyber attack.  Touchnet's January 7, 2015 Data Center Bulletin reads: 

"The TouchNet DataCenter is experiencing a denial of service attack, meaning unknown entities are flooding certain IP addresses to overwhelm server activity. You may have experienced intermittent or spotty services. This is a DataCenter availability issue NOT a security issue.

We are diligently working with our ISPs to resolve the issue and will keep you updated via regular emails. It appears at this time traffic levels have returned to normal. We recommend you put a notice on your business office homepage to alert constituents of potential delays.

We appreciate your patience and continued support."

1/7/2015 10:51

classes.rice.edu server is up and operational again.

Users may access classes.rice.edu.  If you experience any issues with classes after this time, please log a problem with the Helpdesk,

or call the Helpdesk at 713.348.HELP

 

Thanks.

 

1/7/2015 10:03

The server known as classes.rice.edu is down after yesterday's maintenance issues.  Classes.rice.edu provides the owlnet.rice.edu services.

At this time there is NO ETA for when classes.rice.edu will be available.

Please watch IT Alerts for updates on the status of classes.rice.edu.

On Wednesday, January 7, between 6:00 AM and 6:00 PM, IT will fail-over (migrate systems from one server to another) RNAS* and Crate** in order to replace a battery.  Customers using these storage systems during the maintenance may notice a brief connectivity interruption.  Personal and department files saved in the storage.rice.edu system will not affected by this maintenance. Updates will be posted to the IT web site: it.rice.edu.

*RNAS ("r-naz") is a long-term file storage solution, purchased with department funds and shared between of all members in group or department. 

**Crate is a storage solution for Rice researchers.  

The shibboleth services for authentication to Box and other external cloud providers has been restored.

Services Restored

Most of the web and other services that were affected during maintenance today have been restored.  IT staff are still checking service and working to resolve the remaining issues.

We are currently experiencing an outage with a number of services.  The Shibboleth Authentication service is currently one of those services.  Access to Rice Box and other external vendors that use this service are being worked on .

OWL-Space, the university's learning management and collaboration system, is hosted on multiple servers.  Some of these servers have gone offline, affecting some but not all OWL-Space customers.  IT is working on the issue now.  Updates will be reported on the IT web site: it.rice.edu.

IT is experiencing a loss of access to many services due to a maintenance process that appears to have take these services offline.  We are aware of the problem and are addressing it now.

On Wednesday, January 7, between 6:00 AM and 6:00 PM, IT will fail-over (migrate systems from one server to another) RNAS* and Crate** in order to replace a battery.  Customers using these storage systems during the maintenance may notice a brief connectivity interruption.  Personal and department files saved in the storage.rice.edu system will not affected by this maintenance. Updates will be posted to the IT web site: it.rice.edu.

*RNAS ("r-naz") is a long-term file storage solution, purchased with department funds and shared between of all members in group or department.

**Crate is a storage solution for Rice researchers.  

Ensemble, the media services and hosting system supporting OWL-Space videos and webcast archives, will be unavailable Thursday, January 8, between 8:00 AM and noon. During this maintenance window, IT will upgrade the Ensemble system in preparation for the start of classes.  Updates will be posted to the IT web site: it.rice.edu

Today between 9:30 AM and 6:00 PM, IT is working with a vendor to perform maintenance on the infrastructure that supports VMs (Virtual Machines).  The vendor indicates no impact for services running on VMs during this maintenance.  Updates will be posted to the IT web site: it.rice.edu

The diagnostic software load and reboot has been completed.  The wireless network should be back to normal.  Please report any issues to your divisional representative or the IT Service/Help Desk at 713-348-HELP (x4357).  We apologize for any inconvenience and will send a message out upon completion.  Thank you.  - Danny Eaton (x5233, dannyeaton@rice.edu)

IT/Networking in conjunction with our wireless vendor will be performing a diagnostic software load on a wireless controller cluster.  A reboot of the cluster will be needed after the software is applied.  We apologize for any inconvenience and will send a message out upon completion.  Thank you.  - Danny Eaton (x5233, dannyeaton@rice.edu)

The failed network component was replaced.  All networking in Duncan Hall is restored.  - Danny Eaton (x5233, dannyeaton@rice.edu)

One of the network elements in Duncan Hall did not come back up from the FE&P maintenance correctly.  The element hosts the wireless access points in that area.  IT/Networking and Technical Services are onsite and working to restore service.  Thank You.  - Danny Eaton (x5233, dannyeaton@rice.edu)