Blog from August, 2013

The failed vmstorage service has been restored using an alternate cluster node.

The cause of the failure is still being investigated.

System administrators are restarting affected vmclient hosts.

One of the VMstorage servers vmstorage4 has gone offline. The problem is being investigated.

Users in the Graduate Apartments may have experienced a brief network interruption. Network management is investigating. All services (wired and wireless) are online.

Please contact the after hours operation at ext4989 if you continue to experience network issues.

OnBase, the university's document scanning and workflow system, has been returned to full service.

OnBase, the university's document scanning and workflow system has a service disruption for some customers. A few departments are currently unable to upload documents to the OnBase database. IT is working on the issue and updates will be posted to the IT web site.

IT researched reports of some undelivered email messages and discovered that one blacklist service had identified a Rice email server as a spam sender.

Rice has been cleared from that blacklist.

Further updates will be posted to the IT website: it.rice.edu

The RICELink login issues were resolved this morning. Students are again able to log into the Center for Career Development's job and internship database.

IT has had reports of login issues for RICELink, the Center for Career Development's database of jobs and internships. We are researching the problem and will post updates to the IT website: it.rice.edu

IT Networking is currently investigating a possible hardware issue in one of the routers in the PDC Pod A. It does not appear to be service affecting. An update will be provided as soon as possible.

The database for OnBase has been returned to service. IT will continue to research the cause of the previous issue.

The database supporting OnBase, Rice's document scanning and workflow system, is offline. IT is investigating the issue and will post updates to our website: it.rice.edu

IT is investigating the cause of a network switch reboot which momentarily interrupted connections to email and storage. All systems are functional at this time.

Updates will be posted to the IT website: it.rice.edu

Currently, Rice.edu email service is fully functional and no longer blocked by email service providers as a result of blacklisting.

History: on Monday, August 12, two compromised Rice email accounts were used to send a heavy stream of spam internally as well as to external addresses, resulting in Rice.edu mail being blacklisted and then blocked by multiple email service providers. The majority of blacklisting issues were resolved within 24 hours, but several email providers use a 7-day waiting period for blacklisted email servers.

The 7-day period expired on Monday afternoon, August 19. IT has received no further notices that Rice.edu mail is being returned due to blacklisting.

IT Networking has resolved a software problem on one of the routers in George R. Brown Hall. Connectivity to Rice resources should have returned to normal. Please contact the Help Desk at x4357 if you need further assistance.

A limited number of outbound messages are still affected by Rice's August 12 blacklisting.

Blacklisting means a sender is identified as distributing significant amounts of spam, and email providers block all messages from blacklisted email server.

No internal Rice to Rice mail is affected and most of the major email service providers no longer include rice.edu on their blacklist. However, some email providers block messages for a 7-day period after a sender has been placed on a blacklist.

Rice will roll off the remaining blacklists on Monday afternoon. If you received a rejection notice for a message you tried to send to an external, non-rice.edu address this week, please try resending the message now. If you still receive a rejection notice, resend the message Monday afternoon.

For further assistance, contact the Help Desk (helpdesk.rice.edu or helpdesk@ or 713.348.4357).

IT is researching reports of some email delivery rejections from external mail servers due to blacklists. Rice to Rice email delivery is NOT affected.

If you receive a notice that your message to an external address with this subject

Returned Mail: See Transcript for Details

please forward the message to helpdesk@rice.edu with headers in place. If you do not know how to show headers, call 713.348.4357 for assistance before forwarding the message.

IT Networking has completed maintenance on the VPN service. Please contact the Help Desk (x4357) if you have VPN connectivity problems.

IT Networking will be performing maintenance on the VPN service. Connectivity to this service may be intermittent for approximately 30 minutes.

On Thursday, August 15 at 5PM, the student network printing server, owlprinting.rice.edu, will be offline for 15 minutes for an upgrade. During this 15-minute upgrade, no student network printers will accept new print jobs.

At 5:15PM, the faculty and staff printing server, printing.rice.edu, will receive the same upgrade. During the 15-minute upgrade window, no network faculty or staff printers will accept new jobs.

updates will be posted to the IT website: it.rice.edu

On Thursday, August 15, Rice University vendors that require access to systems on our network will use a new address in their VPN connection.

Campus sponsors for vendors using VPN were notified on Monday afternoon and asked to forward the new address to their vendors. To obtain the new address, vendors should contact their Rice sponsor. For assistance modifying the previous address, contact the IT Help Desk: helpdesk@rice.edu or 713.348.4357.

The scheduled storage maintenance for 6:00 AM 8/14 has been completed.

The maintenance on the storage system will begin at 6:00 AM 8/14 as scheduled.

On Monday, a Rice University email account was compromised and used to distribute a heavy stream of spam and phishing messages, both within Rice and to external addresses.

The flood of junk mail interrupted regular email traffic within Rice and led to blacklisting by four external mail servers, who have temporarily blocked all messages from Rice.edu addresses. The IT email administrators are working to get rice.edu off the blacklists.

Please remember to use strong passwords and to NEVER reveal your password to anyone. No one at Rice will ask for your password, and you never have to verify your account.

Email performance updates will be posted to the IT web site: it.rice.edu

The problem with slow Email sending has been identified and corrected. Please report any further issues to the Helpdesk (xHELP).

Regards,
Kenneth Marshall, PhD

Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

IT is investigating a problem with sending Email. Please report any problems to the Helpdesk (xHELP).

Regards,
Kenneth Marshall, PhD

Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

IT received notices that sent messages from Rice email accounts are inconsistent in their delivery. We are researching the issue and will post updates to the Rice IT web site: it.rice.edu.

As of 3:50 PM today, most networking has been restored. If you are still experiencing a network outage, please contact Operations at 713-348-4989. - Danny Eaton (x5233, dannyeaton@rice.edu)

The voicemail service issues from earlier today were resolved around noon today.

There are several network devices still down from the power outage. We are aware of this, and are investigating these devices. Known outages are in GRB, Geology, Autry Court/Tudor Fieldhouse, Southwest Servery, Duncan Hall, Duncan College, and the Brochstein Pavilion.

IT is investigating reports of authentication problems with CAS and wireless.

Regards,
Kenneth Marshall, PhD

Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

A brief authentication problem has been identified and repaired. Please report any authentication problems to the Helpdesk (xHELP).

Regards,
Kenneth Marshall, PhD

Mgr./Middleware, Infrastructure & Development
ktm@rice.edu / 713-348-5294

Campus wireless connectivity is currently offline. IT Networking is investigating and will provide an update as soon as possible.

We are currently experiencing a service outage for the campus voicemail system. We do not yet know the cause, but at this time, voicemails cannot be left by callers or retrieved by users. Additionally, some automated call menu systems may not perform as expected. Updates will be posted here.

We have backed out of the DNS servers upgrade this morning due to issues with the upgrade.

The current DNS servers are to be upgraded to new hardware and software. DNS provides hostname resolution for Rice hosts. During the outage, DNS lookups may be delayed as the servers are upgraded.

The wireless issue was identified and resolved. Wireless connectivity has been restored throughout the campus. Please contact the Help Desk (x4357) if you have any further problems.

Wireless connectivity is offline throughout the campus. IT Networking is investigating.

DNS lookup delay

For a brief period following the DHCP upgrade this morning, one of the campus DNS servers, 10.128.92.32 (dns-c.net.rice.edu) was off-line. This may have caused DNS lookup delays for some clients. The server is back on-line.

The DHCP Servers have been upgraded.

The current DHCP server pair is to be upgraded to new hardware/software. These servers provide dynamic IP addresses to both wired and wireless devices. During the outage Rice users will not be able to obtain new IP leases for wired or wireless connections. Those with an existing connections will not loose it. Network devices with static IP addresses are not affected.

wiki.rice.edu online

Confluence on wiki.rice.edu has been upgraded.

docs.rice.edu online

Confluence on docs.rice.edu has been upgraded.

Wiki.rice.edu, a web content management system used by individuals and groups across Rice, will be offline for a critical security upgrade tonight, August 7, from 8PM until 9PM.

Updates will be posted to the IT web site.

Docs.rice.edu, a web content and documentation system used by several departments, will be offline for a critical security upgrade tonight from 7PM to 8PM.

Updates will be posted to the IT web site.

The rnas.rice.edu file systems were taken offline by a storage network link failure.

The failure was reset and the filesystems have recovered.

Report any residual problems to the helpdesk.

Issues were diagnosed and resolved for connection interruptions to OnBase, the campus system used for document scanning and automated workflows. OnBase returned to full service by 10:30 AM.

IT received reports of intermittent connection interruptions for OnBase, the university's document scanning and automated workflow system. We are researching the issue now and will post alerts to the IT web site.

The recent update to the Radius servers for wireless and VPN connections may result in a prompt to renew your Client Authentication certificate.

If you see this prompt for your Rice wireless or VPN connection, accept the certificate.

Updates will be posted to the IT web site.

The current Radius server pair is to be upgraded to new hardware/software. These servers provide authentication for wireless and VPN users. No service interruption is anticipated.

The RNAS system has been returned to service. A failed hardware component is suspected of causing the problem. File services have resumed. Should you have any problems please notify the helpdesk 713 348 4357

IT has received reports of issues with an archival storage system known as "rnas."

The issue seems to be affecting dropbox.rice.edu most. IT is researching the problem and will report updates on the web page: http://it.rice.edu

IT received a report that docs.rice.edu was unavailable; we researched and resolved the issue.

Rice NetID logins are temporarily required to view content in docs.rice.edu and wiki.rice.edu, two of the systems used to publish internal and public web content for both individuals and groups at Rice.

Both sites are built on an open source platform called Confluence, which recently announced a security vulnerability that targets anonymous (public) access to web content.

By re-routing traffic for docs.rice.edu and wiki.rice.edu through a secure Rice NetID login, the security threat can be mitigated.

However, viewers without a Rice NetID login cannot access the content on the public pages in docs.rice.edu and wiki.rice.edu

IT is researching the recommended security fix, an immediate upgrade to the next version of Confluence.

Updates will be posted to the IT web site: it.rice.edu

The scheduled storage maintenance has been completed. Please report any problems to the help desk 713 348 4357

The schedule outage of the storage systems has begun. Maintenance window is from 7:00 PM until 9:00 PM 8/2/2013. An all clear alert will be issued when services are restored.

Panopto, a system used by students to review lectures previously recorded by their instructors, was replaced with a more robust tool in May and will no longer be available as of Saturday, August 3.

Instructors who utilized Panopto in the past were contacted at the end of the spring semester and have already archived videos they wish to retain.

The Cisco NAC system upgrade has completed successfully. Please report any issues to the Rice IT Help Desk.

The Cisco NAC system, also known as Clean Access, will be upgraded at 7:00p tonight.

On Friday NIGHT, August 2, storage.rice.edu and rnas.rice.edu will be taken offline for upgrades between 7PM and 9PM.

During the upgrade, you will not be able to access the following storage-based services:

  • Your personal storage drive (also called your U: drive).
  • Personal web sites
  • CLEAR.rice.edu directories
  • Department web sites that are connected to storage.rice.edu
  • Rice's PHP web sites
  • blogs.rice.edu

Updates will be posted to the IT web site: it.rice.edu

If you notice issues with your storage connection or web sites after 9PM Friday, please contact the IT Help Desk: helpdesk@rice.edu.

Maintenance to implement a new customer interface for OWL-Space, Rice's course management and collaboration system, was successfully completed at 6:30 AM this morning.

See the new features and the easy to read and navigate design at owlspace.rice.edu.