Skip to main content

Cornell University

Facilities Inventory: Facilities and Campus Services System-Wide Outages

Date and Time Start Time End Time Down Time Error Message TDX Ticket Number Actions Taken to Prevent Recurrence

Friday, April 12, 2024

13:43 14:05 22 min

Find Facility Information web pages showing "500 Internal Server Error"

Previous facilities inventory system down.

Document Archive system down.

Role Management system for setting access permissions down.

Continued as 1305471
Wednesday, April 10, 2024 14:48 15:01 16 min

Find Facility Information web pages showing "500 Internal Server Error"

Previous facilities inventory system down.

Document Archive system down.

Continued as 1305471
Wednesday, April 10, 2024 14:16 14:20 4 min

Document Archive system down.

Outage of other systems not confirmed during brief outage.

Continued as 1305471
Tuesday, April 9, 2024 13:54 14:16 22 min

Find Facility Information web pages showing "500 Internal Server Error"

Previous facilities inventory system down.

Document Archive system down.

Continued as 1305471

Automated message sent to Admin Apps group at 14:01

Friday, April 5, 2024 14:26 14:42 16 min

Find Facility Information web pages showing "500 Internal Server Error"

The previous Facilities Inventory system did not appear to be affected.

Continued as 1305471
Wednesday, April 3rd, 2024 14:09 14:49 40 min

Find Facility Information web pages showing "500 Internal Server Error"

Previous facilities inventory system down.

Document Archive system down.

Continued as 1305471
Thursday, March 28, 2024 14:03 15:23 80 minutes

Find Facility Information web pages showing "500 Internal Server Error"

Previous facilities inventory system down.

Document Archive system down.

Continued as 1305471

Automated error message sent to Admin Apps group at 15:08.

Unknown.  Server restarted to resolve outage.
Friday, March 22, 2024 13:39 13:51 12 minutes

Find Facility Information web pages showing "500 Internal Server Error"

Facilities Inventory System: only triggered if a user is interacting with the system, 13:45

DETAIL: [Macromedia][Oracle JDBC Driver][Oracle]ORA-01034: ORACLE not available ORA-27102: out of memory Linux-x86_64 Error: 12: Cannot allocate memory Additional information: 2673 Additional information: 1 Additional information: 1677721600

FCS Document Archive administrators report an outage during this time frame as well.

Reopened TDX 1305471

Automated error sent to Admin Apps group.

Facilities Inventory Group requested notification of interested parties of outages and inquiry with FCS.
Thursday, March 21, 2024 14:18 14:33 15 minutes Find Facility Information web pages showing "500 Internal Server Error" Continued as 1305471
Thursday, March 21, 2024 14:10 14:17 7 minutes Find Facility Information web pages showing "500 Internal Server Error" Continued as 1305471 Added Pingdom alert to catch all instances and raise visibility of this chronic long-standing issue.
Friday, February 16, 2024

Before 15:08

ColdFusion: Error connecting to web service in security.getFacilityAccessList()

ORA-01034: ORACLE not available ORA-27102: out of memory Linux-x86_64 Error: 12: Cannot allocate memory Additional information: 2673 Additional information: 1 Additional information: 1677721600

Continuation of 1305471
Friday, February 16, 2024 Before 9:47 After 10:29

ColdFusion: Error connecting to web service in security.getFacilityAccessList()

ORA-01034: ORACLE not available ORA-27123: unable to attach to shared memory segment Linux-x86_64 Error: 2: No such file or directory Additional information: 2677 Additional information: 1 Additional information: 1677721600

Automated error sent to Admin Apps group. DBAs added 12G of memory to database server and performed a restart, 2/16/24 before 10:52 am.
Thursday, February 15, 2024 Before 14:54 Unknown, after 15:44

ColdFusion: Error connecting to web service in security.getFacilityAccessList()

ORA-01034: ORACLE not available ORA-27102: out of memory Linux-x86_64 Error: 12: Cannot allocate memory Additional information: 2673 Additional information: 1 Additional information: 1677721600

1305471
Friday, January 26, 2024,  Before 14:20 Unknown, after 14:37

ColdFusion:  Error connecting to web service in security.getFacilityAccessList()

ORA-01034: ORACLE not available ORA-27102: out of memory Linux-x86_64 Error: 12: Cannot allocate memory Additional information: 2673 Additional information: 1 Additional information: 1677721600

Thursday, July 27, 2023 Before 13:29

ColdFusion: Error connecting to web service in security.getOrgAdminList()

ORA-02002: error while writing to audit trail ORA-01652: unable to extend temp segment by 128 in tablespace TEMP

Database Move?
Wednesday, March 15, 2023 Before 14:43 After 18:13

ColdFusion: Error connecting to web service in security.isFIG() and other similar.

Thursday, June 16, 2022 Before 13:51 After 15:14

ColdFusion: Cannot perform web service invocation get_FI_FacilList, and others such as Error connecting to web service in security.isFIG().

Error Executing Database Query.No more data to read from socket

Wednesday, May 18, 2022 Before 14:45 After 14:50

ColdFusion: Error connecting to web service in security.isFIG()

(503)Web service request timed out waiting to execute.

Tuesday, July 14, 2020 Before 9:07 After 9:07

ColdFusion: Error connecting to web service in security.isFIG()

Tuesday, June 30, 2020 Before 14:43 After 14:47

ColdFusion: Error connecting to web service in security.getDepartmentPrivileges()

(503)Web service request timed out waiting to execute.

Friday, July 10, 2020 Before 14:33 After 14:33

ColdFusion: Error connecting to web service in security.getDepartmentPrivileges()

The request has exceeded the allowable time limit Tag: CFLOOP.

Thursday, July 9, 2020 Before 14:22 After 14:22

ColdFusion: Error connecting to web service in security.getDepartmentPrivileges()

(503)Web service request timed out waiting to execute.