Facilities Inventory: Facilities and Campus Services System-Wide Outages
| Date and Time | Start Time | End Time | Down Time | Error Message | TDX Ticket Number | Actions Taken to Prevent Recurrence |
|---|---|---|---|---|---|---|
|
Friday, April 12, 2024 |
13:43 | 14:05 | 22 min |
Find Facility Information web pages showing "500 Internal Server Error" Previous facilities inventory system down. Document Archive system down. Role Management system for setting access permissions down. |
Continued as 1305471 | |
| Wednesday, April 10, 2024 | 14:48 | 15:01 | 16 min |
Find Facility Information web pages showing "500 Internal Server Error" Previous facilities inventory system down. Document Archive system down. |
Continued as 1305471 | |
| Wednesday, April 10, 2024 | 14:16 | 14:20 | 4 min |
Document Archive system down. Outage of other systems not confirmed during brief outage. |
Continued as 1305471 | |
| Tuesday, April 9, 2024 | 13:54 | 14:16 | 22 min |
Find Facility Information web pages showing "500 Internal Server Error" Previous facilities inventory system down. Document Archive system down. |
Continued as 1305471 Automated message sent to Admin Apps group at 14:01 |
|
| Friday, April 5, 2024 | 14:26 | 14:42 | 16 min |
Find Facility Information web pages showing "500 Internal Server Error" The previous Facilities Inventory system did not appear to be affected. |
Continued as 1305471 | |
| Wednesday, April 3rd, 2024 | 14:09 | 14:49 | 40 min |
Find Facility Information web pages showing "500 Internal Server Error" Previous facilities inventory system down. Document Archive system down. |
Continued as 1305471 | |
| Thursday, March 28, 2024 | 14:03 | 15:23 | 80 minutes |
Find Facility Information web pages showing "500 Internal Server Error" Previous facilities inventory system down. Document Archive system down. |
Continued as 1305471 Automated error message sent to Admin Apps group at 15:08. |
Unknown. Server restarted to resolve outage. |
| Friday, March 22, 2024 | 13:39 | 13:51 | 12 minutes |
Find Facility Information web pages showing "500 Internal Server Error" Facilities Inventory System: only triggered if a user is interacting with the system, 13:45 DETAIL: [Macromedia][Oracle JDBC Driver][Oracle]ORA-01034: ORACLE not available ORA-27102: out of memory Linux-x86_64 Error: 12: Cannot allocate memory Additional information: 2673 Additional information: 1 Additional information: 1677721600 FCS Document Archive administrators report an outage during this time frame as well. |
Reopened TDX 1305471 Automated error sent to Admin Apps group. |
Facilities Inventory Group requested notification of interested parties of outages and inquiry with FCS. |
| Thursday, March 21, 2024 | 14:18 | 14:33 | 15 minutes | Find Facility Information web pages showing "500 Internal Server Error" | Continued as 1305471 | |
| Thursday, March 21, 2024 | 14:10 | 14:17 | 7 minutes | Find Facility Information web pages showing "500 Internal Server Error" | Continued as 1305471 | Added Pingdom alert to catch all instances and raise visibility of this chronic long-standing issue. |
| Friday, February 16, 2024 |
Before 15:08 |
ColdFusion: Error connecting to web service in security.getFacilityAccessList() ORA-01034: ORACLE not available ORA-27102: out of memory Linux-x86_64 Error: 12: Cannot allocate memory Additional information: 2673 Additional information: 1 Additional information: 1677721600 |
Continuation of 1305471 | |||
| Friday, February 16, 2024 | Before 9:47 | After 10:29 |
ColdFusion: Error connecting to web service in security.getFacilityAccessList() ORA-01034: ORACLE not available ORA-27123: unable to attach to shared memory segment Linux-x86_64 Error: 2: No such file or directory Additional information: 2677 Additional information: 1 Additional information: 1677721600 |
Automated error sent to Admin Apps group. | DBAs added 12G of memory to database server and performed a restart, 2/16/24 before 10:52 am. | |
| Thursday, February 15, 2024 | Before 14:54 | Unknown, after 15:44 |
ColdFusion: Error connecting to web service in security.getFacilityAccessList() ORA-01034: ORACLE not available ORA-27102: out of memory Linux-x86_64 Error: 12: Cannot allocate memory Additional information: 2673 Additional information: 1 Additional information: 1677721600 |
1305471 | ||
| Friday, January 26, 2024, | Before 14:20 | Unknown, after 14:37 |
ColdFusion: Error connecting to web service in security.getFacilityAccessList() ORA-01034: ORACLE not available ORA-27102: out of memory Linux-x86_64 Error: 12: Cannot allocate memory Additional information: 2673 Additional information: 1 Additional information: 1677721600 |
|||
| Thursday, July 27, 2023 | Before 13:29 |
ColdFusion: Error connecting to web service in security.getOrgAdminList() ORA-02002: error while writing to audit trail ORA-01652: unable to extend temp segment by 128 in tablespace TEMP |
||||
| Database Move? | ||||||
| Wednesday, March 15, 2023 | Before 14:43 | After 18:13 |
ColdFusion: Error connecting to web service in security.isFIG() and other similar. |
|||
| Thursday, June 16, 2022 | Before 13:51 | After 15:14 |
ColdFusion: Cannot perform web service invocation get_FI_FacilList, and others such as Error connecting to web service in security.isFIG(). Error Executing Database Query.No more data to read from socket |
|||
| Wednesday, May 18, 2022 | Before 14:45 | After 14:50 |
ColdFusion: Error connecting to web service in security.isFIG() (503)Web service request timed out waiting to execute. |
|||
| Tuesday, July 14, 2020 | Before 9:07 | After 9:07 |
ColdFusion: Error connecting to web service in security.isFIG() |
|||
| Tuesday, June 30, 2020 | Before 14:43 | After 14:47 |
ColdFusion: Error connecting to web service in security.getDepartmentPrivileges() (503)Web service request timed out waiting to execute. |
|||
| Friday, July 10, 2020 | Before 14:33 | After 14:33 |
ColdFusion: Error connecting to web service in security.getDepartmentPrivileges() The request has exceeded the allowable time limit Tag: CFLOOP. |
|||
| Thursday, July 9, 2020 | Before 14:22 | After 14:22 |
ColdFusion: Error connecting to web service in security.getDepartmentPrivileges() (503)Web service request timed out waiting to execute. |