[01:13:49] [[Special:Log/newusers]] create 10 * GRivera * (New user account) [03:08:11] 2013/08/07 03:07 OK wolfsbane / DISK OK - free space: / 7082 MB (23% inode=93%): [03:09:11] 2013/08/07 03:08 OK wolfsbane /tmp DISK OK - free space: / 7077 MB (23% inode=93%): [04:30:16] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=8099&oldid=8095&rcid=22131 * 185.15.59.202 * (+0) (updated page) [09:50:30] 2013/08/07 09:43 CRIT nightshade aliasd Connection refused [09:57:31] 2013/08/07 09:53 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1983 [10:05:32] 2013/08/07 09:58 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2187 [10:15:33] 2013/08/07 10:11 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1964 [10:25:33] 2013/08/07 10:24 CRIT z-dat-s3-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3625 [10:29:33] 2013/08/07 10:27 CRIT ptolemy Load avg. CRITICAL - load average: 30.77, 23.20, 13.04 [10:30:33] 2013/08/07 10:29 WARN ptolemy Load avg. WARNING - load average: 25.91, 23.35, 13.76 [10:30:34] 2013/08/07 10:29 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [10:30:34] 2013/08/07 10:28 CRIT z-dat-s7-a MySQL slave (Service Check Timed Out) [10:31:33] 2013/08/07 10:29 CRIT z-dat-s4-a MySQL slave (Service Check Timed Out) [10:33:34] 2013/08/07 10:32 CRIT ptolemy Load avg. CRITICAL - load average: 28.21, 25.05, 16.11 [10:33:34] 2013/08/07 10:32 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 63938 MB (10% inode=99%): [10:33:34] 2013/08/07 10:32 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2767 [10:34:35] 2013/08/07 10:33 WARN ptolemy Load avg. WARNING - load average: 25.09, 24.79, 16.59 [10:42:33] 2013/08/07 10:41 OK ptolemy Load avg. OK - load average: 3.89, 12.42, 14.38 [10:42:33] 2013/08/07 10:41 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [10:43:34] 2013/08/07 10:42 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 63933 MB (10% inode=99%): [10:48:35] 2013/08/07 10:46 CRIT z-dat-s4-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3638 [10:57:35] 2013/08/07 10:56 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [10:58:35] 2013/08/07 10:57 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 63925 MB (10% inode=99%): [11:05:36] 2013/08/07 11:04 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3520 [11:17:38] 2013/08/07 11:16 OK z-dat-s4-a MySQL slave Uptime: 1039943 Threads: 1 Questions: 40158754 Slow queries: 14 Opens: 128 Flush tables: 1 Open tables: 95 Queries per second avg: 38.616 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1302 [11:33:39] 2013/08/07 11:26 CRIT z-dat-s7-a MySQL Too many connections [12:08:39] 2013/08/07 12:08 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3567 [12:15:02] [[Special:Log/newusers]] create 10 * FlickJockey918 * (New user account) [12:15:40] 2013/08/07 12:09 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2182 [12:16:40] 2013/08/07 12:16 OK z-dat-s3-a MySQL slave Uptime: 96282 Threads: 58 Questions: 124382164 Slow queries: 18885 Opens: 466246 Flush tables: 1 Open tables: 16384 Queries per second avg: 1291.852 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1739 [12:39:42] 2013/08/07 12:39 CRIT z-dat-s2-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3613 [12:43:43] 2013/08/07 12:37 WARN z-dat-s2-b Load avg. WARNING - load average: 18.50, 16.45, 14.86 [12:51:43] 2013/08/07 12:51 OK z-dat-s2-b Load avg. OK - load average: 13.14, 14.70, 14.69 [12:51:43] 2013/08/07 12:50 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [12:52:43] 2013/08/07 12:51 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 63849 MB (10% inode=99%): [12:58:44] 2013/08/07 12:54 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 16.002 second response time [13:01:44] 2013/08/07 13:00 OK web.amaranth fisheye.toolserver.org HTTP OK: HTTP/1.1 200 OK - 273 bytes in 13.681 second response time [13:07:44] 2013/08/07 13:03 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 16.707 second response time [13:12:45] 2013/08/07 13:11 CRIT web.amaranth fisheye.toolserver.org HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.834 second response time [13:13:46] 2013/08/07 13:12 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 19.214 second response time [13:23:46] 2013/08/07 13:22 CRIT web.amaranth fisheye.toolserver.org CRITICAL - Socket timeout after 21 seconds [13:24:46] 2013/08/07 13:23 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 19.369 second response time [13:32:46] 2013/08/07 13:31 CRIT web.amaranth fisheye.toolserver.org HTTP CRITICAL: HTTP/1.1 200 OK - 274 bytes in 20.206 second response time [13:33:46] 2013/08/07 13:32 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 19.557 second response time [13:34:46] 2013/08/07 13:33 CRIT web.amaranth fisheye.toolserver.org CRITICAL - Socket timeout after 21 seconds [13:44:47] 2013/08/07 13:41 CRIT wolfsbane Load avg. CRITICAL - load average: 37.00, 29.28, 17.09 [13:45:47] 2013/08/07 13:41 CRIT ortelius Load avg. CRITICAL - load average: 79.68, 59.03, 29.60 [13:50:48] 2013/08/07 13:49 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 18.270 second response time [13:55:48] 2013/08/07 13:54 CRIT web.amaranth fisheye.toolserver.org HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.488 second response time [14:00:48] 2013/08/07 13:56 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.478 second response time [14:01:49] 2013/08/07 14:01 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.858 second response time [14:02:49] 2013/08/07 14:01 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 19.010 second response time [14:03:48] 2013/08/07 14:02 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.448 second response time [14:05:50] 2013/08/07 14:05 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.022 second response time [14:05:50] 2013/08/07 14:04 CRIT web.amaranth fisheye.toolserver.org CRITICAL - Socket timeout after 21 seconds [14:10:50] 2013/08/07 14:09 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 19.832 second response time [14:12:50] 2013/08/07 14:11 CRIT web.amaranth fisheye.toolserver.org CRITICAL - Socket timeout after 21 seconds [14:23:52] 2013/08/07 14:22 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 19.440 second response time [14:24:52] 2013/08/07 14:23 CRIT web.amaranth fisheye.toolserver.org HTTP CRITICAL: HTTP/1.1 200 OK - 272 bytes in 20.973 second response time [14:32:53] 2013/08/07 14:32 OK z-dat-s7-a MySQL Uptime: 104105 Threads: 501 Questions: 23298514 Slow queries: 1884 Opens: 177834 Flush tables: 1 Open tables: 3489 Queries per second avg: 223.798 [14:34:52] 2013/08/07 14:33 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 18.044 second response time [14:35:52] 2013/08/07 14:34 CRIT web.amaranth fisheye.toolserver.org CRITICAL - Socket timeout after 21 seconds [14:39:52] 2013/08/07 14:33 CRIT z-dat-s7-a MySQL Too many connections [15:02:56] 2013/08/07 15:01 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 16.173 second response time [15:03:56] 2013/08/07 15:02 OK web.amaranth fisheye.toolserver.org HTTP OK: HTTP/1.1 200 OK - 273 bytes in 10.275 second response time [15:10:56] 2013/08/07 15:04 WARN wolfsbane /tmp DISK WARNING - free space: / 6252 MB (20% inode=93%): [15:11:55] [[Special:Log/newusers]] create 10 * Francisca677 * (New user account) [15:11:56] 2013/08/07 15:04 WARN wolfsbane / DISK WARNING - free space: / 6262 MB (20% inode=93%): [15:16:56] 2013/08/07 15:16 OK wolfsbane /tmp DISK OK - free space: / 6292 MB (21% inode=93%): [15:21:57] 2013/08/07 15:21 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3483 [15:24:57] 2013/08/07 15:18 WARN wolfsbane /tmp DISK WARNING - free space: / 6245 MB (20% inode=93%): [15:28:58] 2013/08/07 15:28 OK z-dat-s2-b MySQL slave Uptime: 245864 Threads: 33 Questions: 190337781 Slow queries: 4339 Opens: 2287991 Flush tables: 1 Open tables: 256 Queries per second avg: 774.158 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1700 [15:46:59] 2013/08/07 15:42 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 8.385 second response time [15:47:59] 2013/08/07 15:47 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [16:14:03] 2013/08/07 16:13 WARN wolfsbane Load avg. WARNING - load average: 7.03, 7.12, 19.49 [16:17:02] 2013/08/07 16:16 WARN ortelius Load avg. WARNING - load average: 2.13, 2.91, 19.33 [16:19:02] 2013/08/07 16:18 OK wolfsbane Load avg. OK - load average: 2.20, 4.23, 14.75 [16:23:03] 2013/08/07 16:22 OK ortelius Load avg. OK - load average: 3.50, 3.75, 14.33 [17:18:04] 2013/08/07 17:10 WARN willow / DISK WARNING - free space: / 16952 MB (16% inode=98%): [17:18:04] 2013/08/07 17:11 WARN willow /tmp DISK WARNING - free space: / 16255 MB (15% inode=98%): [17:24:04] 2013/08/07 17:23 CRIT willow /tmp DISK CRITICAL - free space: / 11057 MB (10% inode=97%): [17:25:05] 2013/08/07 17:23 CRIT willow / DISK CRITICAL - free space: / 10846 MB (10% inode=97%): [18:02:06] 2013/08/07 18:01 OK willow / DISK OK - free space: / 29160 MB (27% inode=99%): [18:02:06] 2013/08/07 18:01 OK willow /tmp DISK OK - free space: / 29315 MB (28% inode=99%): [18:22:08] 2013/08/07 18:15 WARN willow / DISK WARNING - free space: / 17629 MB (16% inode=98%): [18:22:08] 2013/08/07 18:15 WARN willow /tmp DISK WARNING - free space: / 17805 MB (17% inode=98%): [18:33:07] 2013/08/07 18:32 CRIT willow / DISK CRITICAL - free space: / 11049 MB (10% inode=97%): [18:33:07] 2013/08/07 18:32 CRIT willow /tmp DISK CRITICAL - free space: / 11200 MB (10% inode=98%): [19:17:09] 2013/08/07 19:16 WARN rosemary /mnt user-store DISK WARNING - free space: /mnt 320439 MB (6% inode=58%): [19:32:09] 2013/08/07 19:31 WARN willow / DISK WARNING - free space: / 11516 MB (11% inode=98%): [19:33:09] 2013/08/07 19:32 WARN willow /tmp DISK WARNING - free space: / 12312 MB (11% inode=98%): [19:39:39] [[Special:Log/newusers]] create 10 * Delana466 * (New user account) [19:40:10] 2013/08/07 19:39 OK willow / DISK OK - free space: / 22015 MB (21% inode=98%): [19:41:10] 2013/08/07 19:40 OK willow /tmp DISK OK - free space: / 22631 MB (21% inode=99%): [19:59:45] [[Special:Log/newusers]] create 10 * Maye566 * (New user account) [23:47:20] At least willow is down