[00:17:22] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [00:28:19] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [00:28:19] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [00:38:22] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 111 MB (1% inode=61%): [00:42:16] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [00:42:16] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [00:46:46] RECOVERY - Disk space on srv219 is OK: DISK OK [01:26:56] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [03:24:59] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [04:24:12] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [04:55:24] PROBLEM - Disk space on search1021 is CRITICAL: DISK CRITICAL - free space: /a 3220 MB (2% inode=99%): [04:57:30] PROBLEM - Puppet freshness on search1023 is CRITICAL: Puppet has not run in the last 10 hours [05:00:30] PROBLEM - Puppet freshness on search1013 is CRITICAL: Puppet has not run in the last 10 hours [05:02:27] PROBLEM - Puppet freshness on search1011 is CRITICAL: Puppet has not run in the last 10 hours [05:05:27] PROBLEM - Puppet freshness on search1010 is CRITICAL: Puppet has not run in the last 10 hours [05:06:30] PROBLEM - Puppet freshness on search1016 is CRITICAL: Puppet has not run in the last 10 hours [05:08:36] PROBLEM - Puppet freshness on search1002 is CRITICAL: Puppet has not run in the last 10 hours [05:10:52] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 0 seconds [05:11:00] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 0 seconds [05:11:27] PROBLEM - Puppet freshness on search1004 is CRITICAL: Puppet has not run in the last 10 hours [05:12:30] PROBLEM - Puppet freshness on search1014 is CRITICAL: Puppet has not run in the last 10 hours [05:12:30] PROBLEM - Puppet freshness on search1008 is CRITICAL: Puppet has not run in the last 10 hours [05:13:33] PROBLEM - Puppet freshness on search1007 is CRITICAL: Puppet has not run in the last 10 hours [05:14:27] PROBLEM - Puppet freshness on search1005 is CRITICAL: Puppet has not run in the last 10 hours [05:14:27] PROBLEM - Puppet freshness on search1009 is CRITICAL: Puppet has not run in the last 10 hours [05:14:27] PROBLEM - Puppet freshness on search1003 is CRITICAL: Puppet has not run in the last 10 hours [05:14:27] PROBLEM - Puppet freshness on search1017 is CRITICAL: Puppet has not run in the last 10 hours [05:15:30] PROBLEM - Puppet freshness on search1024 is CRITICAL: Puppet has not run in the last 10 hours [05:16:24] PROBLEM - Puppet freshness on search1012 is CRITICAL: Puppet has not run in the last 10 hours [05:19:14] PROBLEM - Puppet freshness on search1006 is CRITICAL: Puppet has not run in the last 10 hours [05:19:14] PROBLEM - Puppet freshness on search1020 is CRITICAL: Puppet has not run in the last 10 hours [05:19:14] PROBLEM - Puppet freshness on search1019 is CRITICAL: Puppet has not run in the last 10 hours [05:20:26] PROBLEM - Puppet freshness on search1018 is CRITICAL: Puppet has not run in the last 10 hours [05:20:26] PROBLEM - Puppet freshness on search1021 is CRITICAL: Puppet has not run in the last 10 hours [05:22:32] PROBLEM - Puppet freshness on search1015 is CRITICAL: Puppet has not run in the last 10 hours [05:22:32] PROBLEM - Puppet freshness on search1022 is CRITICAL: Puppet has not run in the last 10 hours [06:18:00] PROBLEM - Disk space on search1021 is CRITICAL: DISK CRITICAL - free space: /a 3212 MB (2% inode=99%): [06:45:27] PROBLEM - Packetloss_Average on emery is CRITICAL: CRITICAL: packet_loss_average is 8.07419637224 (gt 8.0) [06:50:24] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): [06:51:45] RECOVERY - Packetloss_Average on emery is OK: OK: packet_loss_average is 0.0 [06:54:36] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=57%): [06:55:03] PROBLEM - Disk space on srv224 is CRITICAL: DISK CRITICAL - free space: / 98 MB (1% inode=61%): [06:56:42] RECOVERY - Disk space on srv220 is OK: DISK OK [07:05:42] RECOVERY - Disk space on srv224 is OK: DISK OK [07:07:12] RECOVERY - Disk space on srv221 is OK: DISK OK [08:00:10] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: Puppet has not run in the last 10 hours [09:35:46] PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 277 MB (3% inode=61%): /var/lib/ureadahead/debugfs 277 MB (3% inode=61%): [09:35:46] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 105 MB (1% inode=61%): [09:35:55] PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 270 MB (3% inode=61%): [09:36:04] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 154 MB (2% inode=57%): [09:45:58] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): [09:46:25] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): [09:46:25] PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 277 MB (3% inode=61%): /var/lib/ureadahead/debugfs 277 MB (3% inode=61%): [09:50:46] PROBLEM - Disk space on srv224 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): [09:52:34] PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): [09:52:34] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 100 MB (1% inode=61%): [09:52:43] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 200 MB (2% inode=57%): [09:54:22] RECOVERY - Disk space on srv219 is OK: DISK OK [09:54:40] RECOVERY - Disk space on srv220 is OK: DISK OK [09:56:46] RECOVERY - Disk space on srv222 is OK: DISK OK [09:56:55] RECOVERY - Disk space on srv223 is OK: DISK OK [09:56:55] RECOVERY - Disk space on srv224 is OK: DISK OK [09:58:14] !log nuked /usr/shared/doc on a couple srv's, hey at least 700MB or something, and yes we really should reinstall with a decent partitioning scheme as M ark said [09:58:17] Logged the message, Master [10:05:10] RECOVERY - Disk space on srv221 is OK: DISK OK [10:18:40] PROBLEM - Apache HTTP on srv225 is CRITICAL: Connection refused [10:19:07] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [10:22:18] !log srv222,225 were also upgraded but stopping there for now in favor of reinstalls [10:22:20] Logged the message, Master [10:29:51] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [10:29:51] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [10:40:39] RECOVERY - Apache HTTP on srv225 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.031 second response time [10:43:49] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [10:44:08] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [11:28:02] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [13:26:54] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [14:25:26] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [14:58:33] PROBLEM - Puppet freshness on search1023 is CRITICAL: Puppet has not run in the last 10 hours [15:01:33] PROBLEM - Puppet freshness on search1013 is CRITICAL: Puppet has not run in the last 10 hours [15:03:39] PROBLEM - Puppet freshness on search1011 is CRITICAL: Puppet has not run in the last 10 hours [15:06:39] PROBLEM - Puppet freshness on search1010 is CRITICAL: Puppet has not run in the last 10 hours [15:07:42] PROBLEM - Puppet freshness on search1016 is CRITICAL: Puppet has not run in the last 10 hours [15:10:33] PROBLEM - Puppet freshness on search1002 is CRITICAL: Puppet has not run in the last 10 hours [15:12:39] PROBLEM - Puppet freshness on search1004 is CRITICAL: Puppet has not run in the last 10 hours [15:13:33] PROBLEM - Puppet freshness on search1014 is CRITICAL: Puppet has not run in the last 10 hours [15:13:33] PROBLEM - Puppet freshness on search1008 is CRITICAL: Puppet has not run in the last 10 hours [15:15:39] PROBLEM - Puppet freshness on search1005 is CRITICAL: Puppet has not run in the last 10 hours [15:15:39] PROBLEM - Puppet freshness on search1009 is CRITICAL: Puppet has not run in the last 10 hours [15:15:39] PROBLEM - Puppet freshness on search1007 is CRITICAL: Puppet has not run in the last 10 hours [15:15:39] PROBLEM - Puppet freshness on search1003 is CRITICAL: Puppet has not run in the last 10 hours [15:15:39] PROBLEM - Puppet freshness on search1017 is CRITICAL: Puppet has not run in the last 10 hours [15:16:33] PROBLEM - Puppet freshness on search1024 is CRITICAL: Puppet has not run in the last 10 hours [15:17:36] PROBLEM - Puppet freshness on search1012 is CRITICAL: Puppet has not run in the last 10 hours [15:20:36] PROBLEM - Puppet freshness on search1006 is CRITICAL: Puppet has not run in the last 10 hours [15:20:36] PROBLEM - Puppet freshness on search1020 is CRITICAL: Puppet has not run in the last 10 hours [15:20:36] PROBLEM - Puppet freshness on search1019 is CRITICAL: Puppet has not run in the last 10 hours [15:21:39] PROBLEM - Puppet freshness on search1018 is CRITICAL: Puppet has not run in the last 10 hours [15:21:39] PROBLEM - Puppet freshness on search1021 is CRITICAL: Puppet has not run in the last 10 hours [15:23:45] PROBLEM - Puppet freshness on search1022 is CRITICAL: Puppet has not run in the last 10 hours [15:23:45] PROBLEM - Puppet freshness on search1015 is CRITICAL: Puppet has not run in the last 10 hours [16:28:18] * schoolcraftT slaps Krinkle and starts getting carried away [16:29:56] * schoolcraftT likes slapping people and randomly picks Laaknor to slap. [17:11:41] PROBLEM - MySQL Replication Heartbeat on db24 is CRITICAL: CRIT replication delay 186 seconds [17:12:17] PROBLEM - MySQL Slave Delay on db24 is CRITICAL: CRIT replication delay 193 seconds [17:12:35] PROBLEM - MySQL Replication Heartbeat on db1018 is CRITICAL: CRIT replication delay 181 seconds [17:13:02] PROBLEM - MySQL Slave Delay on db1018 is CRITICAL: CRIT replication delay 181 seconds [17:16:29] RECOVERY - MySQL Slave Delay on db24 is OK: OK replication delay 3 seconds [17:16:47] RECOVERY - MySQL Replication Heartbeat on db1018 is OK: OK replication delay 0 seconds [17:17:14] RECOVERY - MySQL Slave Delay on db1018 is OK: OK replication delay 0 seconds [17:17:59] RECOVERY - MySQL Replication Heartbeat on db24 is OK: OK replication delay 0 seconds [17:23:36] so, somone is DDOSing Special:Search on en.wp [17:24:22] running lots of queries with limit=250 to 1000 [17:29:43] and also running lots of queries via API of single words like: "and", "of", etc [17:29:50] in parallel [18:01:29] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: Puppet has not run in the last 10 hours [18:36:40] rainman-sr, any identifiable User-Agent? [18:36:42] I'd block him [18:37:43] don't have access to user agent field. seems he/she stopped now [20:20:16] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [20:31:25] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [20:31:25] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [20:45:31] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [20:45:31] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [21:28:34] PROBLEM - MySQL Slave Delay on db16 is CRITICAL: CRIT replication delay 181 seconds [21:29:10] PROBLEM - MySQL Replication Heartbeat on db16 is CRITICAL: CRIT replication delay 188 seconds [21:29:28] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [22:59:47] RECOVERY - MySQL Replication Heartbeat on db16 is OK: OK replication delay 6 seconds [23:00:14] RECOVERY - MySQL Slave Delay on db16 is OK: OK replication delay 7 seconds [23:28:15] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours