[00:22:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.800 second response time [00:27:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.036 second response time [00:42:44] /var/www/wiki/mediawiki/extensions/DumpHTML [00:42:44] Permission denied (publickey). [00:42:44] fatal: Could not read from remote repository. [00:42:44] Please make sure you have the correct access rights [00:42:44] and the repository exists. [00:43:10] * Reedy waits for other repos to update [00:44:36] Same for /var/www/wiki/mediawiki/extensions/DataValuesInterfaces [00:44:50] Can't help you, my laptop is nowhere near ATM [00:47:12] Seems to work on a new clone [00:49:45] PROBLEM - Host mw27 is DOWN: PING CRITICAL - Packet loss = 100% [00:50:33] RECOVERY - Host mw27 is UP: PING OK - Packet loss = 0%, RTA = 26.76 ms [01:02:59] (03CR) 10TTO: [C: 04-1] "per MZM" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84102 (owner: 10Ricordisamoa) [02:05:32] !log LocalisationUpdate completed (1.22wmf16) at Sat Sep 14 02:05:31 UTC 2013 [02:05:37] Logged the message, Master [02:13:11] !log LocalisationUpdate completed (1.22wmf17) at Sat Sep 14 02:13:11 UTC 2013 [02:13:14] Logged the message, Master [02:21:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:43] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 14 02:21:42 UTC 2013 [02:21:46] Logged the message, Master [02:23:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [02:38:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:39:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 4.976 second response time [02:42:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:44:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [03:40:34] PROBLEM - MySQL Processlist on db1004 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 112 statistics [03:41:33] RECOVERY - MySQL Processlist on db1004 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 9 statistics [03:51:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:52:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [04:02:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:03:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.144 second response time [04:22:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:23:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [04:31:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.326 second response time [05:36:15] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:40:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 4.729 second response time [05:46:26] (03CR) 10Spage: [C: 031] "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/82766 (owner: 10Matthias Mullie) [05:51:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:52:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [06:01:01] PROBLEM - Memcached on holmium is CRITICAL: Connection refused [06:01:02] PROBLEM - mysqld processes on db64 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [06:01:02] PROBLEM - Parsoid on wtp1008 is CRITICAL: Connection refused [06:01:02] PROBLEM - RAID on db1031 is CRITICAL: CRITICAL: Degraded [06:01:02] PROBLEM - Disk space on cp1060 is CRITICAL: DISK CRITICAL - free space: /srv/sda3 9156 MB (2% inode=99%): /srv/sdb3 9687 MB (3% inode=99%): [06:01:02] PROBLEM - DPKG on cp1062 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [06:01:02] PROBLEM - MySQL disk space on es1003 is CRITICAL: DISK CRITICAL - free space: /a 283455 MB (2% inode=99%): [06:01:11] PROBLEM - Disk space on cp1059 is CRITICAL: DISK CRITICAL - free space: /srv/sda3 10269 MB (3% inode=99%): /srv/sdb3 11102 MB (3% inode=99%): [07:11:36] (03PS1) 10Ori.livneh: 'gmond' is not a Puppet keyword [operations/puppet] - 10https://gerrit.wikimedia.org/r/84191 [08:15:13] (03PS1) 10Ori.livneh: Add .gitreview [operations/software/gdash] - 10https://gerrit.wikimedia.org/r/84192 [08:17:01] (03CR) 10Ori.livneh: [C: 032 V: 032] Add .gitreview [operations/software/gdash] - 10https://gerrit.wikimedia.org/r/84192 (owner: 10Ori.livneh) [08:28:03] (03PS1) 10Ori.livneh: Add 'gdash' to git-deploy [operations/puppet] - 10https://gerrit.wikimedia.org/r/84193 [08:31:32] ori-l: working on gdash integration of eventlogging stuff? :O :D [08:32:27] no, just low-effort feel-good late-night git zen [08:35:01] but who knows [13:49:37] (03CR) 10MZMcBride: "The correct course of action here, if any, would be to make the CleanChanges extension opt-in instead of opt-out. This change currently co" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84102 (owner: 10Ricordisamoa) [14:24:01] (03PS1) 10Ladsgroup: Enabling Echo and Thanks in Persian Wikipedia Bug: 54068 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84203 [18:28:08] (03PS1) 10Ebrahim: PNG instead of SVG rendered for Persian Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84205 [18:58:43] (03PS2) 10Ladsgroup: Enabling Echo and Thanks in Persian Wikipedia Bug: 54068 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84203 [19:01:54] (03Abandoned) 10Ricordisamoa: Revert "Enable CleanChanges extension on Meta-Wiki" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84102 (owner: 10Ricordisamoa) [19:15:20] (03PS1) 10Ladsgroup: Adding option of 'patrolmarks' to autoconfirmed users in ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84206 [19:16:31] (03CR) 10Ladsgroup: [C: 031] "agreed on making the site faster" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84205 (owner: 10Ebrahim) [19:27:39] (03CR) 10Odder: "There is no difference whatsoever between a .svg.png and a .png file, because, guess what, they both are PNGs." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84205 (owner: 10Ebrahim) [19:32:37] (03CR) 10Ebrahim: ""There is no difference whatsoever between a .svg.png and a .png file, because, guess what, they both are PNGs." No, PNG version is lossle" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84205 (owner: 10Ebrahim) [22:58:55] anyone around to take a look at db1049 [22:58:57] ? [23:04:01] Reedy: ^ [23:18:20] paravoid: ^ [23:25:14] !log restarted db1049 slave sql thread after lock wait timeout. investigating cause [23:25:18] Logged the message, Master [23:25:29] hoo: ^ [23:25:48] springle: Thanks a lot :) [23:26:09] Hard to find someone at this time :P [23:28:51] StevenW: ^ [23:30:48] Thx [23:34:29] !log throttling back db1045 pt-table-sync jobs to avoid blocking source-slave sql threads [23:34:33] Logged the message, Master