[00:22:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [00:46:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [00:51:41] (03CR) 10TTO: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76277 (owner: 10TTO) [01:22:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:23:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [01:32:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [01:51:54] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [01:51:54] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:51:54] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [01:51:54] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [01:51:54] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [01:51:55] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [01:52:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [02:02:01] !log LocalisationUpdate completed (1.22wmf11) at Sun Jul 28 02:02:01 UTC 2013 [02:02:12] Logged the message, Master [02:02:52] !log LocalisationUpdate completed (1.22wmf12) at Sun Jul 28 02:02:52 UTC 2013 [02:03:02] Logged the message, Master [02:07:37] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 28 02:07:37 UTC 2013 [02:07:48] Logged the message, Master [02:22:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [02:52:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:54:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [03:01:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:02:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [03:16:03] PROBLEM - Host mw16 is DOWN: PING CRITICAL - Packet loss = 100% [03:16:23] RECOVERY - Host mw16 is UP: PING OK - Packet loss = 0%, RTA = 26.55 ms [03:22:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [03:36:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [03:52:32] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [03:55:41] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:58:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:02:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [04:22:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [04:32:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:33:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [04:53:55] Is wikitech.wikimedia.org part of dumps.wikimedia.org? [05:00:26] PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours [05:22:03] I filed https://bugzilla.wikimedia.org/show_bug.cgi?id=52170 [05:22:03] The grrrit-wm quit/join was me, BTW. [05:22:50] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:24:50] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [05:52:35] PROBLEM - SSH on pdf2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:57:25] RECOVERY - SSH on pdf2 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [06:09:47] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [06:47:11] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [08:22:48] PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours [08:26:07] PROBLEM - SSH on pdf2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:57] RECOVERY - SSH on pdf2 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [08:52:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [09:13:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [09:22:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [09:44:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [09:57:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:58:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [10:12:32] (03CR) 10Liangent: [C: 031] (bug 52145) Set wmgBabelCategoryNames for wikidatawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76309 (owner: 10Odder) [10:14:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [10:58:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [11:06:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:07:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [11:20:26] (03PS1) 10TTO: Clean up headers in CommonSettings and InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 [11:52:43] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [11:52:43] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [11:52:43] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [11:52:43] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [11:52:43] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [11:52:44] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [12:22:27] PROBLEM - Disk space on mw1160 is CRITICAL: DISK CRITICAL - free space: /tmp 358 MB (2% inode=99%): [12:23:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.161 second response time [13:32:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:33:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [13:36:34] PROBLEM - Disk space on mw1160 is CRITICAL: DISK CRITICAL - free space: /tmp 686 MB (3% inode=99%): [13:41:34] PROBLEM - Disk space on mw1160 is CRITICAL: DISK CRITICAL - free space: /tmp 708 MB (3% inode=99%): [13:45:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [13:47:04] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [13:47:24] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.55 ms [13:59:00] (03PS2) 10Alex Monk: Clean up headers in CommonSettings and InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 (owner: 10TTO) [15:01:05] PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours [15:43:39] PROBLEM - SSH on pdf1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:39] RECOVERY - SSH on pdf1 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [15:58:41] (03PS1) 10Reedy: Update wgCodeReviewRepoStatsCacheTime to 1 day [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76348 [16:10:05] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [16:48:02] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [17:28:32] PROBLEM - Disk space on mw1160 is CRITICAL: DISK CRITICAL - free space: /tmp 553 MB (3% inode=99%): [17:31:31] PROBLEM - Disk space on mw1160 is CRITICAL: DISK CRITICAL - free space: /tmp 624 MB (3% inode=99%): [18:20:33] PROBLEM - Disk space on mw1160 is CRITICAL: DISK CRITICAL - free space: /tmp 667 MB (3% inode=99%): [18:23:43] PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours [18:26:33] PROBLEM - Disk space on mw1160 is CRITICAL: DISK CRITICAL - free space: /tmp 712 MB (3% inode=99%): [18:48:30] PROBLEM - Disk space on mw1160 is CRITICAL: DISK CRITICAL - free space: /tmp 406 MB (2% inode=99%): [19:16:45] (03CR) 10MZMcBride: "(5 comments)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 (owner: 10TTO) [19:46:23] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [19:49:32] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:49:34] PROBLEM - Disk space on mw1157 is CRITICAL: DISK CRITICAL - free space: /tmp 712 MB (3% inode=99%): [21:03:34] PROBLEM - Disk space on mw1157 is CRITICAL: DISK CRITICAL - free space: /tmp 355 MB (1% inode=99%): [21:32:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:33:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [21:53:40] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [21:53:40] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [21:53:40] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [21:53:40] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [21:53:40] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [21:53:41] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [22:05:44] Ryan_Lane: Not sure if ops is on it or whether it gracefully degrades nowadays, but mw1157 is running out of disk space quite fast. [22:05:56] Down from 712 MB (3%) to 300MB an hour ago. [22:06:27] icinga's been complaining about it [22:14:24] Krinkle, VIPS appears to be shitting some weird crap into /tmp o_0 [22:14:51] -rw-r--r-- 1 apache apache 419430400 Jul 25 20:36 vips-1-00OR0W.v [22:14:51] -rw-r--r-- 1 apache apache 326987776 Jul 26 10:58 vips-1-11T40W.v [22:19:32] MaxSem: 419430400 is exactly 400MiB [22:19:56] not all files have this size [22:20:02] but they're huge [22:20:13] true, but it is _exact_ 400MiB [22:20:30] 400*1024*0124 == 419430400 [22:20:34] 400*1024*1024 == 419430400 [22:21:01] what's VIPS? [22:21:29] [[mw:Extension:VipsScaler]] [22:22:06] hmm, how do I find an image if I know its SHA1 but not the wiki?:P [22:22:14] not en: or commons: [22:22:42] MaxSem: you mean the File:... part? [22:22:47] or the actual image? [22:22:53] yeah, name and wiki:P [22:23:06] meh, foreachwiki sql.php [22:23:22] heh [22:23:25] finesse? never heard [22:23:33] what SHA1 are you looking for? [22:24:37] 3a4dedd6ea495585aa494d6ca91d4c1f70848cdb [22:27:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:28:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [22:29:06] MaxSem: I think you need to loop [22:31:55] (03CR) 10Ladsgroup: [C: 031] "IIRC We did this setting via JS and nobody objected in the wiki so I think It's ok to be done by initialize settings" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76281 (owner: 10Ebrahim) [22:35:32] RECOVERY - Disk space on mw1157 is OK: DISK OK [22:35:53] !log mw1157 was running out of /tmp space, nuked all *.v older than 1 day. VIPS bug? [22:36:04] Logged the message, Master [22:56:10] (03CR) 10Se4598: [C: 04-1] "dewiki's entry should be left untouched in this change for now according to the comments in bug 49998 and bug 52202." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76199 (owner: 10Jforrester) [23:13:43] created https://bugzilla.wikimedia.org/52203 [23:42:53] https://de.wikipedia.org/wiki/Wikipedia:Umfragen/VisualEditor_Opt-in [23:42:57] wut. [23:46:28] twkozlowski: ? [23:48:14] Krenair: 442 users voted to move VE back to opt-in on de.wp [23:48:24] Yes. [23:48:29] I can only say: wut, o_0, whoa [23:48:46] that's one fucking amazing turnout [23:51:22] Oh, well I agree with you on the numbers. [23:51:23] other than James_F, who else works on VE [23:51:31] I have some much more polite comments than dewiki does [23:51:38] I'm not surprised at all by the result. [23:51:52] Prodego: they lurk in #mediawiki-visualeditor [23:51:58] awesome thanks [23:51:58] Prodego, uhh... I don't have a list but there's a lot in #mediawiki-visualeditor [23:52:12] * Krenair slightly too slow -.- [23:52:42] you put too many words in [23:55:12] Prodego: luckily I'd need Google Translate to understand those impolite comments, I guess same applies for the developers [23:55:23] s/for/to [23:58:39] Actually Gabriel Wicke (Parsoid developer) is from Germany, and so is Jan Eissfeldt [23:59:30] Oh right, I forgot Gabriel. And Jan is a what? [23:59:34] according to http://lists.wikimedia.org/pipermail/wikimedia-l/2013-July/127213.html [23:59:54] * Jan Eissfeldt (remote from Germany) [23:59:57] implies they are living in germany