[00:09:36] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: CRIT replication delay 302 seconds [00:11:36] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: CRIT replication delay 302 seconds [00:13:36] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: CRIT replication delay 302 seconds [00:15:36] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: CRIT replication delay 302 seconds [00:17:36] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: CRIT replication delay 302 seconds [01:11:20] PROBLEM - RAID on analytics1010 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:12:20] RECOVERY - RAID on analytics1010 is OK: OK: Active: 6, Working: 6, Failed: 0, Spare: 0 [01:24:10] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [01:27:20] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:02:03] !log LocalisationUpdate completed (1.22wmf19) at Sun Oct 6 02:02:03 UTC 2013 [02:02:27] Logged the message, Master [02:02:50] !log LocalisationUpdate completed (1.22wmf20) at Sun Oct 6 02:02:50 UTC 2013 [02:03:06] Logged the message, Master [02:07:25] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Oct 6 02:07:25 UTC 2013 [02:07:38] Logged the message, Master [02:09:48] PROBLEM - search indices - check lucene status page on search1016 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 53527 bytes in 0.009 second response time [02:11:08] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [02:14:18] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:17:05] Has anyone notified coren about labs database issues? [03:07:23] Betacommand: are they filed in bugzilla for a start? [03:07:40] p858snake|l: yes, and emailed about too [03:08:01] p858snake|l: Heh, we just did this in -tech. ;-) [03:08:47] https://bugzilla.wikimedia.org/show_bug.cgi?id=54934 [04:02:54] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [04:03:44] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 19.639 second response time [04:30:04] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [04:33:14] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:40:47] RECOVERY - search indices - check lucene status page on search1016 is OK: HTTP OK: HTTP/1.1 200 OK - 53551 bytes in 0.009 second response time [04:44:34] RECOVERY - MySQL Replication Heartbeat on db1046 is OK: OK replication delay 0 seconds [05:08:34] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:09:34] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [05:14:04] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [07:27:54] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [07:28:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.159 second response time [09:02:39] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:03:29] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [09:17:14] PROBLEM - search indices - check lucene status page on search22 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 133 bytes in 0.054 second response time [09:17:44] PROBLEM - search indices - check lucene status page on search23 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 245 bytes in 0.055 second response time [09:36:14] PROBLEM - MySQL Recent Restart on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:37:04] RECOVERY - MySQL Recent Restart on db1021 is OK: OK 416 seconds since restart [09:58:12] Anyone who knows what's wrong here? [09:58:17] $: TEST="1 2" [09:58:19] $: echo $TEST | xargs -d' ' -n1 -I {} sh -c 'echo {}; echo {}' [09:58:21] 1 [09:58:22] 1 [09:58:23] 2 [09:58:24] sh: Syntax error: ";" unexpected [10:27:28] siebrand: Could it be that you have {} two times in sh? [10:28:13] Christian75: If I don't, then the parameter will not be passed to that command. [10:28:44] Christian75: echo $TEST | xargs -d' ' -n1 -I {} sh -c 'echo {}; echo' would result in an empty line for the second command. [10:35:23] Im not sure I can help. Which output do you want, echo $TEST | xargs -d' ' -n1 -I {} sh -c 'echo {}'; gives output: 1 \n 2 [10:56:28] Christian75: It's just a test for something else. I could share the complete use case, but then it would take more time to explain. I just want the output 1\n1\n2\n2 [10:56:53] Here's my complete command: xargs -n1 -P4 -I % bash -c 'echo "Update %"; repoupdate %; echo "Updated %"' [12:10:39] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 5 copy to table, 410 statistics [12:11:39] RECOVERY - MySQL Processlist on db1021 is OK: OK 0 unauthenticated, 0 locked, 5 copy to table, 1 statistics [14:57:46] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 108 statistics [14:58:46] RECOVERY - MySQL Processlist on db1021 is OK: OK 0 unauthenticated, 0 locked, 1 copy to table, 28 statistics [15:14:06] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:02:32] (03PS1) 10Odder: (bug 53373) Enable FlaggedRevs for Module NS on plwiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87720 [16:58:21] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 35 copy to table, 476 statistics [17:04:10] (03CR) 10Reedy: [C: 04-1] "Needs rebasing" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85513 (owner: 10TTO) [17:04:31] (03CR) 10Reedy: [C: 04-1] "Needs rebasing" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85644 (owner: 10TTO) [17:04:41] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:05:32] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [17:06:19] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85838 (owner: 10TTO) [17:06:21] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 35 copy to table, 503 statistics [17:06:38] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85961 (owner: 10TTO) [17:07:10] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86091 (owner: 10TTO) [17:07:21] RECOVERY - MySQL Processlist on db1021 is OK: OK 0 unauthenticated, 0 locked, 2 copy to table, 3 statistics [17:07:27] (03PS2) 10Reedy: Wnable $wgUseRCPatrol on fawiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86093 (owner: 10TTO) [17:07:35] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86093 (owner: 10TTO) [17:07:58] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86414 (owner: 10TTO) [17:08:40] Reedy: merge? https://gerrit.wikimedia.org/r/#/c/87682/ :) [17:09:13] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86660 (owner: 10Dereckson) [17:10:12] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87275 (owner: 10Odder) [17:10:29] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86379 (owner: 10Wizardist) [17:10:46] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87432 (owner: 10Odder) [17:11:04] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86366 (owner: 10Rillke) [17:11:44] YuviPanda: I don't have root and I don't have any merge rights on that repo [17:11:50] aww [17:11:50] ok [17:11:52] Never have.. [17:11:56] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87720 (owner: 10Odder) [17:12:01] In other news... [17:12:07] Who has done what to mediawiki-config!? [17:12:42] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86415 (owner: 10TTO) [17:12:52] (03CR) 10Reedy: [C: 04-1] "Needs rebasing." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86418 (owner: 10TTO) [17:12:57] Or to gerrit [17:13:01] Not sure which is having the issues [17:13:06] gerrit, probably [17:13:46] * Reedy kicks grrrit-wm [17:13:46] nice spam Reedy [17:14:00] probably someone fixed whitespace somewhere [17:14:00] Rebase button says no [17:14:44] The flagged revs one is more curious [17:14:53] File hasn't been touched since July [17:15:47] reedy@ubuntu64-web-esxi:~/git/operations/mediawiki-config$ git review -d 87720 [17:15:47] Downloading refs/changes/20/87720/1 from gerrit [17:15:47] Switched to branch "review/odder/bug/53373" [17:15:47] reedy@ubuntu64-web-esxi:~/git/operations/mediawiki-config$ git rebase origin [17:15:47] First, rewinding head to replay your work on top of it... [17:15:48] Applying: (bug 53373) Enable FlaggedRevs for Module NS on plwiktionary [17:15:50] reedy@ubuntu64-web-esxi:~/git/operations/mediawiki-config$ [17:15:54] Fuck you gerrit [17:16:02] gerrit is using jgit [17:16:04] Computer says no. [17:16:05] (03PS2) 10Reedy: (bug 53373) Enable FlaggedRevs for Module NS on plwiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87720 (owner: 10Odder) [17:16:12] which probably has some crappy merge implementation or something [17:16:15] (03CR) 10Reedy: [C: 032] (bug 53373) Enable FlaggedRevs for Module NS on plwiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87720 (owner: 10Odder) [17:16:20] (and by crappy, i mean extremely crappy) [17:16:20] Is there any way to automate this locally? [17:16:33] ie rebase, commit and push if no merge conflict? [17:16:37] Reedy: use `for`? :P [17:17:00] (03Merged) 10jenkins-bot: (bug 53373) Enable FlaggedRevs for Module NS on plwiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87720 (owner: 10Odder) [17:17:08] gj Gerrit [17:17:09] git review -d $n && git rebase master && git review -f [17:17:53] Theres a couple of those that very liklely won't rebase (doing masses of whitespace changes and such) [17:18:52] does bash support ||? [17:18:56] || git rebase --abort [17:19:22] or just unconditionally do that at the end of the loop, it won't break anything [17:19:33] heh [17:19:52] I hate computers [17:20:03] Amen Reedy [17:20:03] operations is hard, let's go shoppin! [17:20:12] shopping*. bah. [17:20:50] Reedy: hell, i can do that for you if you merge a few patches for me. [17:21:04] haha [17:21:18] Sounds like effort [17:21:35] Reedy: false impression! https://gerrit.wikimedia.org/r/#/c/86319/ and dependency. trivial babies. [17:21:58] twkozlowski can do his own ;) [17:22:25] :-( [17:22:25] hmmm, do i receive notifications about actions on a changeset as a committer? [17:22:27] Your branch is behind 'origin/production' by 157 commits, and can be fast-forwarded. [17:22:30] wtf are ops upto [17:22:39] i'll have to remove myself from reviewers list anyway [17:23:57] Reedy: I'm assuming https://gerrit.wikimedia.org/r/#/c/86320/ needs rebasing as well, right? [17:24:26] Yeah, try pressing the rebase button ;) [17:24:43] I was leaving the collation ones for the moment as they need a bit more work [17:25:46] Ah well. [17:27:10] Reedy: you could always try poking qchris about this, if you iddn't already [17:27:33] Ouch! [17:27:41] hi qchris. [17:27:42] It's not his fault java is shit [17:27:53] * qchris reads the logs [17:27:54] qchris: gerrit refuses to rebase changes which should rebase clearly. :( [17:28:00] Mhmm ... [17:33:44] Yes, gerrit does it's own thing when trying to rebase. I can well imagine git itself is smarter about it ... [17:50:41] (03PS2) 10Odder: (bug 54680) Set $wgCategoryCollation for the French Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86320 [17:53:46] (03PS2) 10Odder: (bug 54229) Add autopatrolled user group on ukwikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87275 [17:54:30] (03PS1) 10Bartosz Dziewoński: Remove old Vector stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 [17:54:46] Reedy: you gonna like this one ^ [17:56:00] +0, -60 [17:57:21] not enough - [17:57:37] (03PS5) 10Rillke: Allow Commons admins self-adding translationadmin group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86366 [17:57:59] (03PS2) 10Bartosz Dziewoński: Remove old Vector stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 [17:58:16] (03CR) 10Siebrand: "I'm not that familiar with Wikimedia deployment files aside from searching in them to see what lives where, so not voting here. I made a c" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 (owner: 10Bartosz Dziewoński) [17:59:12] (03CR) 10Rillke: "And for typing one 4 lines, I now had to launch my dev VM on Sunday evening ;-)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86366 (owner: 10Rillke) [18:01:23] (03PS2) 10Odder: (bug 54922) Add an accountcreator user group on svwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87432 [18:28:35] (03CR) 10Bartosz Dziewoński: "This needs to wait until older branches go out of production, of course. (A week from now, it seems.)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 (owner: 10Bartosz Dziewoński) [18:29:15] (03PS2) 10Hashar: zuul: statsd sent to tungsten.eqiad.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/86744 [18:31:38] (03CR) 10Hashar: "Ori, Faidon: now that tungsten is being a statsd server I guess we can have Zuul to send its metrics to it, can't we ?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86744 (owner: 10Hashar) [18:32:30] (03CR) 10Ori.livneh: [C: 031] "Yep." [operations/puppet] - 10https://gerrit.wikimedia.org/r/86744 (owner: 10Hashar) [19:19:07] (03CR) 10Hashar: "Note that we have enabled that feature so that Chris Steipp can test it out. I dont think we should disable it but instead should work on " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87045 (owner: 10Mattflaschen) [19:27:40] (03CR) 10Hashar: "is this still needed now that statsd is on tungsten.eqiad.wmnet ?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/83603 (owner: 10Ori.livneh) [20:29:47] (03PS3) 10Wizardist: Set Europe/Minsk TZ for bewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86379 [20:30:48] (03CR) 10Wizardist: "I hope I did it right. Git didn't show me any conflicts though." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86379 (owner: 10Wizardist) [21:00:08] (03CR) 10TTO: "Many of them have been sitting here for over two weeks... that would seem to be part of the reason..." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85838 (owner: 10TTO) [21:02:31] (03PS2) 10TTO: Allow crats on outreachwiki to revoke translationadmin group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85513 [21:03:32] (03PS2) 10TTO: Set up patroller user group on frwikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85644 [21:19:02] (03PS1) 10Ori.livneh: Remove EditPageTracking extension [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87896 [21:29:22] (03CR) 10Nemo bis: [C: 031] "Cruft removal FTW." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87896 (owner: 10Ori.livneh) [21:30:29] :) [21:45:50] ori-l: cruft removal in cofig? https://gerrit.wikimedia.org/r/#/c/87731/ :D [21:47:52] * MatmaRex is off to sleep. [21:49:12] (03CR) 10Ori.livneh: [C: 031] Remove old Vector stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 (owner: 10Bartosz Dziewoński) [21:56:12] (03CR) 10Bartosz Dziewoński: [C: 04-1] "(Just in case – this is not safe to deploy just yet, AFAIK.)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 (owner: 10Bartosz Dziewoński) [22:00:22] someone around, who want's to chat about nodejs [22:24:29] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [22:27:29] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:30:34] (03CR) 10MZMcBride: "Filed bug 55380 and bug 55381 for the "notably did not remove" items." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 (owner: 10Bartosz Dziewoński)