[00:10:13] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [00:13:06] * Karol007_ is back (gone 10:17:21) [00:13:25] Karol007_: please disable that script [00:14:44] jeremyb: OK, I forgot about that [00:14:52] danke [00:22:07] jeremyb: in my time zone is 2:21 am, I go sleep. Good night [00:24:40] sounds like CEST [00:29:29] indeed. [00:29:36] my TZ, too. [00:29:37] :) [00:50:16] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [01:05:16] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [01:20:16] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [01:35:16] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [01:41:33] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 228 seconds [01:42:09] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 209 seconds [01:49:03] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 628s [01:54:36] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 15 seconds [01:57:46] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 25s [01:57:54] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 0 seconds [02:00:22] !log LocalisationUpdate failed: git pull of extensions failed [02:00:50] Logged the message, Master [02:01:36] Reedy: ^ [02:01:46] lol. [02:03:01] Not sure how much I care at 3am ;) [02:03:38] i can never remember where you are [02:03:40] ;P [02:04:12] Reedy: apparently broken last night too but fine the night before [02:04:25] * jeremyb didn't notice until now [02:04:34] It'll be something stupid like a file lock [02:04:45] or the new script? [02:04:49] changed recently i thought [02:05:15] anyway, you're welcome to sleep ;) [02:05:26] Not sure, what's changed? [02:07:10] !g 6905 [02:07:10] https://gerrit.wikimedia.org/r/#q,6905,n,z [02:07:11] maybe [02:07:14] Reedy [02:07:49] if cd $GITDIR/$path && git pull && git submodule update --init [02:08:02] does look slightly suspect with regards to the error message beneath it [02:08:18] idk why it took so long to break though [02:08:57] merged in time for the cron ~4 days ago? [02:09:04] it's only been broken for 2 [02:09:29] anyway, could be something unrelated [02:10:15] Mmm [02:11:34] Initialized empty Git repository in /var/lib/l10nupdate/mediawiki/extensions/BlameMaps/.git/ [02:11:35] fatal: remote error: Git repository not found [02:11:35] Clone of 'https://gerrit.wikimedia.org/r/p/mediawiki/extesions/BlameMaps.git' into submodule path 'BlameMaps' failed [02:11:44] I spy typo [02:12:08] Probably my fault [02:14:14] lol! [02:19:36] git seems to have somewhat cached it.. [02:23:14] grrr [02:23:41] were the branches a problem? (that you just added) [02:24:00] No, but it was something else strange [02:24:14] it's still giving me the same error message even though .gitmodule on the disk is now right [02:24:31] what are you running and what's the msg? [02:25:12] exactly same as before [02:25:12] reedy@fenari:/var/lib/l10nupdate/mediawiki/extensions$ sudo -u l10nupdate git submodule update --init [02:25:12] Initialized empty Git repository in /var/lib/l10nupdate/mediawiki/extensions/BlameMaps/.git/ [02:25:12] fatal: remote error: Git repository not found [02:25:12] Clone of 'https://gerrit.wikimedia.org/r/p/mediawiki/extesions/BlameMaps.git' into submodule path 'BlameMaps' failed [02:25:48] ok, let me see [02:25:59] reedy@fenari:/var/lib/l10nupdate/mediawiki/extensions$ cat .gitmodules | grep BlameMaps\.git [02:25:59] url = https://gerrit.wikimedia.org/r/p/mediawiki/extensions/BlameMaps.git [02:27:15] Reedy: `git submodule sync` ? [02:27:36] and then init again [02:27:47] nope [02:28:35] Reedy: it gave some output? [02:28:40] from sync [02:28:57] yeah, nothing related to blamemaps [02:29:12] hmm, git submodule status seems interesting [02:29:25] ad7deaa196a6a84e80684821d09d8d8fe8580016 Blackout (remotes/origin/HEAD) [02:29:25] -818bd2eafb54a815658a75316d690102a83bf666 BlameMaps [02:29:53] rm -rf BlameMaps, rinse repeat? [02:30:14] there's no folder for it [02:30:25] git status -sb [02:30:44] doesn't llike -b [02:30:51] huh? [02:30:57] what git version? [02:30:59] reedy@fenari:/var/lib/l10nupdate/mediawiki/extensions$ sudo -u l10nupdate git status -s [02:31:00] D BlameMaps [02:31:08] 1.7.0.4 [02:31:18] i'm 1.7.2.5 [02:31:27] idk what to do now [02:32:10] well you could manually fix it [02:32:13] i guess [02:33:37] Reedy: `git config submodule.BlameMaps.url thenewurl` [02:34:41] Yay [02:34:47] Stupid git thinking it knows better [02:34:51] heh [02:36:05] welp, concert's over, good night ;) [02:36:11] * jeremyb heads home [02:43:43] jeremyb: [02:43:44] fatal: reference is not a tree: afe5c8437e860f00ba7c0bbdb7f4b1d05301c292 [02:43:44] Unable to checkout 'afe5c8437e860f00ba7c0bbdb7f4b1d05301c292' in submodule path 'SemanticGenealogy' [02:43:46] lol [02:54:10] * jeremyb is back [03:03:09] SemanticGenealogy seems screwed up [03:04:13] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [03:04:13] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [03:04:20] well we can fix it... [03:04:26] * jeremyb is about to submit a change for that [03:04:49] No, it seems fucked up everywhere [03:05:14] i was thinking maybe chad did a force push to overwrite a change that had been in that repo [03:05:26] and the hash there now is from before the force push [03:05:34] yeah [03:06:13] where else is it broke? [03:06:45] reedy@ubuntu64-web-esxi:/var/www/wiki/mediawiki/extensions$ git submodule update SemanticGenealogy/ [03:06:45] fatal: Needed a single revision [03:06:45] Unable to find current revision in submodule path 'SemanticGenealogy' [03:06:46] lol [03:07:45] yeah, that didn't work for me either [03:08:15] but so where else is it broken? [03:08:26] what do you mean? [03:08:53] 12 03:04:49 < Reedy> No, it seems fucked up everywhere [03:08:59] locally [03:09:00] on fenari [03:09:06] ergo, everywhere ;) [03:09:37] ohh [03:09:46] well if gerrit is fixed that should fix most of fenari? [03:18:40] yeah, it is now [03:18:43] just cleaning up [03:19:43] !log Running localisation update manually in screen session on fenari [03:19:53] Logged the message, Master [03:20:45] Running updates for 1.20wmf9 (on aawikibooks) [03:20:45] 1143 MediaWiki messages are updated [03:22:33] I wonder why the meta repo doesn't seem to always update then.. [03:31:18] 4.30am [03:31:35] localisation update is still chugging away in screen [03:38:33] !log LocalisationUpdate completed (1.20wmf9) at Sun Aug 12 03:38:33 UTC 2012 [03:38:42] Logged the message, Master [03:40:11] yay! [03:40:25] Reedy: and the other one now? ;) or is it going already? [03:40:30] (each branch) [03:56:34] !log LocalisationUpdate completed (1.20wmf8) at Sun Aug 12 03:56:33 UTC 2012 [03:56:43] Logged the message, Master [03:58:31] there we go [04:10:13] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [05:39:45] PROBLEM - Host srv266 is DOWN: PING CRITICAL - Packet loss = 100% [07:21:58] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [07:32:01] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [07:35:55] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [07:35:55] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [07:35:55] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [08:12:57] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [09:47:54] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [10:11:01] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [10:37:22] Hmm, so did the Gerrit review period end on August 10, or was it extended? [10:44:21] (I say that because it was due to end on August 10, not because I just picked that date out of thin air..! :) ) [10:50:55] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [10:51:29] epriestly: Hey :) You're the Phabricator guy? [11:05:54] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [11:20:54] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [11:35:54] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [11:48:35] where us extension:echo used on wikimedia? [11:52:38] liangent: I think there's a test deployment on mediawiki.org [11:52:53] Or that was the plan, anyway. [13:05:01] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [13:05:01] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [14:11:00] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [15:38:42] Can someone please silence the damn bot in here? [16:00:10] Brooke: why do you care so much? [16:01:23] It's a pretty annoying bot. [16:04:08] i think not [16:04:32] certainly there's room for improvement... [16:38:34] yes, like a +q [16:39:59] * jeremyb kicks Nemo_bis [16:40:12] hmpf [16:40:18] kick the bot [16:40:22] * jeremyb stabs Nemo_bis [16:40:28] come on! [16:40:59] hmm next blood donation will perhaps have to wait more than usual [16:41:14] * Nemo_bis looks jeremyb disapprovingly [16:42:28] Nemo_bis: whole blood or [[Apheresis]]? [16:42:58] jeremyb: whole [16:43:30] ok [16:44:07] it's nice to feel blood flowing actually [17:04:34] Whoops: [17:04:51] ugzilla has suffered an internal error. Please save this page and send it to noc@wikimedia.org with details of what you were doing at the time this message appeared. [17:04:52] URL: https://bugzilla.wikimedia.org/process_bug.cgi [17:04:53] There was an error sending mail from 'bugzilla-daemon@wikimedia.org' to 'sam@reedyboy.net':Can't send data [17:05:43] jeremyb has weird problems too [17:05:49] had* [17:06:02] siebrand: did the change show up on the web? [17:06:02] long-lasting effects [17:06:23] Nemo_bis: the comment I added was saved. [17:06:32] siebrand: what bug? [17:06:39] https://bugzilla.wikimedia.org/show_bug.cgi?id=39285 [17:07:23] wikibugs got it [17:19:57] oh, look what I've just uploaded http://archive.org/details/wiki-leuksmancom [17:23:00] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [17:23:45] Reedy: did you change your prefs recently? [17:32:54] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [17:33:04] (bugzilla prefs i mean) [17:36:58] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [17:36:58] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [17:36:58] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [17:46:21] jeremyb: I moved email providers earlier [17:46:23] but not my prefs [17:47:33] when? [17:47:51] you seem to have got it unstuck [17:48:04] couple of hours ago [17:48:25] huh [17:52:01] Reedy: i guess there's no ops working today, right? [17:52:11] not unless stuff breaks [17:52:40] well something's broke but just for 1 person... i assume no one will be around until tomorrow [17:53:09] (also spence is not puppet running so nagios isn't getting updates. but that's also not an emergency) [17:53:55] what updates does nagios need? [17:54:05] spence is now ~110 hrs out of date [17:54:24] Reedy: if you add a new host or change the checks for a host then it gets to spence by way of puppet [17:54:33] (the new nagios conf i mean) [17:54:37] yes [17:54:59] I don't think any new critical servers have gone into rotation that it being out of date really causes an issue [17:55:02] anyway, we will wait ;) [17:55:28] right. but i think we also shouldn't just wait until someone decides to deploy a new server [17:55:41] sure [17:56:01] but it's hardly of high priority [17:56:52] it's of "i will let someone know after they show up" priority. or maybe a little less ;) [18:14:01] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [19:48:58] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [20:11:54] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [20:51:58] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [21:06:58] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [21:21:58] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [21:36:57] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [22:15:14] Hmm, slow news week, anyone want to shout about their new extension, etc? [22:20:48] Jarry1250: sure! http://www.mediawiki.org/wiki/Extension:Phalanx & http://www.mediawiki.org/wiki/Admin_tools_development ;) [22:21:16] ashley: Heh, can't promise anything but I'll definitely take a look :) [22:23:52] Jarry1250: surely new anti-spam/anti-vandalism stuff is always interesting [22:24:08] ashley: I remember some i18n discussions about it? [22:24:22] Heh :) What happened this work re: it, I remember reading something but I can't quite recall [22:25:19] Nemo_bis: ah, indeed so (it was the discussion regarding why it was forked from the Wikia repository) [22:25:24] Jarry1250: you could write something about the recently published https://www.mediawiki.org/wiki/Project_Milkshake [22:25:43] ashley: Wikia is such a mess [22:26:06] Nemo_bis: can't disagree with you on that :) wish they'd contribute more to upstream and fork less, but alas... [22:26:10] Brooke is a supporter of "Wikiafication" for this reason, reduce confusion ;) [22:26:11] Jarry1250: what about WLM stuff? [22:26:16] * Nemo_bis j/k [22:26:38] * jeremyb runs away [22:26:57] jeremyb: Oooh, which WLM stuff in particular? I had a bit about the app last week. [22:27:00] jeremyb: monuments to be photographed as soon as possible? [22:27:26] Jarry1250: yeah, the app, whatever. ask #wikilovesmonuments and #wikimedia-mobile [22:27:57] jeremyb: Coolio [22:28:52] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [22:30:22] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [22:43:56] Jarry1250: what about "JS scandals"? https://en.wikipedia.org/wiki/User_talk:Vibhijain#piwiki_JavaScript https://si.wikipedia.org/wiki/%E0%B6%B4%E0%B6%BB%E0%B7%92%E0%B7%81%E0%B7%93%E0%B6%BD%E0%B6%9A_%E0%B7%83%E0%B7%8F%E0%B6%9A%E0%B6%A0%E0%B7%8A%E0%B6%A1%E0%B7%8F%E0%B7%80:%E0%B6%B6%E0%B7%92%E0%B6%9F%E0%B7%94%E0%B7%80%E0%B7%8F [22:47:16] gn8 folks [23:05:54] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [23:05:54] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [23:24:03] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [23:28:33] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [23:36:21] PROBLEM - SSH on nescio is CRITICAL: Server answer: [23:42:03] RECOVERY - SSH on nescio is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:46:24] PROBLEM - SSH on nescio is CRITICAL: Server answer: [23:47:19] PROBLEM - SSH on ssl3001 is CRITICAL: Server answer: [23:53:46] RECOVERY - SSH on nescio is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:59:28] PROBLEM - SSH on nescio is CRITICAL: Server answer: