[02:00:17] New patchset: Reedy; "(bug 45444) Set $wgCategoryCollation to 'uca-uk' for the Ukrainian Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52901 [02:01:37] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52901 [02:07:05] !log reedy synchronized wmf-config/InitialiseSettings.php [02:26:54] !log LocalisationUpdate completed (1.21wmf11) at Sun Mar 10 02:26:53 UTC 2013 [02:50:27] !log LocalisationUpdate completed (1.21wmf10) at Sun Mar 10 02:50:26 UTC 2013 [10:31:12] Could anyone merge and deploy https://gerrit.wikimedia.org/r/52979 please? I don't want to have to many incomplete log entries around... [10:36:59] Reedy: ^ [16:15:55] !log staggered restart of most upload varnishes (cp1036 left intentionally LRU stuck) [16:26:52] where is that bot [16:30:05] I'd log that I restarted ircecho [16:30:10] but more-bots is apparently also down [16:33:19] something went wrong with abusefilter [16:34:39] so morebots runs from wikitech-static [16:35:22] Vito: What? [16:35:36] Don't say logging, please :P [16:35:50] lol actually the filter itself :D [16:36:00] Ok, what's wrong with that [16:36:42] mh can you read filters? [16:36:44] https://it.wikipedia.org/wiki/Speciale:FiltroAntiAbusi/test [16:37:06] see filter #147 against Stratford (Ontario) page [16:37:06] PROBLEM - Puppet freshness on amssq41 is CRITICAL: Puppet has not run in the last 10 hours [16:37:07] [[Stratford (Ontario)]] [16:38:04] Vito: I don't have the global rights for that (though I'm the maintainer of the extension... :P) [16:38:06] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [16:38:18] hoo: lemme do them temporary [16:38:58] done hoo [16:39:06] fine [16:39:14] though you should have it [16:39:37] I need it from time to time... but usually I just fish the filters from the DB :P [16:39:55] Vito: Wrong group, I still can't test [16:40:02] I need the one which was created for Andrew [16:40:46] try now [16:41:30] Ok, got it... so what's wrong? [16:42:26] hoo: filter works fine in test mode [16:42:37] Oh, yes [16:42:46] but it failed in catching edits at 5pm [16:44:53] mh [16:47:34] Vito: I'll import the filter to my test wiki and play around over there... hang on [16:47:35] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 16:47:26 UTC 2013 [16:48:03] hoo: actually there was a similar issue in the past [16:48:17] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [16:48:22] what was the cause back then? [16:48:26] https://bugzilla.wikimedia.org/show_bug.cgi?id=28154 [16:49:25] Vito: So, you suggest that disallow might be the cause? [16:50:10] I'm not sure [16:50:36] but in the past I saw this bug just in coincidence with disallow option [17:01:43] Vito: The filter works locally :/ [17:02:29] Ah, I think I got the cause... hang on [17:06:34] no :/ [17:07:06] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [17:07:07] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [17:07:07] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [17:07:07] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [17:07:07] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [17:13:24] paravoid mutante-away andrewbogott_afk ping [17:13:37] damn where is everyone when I need them [17:14:41] paravoid: what's wikitech-static ? [17:15:29] paravoid take a look in -labs there is ton of troubles [17:17:55] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 17:17:52 UTC 2013 [17:18:02] hmm https://wikitech-static.wikimedia.org/wiki/ [17:18:15] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [17:27:48] Vito: I have another look at the filter in a couple of days and if the issue persists (which I fear it will) I'll open a bug [17:27:59] ok! [17:28:15] I'm not really sure how to debug that as I can't reproduce it right now [17:48:25] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 17:48:22 UTC 2013 [17:49:05] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [17:49:15] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [18:18:55] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 18:18:48 UTC 2013 [18:19:15] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [18:34:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 3.118 second response time [18:46:26] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 181 seconds [18:47:25] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 192 seconds [18:49:27] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 18:49:21 UTC 2013 [18:50:15] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [18:58:27] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [18:58:36] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [19:08:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.889 second response time [19:19:57] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 19:19:52 UTC 2013 [19:20:15] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [19:43:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:35] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 19:51:26 UTC 2013 [19:52:15] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [19:56:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [20:13:05] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [20:20:06] PROBLEM - Puppet freshness on virt0 is CRITICAL: Puppet has not run in the last 10 hours [20:20:06] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Puppet has not run in the last 10 hours [20:26:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:34:05] PROBLEM - Varnish HTCP daemon on cp1034 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:34:55] RECOVERY - Varnish HTCP daemon on cp1034 is OK: PROCS OK: 1 process with UID = 997 (varnishhtcpd), args varnishhtcpd worker [20:35:05] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 20:35:01 UTC 2013 [20:35:16] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [20:37:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [20:53:33] New patchset: Reedy; "(bug 45446) Set $wgCategoryCollation to 'uca-sv' for the Swedish Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52902 [20:53:50] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52902 [20:55:45] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 20:55:44 UTC 2013 [20:56:18] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [21:00:23] !log reedy synchronized wmf-config/InitialiseSettings.php [21:00:29] Logged the message, Master [21:09:53] !log reedy synchronized php-1.21wmf11/extensions/CentralAuth/ [21:09:56] Logged the message, Master [21:11:35] PROBLEM - SSH on cp1034 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:25] RECOVERY - SSH on cp1034 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [21:26:25] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 21:26:22 UTC 2013 [21:27:15] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [21:27:55] thanks, Reedy, btw ;) [21:36:35] PROBLEM - SSH on cp1034 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:38:25] RECOVERY - SSH on cp1034 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [21:40:26] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 196 seconds [21:40:36] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 198 seconds [21:41:13] Reedy (or anyone): Please kill the l10n cache for metawiki [21:41:43] Kill? [21:41:50] Purge [21:41:57] Or rebuild [21:42:03] It can't be done on a site by site basis [21:42:18] dunno how you do that to not get into a big cache stampede [21:42:27] CA is missing one message on meta now [21:43:01] could you simply get an admin to create it in the MediaWiki: NS? [21:43:12] not sure if that's an acceptable workaround [21:43:14] ori-l: Someone already did that [21:43:27] Not sure if that's really the way to go after an incomplete workaround [21:43:36] * s/workarodun/deploy [21:43:43] Sorry, I'm tired [21:45:45] Ok, so I guess we for now just keep the local message and I delete it in a week [21:46:10] Sorry, I don't feel confident enough to do it myself without more people around [21:46:16] :( [21:47:10] Well... we would need to create them locally on all wikis which have wmf11 :/ [21:47:17] * the message [21:48:32] someone just needs to run a scap [21:48:37] ori-l: Can you poke somebody who can do this? Or how long's the lifetime of the l10n cache? [21:48:41] to rebuild the i18n cache [21:49:25] I don't want to run scap, but OK, I'll follow the instructions for rebuilding the cache [21:49:44] I'm running scap now in a screen [21:49:58] thanks Reedy [21:50:01] Reedy: Thanks again [21:50:22] thanks Reedy [21:50:44] * ori-l pours Reedy some tea. [21:56:55] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 21:56:50 UTC 2013 [21:57:15] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [21:58:55] !log reedy Started syncing Wikimedia installation... : Rebuild message cache [21:59:00] Logged the message, Master [22:02:39] PROBLEM - SSH on cp1034 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:26] RECOVERY - SSH on cp1034 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [22:05:00] Hmmph. Anyone with an idea why the message doesn't seem to have made it through to the translations? It's fine in git master and on TWN, but apparently blank on our wikis (except meta where it's been manually created). [22:05:35] James_F: "(10:58:54 PM) logmsgbot_: !log reedy Started syncing Wikimedia installation... : Rebuild message cache" I guess that's going to fix it [22:06:08] hoo: Oh, that should help, yes. :-) [22:16:26] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [22:16:36] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [22:18:53] !log reedy Finished syncing Wikimedia installation... : Rebuild message cache [22:18:58] Logged the message, Master [22:27:15] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 22:27:12 UTC 2013 [22:28:15] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [22:35:17] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [22:37:16] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [22:57:50] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 22:57:42 UTC 2013 [22:59:05] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [22:59:10] PROBLEM - Varnish HTCP daemon on cp1034 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:00:22] RECOVERY - Varnish HTCP daemon on cp1034 is OK: PROCS OK: 1 process with UID = 997 (varnishhtcpd), args varnishhtcpd worker [23:28:27] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 23:28:16 UTC 2013 [23:29:16] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [23:58:47] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Sun Mar 10 23:58:43 UTC 2013 [23:59:15] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours