[00:06:58] RECOVERY - MySQL Replication Heartbeat on db1001 is OK: OK replication delay 1 seconds [00:07:16] RECOVERY - MySQL Slave Delay on db1001 is OK: OK replication delay 0 seconds [00:30:58] PROBLEM - Puppet freshness on cp1017 is CRITICAL: Puppet has not run in the last 10 hours [00:30:58] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [00:43:52] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [01:39:59] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 188 seconds [01:40:17] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 202 seconds [01:44:20] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [01:47:47] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 652s [01:50:38] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 25 seconds [01:52:17] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 4 seconds [01:53:38] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 26s [02:05:38] PROBLEM - Host mw1064 is DOWN: PING CRITICAL - Packet loss = 100% [02:13:53] RECOVERY - Puppet freshness on tarin is OK: puppet ran at Wed Jul 4 02:13:28 UTC 2012 [02:58:04] PROBLEM - Puppet freshness on ms3 is CRITICAL: Puppet has not run in the last 10 hours [03:07:58] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [03:15:27] * jeremyb waves [03:17:28] seems i didn't miss much while i was gone [03:18:01] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [03:30:18] PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours [04:33:42] PROBLEM - Puppet freshness on cp3002 is CRITICAL: Puppet has not run in the last 10 hours [04:33:42] PROBLEM - Puppet freshness on search31 is CRITICAL: Puppet has not run in the last 10 hours [04:35:39] PROBLEM - Puppet freshness on sq69 is CRITICAL: Puppet has not run in the last 10 hours [04:36:42] PROBLEM - Puppet freshness on search34 is CRITICAL: Puppet has not run in the last 10 hours [04:37:45] PROBLEM - Puppet freshness on strontium is CRITICAL: Puppet has not run in the last 10 hours [04:37:45] PROBLEM - Puppet freshness on search24 is CRITICAL: Puppet has not run in the last 10 hours [04:41:39] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [04:41:39] PROBLEM - Puppet freshness on search21 is CRITICAL: Puppet has not run in the last 10 hours [04:41:39] PROBLEM - Puppet freshness on search22 is CRITICAL: Puppet has not run in the last 10 hours [04:41:39] PROBLEM - Puppet freshness on search28 is CRITICAL: Puppet has not run in the last 10 hours [04:41:39] PROBLEM - Puppet freshness on search27 is CRITICAL: Puppet has not run in the last 10 hours [04:41:40] PROBLEM - Puppet freshness on search36 is CRITICAL: Puppet has not run in the last 10 hours [04:42:42] PROBLEM - Puppet freshness on search33 is CRITICAL: Puppet has not run in the last 10 hours [04:43:45] PROBLEM - Puppet freshness on search20 is CRITICAL: Puppet has not run in the last 10 hours [04:43:45] PROBLEM - Puppet freshness on search30 is CRITICAL: Puppet has not run in the last 10 hours [04:43:45] PROBLEM - Puppet freshness on search16 is CRITICAL: Puppet has not run in the last 10 hours [04:45:42] PROBLEM - Puppet freshness on sq70 is CRITICAL: Puppet has not run in the last 10 hours [04:47:39] PROBLEM - Puppet freshness on search26 is CRITICAL: Puppet has not run in the last 10 hours [04:48:42] PROBLEM - Puppet freshness on sq67 is CRITICAL: Puppet has not run in the last 10 hours [04:49:45] PROBLEM - Puppet freshness on sq68 is CRITICAL: Puppet has not run in the last 10 hours [04:49:45] PROBLEM - Puppet freshness on search18 is CRITICAL: Puppet has not run in the last 10 hours [04:50:39] PROBLEM - Puppet freshness on cp3001 is CRITICAL: Puppet has not run in the last 10 hours [04:51:42] PROBLEM - Puppet freshness on search17 is CRITICAL: Puppet has not run in the last 10 hours [04:52:45] PROBLEM - Puppet freshness on search13 is CRITICAL: Puppet has not run in the last 10 hours [04:52:45] PROBLEM - Puppet freshness on search25 is CRITICAL: Puppet has not run in the last 10 hours [04:54:42] PROBLEM - Puppet freshness on search15 is CRITICAL: Puppet has not run in the last 10 hours [04:56:39] PROBLEM - Puppet freshness on search14 is CRITICAL: Puppet has not run in the last 10 hours [04:56:39] PROBLEM - Puppet freshness on search19 is CRITICAL: Puppet has not run in the last 10 hours [04:56:39] PROBLEM - Puppet freshness on search29 is CRITICAL: Puppet has not run in the last 10 hours [04:57:42] PROBLEM - Puppet freshness on search23 is CRITICAL: Puppet has not run in the last 10 hours [04:59:39] PROBLEM - Puppet freshness on arsenic is CRITICAL: Puppet has not run in the last 10 hours [05:01:46] New patchset: Tim Starling; "Adjust sshd and apache nice levels per RT #664" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14166 [05:02:18] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/14166 [05:02:39] PROBLEM - Puppet freshness on palladium is CRITICAL: Puppet has not run in the last 10 hours [05:03:12] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14166 [05:24:07] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [05:43:37] PROBLEM - Host mw1017 is DOWN: PING CRITICAL - Packet loss = 100% [05:48:36] New patchset: Tim Starling; "Set apache nice level to 0" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14167 [05:49:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/14167 [05:49:46] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14167 [06:08:31] PROBLEM - swift-container-auditor on ms-be4 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [06:13:03] RECOVERY - swift-container-auditor on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [06:19:20] New patchset: Tim Starling; "Fix for Id7008681: that's an adjustment, not an absolute value" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14170 [06:19:52] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14170 [06:52:24] New patchset: Tim Starling; "Start apache at nice level 0 regardless of the nice level of the caller" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14172 [06:52:55] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14172 [07:00:55] PROBLEM - Apache HTTP on mw32 is CRITICAL: Connection refused [07:02:07] PROBLEM - Apache HTTP on mw41 is CRITICAL: Connection refused [07:09:12] RECOVERY - Apache HTTP on mw32 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [07:13:51] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [07:17:54] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [07:18:03] RECOVERY - Apache HTTP on mw41 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.040 second response time [07:34:31] !log updating Jenkins copy of integration/jenkins from 0f069c3 to e264d1b. Bring new ant script + update to testswarm fetcher [07:34:40] Logged the message, Master [08:15:50] PROBLEM - swift-container-auditor on ms-be4 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [08:17:21] RECOVERY - swift-container-auditor on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [09:25:47] PROBLEM - swift-container-auditor on ms-be4 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [09:32:52] !log swift-container-auditor seems to get down from time. Nagios reporting 0 processes at 8:15am and 9:25am UTC (I guess it get restarted automatically by puppet) [09:33:01] Logged the message, Master [09:39:35] PROBLEM - swift-container-auditor on ms-be3 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [09:49:11] RECOVERY - swift-container-auditor on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [10:00:08] RECOVERY - swift-container-auditor on ms-be3 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [10:10:38] New patchset: Raimond Spekking; "Bug 38159: Add import source for dewikiversity" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/14180 [10:31:55] PROBLEM - Puppet freshness on cp1017 is CRITICAL: Puppet has not run in the last 10 hours [10:31:55] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [10:44:58] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [11:45:50] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [12:35:20] good morning [12:37:26] good morning jeremyb [12:42:21] jeremyb: hi, guess lets merge the wikidata stuff, just looked at those again [12:42:32] wooot! [12:42:34] ;) [12:43:00] anyone up to sync the interwiki table? We've just discovered that WMDE's blog had no prefix... [12:44:17] hoo: ok, doing so [12:44:30] !log updating/syncing interwiki cache [12:44:39] Logged the message, Master [12:46:10] jeremyb: apache config part i can do, but besides merging i also need to sync the old way. mediawiki config: i can merge it, but i dont know if there needs to be another sync for it [12:46:34] yes, mediawiki config needs syncing to each apache too [12:46:51] or the docroot won't exist [12:47:15] Thanks, mutante :) [12:49:08] jeremyb: ok, merging the new files in docroot first, shouldnt change anything [12:49:22] mutante: right [12:49:43] Change merged: Dzahn; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/14071 [12:51:33] hashar: are you on?:) [12:51:48] i thought that might happen ;P [12:51:56] mutante: what's the question? [12:52:37] what am i supposed to use to sync these? sync-file individually rather than scap [12:53:32] mediawiki not apache? [12:53:36] yes [12:53:49] or sync-dir ? [12:54:00] i think that exists [12:54:10] * jeremyb has no first hand experience of course [12:54:59] there's also sync-docroot? [12:55:11] maybe sync-docroot is needed for the new extract2 [12:55:14] mutante: overwheelmed as usual [12:55:22] writing email about 20% :) [12:55:26] heh [12:56:20] sync-dir just calls sync-common-file [12:56:28] * jeremyb doesn't know what to suggest [12:56:43] morning ottomata! [12:56:53] tell me you're not working today? [12:57:20] joining us for hot dog eating contest spectating? [12:58:43] PROBLEM - Puppet freshness on ms3 is CRITICAL: Puppet has not run in the last 10 hours [12:59:12] mutante: i need to leave in less than 20 mins for ~40 mins... (and then I'm gone for a while from 14:20) [12:59:17] any parting questions? [12:59:25] (UTC) [13:00:29] jeremyb: not really, i'll look at it and if i can confirm i see the new files on cluster then i will merge the redirect [13:00:39] right [13:01:09] although it's not a redirect any more (well it is but it's internal not a 30x) [13:01:33] yes, i should say "the redirect file, removing the redirect and adding the apache virtual host" [13:04:17] mutante: hi, Denny here from wikidata. jeremyb pointed me here in case there are questions [13:04:27] * jeremyb poofs [13:08:32] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [13:11:14] PROBLEM - swift-container-auditor on ms-be4 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [13:14:55] !log git pull in /h/w/common/docroot . adding wikidata.org files on fenari. , then "sync-docroot" [13:15:05] Logged the message, Master [13:15:26] Denny_WMDE: alright, i'm on it [13:18:35] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [13:19:47] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/9874 [13:21:21] !log svn commiting gerrit 987, sync-apache [13:21:29] Logged the message, Master [13:23:17] !log apache-graceful-all to add wikidata.org virtual host [13:23:25] Logged the message, Master [13:26:14] RECOVERY - swift-container-auditor on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [13:31:38] PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours [13:33:00] New patchset: Mark Bergsma; "Make ProxyFetch's stop method more robust" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14186 [13:33:00] New patchset: Mark Bergsma; "Move monitor's cleanup handler install to ancestor" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14187 [13:33:01] New patchset: Mark Bergsma; "Stop IdleConnection monitor from retrying when inactive" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14188 [13:33:02] New patchset: Mark Bergsma; "Catch callbacks from connector.stopConnecting()" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14189 [13:33:03] New patchset: Mark Bergsma; "ReconnectingClientFactory.stopTrying() doesn't disconnect" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14190 [13:33:03] New patchset: Mark Bergsma; "Don't report a successful connection when inactive" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14191 [13:33:04] New patchset: Mark Bergsma; "ReconnectingClientFactory doesn't have .disconnect()" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14192 [13:33:05] New patchset: Mark Bergsma; "Make Server.monitors a set, fix removeMonitors bug changing the list during an iteration" [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14193 [13:34:21] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14186 [13:35:07] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14187 [13:35:36] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14188 [13:36:08] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14189 [13:36:32] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14190 [13:37:03] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14191 [13:37:59] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14192 [13:38:30] Change merged: Mark Bergsma; [operations/debs/pybal] (master) - https://gerrit.wikimedia.org/r/14193 [13:41:23] PROBLEM - swift-container-auditor on ms-be3 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [13:46:41] mutante: your !log is for 9847 not 987, right? [13:54:26] err 9874* [13:54:38] * jeremyb was rushing to hit enter because the train was going underground [13:57:21] jeremyb: yes, fixed SAL. [13:58:28] yah [13:58:58] mutante: so, should be live? [13:59:33] i am off for a while for internal demo time. back in an hour or two [14:02:04] mutante: i think extract2.php isn't live [14:02:10] (or is stale) [14:03:19] jeremyb: gotta disappoint you and stop here for 2 reasons. a) technical issue with the deployment switch svn->git b) there might still be concerns over the URI scheme. Need to write to list to explain and get some consensus [14:03:40] mutante: concern from who? Ryan_Lane said it was approved [14:03:53] mutante: anyway, he seemed to agree this was separate from uri scheme talk [14:05:06] mutante: anyway, the immediate problem i see atm is with extract2.php and that's not the problem git repo, right? (i think only the apache repo is a problem?) [14:05:27] i didnt know that Ryan approved [14:05:44] i don't know that he approved [14:05:53] i think maybe robla did. ryan said that it had been [14:06:53] RECOVERY - swift-container-auditor on ms-be3 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [14:07:50] jeremyb: i can see extract2.php on a server in /usr/local/ [14:08:05] mutante: does it mention wikidata.org in it? [14:08:11] /usr/local/apache/common-local/docroot/www.wikidata.org/w [14:08:42] i mean inside extract2. does it mention wikidata.org [14:08:44] ? [14:09:10] no [14:10:05] so it needs an update. maybe a sync-docroot [14:10:19] which !log says you did already [14:10:34] btw, on the 3rd: [05:20:22] but the uri scheme was already approved [14:11:07] yes, i did that or there wouldn't be /docroot/www.wikidata.org ..ehm [14:11:25] and thanks for that quote [14:11:27] right. but the extract2.php in there is just a symlink [14:11:34] so you have to update the real one [14:16:17] extract2.php -> /apache/common/extract2.php , and it is in there in /h/w/common/extract2.php on fenari, but it does not get from there to /usr/local/apache/common-local and that is where the sync script syncs from.. that is what i meant with technical issue a).. [14:16:38] brb [14:17:01] oh... [14:17:27] not quite seeing the connection to an svn->git migration [14:17:36] but that does sound like a problem [14:20:16] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.026 seconds [14:30:13] * jeremyb is running away in 2 mins. not sure when i'll be back. have my phone with mail and can gchat is needed [14:34:58] PROBLEM - Puppet freshness on search31 is CRITICAL: Puppet has not run in the last 10 hours [14:34:58] PROBLEM - Puppet freshness on cp3002 is CRITICAL: Puppet has not run in the last 10 hours [14:36:55] PROBLEM - Puppet freshness on sq69 is CRITICAL: Puppet has not run in the last 10 hours [14:37:58] PROBLEM - Puppet freshness on search34 is CRITICAL: Puppet has not run in the last 10 hours [14:38:52] PROBLEM - Puppet freshness on search24 is CRITICAL: Puppet has not run in the last 10 hours [14:38:52] PROBLEM - Puppet freshness on strontium is CRITICAL: Puppet has not run in the last 10 hours [14:42:55] PROBLEM - Puppet freshness on search21 is CRITICAL: Puppet has not run in the last 10 hours [14:42:55] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [14:42:55] PROBLEM - Puppet freshness on search22 is CRITICAL: Puppet has not run in the last 10 hours [14:42:55] PROBLEM - Puppet freshness on search27 is CRITICAL: Puppet has not run in the last 10 hours [14:42:55] PROBLEM - Puppet freshness on search36 is CRITICAL: Puppet has not run in the last 10 hours [14:42:56] PROBLEM - Puppet freshness on search28 is CRITICAL: Puppet has not run in the last 10 hours [14:43:58] PROBLEM - Puppet freshness on search33 is CRITICAL: Puppet has not run in the last 10 hours [14:44:53] PROBLEM - Puppet freshness on search16 is CRITICAL: Puppet has not run in the last 10 hours [14:44:53] PROBLEM - Puppet freshness on search20 is CRITICAL: Puppet has not run in the last 10 hours [14:44:53] PROBLEM - Puppet freshness on search30 is CRITICAL: Puppet has not run in the last 10 hours [14:46:58] PROBLEM - Puppet freshness on sq70 is CRITICAL: Puppet has not run in the last 10 hours [14:48:55] PROBLEM - Puppet freshness on search26 is CRITICAL: Puppet has not run in the last 10 hours [14:49:58] PROBLEM - Puppet freshness on sq67 is CRITICAL: Puppet has not run in the last 10 hours [14:50:52] PROBLEM - Puppet freshness on search18 is CRITICAL: Puppet has not run in the last 10 hours [14:50:53] PROBLEM - Puppet freshness on sq68 is CRITICAL: Puppet has not run in the last 10 hours [14:51:55] PROBLEM - Puppet freshness on cp3001 is CRITICAL: Puppet has not run in the last 10 hours [14:52:58] PROBLEM - Puppet freshness on search17 is CRITICAL: Puppet has not run in the last 10 hours [14:53:52] PROBLEM - Puppet freshness on search13 is CRITICAL: Puppet has not run in the last 10 hours [14:53:52] PROBLEM - Puppet freshness on search25 is CRITICAL: Puppet has not run in the last 10 hours [14:54:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:58] PROBLEM - Puppet freshness on search15 is CRITICAL: Puppet has not run in the last 10 hours [14:56:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.188 seconds [14:57:55] PROBLEM - Puppet freshness on search14 is CRITICAL: Puppet has not run in the last 10 hours [14:57:55] PROBLEM - Puppet freshness on search19 is CRITICAL: Puppet has not run in the last 10 hours [14:57:55] PROBLEM - Puppet freshness on search29 is CRITICAL: Puppet has not run in the last 10 hours [14:58:58] PROBLEM - Puppet freshness on search23 is CRITICAL: Puppet has not run in the last 10 hours [15:00:55] PROBLEM - Puppet freshness on arsenic is CRITICAL: Puppet has not run in the last 10 hours [15:02:43] hm arsenic kills cute puppets [15:03:55] PROBLEM - Puppet freshness on palladium is CRITICAL: Puppet has not run in the last 10 hours [15:06:21] !log sync-common-file extract2.php, apache-graceful-all [15:06:29] Logged the message, Master [15:19:20] mutante: nothing we can do right now for that, until the extract2.php issue is fixed? [15:20:36] Denny_WMDE: it's on there, i just have the redirect in cache it seems, just talking about how to purge it in #wikimedia-tech [15:20:56] eg. http://www.wikidata.org/?foo=baarrr works ;) [15:21:12] Denny_WMDE: but i can do: curl -H 'Host:www.wikidata.org' --url "http://www.wikidata.org" srv293 | grep Welcome and i get

Welcome to Wikidata!

:) [15:21:44] mutante: cool! thanks [15:22:07] so i guess at some point the cache will just get stale and move and then it works [15:22:09] awesome, thanks! [15:22:10] aha, yay, i just see that working link myself right now:) joy [15:22:53] Denny_WMDE: re: extract2. needed sync-common-file: 15:06 mutante: sync-common-file extract2.php, apache-graceful-all [15:24:55] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [15:25:31] !log wikidata.org works now, besides old redirect may still be cached on cp* boxes (not purged by purgeList.php via multicast?). http://www.wikidata.org/?notcached [15:25:39] Logged the message, Master [15:27:35] mutante: thanks! I will bug someone here tomorrow if it still taking it from cache :) [15:27:39] but so far: awesome! [15:28:13] PROBLEM - Host mw1136 is DOWN: PING CRITICAL - Packet loss = 100% [15:28:32] Denny_WMDE: yw! [15:29:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.551 seconds [15:38:03] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/12366 [15:38:12] Denny_WMDE: https://www.wikidata.org/?foo works, but people will create bugs about the certificate, this way it currently gets the *.wikimedia.org cert, just for later [15:41:22] mutante: ah thanks for the warning [15:42:20] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/14180 [15:54:10] PROBLEM - swift-container-auditor on ms-be3 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [16:07:04] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:13] PROBLEM - SSH on argon is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:12:37] PROBLEM - Host mw1036 is DOWN: PING CRITICAL - Packet loss = 100% [16:15:55] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.011 seconds [16:16:58] RECOVERY - swift-container-auditor on ms-be3 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [16:21:40] New patchset: Alex Monk; "(bug 38157) Remove liquidthreads_labswikimedia from config" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/14200 [16:27:01] PROBLEM - Host argon is DOWN: PING CRITICAL - Packet loss = 100% [16:32:03] !log wikidata.org on now - redirect purged from squids [16:32:11] Logged the message, Master [16:50:01] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:59] PROBLEM - Host mw1098 is DOWN: PING CRITICAL - Packet loss = 100% [16:59:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.792 seconds [17:13:18] New review: Krinkle; "Maybe add to deleted.dblist?" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/14200 [17:14:23] New review: Alex Monk; "It's already there." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/14200 [17:15:04] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [17:16:58] New review: Krinkle; "k, looks like it was forgotten in that case :)" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/14200 [17:18:58] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [17:27:31] PROBLEM - swift-container-auditor on ms-be3 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [17:28:07] PROBLEM - swift-container-auditor on ms-be4 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [17:33:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:37:07] PROBLEM - Host db1028 is DOWN: PING CRITICAL - Packet loss = 100% [17:40:52] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.989 seconds [17:42:22] RECOVERY - swift-container-auditor on ms-be3 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [17:42:58] RECOVERY - swift-container-auditor on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [18:15:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.647 seconds [18:33:12] PROBLEM - swift-container-auditor on ms-be3 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [18:57:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.378 seconds [19:07:42] RECOVERY - swift-container-auditor on ms-be3 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [19:12:01] mutante: thanks (this is jeremy) [19:12:13] worksforme [19:22:14] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [19:22:50] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.82 ms [19:26:08] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [19:36:29] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.029 second response time [19:39:47] PROBLEM - Host mw1023 is DOWN: PING CRITICAL - Packet loss = 100% [19:40:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.827 seconds [20:24:02] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:31:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.918 seconds [20:32:44] PROBLEM - Puppet freshness on cp1017 is CRITICAL: Puppet has not run in the last 10 hours [20:32:44] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [20:37:50] PROBLEM - swift-container-auditor on ms-be4 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [20:46:22] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [21:05:25] PROBLEM - Host mw1119 is DOWN: PING CRITICAL - Packet loss = 100% [21:05:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:14:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.088 seconds [21:16:49] RECOVERY - swift-container-auditor on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [21:46:22] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [21:48:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:58] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CRIT replication delay 434 seconds [21:53:34] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CRIT replication delay 421 seconds [21:57:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.730 seconds [22:09:01] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 13 seconds [22:09:55] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 1 seconds [22:32:07] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:34:58] PROBLEM - Host db1010 is DOWN: PING CRITICAL - Packet loss = 100% [22:40:58] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.967 seconds [22:43:01] PROBLEM - Host mw1154 is DOWN: PING CRITICAL - Packet loss = 100% [22:59:31] PROBLEM - Puppet freshness on ms3 is CRITICAL: Puppet has not run in the last 10 hours [23:09:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [23:13:19] PROBLEM - Host mw1078 is DOWN: PING CRITICAL - Packet loss = 100% [23:16:01] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:16:46] PROBLEM - swift-container-auditor on ms-be4 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [23:19:28] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [23:23:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.755 seconds [23:25:46] RECOVERY - swift-container-auditor on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [23:32:31] PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours [23:35:13] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CRIT replication delay 482 seconds [23:35:22] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CRIT replication delay 473 seconds [23:48:52] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 18 seconds [23:48:52] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 20 seconds [23:57:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds