[00:01:49] anomie|awayish: So, lighting deploy? Shall I go first given that you're |awayish ? [00:01:55] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Server Error - 1703 bytes in 6.343 second response time [00:02:09] RoanKattouw: I'm just waiting for Jenkins to merge, I'll be done in a few minutes [00:02:17] OK you go first then [00:02:19] My prep will be slower [00:03:31] <^d> !log restarted gitblit service on antimony [00:03:36] !log anomie synchronized php-1.23wmf15/includes/htmlform/ 'Backport [[gerrit:117038]] to fix regression' [00:03:40] Logged the message, Master [00:03:47] Logged the message, Master [00:04:02] !log anomie synchronized php-1.23wmf16/includes/htmlform/ 'Backport [[gerrit:117038]] to fix regression' [00:04:09] RoanKattouw: Ok, I'm done [00:04:09] Logged the message, Master [00:04:15] Sweet [00:04:18] <^d> RoanKattouw: It's booting. [00:05:37] <^d> RoanKattouw: Back. [00:05:44] Cool [00:05:46] Thanks man [00:05:55] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 179008 bytes in 7.271 second response time [00:11:39] !log catrope updated /a/common/php-1.23wmf16 to {{Gerrit|I706751cd2}}: Update VisualEditor to wmf16 branch for cherry-pick [00:11:48] Logged the message, Master [00:20:05] !log catrope updated /a/common/php-1.23wmf16 to {{Gerrit|Id675e756a}}: Update VisualEditor extension, for the right cherry-pick this time [00:20:14] Logged the message, Master [00:20:51] !log catrope synchronized php-1.23wmf16/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWBlockImageNode.js 'Fix 2-pixel image bug' [00:20:59] Logged the message, Master [01:12:37] (03CR) 10Catrope: [C: 031] Remove old Tampa srv* and mw* apaches from dsh groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/108070 (owner: 10Chad) [01:27:40] (03CR) 10Chad: Elasticsearch upgrade starting (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117095 (owner: 10Manybubbles) [01:47:04] !log s6 xtrabackup clone db1022 to db1010 [01:47:13] Logged the message, Master [02:11:40] (03PS2) 10Manybubbles: Elasticsearch upgrade starting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117095 [02:11:55] (03CR) 10Manybubbles: "Good idea, ^d" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117095 (owner: 10Manybubbles) [02:25:35] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [02:26:16] !log LocalisationUpdate completed (1.23wmf15) at 2014-03-06 02:26:16+00:00 [02:26:24] Logged the message, Master [02:26:25] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 35.92 ms [02:50:59] !log LocalisationUpdate completed (1.23wmf16) at 2014-03-06 02:50:59+00:00 [02:51:08] Logged the message, Master [03:05:58] (03CR) 10Ryan Lane: [C: 031] Adding git-fat support for trebuchet deployment [operations/puppet] - 10https://gerrit.wikimedia.org/r/112944 (owner: 10Ottomata) [03:20:43] !log db1034 testing /a ext4 noatime,barrier=0 [03:20:52] Logged the message, Master [03:29:47] !log springle synchronized wmf-config/db-eqiad.php 's1 depool db1049' [03:29:55] Logged the message, Master [03:30:35] !log s1 xtrabackup clone db1049 to db1034 [03:30:44] Logged the message, Master [03:33:45] (03CR) 10Dzahn: turn planet into a module (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/108674 (owner: 10Dzahn) [03:34:41] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-03-06 03:34:41+00:00 [03:34:49] (03PS10) 10Dzahn: turn planet into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108674 [03:34:51] Logged the message, Master [04:19:58] !log springle synchronized wmf-config/db-eqiad.php 's6 repool db1010 warm up' [04:20:05] Logged the message, Master [05:24:44] (03PS2) 10Nemo bis: [gdash] Add some yearly graphs [operations/puppet] - 10https://gerrit.wikimedia.org/r/117020 [05:59:38] (03CR) 10Ori.livneh: [C: 032] "Thank you dearly for the yearly graphs," [operations/puppet] - 10https://gerrit.wikimedia.org/r/117020 (owner: 10Nemo bis) [06:02:38] ori-l, gerrit poet [06:06:27] mutante|away: :) [06:08:38] ori: did you see anomie's mail? [06:08:45] (good morning) [06:44:05] (03CR) 10Dzahn: [C: 031] "tested one more time on fresh eqiad labs instance. as long as i comment the installation of the certificate, all fine, no warnings, create" [operations/puppet] - 10https://gerrit.wikimedia.org/r/108674 (owner: 10Dzahn) [06:45:10] (03CR) 10Dzahn: "paravoid, could you give this one more review given the comments above?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/108070 (owner: 10Chad) [06:45:12] (03CR) 10Nemo bis: "wow, thanks! :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117020 (owner: 10Nemo bis) [06:51:33] James_F|Away: fresh graph at the end of https://gdash.wikimedia.org/dashboards/ve/ , is that normal? [07:34:37] (03PS1) 10Springle: repool db1010 and db1034 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117166 [07:35:04] (03CR) 10Springle: [C: 032] repool db1010 and db1034 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117166 (owner: 10Springle) [07:35:11] (03Merged) 10jenkins-bot: repool db1010 and db1034 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117166 (owner: 10Springle) [07:44:25] !log springle synchronized wmf-config/db-eqiad.php 's6 db1010 full steam, s1 db1034 warm up' [07:44:34] Logged the message, Master [08:24:52] paravoid: no, i hadn't, but i just read it now. thanks for flagging it. i'll look into it right now. [09:04:29] (03PS2) 10Hashar: beta: vary deployment-bastion by ::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/116982 [09:04:31] (03PS2) 10Hashar: beta: on eqiad mount /srv using labs_lvm [operations/puppet] - 10https://gerrit.wikimedia.org/r/116987 [09:37:19] !log springle synchronized wmf-config/db-eqiad.php 's1 db1034 full steam' [09:37:27] Logged the message, Master [09:40:24] (03CR) 10Hashar: "On the beta cluster, the call_geoip cookie cause Varnish to eat the central auth cookies preventing new users from login in. See bug 62244" [operations/puppet] - 10https://gerrit.wikimedia.org/r/113935 (owner: 10Ori.livneh) [09:41:14] <_joe_> /win 15 [09:41:18] <_joe_> ooops sorry [09:41:28] (03PS2) 10Faidon Liambotis: Revert "CentralNotice: set $wgCentralGeoScriptURL to false" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117086 (owner: 10Ori.livneh) [09:41:39] (03PS2) 10Faidon Liambotis: Disable cookie-based geolocation on text varnishes [operations/puppet] - 10https://gerrit.wikimedia.org/r/117004 (owner: 10Ori.livneh) [09:41:49] (03CR) 10Faidon Liambotis: [C: 032] Revert "CentralNotice: set $wgCentralGeoScriptURL to false" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117086 (owner: 10Ori.livneh) [09:41:57] (03Merged) 10jenkins-bot: Revert "CentralNotice: set $wgCentralGeoScriptURL to false" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117086 (owner: 10Ori.livneh) [09:42:52] !log faidon updated /a/common to {{Gerrit|Id235690c6}}: Revert "CentralNotice: set $wgCentralGeoScriptURL to false" [09:43:01] Logged the message, Master [09:43:41] !log faidon synchronized wmf-config/CommonSettings.php 'revert wgCentralGeoScriptURL to false' [09:43:48] Logged the message, Master [09:44:01] (03CR) 10Faidon Liambotis: [C: 032] Disable cookie-based geolocation on text varnishes [operations/puppet] - 10https://gerrit.wikimedia.org/r/117004 (owner: 10Ori.livneh) [10:00:18] (03CR) 10Hashar: "Geoip in cookie has been disabled with https://gerrit.wikimedia.org/r/#/c/117004/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/113935 (owner: 10Ori.livneh) [10:22:50] who has the keys to change mailman passwords? [10:23:14] File a bug in bugzilla under Mailing Lists [10:23:45] yes, but then how do we get the password? [10:24:01] and what is the turn around time? [10:25:00] when I ran the mailing lists at RootsWeb we had a hack over the top so admins could do it themselves [10:26:26] sDrewth: email, depends if people are around [10:26:29] though I suppose with mail-jacking that is less feasible these days [10:26:51] okay, so don't request it on the weekend [10:35:17] sDrewth: i'll handle. the best way is rt request. mutante|away will probably handle [10:53:23] hi! I often get a 503 error trying to query wikidata [10:53:34] for example: curl http://www.wikidata.org/wiki/Special:EntityData/Q52.json [10:54:27] am I doing something wrong? is there a better way to get JSON representations of wikidata entries? [12:44:09] ema: IIRC it's a known bug for some entities, check bugzilla [12:56:09] Nemo_bis: right, bug #60003 [12:56:14] Nemo_bis: thanks [13:18:33] :) [13:55:25] PROBLEM - MySQL Slave Running on db1007 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Table ./metawiki/translate_messageindex is marked as crashed [13:59:25] RECOVERY - MySQL Slave Running on db1007 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [14:13:56] (03PS1) 10Andrew Bogott: Don't hard-code the labs status region to pmtpa [operations/puppet] - 10https://gerrit.wikimedia.org/r/117179 [14:15:46] (03CR) 10Andrew Bogott: [C: 032] Don't hard-code the labs status region to pmtpa [operations/puppet] - 10https://gerrit.wikimedia.org/r/117179 (owner: 10Andrew Bogott) [14:16:15] (03PS5) 10Aude: Setup test.wikidata as repo for test2 and test.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115366 [14:17:44] (03CR) 10Mark Bergsma: [C: 032] DNS setup for legalteamwiki [operations/dns] - 10https://gerrit.wikimedia.org/r/116220 (owner: 10Jalexander) [14:26:07] (03PS3) 10Mark Bergsma: Apache config for legalteamwiki [operations/apache-config] - 10https://gerrit.wikimedia.org/r/116219 (owner: 10Jalexander) [14:27:05] (03CR) 10Mark Bergsma: [C: 032] Apache config for legalteamwiki [operations/apache-config] - 10https://gerrit.wikimedia.org/r/116219 (owner: 10Jalexander) [14:31:37] (03CR) 10Hoo man: [C: 032] "Let's do this" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115366 (owner: 10Aude) [14:32:33] (03Merged) 10jenkins-bot: Setup test.wikidata as repo for test2 and test.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115366 (owner: 10Aude) [14:32:58] :) [14:34:52] !log hoo synchronized wmf-config/ 'Setup test.wikidata as repo for test2 and test.wikipedia {{Gerrit|I6f4c512}}' [14:35:00] Logged the message, Master [14:35:01] \o/ [14:35:34] !log hoo synchronized wikidataclient.dblist 'Setup test.wikidata as repo for test2 and test.wikipedia {{Gerrit|I6f4c512}}' [14:35:41] Logged the message, Master [14:35:52] aude: Ok... shall we bump the parser cache epoch now or not needed? [14:36:23] might not be necessary [14:36:40] ok :) People can purge, if needed [14:36:47] not that many items [14:37:05] we're done, then [14:37:43] all looks ok [14:38:01] except i need to make new test items on test wikidata [14:38:48] aude: Did you already verify? [14:39:17] i verify test2 does not use wikidata [14:39:54] wikipedia still uses wikidata :) [14:40:08] That was the first thing I tested :D [14:40:38] https://test2.wikipedia.org/wiki/Kitten [14:40:40] works [14:41:16] hooray :) [14:41:54] (03PS3) 10Ottomata: Adding git-fat support for trebuchet deployment [operations/puppet] - 10https://gerrit.wikimedia.org/r/112944 [14:41:58] (03PS6) 10TTO: Initial setup for legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112850 [14:42:12] (03CR) 10Ottomata: [C: 032 V: 032] Adding git-fat support for trebuchet deployment [operations/puppet] - 10https://gerrit.wikimedia.org/r/112944 (owner: 10Ottomata) [14:45:53] (03CR) 10Reedy: [C: 032] Initial setup for legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112850 (owner: 10TTO) [14:45:59] (03PS4) 10Jforrester: Enable VisualEditor for legalteamwiki by default [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112717 [14:46:04] (03Merged) 10jenkins-bot: Initial setup for legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112850 (owner: 10TTO) [14:46:10] (03CR) 10Jforrester: [C: 031] Enable VisualEditor for legalteamwiki by default [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112717 (owner: 10Jforrester) [14:46:38] !log reedy synchronized database lists files: [14:46:46] Logged the message, Master [14:48:24] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: add legalteamwiki [14:48:31] Logged the message, Master [14:48:51] !log reedy updated /a/common to {{Gerrit|I495e02449}}: Initial setup for legalteamwiki [14:48:56] (03PS1) 10Reedy: Add legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117188 [14:49:00] Logged the message, Master [14:49:14] (03CR) 10Reedy: [C: 032] Add legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117188 (owner: 10Reedy) [14:49:16] (03CR) 10jenkins-bot: [V: 04-1] Add legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117188 (owner: 10Reedy) [14:49:27] stfu jenkins [14:49:55] (03CR) 10Reedy: [V: 032] Add legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117188 (owner: 10Reedy) [14:49:59] that is most probably Gerrit yielding a 'host mismatch key' :( [14:50:42] !log reedy synchronized wmf-config/ 'legalteamwiki config and touch of IS' [14:50:49] Logged the message, Master [14:51:46] (03PS1) 10Mark Bergsma: Revert "Apache config for legalteamwiki" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/117190 [14:52:30] (03CR) 10Mark Bergsma: [C: 032] Revert "Apache config for legalteamwiki" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/117190 (owner: 10Mark Bergsma) [14:53:30] (03PS2) 10Hashar: beta: natfixup for eqiad public IP addresses [operations/puppet] - 10https://gerrit.wikimedia.org/r/116787 [14:55:32] (03CR) 10coren: [C: 032] "That works." [operations/puppet] - 10https://gerrit.wikimedia.org/r/116987 (owner: 10Hashar) [14:55:52] (03PS3) 10Hashar: beta: vary deployment-bastion by ::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/116982 [14:57:11] (03PS3) 10Hashar: beta: on eqiad mount /srv using labs_lvm [operations/puppet] - 10https://gerrit.wikimedia.org/r/116987 [14:57:39] (03CR) 10coren: [C: 032] "Trivial enough." [operations/puppet] - 10https://gerrit.wikimedia.org/r/116787 (owner: 10Hashar) [14:57:48] !! [14:58:49] (03PS1) 10Mark Bergsma: Revert "Revert "Apache config for legalteamwiki"" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/117191 [14:58:59] (03CR) 10Mark Bergsma: [C: 032] Revert "Revert "Apache config for legalteamwiki"" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/117191 (owner: 10Mark Bergsma) [14:59:35] (03CR) 10coren: [C: 032] "Yeah $::site!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/116982 (owner: 10Hashar) [15:00:33] (03CR) 10coren: [C: 032] "LVM > fixed partitions" [operations/puppet] - 10https://gerrit.wikimedia.org/r/116987 (owner: 10Hashar) [15:01:35] (03CR) 10coren: [C: 032] "Package add." [operations/puppet] - 10https://gerrit.wikimedia.org/r/112202 (owner: 10Tim Landscheidt) [15:02:39] (03PS1) 10Hashar: contint: clone some more repos for labs CI slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/117193 [15:03:45] (03CR) 10coren: [C: 032] "What's the worst that could happen?™" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117193 (owner: 10Hashar) [15:03:54] lol [15:05:00] hashar: pass him over to me when he is done :) [15:05:05] (03CR) 10coren: [C: 04-1] "I see no point to this; puppet in pmtpa tools is disabled until it dies." [operations/puppet] - 10https://gerrit.wikimedia.org/r/115609 (owner: 10Tim Landscheidt) [15:09:03] hashar: jenkins is moving to labs? [15:09:22] the slaves [15:12:02] (03PS1) 10Hashar: labs_lvm: logoutput => on_failure for all exec calls [operations/puppet] - 10https://gerrit.wikimedia.org/r/117199 [15:13:27] aude: some yeah [15:13:37] yay! [15:13:39] aude: because there is a bunch of stuff I am not willing to package for production [15:13:58] would be nice to get rid of the wikidata jenkins and use wmf jenkins on labs [15:14:11] aude: there is no a few slaves in labs for my own use. Would announce that properly once labs is migrated to eqiad [15:14:21] but been nice to make jenkins work the way we need [15:14:21] potentially, we could use composer on them. [15:14:24] yep [15:14:31] for python script I am already using tox / venv / pip [15:14:36] * aude nods [15:14:37] and javascript has a bunch of npm jobs there as well [15:19:33] !log Jenkins: added phpunit/phpcs/kss on the labs slaves. Flagged them with label hasPhpUnit hasPhpcs [15:19:41] Logged the message, Master [15:21:59] (03PS1) 10coren: Labs: Minor fixes to labs_lvm::volume [operations/puppet] - 10https://gerrit.wikimedia.org/r/117200 [15:22:33] (03PS5) 10Reedy: Remove old Tampa srv* and mw* apaches from dsh groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/108070 (owner: 10Chad) [15:22:45] (03CR) 10jenkins-bot: [V: 04-1] Remove old Tampa srv* and mw* apaches from dsh groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/108070 (owner: 10Chad) [15:23:32] (03CR) 10coren: [C: 032] "Minor fixes are minor." [operations/puppet] - 10https://gerrit.wikimedia.org/r/117200 (owner: 10coren) [15:27:27] (03PS5) 10Jforrester: Enable VisualEditor for legalteamwiki by default [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112717 [15:27:39] (03CR) 10Reedy: [C: 032] Enable VisualEditor for legalteamwiki by default [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112717 (owner: 10Jforrester) [15:27:47] (03Merged) 10jenkins-bot: Enable VisualEditor for legalteamwiki by default [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112717 (owner: 10Jforrester) [15:28:58] !log reedy synchronized database lists files: Ia651454e773e6f26f84c334b51fa64cb3dd44762 [15:29:05] Logged the message, Master [15:29:34] !log reedy synchronized wmf-config/InitialiseSettings.php 'touch' [15:29:43] Logged the message, Master [15:30:33] Reedy: Care to review https://gerrit.wikimedia.org/r/#/c/117154/ before I scap a broken l10n file again today? [15:30:45] (03PS1) 10coren: Labs: more minor fixes to labs_lvm [operations/puppet] - 10https://gerrit.wikimedia.org/r/117202 [15:31:17] (03CR) 10coren: [C: 032] Labs: more minor fixes to labs_lvm [operations/puppet] - 10https://gerrit.wikimedia.org/r/117202 (owner: 10coren) [15:33:18] (03Abandoned) 10Tim Landscheidt: Tools: Set group for $sysdir according to $::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/115609 (owner: 10Tim Landscheidt) [15:50:08] (03CR) 10Alexandros Kosiaris: [C: 032] Fix ruby stability hash issue in recursor.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/116978 (owner: 10Alexandros Kosiaris) [15:51:54] (03PS1) 10Reedy: Don't check tampa apaches in apache-fast-test [operations/puppet] - 10https://gerrit.wikimedia.org/r/117204 [16:01:28] (03PS2) 10Hashar: Configuration for beta cluster caches in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/115629 [16:01:42] (03CR) 10Mark Bergsma: [C: 032] Don't check tampa apaches in apache-fast-test [operations/puppet] - 10https://gerrit.wikimedia.org/r/117204 (owner: 10Reedy) [16:02:09] bd808: deploying today? [16:02:25] aude: Yup [16:03:29] (03PS3) 10Hashar: Configuration for beta cluster caches in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/115629 [16:03:41] (03PS1) 10Reedy: Remove pmtpa apaches from site.pp,dhcpd [operations/puppet] - 10https://gerrit.wikimedia.org/r/117206 [16:03:47] (03CR) 10Hashar: "Updated deployment-apache01 instance IP address since it got deleted and recreated." [operations/puppet] - 10https://gerrit.wikimedia.org/r/115629 (owner: 10Hashar) [16:03:54] bd808: nothing of note for wikidata [16:04:13] aude: I merged the version bump patch from hoo [16:04:13] well, deploy to testwikidata [16:04:26] great [16:04:29] (03PS2) 10Dzahn: remove sockpuppet from puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/115527 [16:05:01] hoo: "deploy to testwikidata"? Is that just a normal group0 host? [16:05:15] it is [16:05:16] bd808: Yep, nothing to worry about ;) [16:05:29] :) [16:07:01] (03PS1) 10Alexandros Kosiaris: Stabilize the result of stdlib's keys function [operations/puppet] - 10https://gerrit.wikimedia.org/r/117207 [16:09:36] (03PS6) 10Reedy: Remove old Tampa srv* and mw* apaches from dsh groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/108070 (owner: 10Chad) [16:10:48] (03CR) 10Mark Bergsma: [C: 032] Remove old Tampa srv* and mw* apaches from dsh groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/108070 (owner: 10Chad) [16:12:14] (03PS3) 10Dzahn: remove sockpuppet from puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/115527 [16:12:27] * bd808 applauds mark for making scap twice as fast (in theory) [16:12:59] no [16:13:03] just half as busy ;) [16:13:43] Think of the poor overworked electrons! [16:16:42] (03PS7) 10Reedy: Remove db and job queue pmtpa files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116036 [16:16:47] (03CR) 10Reedy: [C: 032] Remove db and job queue pmtpa files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116036 (owner: 10Reedy) [16:17:01] (03Merged) 10jenkins-bot: Remove db and job queue pmtpa files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116036 (owner: 10Reedy) [16:17:35] (03CR) 10Dzahn: [C: 032] remove sockpuppet from puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/115527 (owner: 10Dzahn) [16:17:46] !log reedy synchronized docroot and w [16:17:56] Logged the message, Master [16:18:10] !log reedy synchronized wmf-config/ [16:18:18] Logged the message, Master [16:18:40] (03CR) 10Alexandros Kosiaris: [C: 032] Stabilize the result of stdlib's keys function [operations/puppet] - 10https://gerrit.wikimedia.org/r/117207 (owner: 10Alexandros Kosiaris) [16:20:39] akosiaris: hashes shouldn't be assumed to be stable though [16:20:54] (03PS1) 10Reedy: Remove pmtpa apaches [operations/dns] - 10https://gerrit.wikimedia.org/r/117210 [16:20:55] mark: depends on the ruby version [16:21:06] 1.9.3+ has stable hashes IIRC [16:21:07] oh of course, ruby [16:21:24] but yeah, safer to not assume stability of hashes anyway [16:21:42] Reedy: leave that part to the ops decommissioning process [16:21:43] but this constant change in the catalog was making my differ unhappy [16:21:49] it would only upset people anyway ;-) [16:22:11] there's a stringent server lifecycle document around that hehe [16:22:20] awww [16:22:34] Reedy: they'll want you to leave the mgmt in there for wiping [16:22:47] but remove the others.. oh well [16:23:03] Should I amend it? [16:23:28] sure, why not, we'll just use it later [16:24:32] touched the post commit hook and puppet-merge to remove sockpuppet [16:24:55] (03PS2) 10Reedy: Remove pmtpa apaches, leaving management entries [operations/dns] - 10https://gerrit.wikimedia.org/r/117210 [16:41:16] Common ruby :( [16:41:16] - result = hash.keys [16:41:16] + result = hash.keys.sort [16:43:12] hashar: That seems like a php-ism. Who besides a php trained dev expects a hash to preserve insertion order? [16:43:56] All though I guess java has LinkedHashSet and friends [16:47:28] <_joe_> bd808|deploy, python has collections.OrderedDict as well [16:47:54] * bd808|deploy nods [16:48:38] * _joe_ goes back to lurking [16:50:28] :) [16:51:11] greg-g: we have a hhvm jenkins job running mw/core unit tests !!!!! [16:51:15] greg-g: thanks to ori :] [16:51:51] on our jenkins? sweet! [16:51:58] before it was just travis [16:52:01] that's great news [16:52:09] bd808|deploy: yeah that is why I hate PHP. It has a bunch of awesome features that are not matched by other languages. [16:52:22] greg-g: yeah ori got a package and figured out the command to run [16:52:31] bd808|deploy: holy dependency chain, batman: https://gerrit.wikimedia.org/r/#/c/117014/ [16:52:34] s/awesome features/constant overhead/ [16:52:52] greg-g: Yeah it got a little out of hand [16:52:55] :) [16:53:00] I got a bunch of PHP culprit on my laptop [16:53:03] need to do a pres [16:53:24] hashar: oh right, the package from fb, since the debian one is stalled (afaict) [16:53:33] the most hated of all is $myArray + $someArray [16:53:47] greg-g: I guess so [16:53:55] (03PS3) 10Alexandros Kosiaris: remove sockpuppet, decom [operations/dns] - 10https://gerrit.wikimedia.org/r/115688 (owner: 10Matanya) [16:54:27] good luck bd808|deploy on the deploy! [16:57:50] * bd808|deploy is prepping 1.23wmf17 on tin [17:00:15] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [17:01:15] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 35.60 ms [17:03:35] PROBLEM - Apache HTTP on mw31 is CRITICAL: HTTP CRITICAL: HTTP/1.0 500 Internal Server Error - 50631 bytes in 0.151 second response time [17:05:26] I am off, see you tomorrow [17:09:51] (03PS1) 10BryanDavis: Add 1.23wmf17 symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117221 [17:09:53] (03PS1) 10BryanDavis: Wikipedias to 1.23wmf16 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117222 [17:09:55] (03PS1) 10BryanDavis: Group0 wikis to 1.23wmf17 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117223 [17:12:37] (03CR) 10BryanDavis: [C: 032] Add 1.23wmf17 symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117221 (owner: 10BryanDavis) [17:13:09] * bd808|deploy waits for gerrit [17:16:03] (03Merged) 10jenkins-bot: Add 1.23wmf17 symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117221 (owner: 10BryanDavis) [17:16:17] 4 minutes?! [17:16:34] waiting on zuul? [17:16:59] Yeah. It took 4 minutes to merge the symlink patch [17:17:06] <^d> gerrit said 3. [17:17:36] My irc timestamps say 4 but tomato tomato [17:17:43] I just want to know what hardware I can try to get approved so its never more than 15 seconds ;] [17:17:48] I was watching zuul when that happened [17:17:59] Looked like loads of stuff was just sitting queued and then suddenly stuff got run [17:18:38] even when its slow, im just happy i can actually watch them run now [17:18:47] recall that wasnt available for a long while =] [17:19:06] at least now i can see if there is a ton of crap in front of my tests [17:19:09] https://gerrit.wikimedia.org/r/#/c/104711/ - 13 minutes [17:20:16] !log sockpuppet - disabling puppet,disabling monitoring,remove from stored configs,revoke puppet cert,delete salt key [17:20:25] Logged the message, Master [17:20:34] Reedy: Killing sockpuppet.pmtpa.wmnet...done. [17:20:39] wooo [17:20:43] Key for minion sockpuppet.pmtpa.wmnet deleted. [17:20:57] another one bites the dust [17:21:10] so that one has a lot of important data on it [17:21:19] make sure it has its very own wipe ticket please =] [17:21:32] (they all should, but just sayin) [17:22:15] (03CR) 10Alexandros Kosiaris: [C: 032] "It is referenced, albeit through a fact." [operations/puppet] - 10https://gerrit.wikimedia.org/r/116953 (owner: 10Matanya) [17:22:41] I am really tired of looking at racks and racks accessories. it may be time to do my backlogged personal expense reports for a bit. (thats how tired of racks i am) [17:22:50] =P [17:23:03] we just need one more question answered here [17:23:06] "Ryan, is multimaster safe to turn on again?" [17:23:09] for salt [17:23:23] apergos: that was your question actually, just saw it [17:23:40] yes, it's my question [17:31:10] (03PS1) 10Ottomata: Adding submodule update support to git::clone [operations/puppet] - 10https://gerrit.wikimedia.org/r/117229 [17:32:08] (03CR) 10Ottomata: "Andrew, I am mostly doing this so that puppet::self instances can automatically have git submodules updated in their clones of operations/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117229 (owner: 10Ottomata) [17:44:32] (03CR) 10Ori.livneh: [C: 04-1] "There's a '--recurse-submodules' argument you can pass the initial git-clone operations. (This suggests that the name of the Puppet parame" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117229 (owner: 10Ottomata) [17:48:56] greg-g: Ready for the scap and testwiki to 1.23wmf17? [17:50:03] you're a bit early :) but nothing else is going on. but, one sec... VE may need a backport... [17:50:45] greg-g: Oh, ok. my notes from Sam say the scap happens before the window. [17:50:53] oh right... [17:51:20] yeah, sorry, the testwiki to 17 part confused me, but if that's what normally happens, fine [17:56:08] greg-g: Imma gonna do it now then [17:56:17] kk [17:56:28] !log bd808 Started scap: testwiki to php-1.23wmf17 and rebuild l10n cache [17:56:36] Logged the message, Master [17:57:09] !log bd808 scap aborted: testwiki to php-1.23wmf17 and rebuild l10n cache (duration: 00m 41s) [17:57:22] Logged the message, Master [17:57:28] ... [17:57:29] Blerg. l10n cache patch not quite right [17:57:33] ah [17:57:43] though you no-op'd yesterday? [17:57:43] rm: cannot remove `/a/common/php-1.23wmf17/cache/l10n/l10n_cache-en.cdb': Permission denied [17:57:46] oh man [17:58:01] I think I can fix... [18:03:09] greg-g: I made a patch. Hopefully ori is looking at it [18:03:19] k [18:04:43] ori: Do you have a second to force that to update on tin? [18:04:53] It's not needed on the other hosts [18:05:56] ori: Nevermind. It pulled for me [18:06:35] !log bd808 Started scap: testwiki to php-1.23wmf17 and rebuild l10n cache (try #2) [18:06:44] Logged the message, Master [18:09:49] !log shut down sockpuppet - bye for good [18:09:58] Logged the message, Master [18:11:28] bd808|deploy: do you notice a difference without tampa ? [18:11:46] Not sure yet. Just getting to that part [18:11:54] ok:) [18:12:59] mutante: (ok: 1; fail: 0; left: 235). That's ~200 fewer hosts. We'll see if it's really faster or not [18:13:19] PROBLEM - Apache HTTP on mw40 is CRITICAL: HTTP CRITICAL: HTTP/1.0 500 Internal Server Error - 50631 bytes in 0.150 second response time [18:13:26] (03PS1) 10Jforrester: Enable VisualEditor by default on French Wikiveristy [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117238 [18:13:28] PROBLEM - Apache HTTP on srv270 is CRITICAL: HTTP CRITICAL: HTTP/1.0 500 Internal Server Error - 50631 bytes in 0.184 second response time [18:13:38] hrmm [18:13:46] what's up with those [18:14:08] PHP Warning: require(/usr/local/apache/common-local/php-1.23wmf15/../wmf-config/db.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 235 [18:14:22] (03CR) 10Jforrester: [C: 031] Enable VisualEditor by default on French Wikiveristy [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117238 (owner: 10Jforrester) [18:16:44] mutante: That error I pasted seems to be coming from srv270, mw31, mw40 mostly [18:17:05] scap-proxies:mw40.pmtpa.wmnet [18:17:22] scap-proxies:srv270.pmtpa.wmnet [18:17:31] mw31 i don't see anymore in dsh [18:17:40] but the other 2 have that in common, they are scap-proxies [18:17:46] what does that mean [18:18:03] * bd808|deploy looks around for Reedy [18:20:08] PROBLEM - Disk space on snapshot1004 is CRITICAL: DISK CRITICAL - free space: / 1055 MB (3% inode=68%): [18:20:18] PROBLEM - Disk space on terbium is CRITICAL: DISK CRITICAL - free space: / 1086 MB (3% inode=75%): [18:20:48] PROBLEM - Disk space on snapshot1003 is CRITICAL: DISK CRITICAL - free space: / 1055 MB (3% inode=69%): [18:21:36] Reedy: bd808|deploy ..so, there is no db.php file there, but instead [18:21:41] db-eqiad.php db-labs.php db-secondary.php [18:21:44] mutante: wmf15 is going to be gone in about 30 minutes... [18:21:58] so it's right about "no such file" [18:21:58] !log bd808 Finished scap: testwiki to php-1.23wmf17 and rebuild l10n cache (try #2) (duration: 15m 22s) [18:22:07] Logged the message, Master [18:22:08] PROBLEM - Disk space on snapshot1001 is CRITICAL: DISK CRITICAL - free space: / 1034 MB (3% inode=69%): [18:22:08] PROBLEM - Disk space on snapshot1002 is CRITICAL: DISK CRITICAL - free space: / 866 MB (3% inode=69%): [18:22:35] * bd808|deploy thinks that's a change Reedy synced this morning... [18:24:01] mutante: https://gerrit.wikimedia.org/r/#/c/116036/ [18:24:15] Sam deleted the db-pmtpa.php file [18:24:34] what it is looking for was just db.php [18:24:59] Sort of. Multiversion falls back to that when it can't see the realm specific file [18:26:23] What's up? [18:27:26] you broke it all Reedy :) [18:27:45] (03PS1) 10BryanDavis: Make an empty db-pmtpa.php file [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117239 [18:27:47] What's new? [18:27:51] Reedy: ^ [18:27:58] What's trying to read it? :| [18:28:19] dp.php from the scap fanout servers [18:28:30] srv270, mw31, mw40 [18:28:40] stop scapping to them then :D [18:28:43] We're not sure why mw31 is in there [18:28:51] too late :) [18:28:52] Reedy: it wanted db.php on hosts like mw31 [18:29:11] (03CR) 10jenkins-bot: [V: 04-1] Enable VisualEditor by default on French Wikiveristy [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/117238 (owner: 10Jforrester) [18:29:17] and the ones marked as scap proxies have errors [18:29:49] Reedy: scap-proxies:srv270.pmtpa.wmnet scap-proxies:mw40.pmtpa.wmnet and they have 500 Internal Server Error since last scap [18:30:09] nothing should be targetting them [18:30:12] bar maybe scap stuff [18:30:18] (03CR) 10Alexandros Kosiaris: [C: 031] "Puppet catalog differ reports no change (apart from the comment). LGTM" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109869 (owner: 10Matanya) [18:30:27] And ganglia/icinga [18:30:47] Reedy: still in dsh, just those 2 though, not mw31 [18:31:01] wasn't sure what it means that they are scap-proxies [18:31:09] solution: add role::appserver::dead_and_buried [18:31:10] It's part of the fanout [18:31:16] It syncs to N servers first [18:31:16] scap uses slave rsync servers [18:31:27] Then the rest of the servers uses those N to sync from, rather than just tin [18:31:33] so they should stay in there for now [18:31:38] even though they are tampa? [18:31:39] mutante: If I create a ticket... which Queue should I use? [18:31:42] Nope [18:31:47] about the terbium low disk thing [18:31:49] (03CR) 10Alexandros Kosiaris: [C: 032] varnish: puppet 3 compatibility fix: correct variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/109869 (owner: 10Matanya) [18:31:49] We should just stop the syncs to them in the scap scripts [18:32:08] hoo: just mail ops-requests@rt.wikimedia.org [18:32:14]