[00:02:03] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Server Error - 1703 bytes in 6.554 second response time [00:12:03] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 212666 bytes in 7.080 second response time [01:32:05] i think jenkins is acting up again [01:35:11] yes, it definitely is. https://integration.wikimedia.org/ci/job/mediawiki-core-regression-master/5085/console 504's and i'm getting builds failing because of timeouts [01:59:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [01:59:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [01:59:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [01:59:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [02:13:42] !log LocalisationUpdate completed (1.23wmf20) at 2014-04-07 02:13:42+00:00 [02:13:54] Logged the message, Master [02:20:10] !log LocalisationUpdate completed (1.23wmf21) at 2014-04-07 02:20:10+00:00 [02:20:15] Logged the message, Master [02:49:33] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [02:56:15] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 7 02:56:12 UTC 2014 (duration 56m 11s) [02:56:19] Logged the message, Master [05:00:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:00:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:00:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:00:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:04:01] !log Zuul is stuck: (617kb image) [05:04:05] Logged the message, Master [05:28:37] (03PS1) 10Ori.livneh: HHVM on beta: add MMV to disabled extensions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124284 [05:29:59] (03CR) 10Ori.livneh: [C: 032 V: 032] HHVM on beta: add MMV to disabled extensions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124284 (owner: 10Ori.livneh) [05:51:53] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:52:53] PROBLEM - MySQL Slave Running on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:53:43] RECOVERY - MySQL Slave Running on db1047 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [05:53:43] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [06:03:00] (03CR) 10Jgreen: [C: 032 V: 031] Jon Robson access to fluorine [operations/puppet] - 10https://gerrit.wikimedia.org/r/123360 (owner: 10RobH) [06:14:13] PROBLEM - HTTP on gallium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:16:03] RECOVERY - HTTP on gallium is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 563 bytes in 0.002 second response time [06:26:47] (03PS1) 10Jgreen: removing spetrea's ssh key per RT #7203 [operations/puppet] - 10https://gerrit.wikimedia.org/r/124286 [06:27:33] PROBLEM - HTTP on carbon is CRITICAL: Connection refused [06:27:33] PROBLEM - udp2log log age for emery on emery is CRITICAL: CRITICAL: log files /a/log/webrequest/packet-loss.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [06:28:33] RECOVERY - udp2log log age for emery on emery is OK: OK: all log files active [06:33:33] RECOVERY - HTTP on carbon is OK: HTTP OK: HTTP/1.1 200 OK - 232 bytes in 0.002 second response time [06:35:33] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [06:37:06] (03PS1) 10Ori.livneh: HHVM on beta: use memc04 & memc05 for ObjectCache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124288 [06:37:48] (03CR) 10Ori.livneh: [C: 032 V: 032] HHVM on beta: use memc04 & memc05 for ObjectCache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124288 (owner: 10Ori.livneh) [06:49:30] I don't know if there's anything to it, but I have the impression that there has been a spate of alerts about unrelated systems that rely on cross-DC UDP data [06:51:31] (03CR) 10Matanya: removing spetrea's ssh key per RT #7203 (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/124286 (owner: 10Jgreen) [06:52:10] (03CR) 10Jgreen: [C: 032 V: 032] "forcing merge since jenkins/linter isn't responding" [operations/puppet] - 10https://gerrit.wikimedia.org/r/124286 (owner: 10Jgreen) [06:56:42] Jeff_Green: have you seen my omment ? [06:57:15] not sure. I did see stuff over the past few days about slowness, and java tweaks [07:00:03] Jeff_Green: regarding the last patch you just merged [07:01:58] * Jeff_Green re-reading comments [07:02:26] matanya: where did you comment? [07:02:38] oic. sec [07:04:01] matanya: I thought about enabled=false. seemed like the wrong thing to do since this appears to be temporary [07:04:18] ok, thanks Jeff_Green [07:04:22] i.e. I don't want to remove his accounts if we're just going to turn around and reenable them [07:04:44] sure [07:04:53] but I might be wrong :-) [07:05:35] sa far as i know that is the stantard thing to do, but meh [07:06:21] * Jeff_Green done with access requests, off/PTO for a while.. [07:06:31] see ya [08:01:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:01:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:01:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:01:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:05:12] (03PS1) 10Ori.livneh: HHVM on labs: use luastandalone as Scribunto engine [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124291 [08:05:44] (03CR) 10Ori.livneh: [C: 032 V: 032] HHVM on labs: use luastandalone as Scribunto engine [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124291 (owner: 10Ori.livneh) [08:40:33] (03PS1) 10Nemo bis: Enhanced recent changes: explicitly disable by default [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124292 [08:40:59] oh men [08:41:03] jenkins is dead again [08:41:12] !log Jenkins being broken for some reason AGAIN ! [08:41:17] Logged the message, Master [08:42:30] !log Restarting Jenkins, out of Java heap space. Something is leaking memory [08:42:34] Logged the message, Master [08:53:59] !log gallium killed console-kit-daemon process which was eating a lot of memory [08:54:04] Logged the message, Master [08:54:38] hi hashar [09:04:21] !log restarted Zuul [09:04:26] Logged the message, Master [09:21:08] !log reactivating peerings with HE, issues reportedly resolved [09:21:12] Logged the message, Master [09:23:05] (03CR) 10Hashar: [C: 032] beta: point wgUDPProfilerHost to eqiad instance [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123624 (owner: 10Hashar) [09:23:20] (03CR) 10Hashar: [C: 032] beta: drop pmtpa configuration for redis job server [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123623 (owner: 10Hashar) [09:23:44] (03CR) 10Hashar: [C: 032] beta: drop wmfUdp2logDest for pmtpa [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123622 (owner: 10Hashar) [09:24:15] (03CR) 10Hashar: [C: 032] beta: drop pmtpa configuration for Parsoid [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123621 (owner: 10Hashar) [09:24:33] (03CR) 10Hashar: [C: 032] beta: drop pmtpa configuration for memcached [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123620 (owner: 10Hashar) [09:24:54] (03CR) 10Hashar: [C: 032] beta: drop pmtpa configuration for databases [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123619 (owner: 10Hashar) [09:25:31] (03CR) 10Hashar: [C: 032] beta: drop pmtpa cache configuration [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123618 (owner: 10Hashar) [09:25:45] (03CR) 10Hashar: [C: 032] beta: drop pmtpa reference for CirrusSearch [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123617 (owner: 10Hashar) [09:26:46] (03CR) 10Hashar: [C: 032] beta: switch $wgEventLoggingFile to eqiad [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123616 (owner: 10Hashar) [09:27:05] (03Merged) 10jenkins-bot: beta: switch $wgEventLoggingFile to eqiad [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123616 (owner: 10Hashar) [09:27:09] (03Merged) 10jenkins-bot: beta: drop pmtpa reference for CirrusSearch [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123617 (owner: 10Hashar) [09:27:13] (03Merged) 10jenkins-bot: beta: drop pmtpa cache configuration [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123618 (owner: 10Hashar) [09:27:15] (03Merged) 10jenkins-bot: beta: drop pmtpa configuration for databases [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123619 (owner: 10Hashar) [09:27:17] (03Merged) 10jenkins-bot: beta: drop pmtpa configuration for memcached [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123620 (owner: 10Hashar) [09:27:19] (03Merged) 10jenkins-bot: beta: drop pmtpa configuration for Parsoid [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123621 (owner: 10Hashar) [09:27:21] (03Merged) 10jenkins-bot: beta: drop wmfUdp2logDest for pmtpa [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123622 (owner: 10Hashar) [09:27:23] (03Merged) 10jenkins-bot: beta: drop pmtpa configuration for redis job server [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123623 (owner: 10Hashar) [09:27:25] (03Merged) 10jenkins-bot: beta: point wgUDPProfilerHost to eqiad instance [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123624 (owner: 10Hashar) [09:30:36] (03Abandoned) 10Hashar: Central OAuth wiki for Labs (metawiki) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104666 (owner: 10CSteipp) [09:31:54] (03CR) 10Hashar: [C: 031] "Already on beta cluster. We needed it to setup the Varnish caches with XFS." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119534 (owner: 10BryanDavis) [09:36:14] (03CR) 10Gilles: [C: 031] Add setting to show a survey for MediaViewer users on some sites [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124036 (owner: 10Gergő Tisza) [10:11:31] (03CR) 10Hashar: "Some SSL cert is wrong:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/124057 (owner: 10Hashar) [10:11:53] (03PS3) 10Alexandros Kosiaris: Add an account for subbu on Parsoid / Cassandra test hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/123433 (owner: 10GWicke) [10:20:59] (03CR) 10Matanya: "I think bug #60833 is the root cause." [operations/puppet] - 10https://gerrit.wikimedia.org/r/124057 (owner: 10Hashar) [10:23:35] (03CR) 10Alexandros Kosiaris: [C: 032] Add an account for subbu on Parsoid / Cassandra test hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/123433 (owner: 10GWicke) [10:43:43] (03PS1) 10Hashar: contint: get composer on Jenkins slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/124305 [10:45:46] !log integration Getting PHP Composer installed on labs slaves. {{gerrit|124305}} [10:45:50] Logged the message, Master [10:46:04] (03CR) 10Hashar: [C: 031 V: 032] "Cherry picked on integration puppet master integration-puppetmaster.eqiad.wmflabs" [operations/puppet] - 10https://gerrit.wikimedia.org/r/124305 (owner: 10Hashar) [11:01:06] !log reedy Started scap: because we're scappy... (rebuilding l10n cache for 1.23wmf21 [11:01:11] Logged the message, Master [11:02:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [11:02:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [11:02:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [11:02:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [11:19:10] !log reedy Finished scap: because we're scappy... (rebuilding l10n cache for 1.23wmf21 (duration: 18m 04s) [11:19:13] Logged the message, Master [11:36:39] (03PS3) 10Alexandros Kosiaris: Purge /etc/apt/apt.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/123628 [11:42:55] (03CR) 10Alexandros Kosiaris: [C: 032] Purge /etc/apt/apt.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/123628 (owner: 10Alexandros Kosiaris) [11:46:41] hmm, our module dependency chain on load.php is getting freaking long. [12:15:14] (03CR) 10Alexandros Kosiaris: "So, temporary is also well known as permanent in this profession which is the vibe I am getting here. I really hope I am wrong." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119754 (owner: 10Ottomata) [12:20:53] (03CR) 10Hashar: "That did not really help on beta :/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/123444 (owner: 10Hashar) [12:23:07] !log disabled puppet on dataset2, testing [12:23:13] Logged the message, Master [12:27:06] Reedy: are you around by any chance ? Got some issue on beta with the periodic tasks [12:27:30] every 2 seconds runJobs.log get the entry: deployment-jobrunner01 simplewiki: Executed 31 periodic queue task(s). [12:29:06] hmm [12:43:18] (03CR) 10Hashar: "That is for the Ubuntu Hardy distribution which we no more use. If that .hardy file ends up being deployed on Precise instance that needs" [operations/puppet] - 10https://gerrit.wikimedia.org/r/121695 (owner: 10Chad) [12:54:18] Reedy: I have no idea how to list the periodic tasks :( [12:54:37] Reedy: the job queue class apparently has some accessor to list them out but showJobs.php does not support it [12:55:01] (03PS1) 10Aude: Re-connect test.wikipedia to wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124321 [13:15:11] !log reenabled puppet on dataset2, testing done [13:15:16] Logged the message, Master [13:23:24] (03CR) 10Alexandros Kosiaris: [C: 032] mail :lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109514 (owner: 10Matanya) [13:24:08] (03PS3) 10Alexandros Kosiaris: ganglia: class defined in class moved to top [operations/puppet] - 10https://gerrit.wikimedia.org/r/123195 (owner: 10Hashar) [13:26:24] (03CR) 10Alexandros Kosiaris: [C: 032] ganglia: class defined in class moved to top [operations/puppet] - 10https://gerrit.wikimedia.org/r/123195 (owner: 10Hashar) [13:26:31] \O/ [13:27:06] (03PS4) 10Alexandros Kosiaris: ganglia: lint manifest! [operations/puppet] - 10https://gerrit.wikimedia.org/r/123196 (owner: 10Hashar) [13:28:33] (03CR) 10Alexandros Kosiaris: [C: 032] ganglia: lint manifest! [operations/puppet] - 10https://gerrit.wikimedia.org/r/123196 (owner: 10Hashar) [13:28:45] (03PS2) 10Alexandros Kosiaris: ganglia: address selector in a define [operations/puppet] - 10https://gerrit.wikimedia.org/r/123422 (owner: 10Hashar) [13:30:24] (03CR) 10Alexandros Kosiaris: [C: 032] ganglia: address selector in a define [operations/puppet] - 10https://gerrit.wikimedia.org/r/123422 (owner: 10Hashar) [14:03:23] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [14:03:23] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [14:03:23] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [14:03:23] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [14:03:51] sweet [14:09:07] !log Jenkins cleared swap on gallium (swapoff -a && swapon -a). Makes ganglia graph nicer :D [14:09:11] Logged the message, Master [14:11:09] (03PS2) 10Aude: Re-connect test.wikipedia to wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124321 [14:34:46] !log Rebuilding GeoData index [14:34:51] Logged the message, Master [15:02:06] Reedy: Do you have any ideas about the cause of the l10n cache breakage for 1.23wmf21 when l10nupdate runs? [15:02:15] nope [15:02:19] have we got a log of it? [15:02:27] or should I manually run it and save the log somewhere? [15:03:10] /var/log/l10nupdatelog has logs of the cron runs [15:03:28] duh [15:03:53] PROBLEM - MySQL Processlist on db1019 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 156 statistics [15:05:03] Roan thinks the LU_Updater::readMessages warnings are a red herring [15:05:53] RECOVERY - MySQL Processlist on db1019 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 0 statistics [15:06:05] (03PS2) 10Reedy: remove extension-list-wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123659 (owner: 10Aude) [15:06:13] (03CR) 10Reedy: [C: 032] remove extension-list-wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123659 (owner: 10Aude) [15:06:28] (03Merged) 10jenkins-bot: remove extension-list-wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123659 (owner: 10Aude) [15:06:34] I note there's a range of errors [15:07:07] really? [15:07:16] oh, not talking to me? [15:07:24] PHP Warning: include(): Failed opening '/a/common/php-1.23wmf21/extensions/Wikidata/extensions/Wikibase/client/WikibaseClient.i18n.php' for inclusion (include_path='/$ [15:07:24] Warning: include(): Failed opening '/a/common/php-1.23wmf21/extensions/Wikidata/extensions/Wikibase/client/WikibaseClient.i18n.php' for inclusion (include_path='/a/com$ [15:07:39] huh [15:07:48] because we have json [15:08:02] Other extensions don't have those errors [15:08:02] PHP Warning: LU_Updater::readMessages: Unable to parse messages from file:///a/common/php-1.23wmf21/extensions/WikimediaIncubator/InfoPage.i18n.php in /a/common/php-1$ [15:08:02] Warning: LU_Updater::readMessages: Unable to parse messages from file:///a/common/php-1.23wmf21/extensions/WikimediaIncubator/InfoPage.i18n.php in /a/common/php-1.23wm$ [15:08:27] It's a warning, but I'm not sure we need to display those - we know they're not php arrays anymore [15:08:40] oh [15:09:04] all we have in extensions-list is the root Wikidata.php entry poing [15:09:07] point* [15:09:16] no longer list the individual things [15:09:45] I guess it's because you've not got fallback shims [15:09:51] Not that you probably need them [15:10:23] might also be an issue that the wikidata "build" has json for wikibase, but the other libraries are still php i18n there [15:10:30] would that confuse localisation update? [15:10:42] or it shoudl be smart enough and robust to handle both [15:10:44] The failed includes are the most worrying bit. ExtensionMessages-1.23wmf21.php doesn't list WikibaseClient.i18n.php; it has WikibaseClient.i18n.alias.php and WikibaseClient.i18n.magic.php [15:10:56] give me 2 min [15:11:11] It's almost like multiversion is picking up some stuff from wfm20 and some from wmf21 [15:11:20] ick [15:12:39] ok, back [15:13:37] bd808: do the aliases and magic words also migrate to json? [15:14:34] I don't think they have yet [15:14:38] (03PS2) 10MarkTraceur: Styled the alias field value differently [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/124140 (owner: 1001tonythomas) [15:14:39] ok [15:15:03] 1 step at a time! :P [15:15:17] (03CR) 10MarkTraceur: [C: 031] "That could work! I'll let Andre weigh in." [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/124140 (owner: 1001tonythomas) [15:15:19] * bd808 sits down to read the whole l10nupdate script chain again [15:15:52] it's somewhat fuzzy for me though think i kind of understand it [15:20:35] Wikibaseclientalias' => "$IP/extensions/Wikidata/extensions/Wikibase/client/WikibaseClient.i18n.alias.php", [15:20:43] 'wikibaseclientmagic' => "$IP/extensions/Wikidata/extensions/Wikibase/client/WikibaseClient.i18n.magic.php", [15:21:05] 'wikibaseclient' => "$IP/extensions/Wikidata/extensions/Wikibase/client/i18n", [15:21:08] looks ok to me [15:21:33] so why is it looking for WikibaseClient.i18n.php? [15:21:43] aude: look at tin:/tmp/l10nupdate.log-20140407 for the whole extracted log [15:21:51] ok [15:23:58] seems to be using extension list from wmf20? [15:25:01] or not even, it's using the extension messages file form wmf20 [15:28:22] (03CR) 10Aklapper: "I only quickly tested this inline in Firefox by using the Developer tools (right-click on element and select "Inspect Element", quick and " [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/124140 (owner: 1001tonythomas) [15:31:42] Another strange thing in that log is that the git update that ran today was still picking up new wmf/1.23wmf21 for some extensions (CreditsSource, EventLogging, GeoCrumbs, OATHAuth, PageImages, PagedTiffHandler, ProofreadPage, Quiz, VectorBeta). [15:32:11] Those branches should have been there since Thursday shouldn't they? [15:32:29] * aude puzzled [15:36:26] (03CR) 10MarkTraceur: "To avoid misleading information: I didn't test, Andre." [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/124140 (owner: 1001tonythomas) [15:38:41] did the other extensions that migrated to json leave the old *.i18n.php files there? [15:39:06] we really don't care at this point about b/w compat so they are gone from wikibase [15:40:04] * bd808 thinks that's a question for siebrand ^ [15:40:23] i think he was ok with us removign the, [15:40:25] them* [15:40:30] aude: Most have. It depends on if you want to provide B/C. [15:40:42] then shouldn't be a problem for localisation update [15:40:46] aude: For example MobileFrontend doesn't have a shim [15:41:21] aude: I'm not sure if JSON i18n LUs to old style i18n. [15:41:36] aude: In any case, that should be resolved within a short while (~2 weeks). [15:42:21] aude: If you remove the shim, don't forget to also remove the corresponding $wgExtensionMessagesFiles entry. [15:42:36] let me check [15:43:41] siebrand: In case you aren't in the loop on this, each time l10nupdate runs against the 1.23wmf21 branch we end up with a completely broken l10n cache. Even core messages are missing. Running a full scap (which includes mw-update-l10n) seems to fix the cache until the next time l10nupdate runs. [15:44:06] bd808: I've seen some posts to wikitech-l. [15:44:12] * bd808 nods [15:44:26] bd808: We've not had this issue on twn on MediaWiki "proper", so it's in the WMF specific infrastructure... [15:44:39] siebrand: see also this summary of the messages on engineering@ https://wikitech.wikimedia.org/wiki/Incident_documentation/20140403-Deploy [15:44:59] * greg-g waves to siebrand  [15:45:20] i am checking our stuff [15:45:42] (03CR) 10Hoo man: [C: 04-1] "We probably want this daily..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120535 (owner: 10Hoo man) [15:45:54] (03CR) 10Hoo man: "uhm, weekly I mean" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120535 (owner: 10Hoo man) [15:46:00] greg-g / bd808: I am totally unfamiliar with the WMF infrastructure. The only thing I know is that there's a script that builds a JSON file that is supposed to have references to the needed i18n message groups for core and all extensions deployed on any cluster wiki. [15:46:18] we have wgExtensionMessagesFiles for aliases and magic / namespaces [15:46:40] aude: Those should be kept. See my use of "corresponding". [15:46:42] then in our dependency components which still use the old format [15:46:45] siebrand: right [15:49:10] i don't see anything obviously wrong in our stuff [15:51:15] (03PS1) 10Ottomata: Setting up misc udp2log instance on analytics1003 to test moving sqstat there [operations/puppet] - 10https://gerrit.wikimedia.org/r/124341 [15:51:22] aude: I'm leaning towards it being a problem in our l10nupdate-1 script rather than any particular extension that has migrated. [15:51:50] i think so but have trouble understanding exactly where [15:52:13] and scared to touch the scripts (do they work on beta?) [15:52:26] That's a really good question [15:53:05] I don't know if l10nupdate runs there. We may just use mw-update-l10n [15:53:25] i know i ran the scripts there at some point [15:53:29] can't remember which ones [15:53:34] I think it doesn't as beta should update l10n when it updates core/extensions [15:54:32] Right. We don't have the need to pull in l10n changes separate from the master branch there [15:56:54] (03CR) 10Ottomata: [C: 032 V: 032] Setting up misc udp2log instance on analytics1003 to test moving sqstat there [operations/puppet] - 10https://gerrit.wikimedia.org/r/124341 (owner: 10Ottomata) [15:56:56] so.... is l10update-1 pulling in master to do updates? [15:57:13] i suppose not an issue for wikidata [15:58:06] but might be possible that master and deployment branch diverge at some point, when it comes to which i18n format is used [15:59:35] i see a copy is maintained in /var/lib/l10nupdate/mediawiki ? [16:00:16] l10nupdate keeps its own clone of mediawiki/core.git and mediawiki/extensions.git. On each run it updates those to the latest master with `git pull && git submodule update --init`. Then it uses the update.php script from the LocalisationUpdate extension to find new messages. [16:00:27] "New Wikidata Build - 06/04/2014 10:00" [16:00:29] that's master [16:00:51] ok and is robust to handle various formats that might diverge [16:01:04] (03PS1) 10Ottomata: Need to include udp2log class since ::misc doesn't inherit [operations/puppet] - 10https://gerrit.wikimedia.org/r/124343 [16:01:17] (03CR) 10Ottomata: [C: 032 V: 032] Need to include udp2log class since ::misc doesn't inherit [operations/puppet] - 10https://gerrit.wikimedia.org/r/124343 (owner: 10Ottomata) [16:01:43] Well… that's what's under debate now I think. It seemed to work of for 1.23wmf20 (although with an initial hiccup) [16:02:02] But it seems to repeatedly break on 1.23wmf21 [16:02:07] i do think what's in our master there is the same, though if it's using "Wikidata" [16:02:36] if they use the extensions individually (e.g. Wikibase or DataValues stuff), then they might diverge [16:03:52] ottomata: did you see my patch for stat-wm-o ? [16:04:35] (03CR) 10Lydia Pintscher: [C: 04-1] "IMHO we should not do this since page moves for example should not be propagated from test.wikipedia.org to wikidata.org." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124321 (owner: 10Aude) [16:04:37] bd808: if we pointed wikidata in wmf21 at the older version of our stuff, then does l10update run? [16:05:22] (03CR) 10Aude: "wmgWikibasePropagateChangesToRepo = false for testwiki" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124321 (owner: 10Aude) [16:06:17] (03PS1) 10Ottomata: Including standard and admins::roots on analytics1003 [operations/puppet] - 10https://gerrit.wikimedia.org/r/124345 [16:06:19] I'm going to make a patch to l10nupdate-1 that allows running it with more verbose output. That may or may not lead us to see where things go off the rails [16:06:42] matanya: yaaaaa, but i haven't really looked at it yet [16:07:01] just checking. in your spare time [16:07:23] (03CR) 10Aude: "when https://bugzilla.wikimedia.org/show_bug.cgi?id=63623 is fixed, then maybe we can consider changing this back to *not* use wikidata" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124321 (owner: 10Aude) [16:08:50] (03CR) 10Ottomata: [C: 032 V: 032] Including standard and admins::roots on analytics1003 [operations/puppet] - 10https://gerrit.wikimedia.org/r/124345 (owner: 10Ottomata) [16:36:04] PROBLEM - udp2log log age for misc on analytics1003 is CRITICAL: NRPE: Command check_udp2log_log_age-misc not defined [16:36:42] psshhh [16:36:43] whatever [16:36:46] there are no logs to check! [16:37:28] (03PS1) 10Ottomata: Not monitoring logs on analytics1003 misc udp2log instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/124349 [16:38:25] (03CR) 10Ottomata: [C: 032 V: 032] Not monitoring logs on analytics1003 misc udp2log instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/124349 (owner: 10Ottomata) [16:40:51] (03PS1) 10BryanDavis: l10nupdate: add support for --verbose flag [operations/puppet] - 10https://gerrit.wikimedia.org/r/124350 [16:58:24] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx [17:04:10] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:04:10] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:04:10] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:04:10] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:05:47] (03PS1) 10Ottomata: Adding swalling, maryana and jforrester to bast1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/124354 [17:07:08] ahh poo [17:07:16] i think analytics ACLs are messing with sqstat now [17:07:17] :/ [17:10:28] (03PS1) 10Ottomata: Moving sqstat filter back to erbium until analytics can talk to statsd [operations/puppet] - 10https://gerrit.wikimedia.org/r/124355 [17:10:45] (03CR) 10Ottomata: [C: 032 V: 032] Moving sqstat filter back to erbium until analytics can talk to statsd [operations/puppet] - 10https://gerrit.wikimedia.org/r/124355 (owner: 10Ottomata) [17:11:23] (03PS2) 10Ottomata: Adding swalling, maryana and jforrester to bast1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/124354 [17:11:30] (03CR) 10Ottomata: [C: 032 V: 032] Adding swalling, maryana and jforrester to bast1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/124354 (owner: 10Ottomata) [17:12:52] greg-g: I have a patch in gerrit to increase logging verbosity for l10nupdate. Who should we poke to merge? [17:13:01] greg-g: https://gerrit.wikimedia.org/r/#/c/124350/ [17:13:08] bd808: looks good to me [17:13:13] I should merge? [17:13:22] Yes please. :) [17:13:33] (03CR) 10Greg Grossmeier: [C: 031] "Yes please." [operations/puppet] - 10https://gerrit.wikimedia.org/r/124350 (owner: 10BryanDavis) [17:13:35] (03PS2) 10Ottomata: l10nupdate: add support for --verbose flag [operations/puppet] - 10https://gerrit.wikimedia.org/r/124350 (owner: 10BryanDavis) [17:13:38] (03CR) 10Ottomata: [C: 032 V: 032] l10nupdate: add support for --verbose flag [operations/puppet] - 10https://gerrit.wikimedia.org/r/124350 (owner: 10BryanDavis) [17:13:41] haha [17:13:50] I'm hoping this may give us some new clues about what's breaking [17:13:52] the "yes pleaes" jinx was unintentional [17:13:54] done [17:14:14] ty ottomata [17:14:24] yw [17:14:30] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [17:14:35] ottomata: Can you force that to apply on tin as well? [17:14:44] sure [17:18:22] bd808: done [17:18:38] Thank you ottomata [17:21:30] greg-g: What are your feelings about me running l10nupdate now with the louder logging to see what it says? testwiki is the only wiki still on 1.23wmf21. [17:22:58] bd808: doit [17:24:20] !log Manually running l10nupdate with new --verbose flag to capture log output [17:24:25] Logged the message, Master [17:27:03] (03CR) 10Siebrand: [C: 031] Fixup whitespace of jOrgChart.js [operations/software] - 10https://gerrit.wikimedia.org/r/118953 (owner: 10Reedy) [17:31:28] (03CR) 10Siebrand: [C: 031] Simplify boolean return [operations/debs/adminbot] - 10https://gerrit.wikimedia.org/r/121973 (owner: 10Reedy) [17:36:02] !log LocalisationUpdate completed (1.23wmf20) at 2014-04-07 17:36:02+00:00 [17:36:07] Logged the message, Master [17:39:12] one down [17:40:55] * bd808 doesn't think --verbose is going to tell us anything new based on what he's seen so far :/ [18:03:48] !log LocalisationUpdate completed (1.23wmf21) at 2014-04-07 18:03:48+00:00 [18:03:53] Logged the message, Master [18:05:02] what?! [18:05:08] why is testwiki working now [18:06:53] * bd808 is just as confused as greg-g  [18:07:21] heisenbug [18:07:42] we tried to know more about it, so it disappared [18:09:37] So one difference is that this was run by logging in as me and calling the wrapper script that uses sudo to run l10nupdate-1 instead of cron running l10nupdate-1 as the l10nupdate user directly. [18:10:05] * greg-g sighs [18:11:19] * bd808 waits for the resource loader cache purge to finish before passing judgement [18:13:38] !log shwiki queue finished emptying out in staggered loop on terbium [18:13:44] Logged the message, Master [18:23:13] (03CR) 10Aaron Schulz: [C: 032] Added "downloadtiff" pool counter config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122437 (owner: 10Aaron Schulz) [18:23:30] (03Merged) 10jenkins-bot: Added "downloadtiff" pool counter config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122437 (owner: 10Aaron Schulz) [18:23:54] !log aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Added "downloadtiff" pool counter config' [18:23:59] Logged the message, Master [18:27:18] hrm, someone keeps triggered Title::countRevisionsBetween timeouts on https://en.wikipedia.org/wiki/Main_Page?curid=536018 [18:27:29] that really needs a LIMIT [18:38:18] (03CR) 10Ori.livneh: [C: 032] "Doesn't apply the check on LVS servers, so it's safe." [operations/puppet] - 10https://gerrit.wikimedia.org/r/111163 (owner: 10Ori.livneh) [18:39:25] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 7 18:39:21 UTC 2014 (duration 15m 30s) [18:39:30] Logged the message, Master [18:44:30] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [18:44:54] lots o' fatals: https://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&title=MediaWiki+errors&vl=errors+%2F+sec&x=0.5&n=&hreg[]=vanadium.eqiad.wmnet&mreg[]=fatal|exception>ype=stack&glegend=show&aggregate=1&embed=1 [18:46:25] navigating to https://en.wikipedia.org/w/index.php?title=Barack_Obama&oldid=256170852 causes a fatal: [18:46:25] Fatal error: Allowed memory size of 230686720 bytes exhausted (tried to allocate 172165567 bytes) at /usr/local/apache/common-local/php-1.23wmf20/includes/parser/StripState.php on line 123 [18:51:34] lol [18:51:44] time to raise the limit again? [18:51:56] to ~0... [18:54:45] hoo, you mean to an eight laying on its side? [18:55:25] MaxSem: You can never have enough memory... right? [18:56:26] We should order the boxes HP developed for SAP Hana... 12Tb or ram... enough for wikitext?" [18:56:35] * of [18:56:37] * ?! [18:58:45] or just switch to hhvm [18:59:03] (03PS1) 10Ori.livneh: HHVM on beta: Override $wgServer and $wgCanonicalServer [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124379 [18:59:45] That's nowhere near as funny... [19:01:53] (03CR) 10Ori.livneh: [C: 032] HHVM on beta: Override $wgServer and $wgCanonicalServer [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124379 (owner: 10Ori.livneh) [19:05:59] (03PS1) 10Ottomata: Adding + 2 Hadoop journalnodes in Row D [operations/puppet] - 10https://gerrit.wikimedia.org/r/124380 [19:06:19] (03PS2) 10Ottomata: Adding + 2 Hadoop journalnodes in Row D [operations/puppet] - 10https://gerrit.wikimedia.org/r/124380 [19:06:40] (03PS3) 10Ottomata: Adding + 2 Hadoop journalnodes in Row D [operations/puppet] - 10https://gerrit.wikimedia.org/r/124380 [19:06:47] (03CR) 10Ottomata: [C: 032 V: 032] Adding + 2 Hadoop journalnodes in Row D [operations/puppet] - 10https://gerrit.wikimedia.org/r/124380 (owner: 10Ottomata) [19:08:43] temporatily disabling puppet on analytics 1009, 1010, 1019, 1020 to bring up new journalnodes [19:08:51] !log temporatily disabling puppet on analytics 1009, 1010, 1019, 1020 to bring up new journalnodes [19:08:51] 3:08 [19:08:55] Logged the message, Master [19:26:51] !log temporarily stopping journalnode on analytics1011 to copy journaldir to analytics1019 and analytics1020 [19:36:33] (03Abandoned) 10Ori.livneh: Add EventLogging Kafka writer plug-in [operations/puppet] - 10https://gerrit.wikimedia.org/r/85337 (owner: 10Ori.livneh) [19:39:47] dawww, abandone ori? [19:40:34] ottomata: mostly because i didn't want to pile work on you. should i just start writing to hadoop? [19:40:37] i'd be pretty happy to [19:41:21] hm, i guess i thought that patch needed more work but i think it's good to go, i just hadn't realized [19:41:27] (03Restored) 10Ori.livneh: Add EventLogging Kafka writer plug-in [operations/puppet] - 10https://gerrit.wikimedia.org/r/85337 (owner: 10Ori.livneh) [19:41:32] (03PS2) 10Ori.livneh: Add EventLogging Kafka writer plug-in [operations/puppet] - 10https://gerrit.wikimedia.org/r/85337 [19:41:53] yeah, ori, we already packaged the python-kafka thing [19:42:01] that's the last we talked abou tit [19:42:05] ottomata: k, let me uncomment and let's go for it [19:42:15] cool, wait, what are we going to write? eventlogging logs to kafka? [19:42:17] that woudl be cool! [19:42:35] yep [19:42:40] as is, that patch just installs the plugin, right? [19:43:00] probably will nee to add package { 'python-kafka': … } eh [19:43:00] ? [19:43:29] ottomata: yes, also -- what's the kafka host/port it should connect to? [19:43:37] hmmmm, [19:43:58] what is the hostname arg to KafkaClient? [19:44:01] is it an array? [19:44:22] usually you give the list of kafkabrokers so it can query any of them for metadata [19:44:38] it then will automatically connect to the proper brokers for the appropriate topic-partitions [19:44:58] anyway, if you include role::analyitcs::kafka;:config [19:45:07] you should be able to get the hostname(s) out of variable there [19:45:36] either $brokers (a hash) or $brokers_array (array) [19:46:03] ottomata: yep, you can pass a list of hosts [19:46:08] i'll update the patch to do that [19:46:17] k cool [19:47:04] what should the topic be? [19:47:06] eventlogging? [19:47:08] i suppose? [19:47:24] I've been realizing recently that I shoudl start numbering topics [19:47:28] (or versioning?) [19:47:45] i've got a use case coming up where I'm going to have to rename my topics, in order to change the replica setting on them [19:47:46] :/ [19:47:48] so, um [19:47:51] evenlogging01? [19:48:00] eventlogging-01? [19:48:04] ergh, i don't like it [19:48:08] which I didn't have to do that [19:48:08] hm [19:48:11] wich [19:48:12] wish* [19:48:13] bah [19:49:21] ottomata: i don't see the package, btw [19:51:15] hm [19:51:56] bah, where'd it go?hm [19:52:00] maybe we didn't add it to apt...looking [19:57:51] greg-g: I'm not sure what to say about the l10nupdate problems. The run at 02:00 UTC today was a laundry list of warnings. The run I did at 17:23 UTC had warnings about 3 extensions (CentralNotice, MultimediaViewer & ZeroRatedMobileAccess). All the happened in between AFAIK is Sam's scap at 11:19 UTC. [19:59:11] testwiki looks unbroken to me at the moment, but I'm not making any bets about what will happen at 02:00 UTC tomorrow [19:59:14] bd808: so I guess the next test is to kick the cronjob to run early so that the permissions/users are the same? [19:59:56] Yeah we could get a root to do that. That would save us waiting 6 hours to see what happens [20:00:02] yeah [20:00:51] (03PS1) 10Ottomata: 0.8.0-1 release [operations/debs/python-kafka] (debian) - 10https://gerrit.wikimedia.org/r/124390 [20:01:03] (03CR) 10Ottomata: [C: 032 V: 032] 0.8.0-1 release [operations/debs/python-kafka] (debian) - 10https://gerrit.wikimedia.org/r/124390 (owner: 10Ottomata) [20:01:40] I suppose ottomata and ori are our two root choices today [20:03:24] ottomata: around ? [20:03:43] jaaa, gimme two mins [20:03:50] sure [20:04:38] ori, there we go [20:04:41] http://apt.wikimedia.org/wikimedia/pool/main/p/python-kafka/ [20:04:42] ook [20:04:59] matanya, bd808, here I am! [20:05:06] waassuppp [20:05:07] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [20:05:07] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [20:05:07] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [20:05:07] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [20:05:21] bd808 has priority over me [20:05:54] ottomata: greg-g and I were hoping you could make the l10nupdate-1 cron job run for us. [20:06:07] we use so many python modules we should probably use a pypi mirror :D [20:06:23] I ran the code manually and got a good result. We are not sure if this was a side effect of me runing via sudo or not [20:06:51] ok, where/how? [20:07:23] where == On tin as the l10nuser. [20:07:47] how == either a temp puppet patch or a manual messing with the crontab I guess [20:08:05] class misc::deployment::l10nupdate defines the existing job [20:08:08] ottomata: woot [20:08:26] i can manually run ja [20:09:20] We'd like to to trigger from cron to rule out any side effects from environmental settings that may bleed through from sudo [20:09:38] bd808, do this [20:09:39] tail -f /var/log/l10nupdatelog/l10nupdate.log [20:09:42] ready? [20:09:48] (03PS1) 10Odder: Modify wgAddGroups, wgRemoveGroups on brwikimedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124393 [20:10:01] Yup [20:11:21] k bd808 runniing [20:11:33] ottomata: Thanks. I see stuff logging [20:11:58] * bd808 settles in for the 25 minutes until something interesting gets logged [20:12:18] oh 25 mins, shoudla run in screeN! [20:12:24] naw it'll be fine :) [20:14:08] nohup :) [20:14:37] hashar: thanks again for puppet-lint on mwv + tox on el! :) [20:15:02] !! [20:15:33] I need to scale out jobs addition to moaaar people [20:16:28] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [20:17:27] matanya: wazsssuuup [20:17:34] hashar: we spoke about that :) [20:17:59] ottomata: regarding the spam in RT, can you please get rid of that? [20:18:11] I did pair with milimetric last week. Took him like 40 minutes to get all setup :] [20:18:20] great! [20:18:20] (oh also, bd808, want me to merge the eqiad labs secondary disk change) [20:18:21] ? [20:18:27] :P [20:18:33] i'm slow [20:18:34] ottomata: Sure :) [20:18:37] matanya: is that education & MOOC? [20:18:40] yes [20:18:46] k [20:18:58] and as well, please update on the status of stat1? [20:19:17] seems stat1003 is almost ready to take over, yeah? [20:19:42] yeah almost, many folks are using it [20:19:58] one of the very last blockers for tampa [20:19:59] we had base::firewall turned on last week, but it distrupted some folks beecause they didn't have bastion access [20:20:04] yeah, that and emery? [20:20:06] sqstat? [20:20:09] i tried moving sqstat this morning [20:20:11] bd808: did we ever find the problem with l10update? [20:20:17] but the analytics cluster is having troulbe talking to statsd.eqiad.wmnet [20:20:19] so it doesn't work [20:20:30] yes, emery was my next question [20:20:38] we need a place to run sqstat that will work [20:20:39] aude: Not yet. We are trying another run right now actually. [20:20:44] ottomata: i noticed that on vanadium, asked faidon to fix it, and he did for that machine [20:20:46] ok [20:20:53] i was going to find out if it didn't work because of the multicast stream, or because of general business on erbium [20:21:01] was going to move it to a relatively powerful and unused analytics box [20:21:02] aude: So far the bug runs and hides anytime I look for it [20:21:03] and see if it worked [20:21:09] and maybe leave it there til it had ab etter home [20:21:10] ottomata: but now apt-get update on vanadium times out (can't connect to any of the apt repos) so caveat emptor [20:21:38] matanya: I reopened 4433 with some ACL changes that we need to be able to move sqstat off of erbium [20:21:56] hm, i was having troulb econnecting to apt repos from analytics nodes today too ori [20:22:07] but i thought something had gone wrong with the analytics ACLs.. [20:22:17] cld be, vanadiun is on the analytics vlan [20:22:26] it is? [20:22:50] yes, afaik [20:22:52] yeah, i had started setting up new journalnodes and couldn't get lvm! [20:23:01] so I manually copied over the .debs adn did what I needed for now [20:23:09] you could do that to ori (you got root there, ja?) [20:23:14] !log LocalisationUpdate completed (1.23wmf20) at 2014-04-07 20:23:14+00:00 [20:23:19] Logged the message, Master [20:23:21] ottomata: you know apt in in carbon, right ? [20:23:35] ottomata: i have root everywhere, hide yo kids [20:23:40] yes [20:23:45] hahah [20:26:40] ottomata: any eta on those ? [20:27:02] also 6144 should be updated :) [20:27:13] (03CR) 10Nikerabbit: "Hmm? Output from the scripts is fairly minimal:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/124350 (owner: 10BryanDavis) [20:28:52] Nikerabbit: Quite possibly --verbose will turn out to be useless but I wanted to try non-invasive changes first. [20:29:26] (03PS1) 10Hashar: webperf: couple lint issues in python scripts [operations/puppet] - 10https://gerrit.wikimedia.org/r/124395 [20:31:20] matanya: eta on what? [20:31:27] emery [20:31:32] no, need mark [20:32:45] thank you [20:32:50] (03PS6) 10Ottomata: Support eqiad labs secondary disk [operations/puppet] - 10https://gerrit.wikimedia.org/r/119534 (owner: 10BryanDavis) [20:32:55] (03CR) 10Ottomata: [C: 032 V: 032] Support eqiad labs secondary disk [operations/puppet] - 10https://gerrit.wikimedia.org/r/119534 (owner: 10BryanDavis) [20:33:46] matanya: re stats-wm-o [20:33:51] i'm so scared! [20:33:52] yes? [20:34:02] hehe [20:34:05] yeah, big time :) [20:34:22] still some directions will help :P [20:34:49] and I know I recommended of all the statistics.pp things, this would make a good module, and it would….>>>…buuuuuuuuuut, i'm left wondering if in this case it is worth it? [20:34:52] i can be convinced that it is [20:35:04] (03CR) 10BryanDavis: "Nikerabbit: It may turn out to be useless. I'm crossing my fingers that it will give slightly more context to what's happening when the l1" [operations/puppet] - 10https://gerrit.wikimedia.org/r/124350 (owner: 10BryanDavis) [20:35:14] it seems like a lot of work with a lot of risk for not much gain [20:36:22] making statistics.pp even a bit more readable is a huge gain [20:36:57] bd808: sure, though I have not found any reason to suspect LU is breaking anything [20:37:18] ok [20:37:21] matanya: looking at it now [20:37:21] so [20:37:29] the concept of a 'statistics' module at all is not a good one [20:37:48] the things in misc::statistics dont' necessarily have anything to do with each other [20:37:59] except that they share some common packages, and run on machines named stat* [20:38:13] so, if you want to modularlize stats.wikimedia.org [20:38:18] you should just do that [20:38:22] actually [20:38:28] Nikerabbit: Well… the symptoms are that testwiki has good l10n, l10nupdate runs, testwiki has bad l10n, someone runs scap and testwiki has good l10n. [20:38:29] a 'wikistats' module is probably what you want [20:38:42] wikistats is the code that is used to generate the site at stats.wikimedia.org [20:38:45] so [20:39:08] modules/wikistats/manifests [20:39:08] init.pp (packages? common stuff? maybe no init.pp at all?) [20:39:17] But I agree that I can't find a reason why this would be happening that I can support with evidence [20:39:19] site.pp (sets up apache and stats.wikimedia.org) [20:39:33] hm, maybe a [20:39:34] ottomata: i did want that name, but it is already taken ... [20:39:38] wha? [20:39:53] there is a module named wikistats [20:40:02] !log LocalisationUpdate completed (1.23wmf21) at 2014-04-07 20:40:02+00:00 [20:40:07] Logged the message, Master [20:40:11] alex offered me to use 'statistics' instead [20:40:41] and that is what this change is about [20:40:50] creating stats.wikimedia.org [20:40:57] as the commit message indicates [20:40:59] whaaat is this? [20:41:01] greg-g: Second manual run with l10n on testwiki looking ok [20:41:08] http://wikistats.wmflabs.org/ [20:41:15] a toollabs thing? [20:41:18] name collision!!!!! [20:41:20] bd808: ... which is quite logical if ExtensionMessages-X.php is incorrect [20:41:26] yup [20:42:01] oof [20:42:07] i think it is one of ori's baybies [20:42:19] LU would need to somehow prevent messages going to the LocalisationCache... which is quite difficult to do with array_merge( existing messages, new messages ) [20:43:08] ok so matanya, we need to figure out name, i think statistics is a bad one, but anyway [20:43:28] you've created a statistics::packages file which would be needed by other stat* machines [20:43:38] this module shoudl be for things pertaining to wikistats only [20:43:45] so, things that wikistats needs [20:43:47] Nikerabbit: Agreed, except I'm still confused about how it would be different for l10nupdate following a successful scap. l10nupdate doesn't seem to change it. Scap does but it does before it populates the l10n cache. Both scripts should see the same version of the file. [20:43:58] clone wikistats, run cron jobs, set up rsyncs (if needed? maybe not?) run the site [20:43:59] etc. [20:44:29] Nikerabbit: But today ExtensionMessages-1.23wmf21.php is definitely different than the version I saved over the weekend. [20:44:31] yeah, no docs whatsoever is needed :) hence all is guesses [20:44:52] did you check the code itself? [20:44:58] bd808: so something changed it again? [20:45:11] Sam ran a full scap this morning [20:45:33] what, wikistats? me? no way [20:45:48] it is a humungo perl thingee that Erik Z has been working on for eons! [20:45:53] bd808: yup, timestamp matches... and this time it seems to even have correct contents [20:46:02] * bd808 agrees [20:46:06] ottomata: sounds like a pain [20:46:07] since before there was an 'ops' team i betcha [20:46:18] * matanya shurgs [20:46:25] can't help it, ha? [20:46:32] yup, but it is the still the canonical source of wikimedia stats [20:46:43] we want to port much the webrequest stats it contains to hadoop generated stats [20:46:48] but, you know how long that is taking.... [20:46:49] heh [20:47:02] yes, sadly i do [20:47:03] and still, wikistats will probably live for a while, even after we get all the webrequests into hadoop [20:47:17] right now, erik zachte mnaully runs it once a month [20:47:25] and copies the new data and html files over to stat1 [20:47:36] i wouldn't want to waste efforts on stuff going to be removed in near future [20:47:39] ottomata> since before there was an 'ops' team i betcha <-- yes, like 5 years earlier ;) [20:47:40] soooo, you know, this is why I'm wondering if this is all worth it? [20:47:58] yes, i see your point [20:48:09] yeah, i mean i doubt we will remove it in the 'near future', [20:48:10] but yeah [20:48:12] well, i'll leave it there for now [20:48:28] ok thanks for looking into it though! [20:48:30] will think if i want to invest more time in that [20:48:54] i'm more concerned over tampa and puppet3 migrations [20:51:23] ugh, and I'm having 40%-50% packet loss to bast1001 [20:52:33] bd808: well... [20:52:54] greg-g: So… l10nupdate has run twice with successful outcome for 1.23wmf21. ExtensionMessages-1.23wmf21.php looks right since Sam's scap this morning. [20:53:29] huh [20:53:29] 8. ae1-103.stk10.ip4.tinet.net 0.0% 148 8.5 12.1 8.4 78.5 12.1 [20:53:32] 9. xe-11-2-1.was10.ip4.tinet.net 0.0% 148 129.4 134.3 127.8 283.4 16.3 [20:53:35] Sam's scap changed ExtensionMessages-1.23wmf21.php from the version I saved on Saturday morning [20:53:35] 10. xe-5-3-1.cr2-eqiad.wikimedia.org 46.9% 148 161.2 164.1 157.4 230.6 10.3 [20:54:30] greg-g: The version that was there on Saturday morning was very different and would have caused a lot of problems. [20:56:11] What I can't account for is how it broke at 2AM UTC Friday. After that I could imagine how I got broken by running mw-update-l10n outside of the full scap. I don't know why that would break things, but I can't rule it out yet. [20:57:26] bd808: I'm just making things up... but perhaps mergeMessageListFiles was called with wrong wiki (say, before any wiki was moved to that version)? [21:13:17] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 7 21:13:14 UTC 2014 (duration 1m 59s) [21:13:21] Logged the message, Master [21:15:29] ori: hey, around? [21:20:31] hoo: hey, in a meeting [21:20:34] hoo: what's up? [21:20:53] ori: mwgrep seems to only show incomplete results :/ [21:21:03] Which is a problem as we try to hunt down security issues [21:21:11] example? [21:21:26] hoo@terbium:~$ mwgrep SCloadGoogleLanguage [21:21:37] doesn't include bewiki, although there's https://be.wikipedia.org/wiki/MediaWiki:Gadget-GoogleTrans.js [21:22:14] hoo: could you file a bug and assign it to me? [21:22:54] hoo: meanwhile, for you: hhvm error on beta: http://p.defau.lt/?6I1zLCVqcM35UWeTkK1Whg :) [21:23:29] ori: Will do [21:23:36] hoo: thanks [21:26:53] vk.com ? [21:27:02] be.wiki is certainly a weird one [21:28:02] Nemo_bis: Like facebook :P They only link to that and don't do any evil stuffs with it, right? [21:28:28] dunno; linking is sitenotice always seems evil enough to me [21:28:53] There's also Facebook and Twitter... meh [21:39:49] (03PS1) 10BryanDavis: l10nupdate: Add temporary debugging captures [operations/puppet] - 10https://gerrit.wikimedia.org/r/124467 [21:46:10] (03PS2) 10Matanya: decom : brewster [operations/puppet] - 10https://gerrit.wikimedia.org/r/123626 [21:49:43] (03PS2) 10BryanDavis: l10nupdate: Add temporary debugging captures [operations/puppet] - 10https://gerrit.wikimedia.org/r/124467 [21:53:00] (03CR) 10Ori.livneh: [C: 032] webperf: couple lint issues in python scripts [operations/puppet] - 10https://gerrit.wikimedia.org/r/124395 (owner: 10Hashar) [21:54:01] greg-g: So what's going to happen to wmf21? [21:54:10] I ask because I'm wondering whether it's worth backporting things to it [21:54:28] If wmf21 is going to be skipped and we're gonna go from wmf20 to wmf22, then backporting regression fixes is pointless [21:54:44] RoanKattouw: it won't be left behind, but it won't be pushed out anymore today, we'll reassess after tonight's auto l10nupdate [21:54:54] OK [21:55:00] right, I want to delay my answer until 3amUTC :) [21:55:16] So if something works in wmf20, is broken in wmf21 and fixed in master, should I still backport the fix to wmf21? [21:55:28] I would for now, assuming the best [21:55:31] I guess that's equivalent to asking whether wmf21 will ever run on more than testwiki [21:55:32] OK [21:55:50] I have high hopes that it'll be fixed by tomorrow afternoon [21:55:57] Cool [21:56:06] * greg-g has faith in bd808  [21:56:30] ah, bd808, btw that update is done [21:56:56] ottomata: Yeah, thanks. Prepping for the next round of tests. [21:59:52] * hoo kills php with a rusty knife [22:00:31] well, this is probably actually hhvm following the documentation to close :P [22:01:17] greg-g: clear to scap a couple of times? [22:01:53] greg-g: https://gerrit.wikimedia.org/r/#/c/124387/ merged and ready for SWAT. I will add to wikitech docs. [22:01:57] Thanks MatmaRex ^ [22:02:43] I think Isarra would want to have a say in this. [22:02:51] And ^d [22:03:16] bd808: yeah [22:03:21] ori: https://github.com/facebook/hhvm/issues/2181 that's the cause of the Wikibase bug [22:03:43] RoanKattouw: At this point I think the l10n problems have likely been caused by bad ExtensionMessages files. What I don't know yet is how they are being messed up. [22:04:00] greg-g: Awesome. [22:04:06] !log bd808 Started scap: Testing 1.23wmf21 l10n changes [22:04:11] Logged the message, Master [22:04:29] Do we have any place where we collect upstream hhvm bugs? [22:04:42] MaxSem: ^ [22:05:22] odder: Can you write a better patch per your review? [22:05:34] Because that might be the most useful at this point. [22:06:16] hoo, deployment-bastion:/data/project/logs/hhvm*.log [22:06:28] err, bugs not logs [22:06:30] hoo: github.com/facebook/hhvm/issues/ :) [22:06:49] ebernhardson: :D ... I mean such which affect WMF ... [22:07:23] hoo: there is no tracking bug that i'm aware of [22:07:30] there's a keyword [22:07:52] it's a bit awkward, but i think filing it as a wikibase bug and linking to the upstream issue is probably the way to go [22:07:55] !log bd808 Finished scap: Testing 1.23wmf21 l10n changes (duration: 03m 49s) [22:08:00] Logged the message, Master [22:08:33] ori: mh... will someone then care about that or do we have to find someone to fix that? [22:09:17] hoo: yes, I regularly look at bugs with the hhvm keyword and I try to nudge things toward a fix [22:09:22] including pinging upstream if appropriate [22:10:07] ori: Ok, will open a bug [22:10:14] hoo: though if it's the sort of thing that could be fixed by simply explicitly checking for a null value, that may be warranted -- it doesn't sound like it would be a dirty workaround; instead it would be making something explicit [22:10:43] !log bd808 Started scap: Testing 1.23wmf21 l10n changes [22:11:00] ori: mh [22:11:18] The php manual is incorrect over here, which made me wtf at first (and complain about PHP) [22:11:41] scappy scap scap [22:11:53] right, so instead of relying on an idiosyncracy of the zend interpreter... i think explicitly handling a null might be better [22:12:02] greg-g: First run was great and had clean diffs [22:12:15] !log bd808 Finished scap: Testing 1.23wmf21 l10n changes (duration: 01m 31s) [22:12:19] Logged the message, Master [22:12:44] greg-g: second run clean diffs too. [22:12:48] wow, and fast [22:13:00] greg-g: Yeah. No changes makes things fast [22:13:02] but dangit [22:13:21] So … group0 to wmf21 and see if that changes anything? [22:13:32] * bd808 is running out of ideas [22:14:18] ori: Why not write a email about the current situation to Lydia_WMDE, so that we can coordinate HHVM testing etc. on our side? [22:14:39] for prioritization and stuff [22:15:03] hoo: it sounds like a good idea, but I think it's on us (=core mediawiki team) to have a better story for how to test with HHVM, first [22:15:42] ori: Ok, still want to drop an email and maybe name a time frame? [22:15:48] (03PS3) 10BryanDavis: l10nupdate: Add temporary debugging captures [operations/puppet] - 10https://gerrit.wikimedia.org/r/124467 [22:16:02] (03CR) 10Greg Grossmeier: [C: 031] "Consensus is there (linked from bug report). Simple enough." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121090 (owner: 10John F. Lewis) [22:17:27] bd808: try test2, not mw.org or testwikidata, just to spare them pain if it happens :) [22:17:38] * bd808 nods [22:18:05] https://gerrit.wikimedia.org/r/#/c/124475/ Isarra [22:18:10] hoo: sure, but I really wouldn't know what to say, other than link to a bug report [22:18:27] ori: Ok, skip it then... but please keep us up to date [22:18:47] We need like two weeks in advance for decent sprint planing [22:19:05] hoo: will do. thanks for being so responsive (on this and other matters). [22:19:46] (03PS1) 10BryanDavis: test2wiki to 1.23wmf21 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124478 [22:20:57] (03CR) 10BryanDavis: [C: 032] test2wiki to 1.23wmf21 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124478 (owner: 10BryanDavis) [22:22:34] * bd808 glares at jenkins [22:24:19] (03Merged) 10jenkins-bot: test2wiki to 1.23wmf21 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124478 (owner: 10BryanDavis) [22:24:33] * bd808 gives jenkins a cookie [22:26:19] !log bd808 Started scap: test2wiki to 1.23wmf21 [22:26:24] Logged the message, Master [22:29:05] odder: Thanks. Hopefully folks'll be able to work something sensible out. I'd say something about the headings, but that's probably a separate discussion [22:29:06] . [22:29:15] So leaving them be here is probably a good idea. [22:29:36] Whatever; the patch is already -2'd [22:29:47] I knew it was a total waste of my time, but since you asked, I just wanted to be nice. [22:30:27] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [22:30:33] ... [22:31:41] Not surprised with the -2, and I won't be surprised if it doesn't get done; clearly we want to promote Arial and Helvetica. [22:34:22] bd808: why does this one take so much longer? [22:34:30] * greg-g is watching the log [22:34:45] greg-g: This time it decided to update l10n files for 1.23wmf21 :/ [22:34:46] (the log you told me about using your scaplog.py) [22:35:04] All 366 lang files were touched [22:35:08] weee [22:35:17] Which makes no sense at all [22:35:24] yeah [22:35:36] The only change was to the wikiversions.json file [22:35:50] So this may actually tell us something [22:35:55] black boxes man, no one knows anything about them [22:36:36] (weak reference to global news stories) [22:36:45] I think I understand everything in the scap now *except* the l10n php code [22:36:55] right, always a time to learn :) [22:38:27] !log bd808 Finished scap: test2wiki to 1.23wmf21 (duration: 12m 07s) [22:38:31] Logged the message, Master [22:42:19] bd808: gdangit [22:42:47] greg-g: So… yeah [22:42:50] though i guess, scap usually works, now we wait until 6pm your time? or kick it again? [22:43:04] (20 minutes before SWAT, btw) [22:43:19] greg-g: !!! wait [22:43:32] ? [22:43:34] ExtensionMessages-1.23wmf21.php looks broken now [22:43:38] :) [22:43:55] so, an l10nupdate would break, theoretically [22:44:03] yeah [22:44:08] badly I presume [22:44:16] * bd808 looks at more diffs [22:44:20] :) [22:47:37] (03PS1) 10BryanDavis: Revert "test2wiki to 1.23wmf21" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124485 [22:48:45] (03CR) 10BryanDavis: [C: 032] "Rolling back to 1.23wmf21 after seeing that ExtensionMessages-1.23wmf21.php generation was broken when this was applied." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124485 (owner: 10BryanDavis) [22:48:52] (03Merged) 10jenkins-bot: Revert "test2wiki to 1.23wmf21" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/124485 (owner: 10BryanDavis) [22:49:37] !log bd808 Started scap: test2wiki to 1.23wmf20 [22:49:41] Logged the message, Master [22:51:22] greg-g: So working theory now is that when test2wiki is used as the wikidb to generate ExtensionMessages something bad happens. It looks like some part of the code used the 1.23wmf20 branch instead of 1.23wmf21. [22:52:12] greg-g: This scap is definitely rebuilding the l10n cache again. [22:52:43] weird... [22:54:01] bd808: test2wiki shouldn't be the root of anything critical (comment: !?!?!?!) [22:55:16] just it's listing in a loooong list of wikis that are a specific version [22:55:48] chrismcmahon: It really shouldn't matter what wikidb is picked for this particular task, but there may be something particularly wrong in this scenario [22:56:56] Selecting the wikidb to go with a version for generic tasks is code that I ported from php to python and I may have missed a subtle issue. [22:57:02] * bd808 goes to read more code [23:02:23] bd808: Are you still touching the cluster right now? [23:02:27] Cause it's SWAT time [23:02:46] RoanKattouw: it's back to what it was an hour ago, I think he's in deep thought mode [23:02:47] RoanKattouw: scap is still running. Probably 3-5 more minutes [23:02:50] oh [23:02:53] minus that [23:03:33] As soon as scap is done I will get out of your way for SWAT/LD [23:05:55] OK cool [23:06:02] I was distracted with other stuff anyway so it's all good [23:06:07] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [23:06:07] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [23:06:07] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [23:06:07] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [23:06:20] RoanKattouw: what's SWAT? [23:06:26] !log bd808 Finished scap: test2wiki to 1.23wmf20 (duration: 16m 48s) [23:06:31] Logged the message, Master [23:07:04] Jasper_Deng: https://wikitech.wikimedia.org/wiki/SWAT_deploys [23:07:08] RoanKattouw: All yours [23:07:34] greg-g: Also… ExtensionMessages is correct again. [23:07:48] bd808: does your python code not deal with numerals? [23:07:49] :P [23:08:00] So I think I know what's happening and probably how to stop it [23:08:19] but there's a time bomb lurking in there somewhere [23:08:52] so you're telling me we need a Bomb Squad in ADDITION to the SWAT Team now? [23:09:02] * greg-g stops, isn't helping [23:09:35] greg-g: Yes. Do you know the number for the closest bomb squad? [23:09:40] The old php code sorted the wikiverisons.json array (which is really a hash/dict) by value. Then it returns the first version=>wikidb pair for each unique version. [23:10:23] The python code doesn't have this sort and just returns the first arbitrary wikidb it finds for each unique version [23:10:46] when the wikidb value is test2wiki, something bad happens [23:11:01] Alright, SWAT time [23:11:30] (03CR) 10Catrope: [C: 032] Add Musées de la Haute-Saône to wgCopyUploadsDomains [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123459 (owner: 10Jean-Frédéric) [23:11:34] At least that's my working theory. I need to change the python code and test again after SWAT to validate. [23:11:44] (03CR) 10Catrope: [C: 032] Add importsources to enwikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121090 (owner: 10John F. Lewis) [23:12:45] greg-g / RoanKattouw: Could we possibly also drag https://gerrit.wikimedia.org/r/#/c/120595/ into SWAT if possible? :) [23:13:46] (03CR) 10Catrope: [C: 032] Activate the "other projects" sidebar managed by Wikibase in frwikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120595 (owner: 10Tpt) [23:14:13] RoanKattouw: feel free to hand any changes to me that you'd like me to deploy [23:14:19] (03Merged) 10jenkins-bot: Add Musées de la Haute-Saône to wgCopyUploadsDomains [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123459 (owner: 10Jean-Frédéric) [23:14:20] JohnLewis: Adding [23:14:24] (03Merged) 10jenkins-bot: Add importsources to enwikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121090 (owner: 10John F. Lewis) [23:14:24] RoanKattouw: Perfect! :) [23:14:26] (03Merged) 10jenkins-bot: Activate the "other projects" sidebar managed by Wikibase in frwikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120595 (owner: 10Tpt) [23:14:45] ori: Today is a day with some complicated VE-related acrobatics so I'd prefer to roll solo today if that's OK [23:17:37] !log catrope synchronized wmf-config/InitialiseSettings.php 'SWAT changes: other projects bar on frwikisource, import sources' [23:17:42] Logged the message, Master [23:17:55] alright, I need to run and catch a bus, godspeed RoanKattouw, and feel free to do more tests after bd808 [23:18:04] s/after/after,/ [23:18:13] Safe travels gr [23:18:16] * greg-g [23:26:04] RoanKattouw: more than OK :) [23:31:05] so the OOM fatal spike for the past seven hours is primarily coming from requests for two URLs: [23:31:07] 2292 URL: http://en.wikipedia.org/w/index.php?title=Barack_Obama&oldid=256170852 [23:31:11] 140 URL: http://pl.wikipedia.org/wiki/Specjalna:Edytuj_obserwowane/raw [23:35:20] Hm 'PHP fatal error in /usr/local/apache/common-local/php-1.23wmf20/includes/parser/StripState.php line 123: Allowed memory size of 230686720 bytes exhausted (tried to allocate 172165197 bytes)' [23:36:06] SWAT update: I'm still wrestling various git things, this next VE thing is a bit complicated [23:41:47] Alright, here goes [23:41:57] jdlrobson: Now pushing your change out to wmf21 first [23:42:04] !log catrope synchronized php-1.23wmf21/skins/vector/variables.less 'Remove troublesome fonts from font stack' [23:42:09] Logged the message, Master [23:42:19] Which I think should be on testwiki [23:42:34] !log catrope synchronized php-1.23wmf21/resources/oojs-ui/ 'Update OOJS-UI for bug fixes' [23:42:39] Logged the message, Master [23:45:01] !log catrope synchronized php-1.23wmf21/extensions/VisualEditor/ 'VisualEditor bug fixes' [23:45:05] Logged the message, Master [23:49:52] !log catrope synchronized php-1.23wmf21/skins/vector/variables.less 'Remove troublesome fonts from font stack' [23:50:04] !log catrope synchronized php-1.23wmf21/extensions/VisualEditor/ 'VisualEditor bug fixes' [23:50:35] !log catrope synchronized php-1.23wmf21/resources/oojs-ui/ 'Update OOJS-UI for bug fixes' [23:53:33] Argh those fonts just don't seem to want to go away [23:53:44] Oh, hold on [23:53:49] Actually testing on a wmf21 wiki helps [23:53:56] OK taht worked [23:56:50] RoanKattouw: so the font change will only go out with 1.21? [23:57:49] !log catrope synchronized php-1.23wmf20/skins/vector/variables.less 'Remove troublesome fonts from font stack' [23:57:53] Logged the message, Master [23:58:06] Never mind :) [23:58:21] !log catrope synchronized php-1.23wmf20/extensions/VisualEditor/ 'VisualEditor bug fixes' [23:58:30] Logged the message, Master [23:58:34] StevenW: Sorry for the delay, I had some trouble getting it to work and making sure it was actually working [23:58:53] No problem. I saw the log roll in right after I asked. [23:58:59] Thanks for doing the swat deploy.