[00:10:59] <wikibugs>	 (03PS2) 10Gergő Tisza: Handle X-MediaWiki-Patrol-Status header in Varnish [puppet] - 10https://gerrit.wikimedia.org/r/402572 (https://phabricator.wikimedia.org/T167400)
[00:11:37] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Handle X-MediaWiki-Patrol-Status header in Varnish [puppet] - 10https://gerrit.wikimedia.org/r/402572 (https://phabricator.wikimedia.org/T167400) (owner: 10Gergő Tisza)
[00:15:49] <wikibugs>	 (03PS3) 10Gergő Tisza: Handle X-MediaWiki-Patrol-Status header in Varnish [puppet] - 10https://gerrit.wikimedia.org/r/402572 (https://phabricator.wikimedia.org/T167400)
[00:16:04] <wikibugs>	 (03CR) 10Gergő Tisza: Handle X-MediaWiki-Patrol-Status header in Varnish (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/402572 (https://phabricator.wikimedia.org/T167400) (owner: 10Gergő Tisza)
[00:22:46] <wikibugs>	 (03CR) 10MarcoAurelio: [C: 04-1] "Namespaces will display in English there. Are you sure about that?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/389433 (https://phabricator.wikimedia.org/T179807) (owner: 10Jayprakash12345)
[01:20:31] <wikibugs>	 (03PS1) 10Odder: Shrink favicon file sizes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402618 (https://phabricator.wikimedia.org/T177726)
[01:36:20] <wikibugs>	 (03PS2) 10Odder: Shrink favicon file sizes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402618 (https://phabricator.wikimedia.org/T177726)
[02:26:58] <icinga-wm>	 PROBLEM - Check health of redis instance on 6479 on rdb2005 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 127.0.0.1 on port 6479
[02:27:58] <icinga-wm>	 RECOVERY - Check health of redis instance on 6479 on rdb2005 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6479 has 1 databases (db0) with 3124931 keys, up 5 minutes 7 seconds - replication_delay is 0
[03:26:28] <icinga-wm>	 PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 777.89 seconds
[03:34:09] <icinga-wm>	 PROBLEM - puppet last run on mw2172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIPCity.dat.gz]
[03:34:38] <icinga-wm>	 PROBLEM - puppet last run on mw1309 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIP2-City.mmdb.gz]
[03:37:48] <icinga-wm>	 PROBLEM - puppet last run on mw2162 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIP2-City.mmdb.gz],File[/usr/share/GeoIP/GeoIP2-City.mmdb.test]
[04:00:38] <icinga-wm>	 RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 92.98 seconds
[04:02:48] <icinga-wm>	 RECOVERY - puppet last run on mw2162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[04:04:08] <icinga-wm>	 RECOVERY - puppet last run on mw2172 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[04:04:29] <icinga-wm>	 RECOVERY - puppet last run on mw1309 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures
[06:00:59] <icinga-wm>	 PROBLEM - Check Varnish expiry mailbox lag on cp4026 is CRITICAL: CRITICAL: expiry mailbox lag is 2116873
[06:29:49] <wikibugs>	 (03PS5) 10Jayprakash12345: Add Translation: namespace on Punjabi Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/389433 (https://phabricator.wikimedia.org/T179807)
[06:31:15] <wikibugs>	 (03CR) 10Jayprakash12345: "@MarcoAurelio I personally contact him, He answered at task." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/389433 (https://phabricator.wikimedia.org/T179807) (owner: 10Jayprakash12345)
[08:52:02] <elukey>	 !log re-enabled puppet on db110[78] - eventlogging_sync restarted on db1108 (analytics-slave) - T168414
[08:52:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:52:14] <stashbot>	 T168414: Purge all old data from EventLogging master - https://phabricator.wikimedia.org/T168414
[08:53:03] <elukey>	 eventlogging data has already caught up on db1107 (eventlogging master db), so now it is the turn of db1108 (analytics-slave) where eventlogging_sync needs to pull data from the master
[08:53:19] <elukey>	 it will take a couple of days probably
[09:13:58] <wikibugs>	 10Operations, 10Developer-Relations (Jan-Mar-2018), 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3881117 (10Qgil)
[09:14:35] <wikibugs>	 10Operations, 10Developer-Relations (Jan-Mar-2018), 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3771512 (10Qgil)
[09:24:30] <wikibugs>	 10Operations, 10Developer-Relations (Jan-Mar-2018), 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3881119 (10Qgil) SMTP is only needed for outbound messages.  Silly me, I hadn't check https. OK, good to know. Discourse has...
[09:29:22] <wikibugs>	 10Operations, 10Developer-Relations (Jan-Mar-2018), 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3881123 (10Samwilson) I think the other installation does it in `templates/web.template.yml` (see T179649#3772701):   ``` +...
[11:08:31] <wikibugs>	 (03PS1) 10ArielGlenn: add dumps config settings stanzas for labs testing [puppet] - 10https://gerrit.wikimedia.org/r/402633
[11:08:55] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] add dumps config settings stanzas for labs testing [puppet] - 10https://gerrit.wikimedia.org/r/402633 (owner: 10ArielGlenn)
[11:12:31] <wikibugs>	 (03PS2) 10ArielGlenn: add dumps config settings stanzas for labs testing [puppet] - 10https://gerrit.wikimedia.org/r/402633
[11:12:53] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] add dumps config settings stanzas for labs testing [puppet] - 10https://gerrit.wikimedia.org/r/402633 (owner: 10ArielGlenn)
[11:15:37] <wikibugs>	 (03PS3) 10ArielGlenn: add dumps config settings stanzas for labs testing [puppet] - 10https://gerrit.wikimedia.org/r/402633
[11:38:35] <wikibugs>	 10Operations, 10Developer-Relations (Jan-Mar-2018), 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3881150 (10Qgil)
[11:41:02] <wikibugs>	 10Operations, 10Developer-Relations (Jan-Mar-2018), 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3881152 (10Qgil) Forced HTTPS works now. Thanks @Samwilson! Another detail that I would not have found alone.   Yes, I will d...
[12:04:56] <wikibugs>	 (03PS4) 10ArielGlenn: add dumps config settings stanzas for labs testing [puppet] - 10https://gerrit.wikimedia.org/r/402633
[12:11:31] <wikibugs>	 (03PS5) 10ArielGlenn: add dumps config settings stanzas for labs testing [puppet] - 10https://gerrit.wikimedia.org/r/402633
[12:15:22] <wikibugs>	 (03PS6) 10ArielGlenn: add dumps config settings stanzas for labs testing [puppet] - 10https://gerrit.wikimedia.org/r/402633
[12:20:55] <wikibugs>	 (03CR) 10ArielGlenn: [C: 032] add dumps config settings stanzas for labs testing [puppet] - 10https://gerrit.wikimedia.org/r/402633 (owner: 10ArielGlenn)
[12:31:01] <wikibugs>	 (03CR) 10ArielGlenn: [C: 031] "Someone please ping me on irc when you're about to merge this, I have it cherry-picked on the standalone dumps puppetmaster in beta and I'" [puppet] - 10https://gerrit.wikimedia.org/r/402432 (https://phabricator.wikimedia.org/T184239) (owner: 10Paladox)
[12:45:50] <wikibugs>	 (03PS1) 10Albert221: Remove 文A icon from Wikidata and MediaWiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402643 (https://phabricator.wikimedia.org/T152115)
[12:47:18] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Remove 文A icon from Wikidata and MediaWiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402643 (https://phabricator.wikimedia.org/T152115) (owner: 10Albert221)
[12:48:04] <wikibugs>	 (03PS2) 10Albert221: Remove language button from Wikidata and MediaWiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402643 (https://phabricator.wikimedia.org/T152115)
[12:49:34] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Remove language button from Wikidata and MediaWiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402643 (https://phabricator.wikimedia.org/T152115) (owner: 10Albert221)
[12:50:00] <wikibugs>	 (03PS3) 10Albert221: Remove language button from Wikidata and MediaWiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402643 (https://phabricator.wikimedia.org/T152115)
[13:01:01] <wikibugs>	 (03PS4) 10Albert221: Remove language button from Wikidata and MediaWiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402643 (https://phabricator.wikimedia.org/T152115)
[15:10:58] <wikibugs>	 (03CR) 10Dzahn: [C: 04-1] "there's a typo in it - "priviledges"" [puppet] - 10https://gerrit.wikimedia.org/r/399123 (https://phabricator.wikimedia.org/T183972) (owner: 10Chad)
[16:15:58] <icinga-wm>	 PROBLEM - puppet last run on mc1031 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:16:29] <icinga-wm>	 PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:16:38] <icinga-wm>	 PROBLEM - puppet last run on analytics1052 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:16:59] <icinga-wm>	 PROBLEM - puppet last run on elastic1023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:17:49] <icinga-wm>	 PROBLEM - puppet last run on kraz is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:18:09] <icinga-wm>	 PROBLEM - puppet last run on labvirt1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:18:09] <icinga-wm>	 PROBLEM - puppet last run on elastic1049 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:18:28] <icinga-wm>	 PROBLEM - puppet last run on restbase1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:18:29] <icinga-wm>	 PROBLEM - puppet last run on roentgenium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:18:29] <icinga-wm>	 PROBLEM - puppet last run on mw1269 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:18:33] <paladox>	 maybe puppetdb exploded ^^?
[16:18:49] <icinga-wm>	 PROBLEM - puppet last run on notebook1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:19:08] <icinga-wm>	 PROBLEM - puppet last run on mwdebug1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:19:16] <paladox>	 !ops ^^
[16:20:18] <icinga-wm>	 PROBLEM - puppet last run on kafka1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:21:32] <apergos>	 oom and restarted
[16:24:33] <apergos>	 the restart is automatic btw
[16:26:42] <wikibugs>	 10Operations, 10Developer-Relations (Jan-Mar-2018), 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3881338 (10Qgil)
[16:32:28] <wikibugs>	 10Operations, 10Developer-Relations (Jan-Mar-2018), 10cloud-services-team (Kanban): Create discourse-mediawiki.wmflabs.org (pilot instance) - https://phabricator.wikimedia.org/T180854#3881339 (10Qgil) Turns out that post-installation I could edit the SMTP parameters and just copy what discourse.wmflabs.org h...
[16:43:09] <icinga-wm>	 RECOVERY - puppet last run on elastic1049 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[16:43:28] <icinga-wm>	 RECOVERY - puppet last run on restbase1008 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[16:43:28] <icinga-wm>	 RECOVERY - puppet last run on mw1269 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[16:44:08] <icinga-wm>	 RECOVERY - puppet last run on mwdebug1002 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[16:45:18] <icinga-wm>	 RECOVERY - puppet last run on kafka1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:45:58] <icinga-wm>	 RECOVERY - puppet last run on mc1031 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[16:46:38] <icinga-wm>	 RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[16:46:38] <icinga-wm>	 RECOVERY - puppet last run on analytics1052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:46:59] <icinga-wm>	 RECOVERY - puppet last run on elastic1023 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[16:47:49] <icinga-wm>	 RECOVERY - puppet last run on kraz is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[16:48:09] <icinga-wm>	 RECOVERY - puppet last run on labvirt1004 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[16:48:25] <Platonides>	 paladox: not something I can help with...
[16:48:29] <icinga-wm>	 RECOVERY - puppet last run on roentgenium is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures
[16:48:41] <paladox>	 Platonides not sure what you mean.
[16:48:49] <icinga-wm>	 RECOVERY - puppet last run on notebook1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures
[16:59:41] <wikibugs>	 (03PS1) 10Urbanecm: Add inh to langs.tmpl [dns] - 10https://gerrit.wikimedia.org/r/402656 (https://phabricator.wikimedia.org/T184374)
[16:59:57] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add inh to langs.tmpl [dns] - 10https://gerrit.wikimedia.org/r/402656 (https://phabricator.wikimedia.org/T184374) (owner: 10Urbanecm)
[17:02:04] <Platonides>	 the bot pinged me due to your ops calll :P
[17:02:21] <wikibugs>	 (03PS2) 10Urbanecm: Add inh to langs.tmpl [dns] - 10https://gerrit.wikimedia.org/r/402656 (https://phabricator.wikimedia.org/T184374)
[17:02:38] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add inh to langs.tmpl [dns] - 10https://gerrit.wikimedia.org/r/402656 (https://phabricator.wikimedia.org/T184374) (owner: 10Urbanecm)
[17:03:34] <wikibugs>	 (03PS3) 10Urbanecm: Add inh to langs.tmpl [dns] - 10https://gerrit.wikimedia.org/r/402656 (https://phabricator.wikimedia.org/T184374)
[17:07:47] <wikibugs>	 (03PS1) 10ArielGlenn: fix up dblist and related config setting of bigwikis for dumps in labs [puppet] - 10https://gerrit.wikimedia.org/r/402657
[17:11:07] <wikibugs>	 (03PS1) 10Urbanecm: Initial configuration for inhwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402658 (https://phabricator.wikimedia.org/T184374)
[17:12:04] <wikibugs>	 (03CR) 10ArielGlenn: [C: 032] fix up dblist and related config setting of bigwikis for dumps in labs [puppet] - 10https://gerrit.wikimedia.org/r/402657 (owner: 10ArielGlenn)
[17:24:09] <icinga-wm>	 PROBLEM - nova-compute process on labvirt1017 is CRITICAL: PROCS CRITICAL: 2 processes with regex args ^/usr/bin/pytho[n] /usr/bin/nova-compute
[17:25:09] <icinga-wm>	 RECOVERY - nova-compute process on labvirt1017 is OK: PROCS OK: 1 process with regex args ^/usr/bin/pytho[n] /usr/bin/nova-compute
[17:41:32] <wikibugs>	 10Operations, 10Traffic, 10media-storage: "Error: 404, Requested domainname does not exist" when accessing Commons categories/images; works on mobile page - https://phabricator.wikimedia.org/T181801#3881431 (10Aklapper) 05Open>03stalled @Sumitsurai: Any news, following https://wikitech.wikimedia.org/wiki...
[17:51:34] <Sagan>	 paladox: !o.ps trigger notification for the channel operators, and that's why Platonides and me got pinged. but that's something for the server ops, not us
[17:51:43] <paladox>	 oh i see
[17:52:34] <Platonides>	 well, not a big issue
[17:52:42] <Platonides>	 I was just replying
[18:38:48] <HakanIST>	 Unknown database 'alswikibooks_p'
[18:40:30] <HakanIST>	 can't query  als.wikibooks.org on quarry
[18:41:13] <HakanIST>	 oh project is now a redirect
[18:43:27] <wikibugs>	 (03CR) 10Jdlrobson: "Wrong bug number..?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402643 (https://phabricator.wikimedia.org/T152115) (owner: 10Albert221)
[18:43:55] <andre__>	 HakanIST, not related to #operations I'd say?
[18:44:30] <HakanIST>	 andre__: yep
[18:55:05] <wikibugs>	 (03CR) 10MarcoAurelio: "Manual rebase required before merging." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/394846 (https://phabricator.wikimedia.org/T181923) (owner: 10MarcoAurelio)
[18:55:54] <Hauskatze>	 no_justification: hi, any chances that ^^ be done this week?
[18:56:12] <Hauskatze>	 HakanIST reports issues with tools fetching the sitematrix and failing due to missing replicas
[18:56:55] <Hauskatze>	 labs replicas were of course already deleted, but since the projects continue to exist in the background...
[18:59:12] <wikibugs>	 (03CR) 10MarcoAurelio: "Good to go then." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/389433 (https://phabricator.wikimedia.org/T179807) (owner: 10Jayprakash12345)
[19:01:04] <wikibugs>	 (03CR) 10Urbanecm: [C: 031] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/389433 (https://phabricator.wikimedia.org/T179807) (owner: 10Jayprakash12345)
[19:02:58] <wikibugs>	 (03PS6) 10Chad: Delete alswik(ibooks|iquote|tionary), mowik(ipedia|tionary) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/394846 (https://phabricator.wikimedia.org/T181923) (owner: 10MarcoAurelio)
[19:03:06] <no_justification>	 Hauskatze: I'm bored lets do it now
[19:03:43] <Hauskatze>	 no_justification: thanks
[19:03:52] <Hauskatze>	 beers are on mine afterwards :-)
[19:06:48] <wikibugs>	 (03CR) 10Chad: [C: 032] Delete alswik(ibooks|iquote|tionary), mowik(ipedia|tionary) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/394846 (https://phabricator.wikimedia.org/T181923) (owner: 10MarcoAurelio)
[19:06:52] <wikibugs>	 (03CR) 10Urbanecm: "Is it really needed to delete that wikis?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/394846 (https://phabricator.wikimedia.org/T181923) (owner: 10MarcoAurelio)
[19:07:31] <no_justification>	 Urbanecm: A little late to this one....
[19:07:35] <wikibugs>	 (03CR) 10Jayprakash12345: "In dblists/all.dblist, inhwiki should come before internalwiki." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402658 (https://phabricator.wikimedia.org/T184374) (owner: 10Urbanecm)
[19:07:49] <Urbanecm>	 no_justification, just noticed that chad CR+2ed it...
[19:07:58] <no_justification>	 I am Chad ;-)
[19:08:05] <Urbanecm>	 How can I know it? :D
[19:08:19] <wikibugs>	 (03Merged) 10jenkins-bot: Delete alswik(ibooks|iquote|tionary), mowik(ipedia|tionary) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/394846 (https://phabricator.wikimedia.org/T181923) (owner: 10MarcoAurelio)
[19:08:30] <no_justification>	 /whois no_justification
[19:08:31] <wikibugs>	 (03CR) 10jenkins-bot: Delete alswik(ibooks|iquote|tionary), mowik(ipedia|tionary) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/394846 (https://phabricator.wikimedia.org/T181923) (owner: 10MarcoAurelio)
[19:08:31] <no_justification>	 :p
[19:08:45] <Urbanecm>	 no_justification, do you really think that I whois everybody I talk with? ;)
[19:08:51] <no_justification>	 I do :p
[19:09:02] <no_justification>	 Anyway, most wikis aren't deleted, just closed.
[19:09:12] <no_justification>	 But this one has consensus for deletion
[19:09:12] <Urbanecm>	 That's why I wondered why those are deleted
[19:09:15] <no_justification>	 So here we go :)
[19:09:17] <Hauskatze>	 BoT orders so
[19:09:21] <Hauskatze>	 so we comply
[19:09:40] <Urbanecm>	 Hauskatze, what is BoT?
[19:09:45] <no_justification>	 Better phrasing is: board signed off on the community + langcom consensus :)
[19:09:46] <Hauskatze>	 Board of Trustees
[19:10:49] <logmsgbot>	 !log demon@tin Started scap: Delete alswik(ibooks|iquote|tionary), mowik(ipedia|tionary)
[19:10:59] <Hauskatze>	 eddiegp: ^^
[19:10:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:11:49] * eddiegp waves
[19:12:44] <no_justification>	 Once the scap is done, we should recompile the interwiki list
[19:12:50] <no_justification>	 It still has the now-deleted sites
[19:12:58] <no_justification>	 (I think the todo list lacks that step)
[19:13:04] <eddiegp>	 It does.
[19:13:21] <eddiegp>	 Which made me unsure whether I was supposed to remove the wiki from there by hand.
[19:13:23] <Hauskatze>	 dumpInterwiki.php can do so real quick :)
[19:13:35] <no_justification>	 I have a wrapper for dumpInterwiki that handles everything
[19:13:38] <no_justification>	 So I'll do it :)
[19:13:42] <Hauskatze>	 :)
[19:13:43] <eddiegp>	 :)
[19:13:58] * no_justification twiddles thumbs while L10n recompiles
[19:14:10] <Hauskatze>	 I don't think we list those wikis on m:Interwiki map
[19:14:17] * Hauskatze checks just to be sure
[19:16:19] <icinga-wm>	 PROBLEM - HHVM rendering on mw2128 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:16:54] <no_justification>	 Hauskatze: The meta page is just for additions
[19:17:09] <icinga-wm>	 RECOVERY - HHVM rendering on mw2128 is OK: HTTP OK: HTTP/1.1 200 OK - 74855 bytes in 0.309 second response time
[19:17:12] <no_justification>	 Well, should be. If meta's adding all wikis to that manually they're wasting time ;-)
[19:17:25] <no_justification>	 We handle all the stuff in all.dblist automatically in dumpInterwiki
[19:17:39] <Hauskatze>	 no_justification: yep, indeed; we just add there non-wmf sites
[19:17:50] <Hauskatze>	 mainly
[19:18:39] <Hauskatze>	 I guess (cu_changes table, rc_ip field of recentchanges isn't needed given that such data should be gone eddiegp ?
[19:19:31] <eddiegp>	 I think so, as it was closed for more than 30 days.
[19:19:54] <Hauskatze>	 90
[19:20:00] <Hauskatze>	 cu data expires at 90
[19:20:13] <eddiegp>	 Whatever, it's also closed for 90 days :P
[19:20:37] <Hauskatze>	 indee
[19:20:40] <Hauskatze>	 *indeed
[19:21:15] <no_justification>	 We'll still double check :)
[19:23:04] <eddiegp>	 I guess that "delete privacy data" would mean to TRUNCATE the table/UPDATE column=NULL, not actually "DROP DATABASE" or removing the column? The wiki page is a bit unclear on that part, but I'd bet the schema isn't meant to be changed at all.
[19:23:19] <no_justification>	 We'd truncate
[19:23:30] <no_justification>	 Cuz we leave the DBs in place, actually
[19:24:47] <eddiegp>	 I know the dbs stay, there was some far too long discussion/confusion about that. :D
[19:27:43] <no_justification>	 Centralauth cleanup I can do too, not many entries
[19:27:53] <no_justification>	 (sync-apaches at 99%)
[19:29:01] * Hauskatze crosses fingers
[19:32:21] <logmsgbot>	 !log demon@tin Finished scap: Delete alswik(ibooks|iquote|tionary), mowik(ipedia|tionary) (duration: 21m 32s)
[19:32:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:32:33] * no_justification looks for explosions
[19:34:54] <no_justification>	 CA tables cleaned up, globalusage tables had no relevant entries
[19:35:24] <Hauskatze>	 and jobqueue?
[19:35:53] <no_justification>	 Can't.
[19:35:57] <no_justification>	 Script it busted
[19:36:01] <no_justification>	 PHP Fatal error:  Class 'MediaWiki\MediaWikiServices' not found in /srv/mediawiki-staging/php-1.31.0-wmf.15/maintenance/Maintenance.php on line 385
[19:37:28] <Hauskatze>	 https://phabricator.wikimedia.org/diffusion/EWMA/browse/master/wmfManageJobs.php
[19:38:25] <no_justification>	 Maintenance script busted in master?
[19:38:31] <no_justification>	 (scripts, broadly?)
[19:38:38] <no_justification>	 PHP Fatal error:  Class 'MediaWiki\MediaWikiServices' not found in /srv/mediawiki-staging/php-1.31.0-wmf.15/maintenance/Maintenance.php on line 385
[19:41:16] <Hauskatze>	 I'll file a ticket for wmfManageJobs.php no_justification then
[19:41:31] <no_justification>	 I could maybeeeeee do it via eval.php, basically duplicate the logic...
[19:41:33] <no_justification>	 But that's ugly
[19:42:54] <no_justification>	 I'm wondering if we've got maintenance scripts broadly being busted
[19:43:41] <apergos>	 well the abstracts thing was specific to that one cleanup, someone trying to get rid of extra requires, moving classes into their own files
[19:43:54] <apergos>	 dunno if something like that happened for the manage jobs script
[19:43:55] <eddiegp>	 no_justification: I see master having a change around those line that .15 hasn't.
[19:44:07] <eddiegp>	 Looking what gerrit patch that was, but it looks exactly like what we'd want.
[19:44:09] <no_justification>	 Link?
[19:44:21] <eddiegp>	 if ( class_exists( MediaWikiServices::class ) ) { added that is
[19:44:38] <no_justification>	 Looks like a class_exists() hack to force loading of the class
[19:44:45] <no_justification>	 If autoloader is being dumb
[19:44:55] <no_justification>	 (this used to be a thing, iirc)
[19:45:02] <eddiegp>	 no_justification: https://gerrit.wikimedia.org/r/#/c/402138/
[19:45:09] <Hauskatze>	 T184385
[19:45:12] <stashbot>	 T184385: wmfManageJobs.php is broken - https://phabricator.wikimedia.org/T184385
[19:45:20] <Hauskatze>	 [ feel free to correct ]
[19:46:39] <no_justification>	 Let's try that
[19:46:59] <no_justification>	 apergos: I wonder if the issue in Abstracts, with ^^ fix would work again
[19:47:01] <no_justification>	 It seems...likely
[19:47:57] <apergos>	 looking
[19:50:58] <apergos>	 oh you mean to force class_exists() call someplace? maybe 
[19:51:10] <apergos>	 that's ugly but better than nothing
[19:51:39] <no_justification>	 Wait, nvm
[19:56:17] <logmsgbot>	 !log demon@tin Synchronized php-1.31.0-wmf.15/maintenance/Maintenance.php: fix stuff (duration: 00m 51s)
[19:56:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:57:21] <eddiegp>	 Yeah obvoiusly, for SAL, "fix stuff" is much more informative than "no justification given" :P
[19:58:04] <Hauskatze>	 hehehe
[19:58:57] <Hauskatze>	 dinner time
[19:59:00] <Hauskatze>	 bbl
[20:03:10] <apergos>	 eyeroll
[20:03:37] <no_justification>	 Of course it's more informative. I could've put "break stuff"
[20:03:53] <no_justification>	 That's already more differentiation than "no justification given"
[20:04:05] <no_justification>	 :p
[20:04:13] <apergos>	 well from that we would have known you broke stuff instead of fixing it
[20:04:20] <apergos>	 because there's always a justification for fixing stuff
[20:04:24] <apergos>	 and never for breaking stuff
[20:04:26] <apergos>	 :-P :-P :-P
[20:04:40] <p858snake>	 i don't know, breaking stuff can be pretty fun
[20:04:52] <eddiegp>	 So "break stuff" is actually synonymous to "no justification given"? :D
[20:05:05] <apergos>	 it might as well be :-p
[20:06:17] <no_justification>	 Sometimes you break stuff on purpose :)
[20:06:37] <no_justification>	 Interestingly: can't run wmfManageJobs with --delete if all job types don't implement a delete function
[20:06:59] <no_justification>	 Luckily it reports that it's going to delete 0 jobs beforehand, so you can just let it error out
[20:09:23] <p858snake>	 ugh i should get up, i'm opening at work this morning
[20:13:45] <no_justification>	 Ok, I think we're done here
[20:13:54] <no_justification>	 Oh yeah, interwiki list
[20:17:35] <wikibugs>	 (03PS1) 10Chad: updateinterwikicache: check_valid_syntax was moved to lint [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402663
[20:17:37] <wikibugs>	 (03CR) 10Chad: [C: 032] updateinterwikicache: check_valid_syntax was moved to lint [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402663 (owner: 10Chad)
[20:19:01] <wikibugs>	 (03Merged) 10jenkins-bot: updateinterwikicache: check_valid_syntax was moved to lint [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402663 (owner: 10Chad)
[20:19:11] <wikibugs>	 (03CR) 10jenkins-bot: updateinterwikicache: check_valid_syntax was moved to lint [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402663 (owner: 10Chad)
[20:19:34] <wikibugs>	 (03PS1) 10Chad: Updating interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402664
[20:19:37] <wikibugs>	 (03CR) 10Chad: [C: 032] Updating interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402664 (owner: 10Chad)
[20:21:13] <wikibugs>	 (03Merged) 10jenkins-bot: Updating interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402664 (owner: 10Chad)
[20:22:07] <wikibugs>	 (03CR) 10jenkins-bot: Updating interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402664 (owner: 10Chad)
[20:22:59] <icinga-wm>	 PROBLEM - Varnish HTTP text-backend - port 3128 on cp4029 is CRITICAL: connect to address 10.128.0.129 and port 3128: Connection refused
[20:23:59] <icinga-wm>	 RECOVERY - Varnish HTTP text-backend - port 3128 on cp4029 is OK: HTTP OK: HTTP/1.1 200 OK - 218 bytes in 0.157 second response time
[20:25:16] <logmsgbot>	 !log demon@tin Synchronized wmf-config/interwiki.php: auto-sync with my plugin was busted 🙃 (duration: 00m 50s)
[20:25:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:31:25] <eddiegp>	 no_justification: What's the scap command for your plugin updating the interwiki cache? Would be nice to have it in the wikitech docs.
[20:38:52] <no_justification>	 eddiegp: It should be in the wikitech docs :)
[20:39:13] <no_justification>	 `scap update-interwiki-cache`
[20:39:43] <eddiegp>	 Probably, I meant specifically the one for deleting a wiki :)
[20:39:54] <eddiegp>	 Thanks, I'm going to add it there.
[20:40:21] <no_justification>	 I lied
[20:40:22] <no_justification>	 https://wikitech.wikimedia.org/w/index.php?title=Update_the_interwiki_cache&type=revision&diff=1779673&oldid=1309257
[20:40:24] <no_justification>	 :)
[20:40:30] <no_justification>	 I'd just link to that page
[20:40:34] <no_justification>	 So we don't have to dupe docs
[20:40:41] <no_justification>	 * [[Update the interwiki cache]]
[20:43:27] <no_justification>	 Eh, did it myself :)
[20:45:16] <eddiegp>	 For whatever reason that didn't give me an edit conflict but decided my change was compatible with yours. That's awesome, I didn't know mediawiki can do this. :D
[20:47:47] <eddiegp>	 no_justification: So it seems we're done then, thanks again :)
[20:48:41] <bawolff>	 eddiegp: We use GNU diff3 to try and resolve edit conflicts. So if you edit a paragraph separate from the other person, and there is a paragraph between the paragraphs you edited, it should work out fine
[20:49:53] <eddiegp>	 Actually I edited another paragraph, plus the did the very same change Chad did. But that's proably equivalent to just changing another paragraph. :)
[20:53:50] <bawolff>	 This is also the reason that people who use mediawiki on windows tend to get edit conflicts everywhere, because GNU diff3 generally isn't installed on windows
[21:06:12] <wikibugs>	 10Operations, 10Puppet, 10Wikimedia-Apache-configuration, 10Wikimedia-Language-setup, and 2 others: Redirect several wikis - https://phabricator.wikimedia.org/T169450#3881625 (10EddieGP) a:03MarcoAurelio
[21:24:27] <wikibugs>	 (03Draft1) 10Paladox: WIP: Update gerrit login display [puppet] - 10https://gerrit.wikimedia.org/r/402665
[21:24:30] <wikibugs>	 (03PS2) 10Paladox: WIP: Update gerrit login display [puppet] - 10https://gerrit.wikimedia.org/r/402665
[22:03:02] <wikibugs>	 10Operations, 10Cloud-VPS, 10cloud-services-team: 2018-01-02: labstore Tools and Misc share very full - https://phabricator.wikimedia.org/T183920#3881655 (10notconfusing)
[22:03:05] <wikibugs>	 10Operations, 10Cloud-VPS, 10cloud-services-team: wikidumpparse is using 1.2TB of 5T available NFS misc storage - https://phabricator.wikimedia.org/T183970#3881653 (10notconfusing) 05Open>03Resolved OK, I'm now using <100G. Deleted weekly Wikidata dumps going back to 2015.
[22:03:23] <wikibugs>	 (03PS3) 10Paladox: WIP: Update gerrit login display [puppet] - 10https://gerrit.wikimedia.org/r/402665
[22:26:26] <wikibugs>	 (03PS5) 10Albert221: Remove language button from Wikidata and MediaWiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402643 (https://phabricator.wikimedia.org/T183665)
[22:26:31] <wikibugs>	 (03CR) 10Albert221: "oh, indeed, fixed! :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402643 (https://phabricator.wikimedia.org/T183665) (owner: 10Albert221)
[22:26:47] <wikibugs>	 (03PS4) 10Paladox: Update gerrit login display [puppet] - 10https://gerrit.wikimedia.org/r/402665
[22:29:09] <paladox>	 new user frendly login https://phabricator.wikimedia.org/F12304482 :)
[22:36:07] <Wiki13>	 paladox: looks nice
[22:40:41] <wikibugs>	 (03PS5) 10Paladox: Update gerrit login display [puppet] - 10https://gerrit.wikimedia.org/r/402665
[22:40:51] <paladox>	 Yep.
[22:41:49] <Wiki13>	 I serious went to look in the code where that background came from :P
[22:45:37] <paladox>	 Wiki13 https://gerrit.wikimedia.org/r/402665
[22:45:44] <paladox>	 it came from review.typo3.com/login/
[22:45:58] <paladox>	 i mean review.typo3.org/login/
[22:46:04] <wikibugs>	 (03PS6) 10Albert221: Remove language button from Wikidata and MediaWiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/402643 (https://phabricator.wikimedia.org/T183665)
[22:46:19] <wikibugs>	 (03PS6) 10Paladox: Update gerrit login display [puppet] - 10https://gerrit.wikimedia.org/r/402665
[22:46:31] <Wiki13>	 and the image came from commons I saw
[22:47:51] <paladox>	 Yep.
[22:48:57] <Krenair>	 paladox, if we have the opportunity it might be nice to put a bit of explanatory text
[22:49:05] <paladox>	 Ah ok.
[22:49:08] <paladox>	 will do that now.
[22:49:08] <Krenair>	 saying you need your LDAP CN and password, where to register those, etc.
[22:49:14] <Krenair>	 just a suggestion
[22:49:22] <paladox>	 Oh
[22:49:26] <paladox>	 there's a task for that
[22:49:33] <Krenair>	 cool
[22:49:45] <paladox>	 Krenair https://phabricator.wikimedia.org/T60205
[22:49:58] <paladox>	 it's not easy though as the html is built into gerrit.
[22:50:14] <icinga-wm>	 PROBLEM - MegaRAID on db1011 is CRITICAL: CRITICAL: 1 LD(s) must have write cache policy WriteBack, currently using: WriteThrough
[22:50:18] <paladox>	 im hoping with polygerrit as they move to remove GWTUI polygerrit will gain /login/ ui.
[22:51:07] <Krenair>	 they're ditching GWTUI?
[22:51:43] <paladox>	 Krenair yeh
[22:51:50] <paladox>	 Krenair it's being replaced with a new ui
[22:52:07] <paladox>	 there's a patch already for it's removal.
[22:52:20] <paladox>	 https://gerrit-review.googlesource.com/#/c/gerrit/+/116790/
[22:52:52] <Krenair>	 is it just me, or did gerrit's HTML output used to be obfuscated like crazy?
[22:52:59] <Krenair>	 I might be thinking of some other system
[22:53:29] <bawolff>	 didn't they used to use some sort of google compile java to html tool thing?
[22:53:47] <paladox>	 bawolf they do
[22:53:50] <paladox>	 soy
[22:54:03] <paladox>	 closure compiler
[22:54:12] <paladox>	 competer to velocity.
[22:54:32] <paladox>	 they used that in polygerrit + html templates.
[22:57:18] <paladox>	 Krenair anyways if you have any ui feedback for polygerrit, they have a team on it so you will likly get a reply. They also have someone behind the sceens listening to all feedback.
[22:58:37] <Krenair>	 where can I see a demo? do you have one in labs?
[22:59:11] <paladox>	 Yep
[22:59:23] <paladox>	 Krenair https://gerrit.git.wmflabs.org/r/?polygerrit=1
[22:59:41] <paladox>	 and the current updated one running on the very latest is https://gerrit-new.wmflabs.org/r/
[22:59:49] <paladox>	 https://gerrit-new.wmflabs.org/r/?polygerrit=1
[23:00:00] <paladox>	 Krenair it will eventually look like https://docs.google.com/presentation/d/17q-ygGioZi_5DITLyELa8oaOr22e15AHy8cq6XTZ0nY/edit#slide=id.g27f16618ec_0_139
[23:08:49] <paladox>	 Krenair polygerrit will be available on gerrit.wikimedia.org once we upgrade to 2.14 https://phabricator.wikimedia.org/T156120
[23:09:32] <Krenair>	 heh so we're doing another gerrit upgrade
[23:09:33] <Krenair>	 nice
[23:09:37] <Krenair>	 so much for migrating away from it
[23:09:43] <Krenair>	 (not that I'm complaining)
[23:11:00] <paladox>	 yep.
[23:11:56] <paladox>	 Krenair 2.15 introduces wip mode, and drops drafts (replaces it with private).
[23:12:29] <Krenair>	 'private'?
[23:12:33] <Krenair>	 what's that going to be like?
[23:13:20] <paladox>	 Krenair easier to switch the change from private to open and back to private if needed.
[23:13:30] <paladox>	 only reviewers can see the change.
[23:13:42] <Krenair>	 actual access controls on them?
[23:13:57] <paladox>	 Kind of, if a reviewer is added then they can see the change.
[23:14:05] <Krenair>	 no
[23:14:18] <Krenair>	 I mean can you see the private entries in the public copy of the repository
[23:15:10] <paladox>	 yes i think so.
[23:15:20] <Krenair>	 heh
[23:15:54] <Krenair>	 can you make private a change that's already been merged?
[23:16:48] <paladox>	 yes
[23:16:57] <paladox>	 i think so
[23:16:58] <Krenair>	 wow wtf really?
[23:17:11] <paladox>	 yep
[23:17:16] <paladox>	 see http://gerrit-new.wmflabs.org/r/c/avatar-gerrit/+/10
[23:17:27] <Krenair>	 that's a giant red flag
[23:17:39] <Krenair>	 what happens when you merge something marked private?
[23:18:31] <paladox>	 It mergesit aallows you to merge, but it will show up in diffusion / github
[23:18:36] <paladox>	 mergesit = merges
[23:18:41] <paladox>	 it
[23:18:57] <paladox>	 It's so you can merge a security patch
[23:19:21] <paladox>	 like gerrit had to do i think a month or two ago, though i carn't remeber the exact change.
[23:19:44] <Krenair>	 but the review comments do not get forcefully released?
[23:20:04] <paladox>	 nope.
[23:20:30] <Krenair>	 sigh
[23:20:43] <wikibugs>	 10Puppet, 10Beta-Cluster-Infrastructure: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing - https://phabricator.wikimedia.org/T184240#3881692 (10Krenair)
[23:21:22] <Krenair>	 I can see this being horrendously abused on wikimedia projects
[23:22:04] <paladox>	 https://gerrit-review.googlesource.com/Documentation/access-control.html#category_view_private_changes
[23:22:07] <paladox>	 Krenair there's ^^
[23:22:57] <wikibugs>	 10Puppet, 10Beta-Cluster-Infrastructure: Puppet broken on deployment-kafka-jumbo-[12] due to version of a package being missing - https://phabricator.wikimedia.org/T184240#3881693 (10Krenair) a:05Ottomata>03None I have a feeling that https://gerrit.wikimedia.org/r/402432 which we cherry-picked for T184239...
[23:23:17] <wikibugs>	 (03CR) 10Alex Monk: [C: 031] "May also fix T184240" [puppet] - 10https://gerrit.wikimedia.org/r/402432 (https://phabricator.wikimedia.org/T184239) (owner: 10Paladox)
[23:32:01] <wikibugs>	 10Puppet, 10Beta-Cluster-Infrastructure: Puppet broken on deployment-eventlogging04 due to missing repo on deployment-tin? - https://phabricator.wikimedia.org/T184238#3881706 (10Krenair) Probably not helping matters is puppet failing on deployment-tin - cf T184176
[23:35:43] <wikibugs>	 10Puppet, 10Beta-Cluster-Infrastructure: Puppet broken on deployment-eventlogging04 due to missing repo on deployment-tin? - https://phabricator.wikimedia.org/T184238#3881708 (10Krenair) have tried cloning to /srv/deployment/analytics/superset/deploy/ but then it needs some missing DEPLOY_HEAD file, no idea wh...
[23:46:35] <wikibugs>	 10Operations, 10Puppet, 10Beta-Cluster-Infrastructure, 10media-storage: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module - https://phabricator.wikimedia.org/T184236#3881709 (10Krenair) I think the code was always broken and it actually wanted to do this: ```lang=diff diff --git...