[00:00:07] (03Abandoned) 10Alexandros Kosiaris: grafana: Hieraize parameters [puppet] - 10https://gerrit.wikimedia.org/r/404309 (https://phabricator.wikimedia.org/T170150) (owner: 10Alexandros Kosiaris) [00:01:14] 10Operations, 10Phabricator, 10Patch-For-Review: Switch phabricator from using apache to nginx - https://phabricator.wikimedia.org/T185644#3923862 (10Paladox) p:05Triage>03High Phabricator needs to be restarted every week due to apache stalling every so often. So i am prioritising this as we shoulden't n... [00:11:26] 10Operations, 10Analytics-Data-Quality, 10Traffic: Vet reliability of the response_size field for data analysis purposes - https://phabricator.wikimedia.org/T185350#3923867 (10Tbayer) Thanks @BBlack , these assessments are really useful! It would still be interesting how issue 3. (weird video request pattern... [01:19:09] 10Operations, 10Performance-Team, 10Traffic: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3923968 (10Imarlier) @BBlack As promised, graph of frequency over time: https://logstash.wikimedia.org/app/kibana#/discover?_g=(refreshInterval:(display:Off,... [01:28:23] (03PS1) 10Rxy: Add 'rollbacker' group at arwikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406256 (https://phabricator.wikimedia.org/T185720) [02:32:42] !log bootstrapping restbase2009-b - T184100 [02:32:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:32:57] T184100: Reprovision legacy Cassandra nodes into new cluster - https://phabricator.wikimedia.org/T184100 [03:28:40] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 900.40 seconds [03:52:50] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 197.90 seconds [04:21:01] PROBLEM - HHVM jobrunner on mw1300 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 473 bytes in 0.001 second response time [04:22:01] RECOVERY - HHVM jobrunner on mw1300 is OK: HTTP OK: HTTP/1.1 200 OK - 206 bytes in 0.001 second response time [04:31:49] !log bootstrapping restbase2009-c - T184100 [04:32:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:32:03] T184100: Reprovision legacy Cassandra nodes into new cluster - https://phabricator.wikimedia.org/T184100 [04:44:17] (03CR) 10TerraCodes: Add 'rollbacker' group at arwikibooks (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406256 (https://phabricator.wikimedia.org/T185720) (owner: 10Rxy) [06:39:48] !log niharika29@tin Started deploy [scholarships/scholarships@5d2fca4]: Update deadline for scholarships application [06:39:50] !log niharika29@tin Finished deploy [scholarships/scholarships@5d2fca4]: Update deadline for scholarships application (duration: 00m 03s) [06:40:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:40:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:48:29] (03PS1) 10Urbanecm: Change logos of hi.wikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406263 (https://phabricator.wikimedia.org/T185347) [07:50:20] (03PS1) 10Urbanecm: Change wgMetaNamespace and wgSitename for hiwikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406264 (https://phabricator.wikimedia.org/T185347) [07:56:57] (03PS4) 10Urbanecm: Add "Portal" namespace on it.wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405024 (https://phabricator.wikimedia.org/T185232) (owner: 10Jayprakash12345) [07:57:33] (03PS3) 10Urbanecm: Allow bureaucrats to add/remove 'accountcreator' permission on Wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406012 (https://phabricator.wikimedia.org/T185597) (owner: 10Jayprakash12345) [07:58:34] (03CR) 10Urbanecm: [C: 031] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/405024 (https://phabricator.wikimedia.org/T185232) (owner: 10Jayprakash12345) [07:58:52] (03CR) 10Urbanecm: [C: 031] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406012 (https://phabricator.wikimedia.org/T185597) (owner: 10Jayprakash12345) [08:09:14] (03CR) 10Rxy: ">" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406256 (https://phabricator.wikimedia.org/T185720) (owner: 10Rxy) [08:15:03] (03CR) 10Rxy: [C: 031] Allow bureaucrats to add/remove 'accountcreator' permission on Wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406012 (https://phabricator.wikimedia.org/T185597) (owner: 10Jayprakash12345) [10:30:49] (03CR) 10TerraCodes: [C: 031] "> >" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406256 (https://phabricator.wikimedia.org/T185720) (owner: 10Rxy) [11:47:52] 10Operations, 10Ops-Access-Requests, 10Research, 10Research-collaborations, 10Patch-For-Review: Analytics cluster access request for ISI Foundation team - https://phabricator.wikimedia.org/T141634#3924377 (10Simonjoylet) @DarTar : I'd like to grant the access to the analytics cluster for a research proje... [12:12:45] (03PS1) 10WMDE-leszek: Labs: switch to new repository setting style for Wikibase client [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406276 (https://phabricator.wikimedia.org/T185200) [12:18:54] 10Operations, 10Ops-Access-Requests, 10Research, 10Research-collaborations, 10Patch-For-Review: Analytics cluster access request for ISI Foundation team - https://phabricator.wikimedia.org/T141634#3924422 (10MoritzMuehlenhoff) @Simonjoylet: Please create a separate access request ticket in Phabricator, s... [13:07:10] (03CR) 10Jayprakash12345: "Make alias as well, so that existing links don't break." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406264 (https://phabricator.wikimedia.org/T185347) (owner: 10Urbanecm) [13:07:28] (03CR) 10Jayprakash12345: [C: 031] Change logos of hi.wikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406263 (https://phabricator.wikimedia.org/T185347) (owner: 10Urbanecm) [13:16:12] 10Operations, 10Gerrit, 10Patch-For-Review, 10Release-Engineering-Team (Kanban): Investigate seemingly random Gerrit slow-downs - https://phabricator.wikimedia.org/T148478#3924456 (10Paladox) This feels like it’s mostly resolved now. [13:35:04] (03CR) 10Ladsgroup: [C: 032] Labs: switch to new repository setting style for Wikibase client [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406276 (https://phabricator.wikimedia.org/T185200) (owner: 10WMDE-leszek) [13:36:36] (03Merged) 10jenkins-bot: Labs: switch to new repository setting style for Wikibase client [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406276 (https://phabricator.wikimedia.org/T185200) (owner: 10WMDE-leszek) [13:37:17] (03CR) 10jenkins-bot: Labs: switch to new repository setting style for Wikibase client [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406276 (https://phabricator.wikimedia.org/T185200) (owner: 10WMDE-leszek) [13:38:43] ^ rebased on tin [16:24:36] (03PS1) 10Addshore: BETA: Enable FileImporter on testwiki with open config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406299 (https://phabricator.wikimedia.org/T185671) [16:26:34] (03CR) 10Addshore: [C: 032] BETA: Enable FileImporter on testwiki with open config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406299 (https://phabricator.wikimedia.org/T185671) (owner: 10Addshore) [16:26:40] (03PS2) 10Addshore: BETA: Enable FileImporter on testwiki with open config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406299 (https://phabricator.wikimedia.org/T185671) [16:26:43] (03CR) 10Addshore: [C: 032] BETA: Enable FileImporter on testwiki with open config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406299 (https://phabricator.wikimedia.org/T185671) (owner: 10Addshore) [16:28:17] (03Merged) 10jenkins-bot: BETA: Enable FileImporter on testwiki with open config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406299 (https://phabricator.wikimedia.org/T185671) (owner: 10Addshore) [16:28:43] (03CR) 10jenkins-bot: BETA: Enable FileImporter on testwiki with open config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406299 (https://phabricator.wikimedia.org/T185671) (owner: 10Addshore) [16:30:37] !log addshore@tin Synchronized wmf-config/CommonSettings-labs.php: BETA: Enable FileImporter on testwiki with open config PT 1/2 (duration: 00m 58s) [16:30:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:32:15] !log addshore@tin Synchronized wmf-config/InitialiseSettings-labs.php: BETA: Enable FileImporter on testwiki with open config PT 2/2 (duration: 00m 56s) [16:32:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:49:03] 10Operations, 10Analytics: setup/install eventlog1002.eqiad.wmnet - https://phabricator.wikimedia.org/T185667#3925028 (10RobH) a:05Ottomata>03RobH [17:53:56] 10Operations, 10cloud-services-team: silver: / partition low on space - https://phabricator.wikimedia.org/T151493#3925033 (10jcrespo) [18:32:30] (03CR) 10Alexandros Kosiaris: [C: 04-1] "This is better already, some more comments inline." (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/405591 (owner: 10Paladox) [18:34:38] (03CR) 10Reedy: ircecho: Support ssl when connecting to irc (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/405591 (owner: 10Paladox) [18:36:26] (03Abandoned) 10Alexandros Kosiaris: Test the future parser in puppet compiler [puppet] - 10https://gerrit.wikimedia.org/r/322898 (https://phabricator.wikimedia.org/T154915) (owner: 10Alexandros Kosiaris) [18:36:55] (03Abandoned) 10Alexandros Kosiaris: motd::script: Don't use validate_re on an integer [puppet] - 10https://gerrit.wikimedia.org/r/356483 (owner: 10Alexandros Kosiaris) [18:37:22] (03PS2) 10Alexandros Kosiaris: ganeti: Remove long absented resource [puppet] - 10https://gerrit.wikimedia.org/r/404978 [18:37:30] (03CR) 10Alexandros Kosiaris: [C: 032] ganeti: Remove long absented resource [puppet] - 10https://gerrit.wikimedia.org/r/404978 (owner: 10Alexandros Kosiaris) [18:45:01] 10Operations, 10Traffic, 10Accessibility, 10Browser-Support-Internet-Explorer: Wikipedia no longer accessible to those using some braille devices - https://phabricator.wikimedia.org/T185582#3920199 (10BBlack) From the #Traffic perspective, to recap and clarify some of the bits above from the general/broad... [20:59:58] (03PS7) 10Thcipriani: Scap canary: cache last good deploy time [puppet] - 10https://gerrit.wikimedia.org/r/403574 (https://phabricator.wikimedia.org/T183999) [21:23:16] 10Operations, 10cloud-services-team: silver: / partition low on space - https://phabricator.wikimedia.org/T151493#2819106 (10aborrero) >>! In T151493#3533580, @herron wrote: > In addition to the directories outlined in the description /usr/share/texlive appears to be large as well. Is texlive required for thi... [21:42:47] 10Operations, 10Wikimedia-Mailing-lists: Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 - https://phabricator.wikimedia.org/T52864#3925248 (10Legoktm) I briefly talked with @herron about this today. I think we are still blocked on the lack of Debian packaging. Looking at 10Operations, 10ops-eqiad, 10DBA, 10procurement: Rack and setup db1115 - https://phabricator.wikimedia.org/T185788#3925325 (10Marostegui) p:05Triage>03Normal [23:25:53] 10Operations, 10ops-eqiad, 10DBA, 10procurement: Rack and setup db1115 - https://phabricator.wikimedia.org/T185788#3925341 (10Marostegui) [23:26:20] 10Operations, 10ops-eqiad, 10DBA, 10procurement: Rack and setup db1115 (tendril replacement database) - https://phabricator.wikimedia.org/T185788#3925325 (10Marostegui) [23:48:46] (03PS1) 10Marostegui: install_server: Allow install db1115 [puppet] - 10https://gerrit.wikimedia.org/r/406375 (https://phabricator.wikimedia.org/T185788) [23:49:55] (03PS4) 10Dzahn: ci: replace apache with httpd for proxy/website [puppet] - 10https://gerrit.wikimedia.org/r/403730 [23:50:27] (03PS2) 10Marostegui: install_server: Allow install db1115 [puppet] - 10https://gerrit.wikimedia.org/r/406375 (https://phabricator.wikimedia.org/T185788) [23:53:00] (03CR) 10Dzahn: [C: 032] ci: replace apache with httpd for proxy/website [puppet] - 10https://gerrit.wikimedia.org/r/403730 (owner: 10Dzahn) [23:55:50] (03CR) 10Dzahn: "done. noop on contint1001/2001" [puppet] - 10https://gerrit.wikimedia.org/r/403730 (owner: 10Dzahn)