[02:29:29] !log l10nupdate@tin scap sync-l10n completed (1.30.0-wmf.15) (duration: 09m 36s) [02:29:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:36:24] !log l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 28 02:36:23 UTC 2017 (duration 6m 54s) [02:36:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:47:17] (03CR) 10Zoranzoki21: [C: 031] Make both LoginNotify email features default for Hewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374082 (https://phabricator.wikimedia.org/T174263) (owner: 10Samtar) [03:26:36] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 822.25 seconds [03:45:45] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 241.62 seconds [06:53:31] (03PS1) 10ArielGlenn: Advertise globalblocks table dumps on dataset hosts [puppet] - 10https://gerrit.wikimedia.org/r/374131 (https://phabricator.wikimedia.org/T173468) [06:54:32] (03CR) 10ArielGlenn: [C: 032] Advertise globalblocks table dumps on dataset hosts [puppet] - 10https://gerrit.wikimedia.org/r/374131 (https://phabricator.wikimedia.org/T173468) (owner: 10ArielGlenn) [06:55:57] (03PS1) 10Volans: CLI: fix --version option [software/cumin] - 10https://gerrit.wikimedia.org/r/374132 [06:55:59] (03PS1) 10Volans: Fix data_files installation directory [software/cumin] - 10https://gerrit.wikimedia.org/r/374133 (https://phabricator.wikimedia.org/T174008) [07:05:09] !log installing openjdk security updates on meitnerium [07:05:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:12:24] 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Kanban): setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3557575 (10mmodell) [07:12:46] !log installing openjdk security updates on notebook* hosts [07:12:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:14:05] PROBLEM - MegaRAID on db1055 is CRITICAL: CRITICAL: 1 LD(s) must have write cache policy WriteBack, currently using: WriteThrough [07:14:17] (03PS1) 10Muehlenhoff: Remove expiry date [puppet] - 10https://gerrit.wikimedia.org/r/374135 [07:14:38] (03CR) 10jerkins-bot: [V: 04-1] Remove expiry date [puppet] - 10https://gerrit.wikimedia.org/r/374135 (owner: 10Muehlenhoff) [07:18:58] (03PS2) 10Muehlenhoff: Remove expiry date [puppet] - 10https://gerrit.wikimedia.org/r/374135 [07:19:21] (03CR) 10jerkins-bot: [V: 04-1] Remove expiry date [puppet] - 10https://gerrit.wikimedia.org/r/374135 (owner: 10Muehlenhoff) [07:20:18] (03PS3) 10Muehlenhoff: Remove expiry date [puppet] - 10https://gerrit.wikimedia.org/r/374135 [07:21:31] 10Operations, 10ops-eqiad, 10DBA: BBU issues on db1055, RAID cache on WriteThrough - https://phabricator.wikimedia.org/T174265#3557580 (10Marostegui) This happened again, we definitely need to change the BBU //cc @Cmjohnson ``` root@db1055:~# megacli -AdpBbuCmd -a0 BBU status for Adapter: 0 BatteryType:... [07:22:06] * elukey looks for the morning alter tables from marostegui [07:22:13] !log Force re-learn cycle on db1055 - https://phabricator.wikimedia.org/T174265 [07:22:22] elukey: none today!! [07:22:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:22:38] ACtually, yes, I have to do some, but it is too late for them, as they will block a table [07:23:00] elukey: we do have a database alter pending though ;-) [07:24:08] marostegui: \o/ [07:24:09] :D [07:28:38] (03PS1) 10Marostegui: db-eqiad.php: Depool db1045 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374136 (https://phabricator.wikimedia.org/T172679) [07:30:06] (03CR) 10jerkins-bot: [V: 04-1] db-eqiad.php: Depool db1045 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374136 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [07:31:29] (03PS2) 10Marostegui: db-eqiad.php: Depool db1045 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374136 (https://phabricator.wikimedia.org/T172679) [07:34:05] RECOVERY - MegaRAID on db1055 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy [07:35:48] 10Operations, 10ops-eqiad, 10DBA: BBU issues on db1055, RAID cache on WriteThrough - https://phabricator.wikimedia.org/T174265#3557608 (10Marostegui) After forcing the re-learn again: ``` ˜/icinga-wm 9:34> RECOVERY - MegaRAID on db1055 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy ``` Let's ju... [07:36:59] 10Operations, 10Multimedia, 10TimedMediaHandler, 10HHVM, 10Patch-For-Review: Migrate video scalers to jessie - https://phabricator.wikimedia.org/T145742#3557611 (10MoritzMuehlenhoff) With Theora disabled and mw1260 depooled on Friday, there were no further transcoding errors; all 1699 jobs were processed... [07:38:26] (03PS1) 10Marostegui: mariadb: Add db1099 to s5 [puppet] - 10https://gerrit.wikimedia.org/r/374138 (https://phabricator.wikimedia.org/T172679) [07:44:44] (03CR) 10Giuseppe Lavagetto: [C: 031] Remove ffmpeg2theora from package list [puppet] - 10https://gerrit.wikimedia.org/r/373733 (https://phabricator.wikimedia.org/T172445) (owner: 10Muehlenhoff) [07:50:40] 10Operations, 10ops-eqiad, 10DBA: BBU issues on db1055, RAID cache on WriteThrough - https://phabricator.wikimedia.org/T174265#3557621 (10Marostegui) p:05Triage>03High And failed again. Let's not spend more time on this and just replace it. [07:51:11] 10Operations, 10ops-eqiad, 10User-Joe: Decom mw1170-mw1179, and replace them with new systems. - https://phabricator.wikimedia.org/T167130#3557623 (10Joe) p:05Normal>03High [07:51:42] !log reimaging mw1259 to jessie (T145742) [07:51:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:51:55] T145742: Migrate video scalers to jessie - https://phabricator.wikimedia.org/T145742 [07:52:08] 10Operations, 10ops-eqiad, 10User-Joe: Decom mw1170-mw1179, and replace them with new systems. - https://phabricator.wikimedia.org/T167130#3318501 (10Joe) Any news on this? We do need to rack the new appservers as putting them in production is needed in order to go on with the eqiad row D switch upgrade T172459 [07:52:45] 10Operations, 10ops-eqiad, 10hardware-requests, 10Patch-For-Review, 10User-Joe: Decommission mw1170-mw1179 - https://phabricator.wikimedia.org/T168271#3557629 (10Joe) p:05Normal>03High [07:54:04] PROBLEM - MegaRAID on db1055 is CRITICAL: CRITICAL: 1 LD(s) must have write cache policy WriteBack, currently using: WriteThrough [07:54:57] (03CR) 10Marostegui: "https://puppet-compiler.wmflabs.org/compiler02/7603/" [puppet] - 10https://gerrit.wikimedia.org/r/374138 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [07:56:37] (03CR) 10Marostegui: [C: 032] mariadb: Add db1099 to s5 [puppet] - 10https://gerrit.wikimedia.org/r/374138 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [08:01:41] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1045 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374136 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [08:03:10] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1045 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374136 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [08:03:24] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1045 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374136 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [08:04:54] !log marostegui@tin Synchronized wmf-config/db-eqiad.php: Depool db1045 to clone db1099 from it - T172679 (duration: 00m 46s) [08:05:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:05:05] T172679: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679 [08:10:37] (03PS2) 10ArielGlenn: minimal manifest for dumpsdata hosts [puppet] - 10https://gerrit.wikimedia.org/r/373271 [08:10:46] !log Stop MySQL on db1045 - T172679 [08:10:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:10:58] T172679: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679 [08:11:54] (03CR) 10ArielGlenn: [C: 032] minimal manifest for dumpsdata hosts [puppet] - 10https://gerrit.wikimedia.org/r/373271 (owner: 10ArielGlenn) [08:16:22] !log Ugprade MariaDB on s4 codfw master - db2051 to 10.0.32 - T168661 [08:16:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:16:35] T168661: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661 [08:21:13] 10Operations, 10MediaWiki-Platform-Team, 10Performance-Team, 10HHVM: Convert Wikimedia production HHVM instances to have hhvm.php7.all set true - https://phabricator.wikimedia.org/T173786#3539734 (10Joe) Beware that, as HHVM developers declared themselves, the php 7 implementation in HHVM will never be 100... [08:22:14] (03PS4) 10ArielGlenn: start of setup of dumpsdata hosts [puppet] - 10https://gerrit.wikimedia.org/r/373117 (https://phabricator.wikimedia.org/T169849) [08:36:01] (03CR) 10ArielGlenn: [C: 032] start of setup of dumpsdata hosts [puppet] - 10https://gerrit.wikimedia.org/r/373117 (https://phabricator.wikimedia.org/T169849) (owner: 10ArielGlenn) [08:40:54] 10Operations, 10ops-codfw: mw2256 - hardware issue - https://phabricator.wikimedia.org/T163346#3557695 (10elukey) `racadm getsysinfo` reports: ``` Embedded NIC MAC Addresses: NIC.Embedded.1-1-1 Ethernet = 14:18:77:5F:43:64 NIC.Embedded.2-1-1 Ethernet = 14:18:77:5F:43:65... [08:41:54] !log restarting squid3 on install1002 [08:42:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:42:49] (03PS1) 10Elukey: linux-host-entries: change MAC address of mw2256 [puppet] - 10https://gerrit.wikimedia.org/r/374168 (https://phabricator.wikimedia.org/T163346) [08:43:54] (03PS2) 10Elukey: linux-host-entries: change MAC address of mw2256 [puppet] - 10https://gerrit.wikimedia.org/r/374168 (https://phabricator.wikimedia.org/T163346) [08:48:28] (03PS6) 10Paladox: contint: Make php.pp compatible with stretch [puppet] - 10https://gerrit.wikimedia.org/r/361680 (https://phabricator.wikimedia.org/T166611) [08:48:39] (03PS20) 10Paladox: Zuul: Add systemd script for zuul [puppet] - 10https://gerrit.wikimedia.org/r/359016 (https://phabricator.wikimedia.org/T167833) [08:49:10] (03PS7) 10Paladox: Phabricator: Redirect all http traffic to https [puppet] - 10https://gerrit.wikimedia.org/r/354247 (https://phabricator.wikimedia.org/T165643) [08:49:50] 10Operations, 10Ops-Access-Requests, 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10User-Addshore: Requesting access to contint-admins for addshore - https://phabricator.wikimedia.org/T173233#3557706 (10hashar) The modules/admin `contint-admins` grants shell access to the contint machine... [08:50:51] 10Operations, 10Ops-Access-Requests, 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10User-Addshore: Requesting access to contint-admins for addshore - https://phabricator.wikimedia.org/T173233#3557712 (10Addshore) 05Resolved>03Open [08:51:17] for anyone that can write to / add me to ldap groups ^^ [08:51:38] moritzm: perhaps? :) [08:52:26] (03PS1) 10Filippo Giunchedi: ferm: add return traffic for ferm::client notrack [puppet] - 10https://gerrit.wikimedia.org/r/374169 (https://phabricator.wikimedia.org/T173731) [08:54:04] addshore: sure, I can do that, but Greg should ack this on the Phab task. they have requested the addition of the group last Friday and the initial set of ciadmin members were all limited to RelEng [08:54:04] RECOVERY - MegaRAID on db1055 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy [08:54:28] (03PS2) 10Filippo Giunchedi: ferm: add return traffic for ferm::client notrack [puppet] - 10https://gerrit.wikimedia.org/r/374169 (https://phabricator.wikimedia.org/T173731) [08:54:30] (03PS1) 10Filippo Giunchedi: swift: don't track connections to swift backend services on frontend machines [puppet] - 10https://gerrit.wikimedia.org/r/374170 (https://phabricator.wikimedia.org/T173731) [08:56:56] (03PS9) 10Paladox: Gerrit: Enable logstash by default for prod gerrit [puppet] - 10https://gerrit.wikimedia.org/r/332531 (https://phabricator.wikimedia.org/T141324) [08:59:47] althgouh, hmm, moritzm I would already say I have a 'higher' level of access than that ciadmin ladap group, I mean, I could already deploy the changes on the hosts themselves etc, I just can't use my api key to use jenkins job builder [09:00:41] I guess this means legoktm also can't update jenkins jobs now [09:00:58] (03CR) 10Filippo Giunchedi: "This change was missing from https://gerrit.wikimedia.org/r/#/c/373039 and I had to revert it." [puppet] - 10https://gerrit.wikimedia.org/r/374169 (https://phabricator.wikimedia.org/T173731) (owner: 10Filippo Giunchedi) [09:02:38] But I will wait for greg. *bails on his third attempt at updating this jenkins job* [09:02:47] (03CR) 10Filippo Giunchedi: "> LGTM, but we cannot merge this until all of the nodes are" [puppet] - 10https://gerrit.wikimedia.org/r/373863 (https://phabricator.wikimedia.org/T169939) (owner: 10Filippo Giunchedi) [09:04:08] 10Operations, 10Ops-Access-Requests, 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10User-Addshore: Requesting access to contint-admins for addshore - https://phabricator.wikimedia.org/T173233#3557721 (10Addshore) Per IRC perhaps @greg needs to sign off on this. > <•moritzm> addshore: su... [09:07:00] (03PS9) 10Paladox: Gerrit: Set auth.userNameToLowerCase [puppet] - 10https://gerrit.wikimedia.org/r/368196 [09:07:04] (03PS7) 10Paladox: Gerrit: Remove ldap user and password from secure.config [puppet] - 10https://gerrit.wikimedia.org/r/366910 [09:12:44] (03PS8) 10Paladox: Gerrit: Reveal the author in the title of the email [puppet] - 10https://gerrit.wikimedia.org/r/356645 (https://phabricator.wikimedia.org/T43608) [09:30:16] (03CR) 10Jcrespo: [C: 031] mariadb: Decommission db1028 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373869 (https://phabricator.wikimedia.org/T174076) (owner: 10Jcrespo) [09:30:34] (03PS2) 10Jcrespo: mariadb: Decommission db1028 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373869 (https://phabricator.wikimedia.org/T174076) [09:32:51] (03CR) 10Marostegui: [C: 031] mariadb: Decommission db1028 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373869 (https://phabricator.wikimedia.org/T174076) (owner: 10Jcrespo) [09:42:24] (03PS1) 10Giuseppe Lavagetto: base::service_unit: remove unused parameter [puppet] - 10https://gerrit.wikimedia.org/r/374181 [09:44:12] (03CR) 10Alexandros Kosiaris: [C: 032] Generate kubernetes manpages for kubectl [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/373917 (https://phabricator.wikimedia.org/T170346) (owner: 10Alexandros Kosiaris) [09:51:05] 10Operations, 10Performance-Team, 10Thumbor, 10Patch-For-Review, 10User-fgiunchedi: Track incoming HTTP request count on the Thumbor boxes - https://phabricator.wikimedia.org/T151554#3557858 (10fgiunchedi) I've added request latency percentiles to the thumbor dashboard as well. Note we'll need to add mor... [09:56:29] (03PS2) 10Alexandros Kosiaris: WIP: Upgrade to kubernetes 1.7.4 [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/373554 (https://phabricator.wikimedia.org/T170119) [09:57:06] (03CR) 10Gehel: [C: 04-1] "Suspicious line (probably a copy / paste error), otherwise looks good." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/373404 (https://phabricator.wikimedia.org/T157676) (owner: 10Smalyshev) [10:08:30] (03CR) 10Muehlenhoff: [C: 032] Remove expiry date [puppet] - 10https://gerrit.wikimedia.org/r/374135 (owner: 10Muehlenhoff) [10:08:34] (03PS4) 10Muehlenhoff: Remove expiry date [puppet] - 10https://gerrit.wikimedia.org/r/374135 [10:13:55] (03PS1) 10Filippo Giunchedi: thumbor: increase check timeout [puppet] - 10https://gerrit.wikimedia.org/r/374204 (https://phabricator.wikimedia.org/T172930) [10:16:54] (03CR) 10Elukey: [C: 032] linux-host-entries: change MAC address of mw2256 [puppet] - 10https://gerrit.wikimedia.org/r/374168 (https://phabricator.wikimedia.org/T163346) (owner: 10Elukey) [10:16:58] (03PS3) 10Elukey: linux-host-entries: change MAC address of mw2256 [puppet] - 10https://gerrit.wikimedia.org/r/374168 (https://phabricator.wikimedia.org/T163346) [10:17:00] (03CR) 10Elukey: [V: 032 C: 032] linux-host-entries: change MAC address of mw2256 [puppet] - 10https://gerrit.wikimedia.org/r/374168 (https://phabricator.wikimedia.org/T163346) (owner: 10Elukey) [10:24:54] (03PS1) 10Filippo Giunchedi: thumbor: tune histogram buckets for nginx request duration [puppet] - 10https://gerrit.wikimedia.org/r/374208 (https://phabricator.wikimedia.org/T151554) [10:32:20] 10Operations, 10ops-codfw, 10Patch-For-Review: mw2256 - hardware issue - https://phabricator.wikimedia.org/T163346#3194210 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['mw2256.codfw.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimag... [10:37:10] 10Operations, 10Ops-Access-Requests, 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10User-Addshore: Requesting access to contint-admins for addshore - https://phabricator.wikimedia.org/T173233#3557926 (10hashar) @MoritzMuehlenhoff this task is to grant @addshore access to the CI machines.... [10:47:51] (03CR) 10Muehlenhoff: Add support for querying reverse dependencies (033 comments) [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/373865 (owner: 10Muehlenhoff) [10:48:04] (03PS2) 10Muehlenhoff: Add support for querying reverse dependencies [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/373865 [11:00:35] 10Operations, 10Ops-Access-Requests, 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10User-Addshore: Requesting access to contint-admins for addshore - https://phabricator.wikimedia.org/T173233#3558082 (10MoritzMuehlenhoff) Ok, makes sense. I've added @addhore to cn=ciadmin. [11:04:47] (03PS1) 10ArielGlenn: add user and directory setup to dumpsdata hosts [puppet] - 10https://gerrit.wikimedia.org/r/374242 (https://phabricator.wikimedia.org/T169849) [11:05:54] !log installing libxml2 security updates [11:06:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:06:34] (03CR) 10Volans: [C: 031] "Much nicer, thanks for the fixes!" (031 comment) [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/373865 (owner: 10Muehlenhoff) [11:13:55] PROBLEM - Check whether ferm is active by checking the default input chain on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:13:55] PROBLEM - mediawiki-installation DSH group on mw2256 is CRITICAL: Host mw2256 is not in mediawiki-installation dsh group [11:14:55] PROBLEM - DPKG on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:14:55] PROBLEM - nutcracker port on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:15:44] PROBLEM - Disk space on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:15:45] PROBLEM - nutcracker process on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:16:16] 10Operations, 10Dumps-Generation, 10Patch-For-Review: Architecture and puppetize setup for dumpsdata boxes - https://phabricator.wikimedia.org/T169849#3558220 (10ArielGlenn) Notes from today's irc chat with @madhuvishy about the rsync that will happen from dumpsdata hosts to labstore hosts: There are severa... [11:16:35] PROBLEM - HHVM processes on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:16:35] PROBLEM - puppet last run on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:17:34] PROBLEM - HHVM rendering on mw2256 is CRITICAL: connect to address 10.192.16.55 and port 80: Connection refused [11:17:34] PROBLEM - salt-minion processes on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:19:15] PROBLEM - MD RAID on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:20:14] PROBLEM - Apache HTTP on mw2256 is CRITICAL: connect to address 10.192.16.55 and port 80: Connection refused [11:20:14] PROBLEM - Nginx local proxy to apache on mw2256 is CRITICAL: connect to address 10.192.16.55 and port 443: Connection refused [11:21:04] PROBLEM - Check size of conntrack table on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:21:55] PROBLEM - Check systemd state on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:21:55] PROBLEM - configured eth on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:22:28] 10Operations, 10Ops-Access-Requests, 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10User-Addshore: Requesting access to contint-admins for addshore - https://phabricator.wikimedia.org/T173233#3558273 (10Addshore) 05Open>03Resolved Looks like everything is now working! [11:22:55] PROBLEM - Check the NTP synchronisation status of timesyncd on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:22:55] PROBLEM - dhclient process on mw2256 is CRITICAL: Return code of 255 is out of bounds [11:23:22] sigh I thought it was downtimed, sorry :( [11:28:53] (03PS3) 10Muehlenhoff: Add support for querying reverse dependencies [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/373865 [11:31:46] (03PS3) 10Alexandros Kosiaris: WIP: Upgrade to kubernetes 1.7.4 [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/373554 (https://phabricator.wikimedia.org/T170119) [11:32:05] PROBLEM - Check Varnish expiry mailbox lag on cp1099 is CRITICAL: CRITICAL: expiry mailbox lag is 2018852 [11:37:24] PROBLEM - IPMI Temperature on mw2256 is CRITICAL: CHECK_NRPE: Socket timeout after 60 seconds. [11:54:10] (03PS1) 10Marostegui: s5.hosts: Add db1099 to s5 [software] - 10https://gerrit.wikimedia.org/r/374311 (https://phabricator.wikimedia.org/T172679) [11:55:42] (03CR) 10Marostegui: [C: 032] s5.hosts: Add db1099 to s5 [software] - 10https://gerrit.wikimedia.org/r/374311 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [11:56:29] (03Merged) 10jenkins-bot: s5.hosts: Add db1099 to s5 [software] - 10https://gerrit.wikimedia.org/r/374311 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [11:58:16] (03PS1) 10Marostegui: db-eqiad.php: Add db1099 to s5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374312 (https://phabricator.wikimedia.org/T172679) [12:00:51] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Add db1099 to s5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374312 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [12:02:18] (03Merged) 10jenkins-bot: db-eqiad.php: Add db1099 to s5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374312 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [12:02:31] (03CR) 10jenkins-bot: db-eqiad.php: Add db1099 to s5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374312 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [12:03:41] !log marostegui@tin Synchronized wmf-config/db-eqiad.php: Add db1099 pooled with low weight to s5 - T172679 (duration: 00m 45s) [12:03:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:03:55] T172679: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679 [12:05:56] !log restarting nginx for upgrade on elastic* / relforge* [12:06:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:07:46] !log restarting nginx for upgrade on wdqs* [12:07:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:09:33] (03PS12) 10Gehel: wdqs - moving to role / profiles [puppet] - 10https://gerrit.wikimedia.org/r/369682 (https://phabricator.wikimedia.org/T171704) [12:11:46] (03CR) 10Gehel: [C: 032] wdqs - moving to role / profiles [puppet] - 10https://gerrit.wikimedia.org/r/369682 (https://phabricator.wikimedia.org/T171704) (owner: 10Gehel) [12:17:38] (03PS1) 10Marostegui: db-eqiad.php: Give more weight to db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374314 [12:17:54] RECOVERY - Apache HTTP on mw2256 is OK: HTTP OK: HTTP/1.1 200 OK - 10975 bytes in 0.074 second response time [12:18:04] RECOVERY - Nginx local proxy to apache on mw2256 is OK: HTTP OK: HTTP/1.1 200 OK - 10975 bytes in 0.151 second response time [12:18:47] 10Operations, 10ops-codfw, 10Patch-For-Review: mw2256 - hardware issue - https://phabricator.wikimedia.org/T163346#3558332 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['mw2256.codfw.wmnet'] ``` Of which those **FAILED**: ``` set(['mw2256.codfw.wmnet']) ``` [12:22:28] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Give more weight to db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374314 (owner: 10Marostegui) [12:23:59] (03Merged) 10jenkins-bot: db-eqiad.php: Give more weight to db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374314 (owner: 10Marostegui) [12:25:22] (03PS1) 10Muehlenhoff: mw1168/mw1169: Reimage with jessie [puppet] - 10https://gerrit.wikimedia.org/r/374315 [12:26:03] (03CR) 10jenkins-bot: db-eqiad.php: Give more weight to db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374314 (owner: 10Marostegui) [12:27:07] !log marostegui@tin Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s) [12:27:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:27:20] T172679: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679 [12:31:24] RECOVERY - MD RAID on mw2256 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [12:31:25] RECOVERY - salt-minion processes on mw2256 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [12:31:44] RECOVERY - HHVM processes on mw2256 is OK: PROCS OK: 6 processes with command name hhvm [12:31:54] RECOVERY - Disk space on mw2256 is OK: DISK OK [12:31:54] RECOVERY - dhclient process on mw2256 is OK: PROCS OK: 0 processes with command name dhclient [12:31:55] RECOVERY - configured eth on mw2256 is OK: OK - interfaces up [12:32:04] RECOVERY - Check whether ferm is active by checking the default input chain on mw2256 is OK: OK ferm input default policy is set [12:32:05] RECOVERY - Check size of conntrack table on mw2256 is OK: OK: nf_conntrack is 0 % full [12:32:30] (03CR) 10Muehlenhoff: [C: 032] mw1168/mw1169: Reimage with jessie [puppet] - 10https://gerrit.wikimedia.org/r/374315 (owner: 10Muehlenhoff) [12:32:39] (03PS3) 10Gehel: wdqs - allow wdqs-admins to pool / depool servers [puppet] - 10https://gerrit.wikimedia.org/r/370198 (https://phabricator.wikimedia.org/T172798) [12:33:56] (03CR) 10Gehel: [C: 032] wdqs - allow wdqs-admins to pool / depool servers [puppet] - 10https://gerrit.wikimedia.org/r/370198 (https://phabricator.wikimedia.org/T172798) (owner: 10Gehel) [12:34:55] RECOVERY - DPKG on mw2256 is OK: All packages OK [12:36:24] RECOVERY - IPMI Temperature on mw2256 is OK: Sensor Type(s) Temperature Status: OK [12:37:55] PROBLEM - DPKG on mw2256 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [12:39:28] (03CR) 10Giuseppe Lavagetto: [C: 032] "https://puppet-compiler.wmflabs.org/compiler02/7609/" [puppet] - 10https://gerrit.wikimedia.org/r/374181 (owner: 10Giuseppe Lavagetto) [12:39:34] (03PS2) 10Giuseppe Lavagetto: base::service_unit: remove unused parameter [puppet] - 10https://gerrit.wikimedia.org/r/374181 [12:41:04] RECOVERY - DPKG on mw2256 is OK: All packages OK [12:42:23] jouncebot: next [12:42:24] In 0 hour(s) and 17 minute(s): European Mid-day SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170828T1300) [12:42:50] hashar: I can swat today, unless you really really want to :) [12:43:17] !log jmm@puppetmaster1001 conftool action : set/pooled=inactive; selector: mw1259.eqiad.wmnet [12:43:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:43:54] PROBLEM - puppet last run on mw2256 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 1 minute ago with 11 failures. Failed resources (up to 3 shown): File_line[login.defs-SYS_GID_MAX],File[/etc/apache2/mods-available/setenvif.conf],File[/etc/apache2/mods-available/userdir.conf],File[/etc/apache2/mods-available/autoindex.conf] [12:44:58] (03PS11) 10Gehel: wdqs - remove upstart configuration files [puppet] - 10https://gerrit.wikimedia.org/r/369688 (https://phabricator.wikimedia.org/T171704) [12:45:10] (03CR) 10jerkins-bot: [V: 04-1] wdqs - remove upstart configuration files [puppet] - 10https://gerrit.wikimedia.org/r/369688 (https://phabricator.wikimedia.org/T171704) (owner: 10Gehel) [12:47:13] (03PS1) 10Marostegui: db-eqiad.php: Add more weight to db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374316 [12:47:15] (03PS2) 10Filippo Giunchedi: thumbor: increase check timeout [puppet] - 10https://gerrit.wikimedia.org/r/374204 (https://phabricator.wikimedia.org/T172930) [12:47:55] (03CR) 10Filippo Giunchedi: [C: 032] thumbor: increase check timeout [puppet] - 10https://gerrit.wikimedia.org/r/374204 (https://phabricator.wikimedia.org/T172930) (owner: 10Filippo Giunchedi) [12:49:07] (03CR) 10Zfilipin: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373698 (https://phabricator.wikimedia.org/T163872) (owner: 10Urbanecm) [12:49:37] Urbanecm: um, this has -1 from jenkins-bot? https://gerrit.wikimedia.org/r/#/c/373698/4 [12:49:40] (03PS12) 10Gehel: wdqs - remove upstart configuration files [puppet] - 10https://gerrit.wikimedia.org/r/369688 (https://phabricator.wikimedia.org/T171704) [12:49:54] (03CR) 10jerkins-bot: [V: 04-1] wdqs - remove upstart configuration files [puppet] - 10https://gerrit.wikimedia.org/r/369688 (https://phabricator.wikimedia.org/T171704) (owner: 10Gehel) [12:50:08] zeljkof, will have a look into it [12:50:32] (03CR) 10jerkins-bot: [V: 04-1] Automatically include commons and wikidata in $wmgThrottlingExceptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373698 (https://phabricator.wikimedia.org/T163872) (owner: 10Urbanecm) [12:50:42] Urbanecm: looks like it's just lint problem https://integration.wikimedia.org/ci/job/operations-mw-config-composer-hhvm-jessie/5954/console [12:50:58] RECOVERY - nutcracker process on mw2256 is OK: PROCS OK: 1 process with UID = 111 (nutcracker), command name nutcracker [12:51:04] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Add more weight to db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374316 (owner: 10Marostegui) [12:51:07] RECOVERY - nutcracker port on mw2256 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 11212 [12:52:06] (03PS2) 10Filippo Giunchedi: thumbor: tune histogram buckets for nginx request duration [puppet] - 10https://gerrit.wikimedia.org/r/374208 (https://phabricator.wikimedia.org/T151554) [12:52:22] 10Operations, 10Phabricator, 10Traffic, 10Zero: Missing IP addresses for Maroc Telecom - https://phabricator.wikimedia.org/T174342#3558443 (10Dispenser) [12:52:29] (03Merged) 10jenkins-bot: db-eqiad.php: Add more weight to db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374316 (owner: 10Marostegui) [12:52:41] (03CR) 10jenkins-bot: db-eqiad.php: Add more weight to db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374316 (owner: 10Marostegui) [12:52:48] RECOVERY - Check the NTP synchronisation status of timesyncd on mw2256 is OK: OK: synced at Mon 2017-08-28 12:52:46 UTC. [12:53:17] (03PS5) 10Urbanecm: Automatically include commons and wikidata in $wmgThrottlingExceptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373698 (https://phabricator.wikimedia.org/T163872) [12:53:23] zeljkof, ^^ [12:53:33] !log marostegui@tin Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s) [12:53:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:53:45] T172679: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679 [12:53:51] (03CR) 10Filippo Giunchedi: [C: 032] thumbor: tune histogram buckets for nginx request duration [puppet] - 10https://gerrit.wikimedia.org/r/374208 (https://phabricator.wikimedia.org/T151554) (owner: 10Filippo Giunchedi) [12:54:08] PROBLEM - DPKG on mw2256 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [12:54:56] Urbanecm: great, thanks [12:55:00] Yw [12:55:19] (03PS13) 10Gehel: wdqs - remove upstart configuration files [puppet] - 10https://gerrit.wikimedia.org/r/369688 (https://phabricator.wikimedia.org/T171704) [12:56:00] Krinkle: [12:56:40] PROBLEM - HHVM rendering on mw2256 is CRITICAL: connect to address 10.192.16.55 and port 80: Connection refused [12:56:49] Krinkle: sorry, clicked enter to quickly, want to deploy https://gerrit.wikimedia.org/r/#/c/373984/ yourself? [12:57:06] (03PS4) 10Alexandros Kosiaris: WIP: Upgrade to kubernetes 1.7.4 [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/373554 (https://phabricator.wikimedia.org/T170119) [12:57:08] RECOVERY - puppet last run on mw2256 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [12:57:17] RECOVERY - DPKG on mw2256 is OK: All packages OK [12:58:36] (03PS14) 10Gehel: wdqs - remove upstart configuration files [puppet] - 10https://gerrit.wikimedia.org/r/369688 (https://phabricator.wikimedia.org/T171704) [12:59:18] PROBLEM - Nginx local proxy to apache on mw2256 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 327 bytes in 0.151 second response time [12:59:38] RECOVERY - HHVM rendering on mw2256 is OK: HTTP OK: HTTP/1.1 200 OK - 79209 bytes in 0.380 second response time [12:59:45] 10Operations, 10Ops-Access-Requests: NDA request for Samtar - https://phabricator.wikimedia.org/T174316#3558461 (10Aklapper) I believe this is an #ops-access-requests if you already have an NDA. [13:00:05] addshore, hashar, anomie, RainbowSprinkles, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: Respected human, time to deploy European Mid-day SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170828T1300). Please do the needful. [13:00:05] Urbanecm, Lucas_WMDE, and Krinkle: A patch you scheduled for European Mid-day SWAT(Max 8 patches) is about to be deployed. Please be available during the process. [13:00:17] I can SWAT today! [13:00:18] RECOVERY - Nginx local proxy to apache on mw2256 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.203 second response time [13:00:30] (03PS15) 10Gehel: wdqs - remove upstart configuration files [puppet] - 10https://gerrit.wikimedia.org/r/369688 (https://phabricator.wikimedia.org/T171704) [13:01:08] I'm here, surprisingly :D [13:01:14] Urbanecm: :D [13:01:21] I am reviewing 373694 [13:01:24] ack [13:01:59] I don't see Lucas_WMDE :| [13:02:27] (03PS16) 10Gehel: wdqs - remove upstart configuration files [puppet] - 10https://gerrit.wikimedia.org/r/369688 (https://phabricator.wikimedia.org/T171704) [13:02:32] zeljkof, btw, note that 373695 must be deployed before 373698 [13:02:38] BTW, he's here [13:02:43] Lucas_WMDE1, ping, SWAT started [13:02:51] Krinkle, Lucas_WMDE1: let me know if your patches are urgent, since they are at the end there is a chance I will run out of time [13:03:10] (03CR) 10Gehel: [C: 032] wdqs - remove upstart configuration files [puppet] - 10https://gerrit.wikimedia.org/r/369688 (https://phabricator.wikimedia.org/T171704) (owner: 10Gehel) [13:03:16] Urbanecm: ok, will deploy according to the order on the page [13:03:16] mine isn’t urgent, no [13:04:33] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373694 (https://phabricator.wikimedia.org/T174098) (owner: 10Urbanecm) [13:05:58] (03Merged) 10jenkins-bot: Remove non-transparent background from dty.wiki logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373694 (https://phabricator.wikimedia.org/T174098) (owner: 10Urbanecm) [13:06:08] (03CR) 10jenkins-bot: Remove non-transparent background from dty.wiki logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373694 (https://phabricator.wikimedia.org/T174098) (owner: 10Urbanecm) [13:06:43] (03CR) 10Hashar: [C: 031] Log 'WikibaseQualityConstraints' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/367914 (https://phabricator.wikimedia.org/T171281) (owner: 10Lucas Werkmeister (WMDE)) [13:07:16] zeljkof: the log patch for WMDE is all fine [13:07:36] hashar: feel free to review the patches and +1 :D [13:07:40] did :) [13:07:46] hashar: thanks! [13:08:01] Urbanecm: 373694 is at mwdebug1002 [13:08:08] hashar: thanks :) [13:08:14] ack [13:08:17] RECOVERY - Check systemd state on mw2256 is OK: OK - running: The system is fully operational [13:08:49] zeljkof, please deploy [13:08:54] Urbanecm: ok [13:09:53] !log zfilipin@tin Synchronized static/images/project-logos/: SWAT: [[gerrit:373694|Remove non-transparent background from dty.wiki logos (T174098)]] (duration: 00m 45s) [13:10:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:10:05] T174098: Fix dty.wikipedia logo - https://phabricator.wikimedia.org/T174098 [13:10:30] Urbanecm: 373694 deployed, reviewing 373873 [13:11:25] 10Operations, 10ops-codfw, 10Patch-For-Review: mw2256 - hardware issue - https://phabricator.wikimedia.org/T163346#3558502 (10elukey) Host reimaged and re-pooled, everything looks good. Let's keep this task open for a couple of days to see if anything weird comes up but I'd say that we are good. [13:11:37] Urbanecm: zeljkof the background is still there, and I've purged the caché [13:11:54] zeljkof, please purge the URLs [13:12:05] tabbycat, with mwdebug it wasn't there [13:12:05] tabbycat: I'll purge the URLs [13:12:09] thank you zeljkof [13:12:10] mwscript purgeList.php [13:12:16] thank you zeljkof [13:12:24] Urbanecm: I was late for mwdebug :) [13:12:26] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373873 (https://phabricator.wikimedia.org/T174152) (owner: 10Urbanecm) [13:12:44] tabbycat, you can use mwdebug all the time, as it is the same like prod, sometime with added changes [13:13:55] (03Merged) 10jenkins-bot: Add 4 URLs to $wgCopyUploadsDomain whitelist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373873 (https://phabricator.wikimedia.org/T174152) (owner: 10Urbanecm) [13:14:02] RECOVERY - mediawiki-installation DSH group on mw2256 is OK: OK [13:14:30] on mwdebug1001 and mwdebug1002 I see the non-transparent background :S [13:14:37] maybe a computer issue? [13:15:31] tabbycat, I see transparent background with and without mwdebug [13:15:47] now I do as well [13:15:54] caching [13:15:55] Urbanecm: tabbycat: try now [13:16:03] yep, not it works [13:16:05] (03CR) 10jenkins-bot: Add 4 URLs to $wgCopyUploadsDomain whitelist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373873 (https://phabricator.wikimedia.org/T174152) (owner: 10Urbanecm) [13:16:14] I purged again for the 10th time :) [13:16:19] this time it worked [13:17:25] Urbanecm: 373873 is at mwdebug [13:18:32] ack [13:20:06] (03CR) 10Zfilipin: [C: 031] Allow sysops to grant/remove transwiki user group in dtywiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374066 (https://phabricator.wikimedia.org/T174226) (owner: 10Urbanecm) [13:20:11] working [13:20:14] Please deploy zeljkof [13:20:15] (03PS3) 10Zfilipin: Allow sysops to grant/remove transwiki user group in dtywiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374066 (https://phabricator.wikimedia.org/T174226) (owner: 10Urbanecm) [13:20:24] Urbanecm: deploying [13:21:25] !log zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:373873|Add 4 URLs to $wgCopyUploadsDomain whitelist (T174152)]] (duration: 00m 45s) [13:21:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:21:38] T174152: Please add *.geograph.org.uk to the wgCopyUploadsDomains whitelist of Wikimedia Commons - https://phabricator.wikimedia.org/T174152 [13:21:59] Urbanecm: 373873 deployed, reviewing 374066 [13:22:32] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374066 (https://phabricator.wikimedia.org/T174226) (owner: 10Urbanecm) [13:24:08] (03PS1) 10Marostegui: db-eqiad.php: Fully pool db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374318 (https://phabricator.wikimedia.org/T172679) [13:24:13] (03Merged) 10jenkins-bot: Allow sysops to grant/remove transwiki user group in dtywiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374066 (https://phabricator.wikimedia.org/T174226) (owner: 10Urbanecm) [13:25:12] Urbanecm: 374066 is at mwdebug [13:25:48] working, please deploy [13:26:13] (03CR) 10jenkins-bot: Allow sysops to grant/remove transwiki user group in dtywiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374066 (https://phabricator.wikimedia.org/T174226) (owner: 10Urbanecm) [13:26:52] (03CR) 10Zfilipin: [C: 031] throttle.php: Separate the throttling definitions from the exception values itself [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373695 (https://phabricator.wikimedia.org/T167040) (owner: 10Urbanecm) [13:27:01] Urbanecm: deploying [13:27:50] !log zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:374066|Allow sysops to grant/remove transwiki user group in dtywiki (T174226)]] (duration: 00m 44s) [13:28:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:28:03] T174226: Enable Importer User Group For Doteli Wikipedia - https://phabricator.wikimedia.org/T174226 [13:28:20] Urbanecm: 374066 deployed, reviewing 373695 [13:28:31] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373695 (https://phabricator.wikimedia.org/T167040) (owner: 10Urbanecm) [13:28:36] (03PS10) 10Gehel: wdqs - send logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/371939 (https://phabricator.wikimedia.org/T172710) [13:28:44] Please deploy it right into the production cluster, I can't test it as well as the other one [13:28:58] (03CR) 10jerkins-bot: [V: 04-1] wdqs - send logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/371939 (https://phabricator.wikimedia.org/T172710) (owner: 10Gehel) [13:29:23] Urbanecm: so 373695 and 373698 can not be tested at mwdebug? [13:29:27] Yes. [13:29:54] (03PS11) 10Gehel: wdqs - send logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/371939 (https://phabricator.wikimedia.org/T172710) [13:30:00] (03Merged) 10jenkins-bot: throttle.php: Separate the throttling definitions from the exception values itself [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373695 (https://phabricator.wikimedia.org/T167040) (owner: 10Urbanecm) [13:30:11] (03CR) 10jenkins-bot: throttle.php: Separate the throttling definitions from the exception values itself [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373695 (https://phabricator.wikimedia.org/T167040) (owner: 10Urbanecm) [13:30:20] Urbanecm: OK. For 373695, the deploy order is throttle-analyze, throttle, common settings? [13:30:53] Yes. [13:32:15] Urbanecm: ok, deploying [13:32:21] ack [13:33:48] (03PS1) 10Giuseppe Lavagetto: profile::base: stringify hiera defaults [puppet] - 10https://gerrit.wikimedia.org/r/374319 (https://phabricator.wikimedia.org/T171704) [13:33:58] !log zfilipin@tin Synchronized wmf-config/throttle-analyze.php: SWAT: [[gerrit:373695|throttle.php: Separate the throttling definitions from the exception values itself (T167040)]] (duration: 00m 44s) [13:34:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:34:09] T167040: throttle.php: Separate the throttling definitions from the exception values itself - https://phabricator.wikimedia.org/T167040 [13:34:58] !log zfilipin@tin Synchronized wmf-config/throttle.php: SWAT: [[gerrit:373695|throttle.php: Separate the throttling definitions from the exception values itself (T167040)]] (duration: 00m 44s) [13:35:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:36:09] !log zfilipin@tin Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:373695|throttle.php: Separate the throttling definitions from the exception values itself (T167040)]] (duration: 00m 44s) [13:36:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:36:31] Urbanecm: 373695 is deployed, can you check on prod? [13:36:55] N [13:36:56] No [13:37:00] It is untestable at all [13:37:05] (as well as the other one) [13:37:18] <_joe_> win 25 [13:37:20] Urbanecm: ok, logs look fine, so at least things are not going south :) [13:37:38] Urbanecm: reviewing 373698 [13:38:50] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373698 (https://phabricator.wikimedia.org/T163872) (owner: 10Urbanecm) [13:38:58] (03CR) 10jerkins-bot: [V: 04-1] Automatically include commons and wikidata in $wmgThrottlingExceptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373698 (https://phabricator.wikimedia.org/T163872) (owner: 10Urbanecm) [13:39:04] (03PS7) 10Zfilipin: Automatically include commons and wikidata in $wmgThrottlingExceptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373698 (https://phabricator.wikimedia.org/T163872) (owner: 10Urbanecm) [13:39:16] (03CR) 10Zfilipin: Automatically include commons and wikidata in $wmgThrottlingExceptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373698 (https://phabricator.wikimedia.org/T163872) (owner: 10Urbanecm) [13:39:25] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373698 (https://phabricator.wikimedia.org/T163872) (owner: 10Urbanecm) [13:40:55] (03Merged) 10jenkins-bot: Automatically include commons and wikidata in $wmgThrottlingExceptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373698 (https://phabricator.wikimedia.org/T163872) (owner: 10Urbanecm) [13:40:57] Lucas_WMDE: your commit is next for swat, will ping you in a few minutes, please be available [13:41:09] (03CR) 10jenkins-bot: Automatically include commons and wikidata in $wmgThrottlingExceptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373698 (https://phabricator.wikimedia.org/T163872) (owner: 10Urbanecm) [13:41:11] zeljkof: thanks, I’ll be here [13:42:12] (03PS12) 10Gehel: wdqs - send logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/371939 (https://phabricator.wikimedia.org/T172710) [13:42:49] !log zfilipin@tin Synchronized wmf-config/throttle-analyze.php: SWAT: [[gerrit:373698|Automatically include commons and wikidata in $wmgThrottlingExceptions (T163872)]] (duration: 00m 44s) [13:42:57] (03PS4) 10Zfilipin: Log 'WikibaseQualityConstraints' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/367914 (https://phabricator.wikimedia.org/T171281) (owner: 10Lucas Werkmeister (WMDE)) [13:43:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:43:02] T163872: Automatically include commons and wikidata in $wmgThrottlingExceptions - https://phabricator.wikimedia.org/T163872 [13:43:42] Urbanecm: 373698 is deployed, I guess there is nothing for you to do :) enjoy your day and thanks for deploying with #releng ;) [13:43:54] PROBLEM - Check the NTP synchronisation status of timesyncd on mw1259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:43:54] PROBLEM - nutcracker port on mw1259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:43:58] Lucas_WMDE: reviewing 367914 [13:44:01] thank you for deploys [13:44:24] (03CR) 10Thcipriani: "Looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/374054 (https://phabricator.wikimedia.org/T172847) (owner: 1020after4) [13:44:54] PROBLEM - Check whether ferm is active by checking the default input chain on mw1259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:44:54] PROBLEM - nutcracker process on mw1259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:45:45] PROBLEM - DPKG on mw1259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:45:45] PROBLEM - puppet last run on mw1259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:46:34] PROBLEM - HHVM jobrunner on mw1259 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:44] PROBLEM - Disk space on mw1259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:46:44] PROBLEM - salt-minion processes on mw1259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:46:47] (03CR) 10Gehel: "puppet compiler looks happy: https://puppet-compiler.wmflabs.org/compiler03/7617/" [puppet] - 10https://gerrit.wikimedia.org/r/371939 (https://phabricator.wikimedia.org/T172710) (owner: 10Gehel) [13:46:50] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/367914 (https://phabricator.wikimedia.org/T171281) (owner: 10Lucas Werkmeister (WMDE)) [13:47:04] Urbanecm: I am glad I could help! :D [13:47:39] Krinkle: around for EU SWAT (commit 373984)? [13:47:40] ^mw1259 is being reimaged, fixing downtime [13:48:14] (03Merged) 10jenkins-bot: Log 'WikibaseQualityConstraints' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/367914 (https://phabricator.wikimedia.org/T171281) (owner: 10Lucas Werkmeister (WMDE)) [13:48:24] (03CR) 10jenkins-bot: Log 'WikibaseQualityConstraints' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/367914 (https://phabricator.wikimedia.org/T171281) (owner: 10Lucas Werkmeister (WMDE)) [13:48:34] PROBLEM - puppet last run on mw1243 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:49:40] Lucas_WMDE: 367914 is at mwdebug1002, can you test there? [13:51:12] zeljkof: not sure how to test it… I don’t even have access to the logs, only my coworkers :) [13:51:32] Lucas_WMDE: ok, should I do a full deploy then? [13:51:33] and I don’t think we’re currently logging any non-exceptional messages [13:51:48] zeljkof: that would be best, I think [13:51:52] if you think the change isn’t too risky :) [13:51:58] Lucas_WMDE: ok, deploying [13:52:19] just gotta make sure it is not too spammy :] [13:52:36] hashar: looks like Krinkle is not around, I should not deploy his commit, right? [13:52:45] !log zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:367914|Log WikibaseQualityConstraints (T171281)]] (duration: 00m 44s) [13:52:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:52:57] T171281: Log details of long-running constraint checks - https://phabricator.wikimedia.org/T171281 [13:53:28] Lucas_WMDE: 367914 is deployed, let your coworkers now :) and thanks for deploying with #releng [13:53:36] (03CR) 10Alexandros Kosiaris: [C: 04-1] "Overall OK, see inline comments" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/374319 (https://phabricator.wikimedia.org/T171704) (owner: 10Giuseppe Lavagetto) [13:53:56] zeljkof: I cant tell what that hacky patch is doing. Then Aaron / Timo / Erik all looked at it :] [13:54:06] that is to reduce some job queue spam and I guess we can land it [13:54:07] (03PS1) 10Muehlenhoff: Put ganglia behind LDAP authentication [puppet] - 10https://gerrit.wikimedia.org/r/374320 [13:54:10] zeljkof: great, thanks for your assistance :) [13:54:17] if the queue explose somehow in the next hour, we can always revert [13:54:28] hashar: so I should deploy it? [13:54:33] yes :) [13:54:34] (queue explosion lol) [13:54:42] hashar: ok, deploying [13:55:41] (03CR) 10Giuseppe Lavagetto: profile::base: stringify hiera defaults (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/374319 (https://phabricator.wikimedia.org/T171704) (owner: 10Giuseppe Lavagetto) [13:55:41] !log restart kafka* daemons on kafka1012 for openjdk security updates (canary) [13:55:51] ottomata: --^ [13:55:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:58:25] +1 [13:58:53] (03PS2) 10Giuseppe Lavagetto: profile::base: stringify hiera defaults [puppet] - 10https://gerrit.wikimedia.org/r/374319 (https://phabricator.wikimedia.org/T171704) [13:58:55] (03PS1) 10Giuseppe Lavagetto: git::clone: enhance compatibility with the future parser [puppet] - 10https://gerrit.wikimedia.org/r/374321 (https://phabricator.wikimedia.org/T171704) [13:59:22] zeljkof: there's a logo change as well [14:00:02] (03CR) 10Alexandros Kosiaris: [C: 031] "Let's send out a notification at least to wikitech-l before merging this." [puppet] - 10https://gerrit.wikimedia.org/r/374320 (owner: 10Muehlenhoff) [14:00:09] tabbycat: did not refresh the page, so I did not notice it [14:00:25] will deploy if this core deploy does not take forever [14:00:36] ktnx :) [14:00:46] (03CR) 10Muehlenhoff: [C: 032] Add support for querying reverse dependencies [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/373865 (owner: 10Muehlenhoff) [14:01:06] (03PS4) 10Ladsgroup: mediawiki: fix logrotating in wikidata cronjob [puppet] - 10https://gerrit.wikimedia.org/r/373854 (https://phabricator.wikimedia.org/T171460) [14:01:20] !log extending EU SWAT for 10 or so minutes to deploy two more patches [14:01:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:01:48] (03CR) 10jerkins-bot: [V: 04-1] mediawiki: fix logrotating in wikidata cronjob [puppet] - 10https://gerrit.wikimedia.org/r/373854 (https://phabricator.wikimedia.org/T171460) (owner: 10Ladsgroup) [14:01:52] (03PS1) 10Gehel: role::wdqs: move to the future parser [puppet] - 10https://gerrit.wikimedia.org/r/374322 (https://phabricator.wikimedia.org/T171704) [14:02:01] (03CR) 10Muehlenhoff: "Ack, I was planning to do that." [puppet] - 10https://gerrit.wikimedia.org/r/374320 (owner: 10Muehlenhoff) [14:02:09] (03PS5) 10Ladsgroup: mediawiki: fix logrotating in wikidata cronjob [puppet] - 10https://gerrit.wikimedia.org/r/373854 (https://phabricator.wikimedia.org/T171460) [14:03:13] (03CR) 10Zfilipin: [C: 031] SVG logo for es.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374065 (https://phabricator.wikimedia.org/T170604) (owner: 10MarcoAurelio) [14:04:35] (03CR) 10Gehel: "The last changes related to future parser seem to be trivial enough: https://puppet-compiler.wmflabs.org/compiler03/7620/wdqs1001.eqiad.wm" [puppet] - 10https://gerrit.wikimedia.org/r/374322 (https://phabricator.wikimedia.org/T171704) (owner: 10Gehel) [14:05:45] RECOVERY - Disk space on mw1259 is OK: DISK OK [14:05:46] 10Operations, 10ops-eqiad: Pending sectors for one disk on tin.eqiad.wmnet - https://phabricator.wikimedia.org/T174347#3558632 (10fgiunchedi) [14:05:54] RECOVERY - salt-minion processes on mw1259 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [14:06:05] RECOVERY - Check whether ferm is active by checking the default input chain on mw1259 is OK: OK ferm input default policy is set [14:06:25] zeljkof: you can deploy the logo change in parallel :) [14:06:54] hashar: I know, but this one is almost done... I rarely deploy core so I am a bit nervous o.O [14:08:04] RECOVERY - DPKG on mw1259 is OK: All packages OK [14:09:42] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374065 (https://phabricator.wikimedia.org/T170604) (owner: 10MarcoAurelio) [14:09:55] ok, core jobs are taking forever... [14:11:16] (03Merged) 10jenkins-bot: SVG logo for es.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374065 (https://phabricator.wikimedia.org/T170604) (owner: 10MarcoAurelio) [14:11:21] (03CR) 10Giuseppe Lavagetto: [C: 031] role::wdqs: move to the future parser [puppet] - 10https://gerrit.wikimedia.org/r/374322 (https://phabricator.wikimedia.org/T171704) (owner: 10Gehel) [14:11:26] (03CR) 10jenkins-bot: SVG logo for es.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374065 (https://phabricator.wikimedia.org/T170604) (owner: 10MarcoAurelio) [14:11:53] (03PS6) 10Ladsgroup: mediawiki: fix logrotating in wikidata cronjob [puppet] - 10https://gerrit.wikimedia.org/r/373854 (https://phabricator.wikimedia.org/T171460) [14:12:15] (03CR) 10jerkins-bot: [V: 04-1] mediawiki: fix logrotating in wikidata cronjob [puppet] - 10https://gerrit.wikimedia.org/r/373854 (https://phabricator.wikimedia.org/T171460) (owner: 10Ladsgroup) [14:13:18] tabbycat: 374065 is at mwdebug1002, please check and let me know if I can continue [14:13:54] RECOVERY - Check the NTP synchronisation status of timesyncd on mw1259 is OK: OK: synced at Mon 2017-08-28 14:13:45 UTC. [14:14:08] zeljkof: please revert, oversize [14:14:44] tabbycat: ok [14:14:52] otherwise looked good to me, it's just a size problem [14:15:33] (03PS7) 10Ladsgroup: mediawiki: fix logrotating in wikidata cronjob [puppet] - 10https://gerrit.wikimedia.org/r/373854 (https://phabricator.wikimedia.org/T171460) [14:15:44] RECOVERY - HHVM jobrunner on mw1259 is OK: HTTP OK: HTTP/1.1 200 OK - 202 bytes in 0.006 second response time [14:17:44] RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:18:20] (03CR) 10Giuseppe Lavagetto: [C: 032] profile::base: stringify hiera defaults [puppet] - 10https://gerrit.wikimedia.org/r/374319 (https://phabricator.wikimedia.org/T171704) (owner: 10Giuseppe Lavagetto) [14:18:30] 10Operations, 10monitoring: Investigate check_nrpe -u option to reduce critical alerts - https://phabricator.wikimedia.org/T172131#3558686 (10herron) a:03herron [14:19:45] (03PS3) 10Faidon Liambotis: Add SPF and DKIM perl package requires to spamassassin class [puppet] - 10https://gerrit.wikimedia.org/r/370487 (https://phabricator.wikimedia.org/T172689) (owner: 10Herron) [14:19:52] (03CR) 10Faidon Liambotis: [C: 032] Add SPF and DKIM perl package requires to spamassassin class [puppet] - 10https://gerrit.wikimedia.org/r/370487 (https://phabricator.wikimedia.org/T172689) (owner: 10Herron) [14:19:57] (03PS1) 10MarcoAurelio: Revert "SVG logo for es.wiktionary" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374326 [14:20:13] zeljkof: ^ [14:20:49] (03PS1) 10Filippo Giunchedi: thumbor: more latency buckets for nginx requests [puppet] - 10https://gerrit.wikimedia.org/r/374327 (https://phabricator.wikimedia.org/T151554) [14:21:07] (03PS8) 10Volans: mediawiki: fix logrotating in wikidata cronjob [puppet] - 10https://gerrit.wikimedia.org/r/373854 (https://phabricator.wikimedia.org/T171460) (owner: 10Ladsgroup) [14:21:09] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374326 (owner: 10MarcoAurelio) [14:21:16] tabbycat: merging [14:21:36] (03CR) 10Volans: [C: 032] mediawiki: fix logrotating in wikidata cronjob [puppet] - 10https://gerrit.wikimedia.org/r/373854 (https://phabricator.wikimedia.org/T171460) (owner: 10Ladsgroup) [14:21:48] (03CR) 10Faidon Liambotis: [C: 04-1] Add 5 second "greet pause" delay to lists.wikimedia.org SMTP (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/371958 (https://phabricator.wikimedia.org/T173143) (owner: 10Herron) [14:21:57] okay, sorry for the inconvenience; there's not a list of sizes and sometimes this happen [14:22:03] (03PS4) 10Faidon Liambotis: Add SPF and DKIM perl package requires to spamassassin class [puppet] - 10https://gerrit.wikimedia.org/r/370487 (https://phabricator.wikimedia.org/T172689) (owner: 10Herron) [14:22:18] (03CR) 10Zfilipin: Revert "SVG logo for es.wiktionary" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374326 (owner: 10MarcoAurelio) [14:22:23] (03PS2) 10Filippo Giunchedi: thumbor: more latency buckets for nginx requests [puppet] - 10https://gerrit.wikimedia.org/r/374327 (https://phabricator.wikimedia.org/T151554) [14:22:29] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374326 (owner: 10MarcoAurelio) [14:22:45] (03Merged) 10jenkins-bot: Revert "SVG logo for es.wiktionary" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374326 (owner: 10MarcoAurelio) [14:22:46] tabbycat: no problem, reverting [14:22:54] (03CR) 10jenkins-bot: Revert "SVG logo for es.wiktionary" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374326 (owner: 10MarcoAurelio) [14:23:20] (03CR) 10Filippo Giunchedi: [C: 032] thumbor: more latency buckets for nginx requests [puppet] - 10https://gerrit.wikimedia.org/r/374327 (https://phabricator.wikimedia.org/T151554) (owner: 10Filippo Giunchedi) [14:23:23] (03PS3) 10Filippo Giunchedi: thumbor: more latency buckets for nginx requests [puppet] - 10https://gerrit.wikimedia.org/r/374327 (https://phabricator.wikimedia.org/T151554) [14:23:51] tabbycat: I have pulled the revert to mwdebug1002 [14:24:21] looks good, as it was [14:27:30] zeljkof: ^^ [14:27:55] tabbycat: thanks [14:28:13] !log zfilipin@tin Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: SWAT: [[gerrit:373984|Disable rebound CDN purges for backlinks in HTMLCacheUpdateJob (T173710)]] (duration: 00m 45s) [14:28:19] okay, so everything seems to be done [14:28:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:28:26] T173710: Job queue is increasing non-stop - https://phabricator.wikimedia.org/T173710 [14:28:37] hashar: deployed 373984 [14:28:44] if you have any idea what to check, please do :) [14:28:49] !log EU SWAT finished [14:29:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:29:09] (03PS2) 10Marostegui: db-eqiad.php: Fully pool db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374318 (https://phabricator.wikimedia.org/T172679) [14:31:00] !log drop PageContentSaveComplete_5588433_15423246 from the log database on db1046 (m4-master) - T170720 [14:31:12] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Fully pool db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374318 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [14:31:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:31:14] T170720: Archive PageContentSaveComplete in hdfs while we continue collecting data - https://phabricator.wikimedia.org/T170720 [14:32:45] PROBLEM - eventlogging_sync processes on dbstore1002 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 0 (root), args /bin/bash /usr/local/bin/eventlogging_sync.sh [14:33:31] (03Merged) 10jenkins-bot: db-eqiad.php: Fully pool db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374318 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [14:33:43] (03CR) 10jenkins-bot: db-eqiad.php: Fully pool db1099 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374318 (https://phabricator.wikimedia.org/T172679) (owner: 10Marostegui) [14:33:56] checking eventlogging_sync, it might have been me [14:34:48] !log marostegui@tin Synchronized wmf-config/db-eqiad.php: Pool db1099 with normal weight on s5 - T172679 (duration: 00m 44s) [14:34:59] (03PS1) 10Urbanecm: Restrict merging rights to autoconfirmed users on wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374328 (https://phabricator.wikimedia.org/T174345) [14:34:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:35:00] T172679: Productionize 11 new eqiad database servers - https://phabricator.wikimedia.org/T172679 [14:37:45] RECOVERY - eventlogging_sync processes on dbstore1002 is OK: PROCS OK: 1 process with UID = 0 (root), args /bin/bash /usr/local/bin/eventlogging_sync.sh [14:39:28] !log restart eventlogging_sync on dbstore1002 - issue after drop of old table [14:39:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:39:55] data was already removed from the slaves and I mistakenly assumed that it wouldn't have been queried by eventlogging_sync [14:40:15] RECOVERY - nutcracker port on mw1259 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 11212 [14:40:15] RECOVERY - nutcracker process on mw1259 is OK: PROCS OK: 1 process with UID = 111 (nutcracker), command name nutcracker [14:44:26] RECOVERY - puppet last run on mw1259 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:49:01] (03CR) 10Hashar: "Thank you :]]" [puppet] - 10https://gerrit.wikimedia.org/r/369873 (owner: 10Hashar) [14:52:28] (03PS1) 10Elukey: stat1003: remove puppet configuration as part of decom [puppet] - 10https://gerrit.wikimedia.org/r/374332 (https://phabricator.wikimedia.org/T152712) [14:52:42] (03CR) 10Hashar: [C: 031] "Current patchset is on the CI puppet master and it seems to pass puppet just fine on integration-r-lang-01." [puppet] - 10https://gerrit.wikimedia.org/r/363337 (https://phabricator.wikimedia.org/T153856) (owner: 10Hashar) [14:54:30] (03PS2) 10Elukey: stat1003: remove puppet configuration as part of decom [puppet] - 10https://gerrit.wikimedia.org/r/374332 (https://phabricator.wikimedia.org/T152712) [14:55:27] (03CR) 10Herron: "Thanks Faidon!" [puppet] - 10https://gerrit.wikimedia.org/r/370487 (https://phabricator.wikimedia.org/T172689) (owner: 10Herron) [15:13:05] 10Operations, 10Puppet, 10Patch-For-Review, 10User-Joe: Switch all hosts to the future parser - https://phabricator.wikimedia.org/T171704#3558822 (10Joe) https://puppet-compiler.wmflabs.org/compiler02/7622/index-future.html has a list with most spurious differences removed. I'll post new puppet-compiler r... [15:14:37] 10Operations, 10Puppet, 10Patch-For-Review, 10User-Joe: Switch all hosts to the future parser - https://phabricator.wikimedia.org/T171704#3558827 (10Joe) [15:16:26] (03CR) 10Gehel: [C: 04-1] git::clone: enhance compatibility with the future parser (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/374321 (https://phabricator.wikimedia.org/T171704) (owner: 10Giuseppe Lavagetto) [15:20:10] godog: I have some monitoring questions if you have a moment. Specifically, I have a bunch of icinga monitors for http services and I want to gather uptime stats. I'm guessing that the right approach is "throw away the existing alerts, reimplement the checks in prometheus, have icinga monitor the new metric" — is that right? And, if so, are there examples you can point me to? [15:31:14] andrewbogott: if you are interested in the uptime you could add a probe with prometheus alongside the existing icinga alert yeah, there's basic scaffolding in puppet as a consequence of T169860 to setup e.g. http probes [15:31:15] T169860: Investigate/setup prometheus blackbox_exporter - https://phabricator.wikimedia.org/T169860 [15:31:41] andrewbogott: though I'm not 100% sure of the use case you have in mind [15:32:48] godog: the use base is pretty simple, I just want to gather some stats about how much of the time nova/glance/etc. apis are down. [15:33:29] T169860 is about using prometheus for alerting, right? [15:34:36] more titled towards a smokeping replacement than alerting, that task specifically [15:35:27] I don't think I know what smokeping is used for [15:35:29] but, I will read! [15:35:42] andrewbogott: define "down" ;) slow, 5xx, no reply, etc... [15:36:05] for starters, 'no reply' [15:36:33] icinga / nagios has built in availability report. Though our setup apparently doesn't have it (or I dont have access to it) [15:38:38] andrewbogott: ok! adding a probe to check http status should be fairly straightfoward, that will get you e.g. success/failure/latency in grafana [15:39:53] godog: and it sounds like you think I should just keep the existing icinga check rather than trying to connect the two? That's easy. [15:41:00] andrewbogott: yeah I'd say add the probing first and see what data/value you get out of it [15:41:05] ok [15:41:06] thanks! [15:42:15] RECOVERY - Check Varnish expiry mailbox lag on cp1099 is OK: OK: expiry mailbox lag is 76168 [15:43:44] andrewbogott: np, feel free to send the puppet reviews [15:43:57] 10Operations, 10Deployment-Systems, 10JobRunner-Service, 10Release-Engineering-Team (Next), 10Scap (Scap3-Adoption-Phase1): Figure out how to disable starting of jobrunner/jobchron in the non-active DC - https://phabricator.wikimedia.org/T167104#3558930 (10thcipriani) @fgiunchedi gave me feedback on my p... [15:46:12] (03CR) 10Gehel: [C: 032] role::wdqs: move to the future parser [puppet] - 10https://gerrit.wikimedia.org/r/374322 (https://phabricator.wikimedia.org/T171704) (owner: 10Gehel) [15:46:17] (03PS2) 10Gehel: role::wdqs: move to the future parser [puppet] - 10https://gerrit.wikimedia.org/r/374322 (https://phabricator.wikimedia.org/T171704) [15:49:00] 10Operations, 10Traffic, 10Wikidata, 10wikiba.se, 10Wikidata-Sprint-2016-11-08: [Task] move wikiba.se webhosting to wikimedia misc-cluster - https://phabricator.wikimedia.org/T99531#3558970 (10Lydia_Pintscher) [15:50:35] (03PS1) 10Gehel: role::elasticsearch::(cirrus|relforge): move to the future parser [puppet] - 10https://gerrit.wikimedia.org/r/374341 (https://phabricator.wikimedia.org/T171704) [15:51:17] 10Operations, 10Puppet, 10Patch-For-Review, 10User-Joe: Switch all hosts to the future parser - https://phabricator.wikimedia.org/T171704#3558973 (10Gehel) [15:53:42] (03PS1) 10Volans: mediawiki: fix logrotating in wikidata cronjob (2) [puppet] - 10https://gerrit.wikimedia.org/r/374342 (https://phabricator.wikimedia.org/T171460) [15:57:27] T169939: Decommission Cassandra: restbase2005-a.codfw.wmnet [15:57:27] T169939: End of August milestone: Cassandra 3 cluster in production - https://phabricator.wikimedia.org/T169939 [16:01:19] (03CR) 10Alexandros Kosiaris: "the entire commit message is wrong on this one. It's all about logstash ofc" [dns] - 10https://gerrit.wikimedia.org/r/372857 (https://phabricator.wikimedia.org/T173565) (owner: 10Alexandros Kosiaris) [16:05:05] (03PS2) 10Awight: Phabricator: Override the frog token's label [puppet] - 10https://gerrit.wikimedia.org/r/371660 (https://phabricator.wikimedia.org/T173208) (owner: 10Greg Grossmeier) [16:05:31] (03Abandoned) 10Reedy: test [puppet] - 10https://gerrit.wikimedia.org/r/373107 (owner: 10Reedy) [16:15:20] (03CR) 10Ladsgroup: [C: 031] mediawiki: fix logrotating in wikidata cronjob (2) [puppet] - 10https://gerrit.wikimedia.org/r/374342 (https://phabricator.wikimedia.org/T171460) (owner: 10Volans) [16:19:23] (03PS2) 10Volans: mediawiki: fix logrotating in wikidata cronjob (2) [puppet] - 10https://gerrit.wikimedia.org/r/374342 (https://phabricator.wikimedia.org/T171460) [16:21:47] (03CR) 10Ladsgroup: mediawiki: fix logrotating in wikidata cronjob (2) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/374342 (https://phabricator.wikimedia.org/T171460) (owner: 10Volans) [16:22:34] (03PS3) 10Volans: mediawiki: fix logrotating in wikidata cronjob (2) [puppet] - 10https://gerrit.wikimedia.org/r/374342 (https://phabricator.wikimedia.org/T171460) [16:24:30] anyone have a moment to possibly review https://gerrit.wikimedia.org/r/#/c/373676/2 ? 's a small patch, I'd appreciate it :) [16:25:02] (03CR) 10Ladsgroup: [C: 031] mediawiki: fix logrotating in wikidata cronjob (2) [puppet] - 10https://gerrit.wikimedia.org/r/374342 (https://phabricator.wikimedia.org/T171460) (owner: 10Volans) [16:25:55] (03CR) 10Muehlenhoff: [C: 031] "Looks good to me" [puppet] - 10https://gerrit.wikimedia.org/r/374332 (https://phabricator.wikimedia.org/T152712) (owner: 10Elukey) [16:26:00] (03PS4) 10Volans: mediawiki: fix logrotating in wikidata cronjob (2) [puppet] - 10https://gerrit.wikimedia.org/r/374342 (https://phabricator.wikimedia.org/T171460) [16:28:37] (03CR) 10Chad: [C: 031] "This is fine to land whenever, it doesn't do anything until 2.15.x" [puppet] - 10https://gerrit.wikimedia.org/r/373520 (owner: 10Paladox) [16:29:20] (03CR) 10Chad: [C: 031] "Let's do this this week" [puppet] - 10https://gerrit.wikimedia.org/r/366910 (owner: 10Paladox) [16:29:49] 10Operations, 10Ops-Access-Requests, 10Discovery, 10Wikidata, and 3 others: allow wdqs-admins to pool / depool wdqs servers - https://phabricator.wikimedia.org/T172798#3559082 (10Gehel) 05Open>03Resolved Access has been granted [16:31:33] 10Operations, 10monitoring: Review check_puppetrun frequency - https://phabricator.wikimedia.org/T173427#3559091 (10faidon) [16:31:44] (03CR) 10Chad: [C: 031] "This is fine, let's do it." [puppet] - 10https://gerrit.wikimedia.org/r/356645 (https://phabricator.wikimedia.org/T43608) (owner: 10Paladox) [16:32:15] 10Operations, 10monitoring, 10netops: pmacct should be upgraded to 1.6.2 on Stretch - https://phabricator.wikimedia.org/T173489#3559093 (10elukey) As Faidon pointed out in the other task, it is `stretch-backports` that offers librdkafka 0.11, not the regular stretch repo, so a quick solution for rhenium woul... [16:32:26] 10Operations, 10monitoring, 10netops, 10User-Elukey: pmacct should be upgraded to 1.6.2 on Stretch - https://phabricator.wikimedia.org/T173489#3559097 (10elukey) [16:32:51] (03PS2) 10Giuseppe Lavagetto: git::clone: enhance compatibility with the future parser [puppet] - 10https://gerrit.wikimedia.org/r/374321 (https://phabricator.wikimedia.org/T171704) [16:32:53] (03PS1) 10Giuseppe Lavagetto: role::mariadb::misc: fix template scoping [puppet] - 10https://gerrit.wikimedia.org/r/374349 (https://phabricator.wikimedia.org/T171704) [16:32:55] (03PS1) 10Giuseppe Lavagetto: phabricator::logmail: fix scoping of templates [puppet] - 10https://gerrit.wikimedia.org/r/374350 (https://phabricator.wikimedia.org/T171704) [16:32:57] (03PS1) 10Giuseppe Lavagetto: role::mariadb::misc::phabricator: fix template scoping [puppet] - 10https://gerrit.wikimedia.org/r/374351 (https://phabricator.wikimedia.org/T171704) [16:32:59] (03PS1) 10Giuseppe Lavagetto: requesttracker::config: fix template scoping [puppet] - 10https://gerrit.wikimedia.org/r/374352 (https://phabricator.wikimedia.org/T171704) [16:33:01] (03PS1) 10Giuseppe Lavagetto: ganglia::gmetad::rrdcached: fix template scoping [puppet] - 10https://gerrit.wikimedia.org/r/374353 (https://phabricator.wikimedia.org/T171704) [16:33:04] (03CR) 10Volans: [C: 032] mediawiki: fix logrotating in wikidata cronjob (2) [puppet] - 10https://gerrit.wikimedia.org/r/374342 (https://phabricator.wikimedia.org/T171460) (owner: 10Volans) [16:34:07] 10Operations, 10monitoring, 10netops, 10User-fgiunchedi: Grafana dashboards for librenms graphite data - https://phabricator.wikimedia.org/T171823#3559109 (10fgiunchedi) [16:39:07] (03PS2) 10Matthias Mullie: Add missing THREED2PNG_PATH [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/373595 (https://phabricator.wikimedia.org/T161719) [16:39:13] !log demon@tin Pruned MediaWiki: 1.30.0-wmf.14 [keeping static files] (duration: 02m 00s) [16:39:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:46:02] !log demon@tin Pruned MediaWiki: 1.30.0-wmf.10 (duration: 02m 41s) [16:46:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:53:36] (03Abandoned) 10Chad: Install Extension:Translate on labswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/214893 (https://phabricator.wikimedia.org/T100313) (owner: 10Ladsgroup) [16:53:40] (03Abandoned) 10Chad: Cleanup: squid.php → ReverseProxy.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/309742 (https://phabricator.wikimedia.org/T104148) (owner: 10Dereckson) [16:53:58] (03PS1) 10Urbanecm: Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374358 (https://phabricator.wikimedia.org/T174357) [16:54:07] (03Abandoned) 10Chad: Localisation of Babel categories on nap.wikipedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/263342 (https://phabricator.wikimedia.org/T123188) (owner: 10Mdann52) [16:54:10] (03Abandoned) 10Chad: Disable wgIncludeLegacyJavaScript on all sites [mediawiki-config] - 10https://gerrit.wikimedia.org/r/277823 (owner: 10Jforrester) [16:54:41] (03Abandoned) 10Chad: WIP: `scap scrape` plugin split out from change 306259 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312016 (owner: 1020after4) [16:54:43] (03Abandoned) 10Chad: Enable Flow on wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/309499 (https://phabricator.wikimedia.org/T127792) (owner: 10Dereckson) [16:54:46] (03Abandoned) 10Chad: Apply rate limit to edits for normal users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/280002 (https://phabricator.wikimedia.org/T56515) (owner: 10Jforrester) [16:54:50] (03Abandoned) 10Chad: [DO NOT MERGE] Switch FlaggedRevs to "flagged protection" mode on huwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/260224 (https://phabricator.wikimedia.org/T121995) (owner: 10Gergő Tisza) [16:54:51] (03Abandoned) 10Chad: Dynamically fiddle with wgLocalDatabases to recognise wikitech separation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/280704 (https://phabricator.wikimedia.org/T131385) (owner: 10Alex Monk) [16:54:53] (03Abandoned) 10Chad: Revert "Revert "Move sourceswiki special.dblist->wikisource.dblist"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/227738 (owner: 10Alex Monk) [16:54:56] (03Abandoned) 10Chad: VisualEditor: Enabled for logged-out users on the English Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242042 (https://phabricator.wikimedia.org/T90662) (owner: 10Jforrester) [16:54:59] (03Abandoned) 10Chad: [WIP] Make VisualEditor access RESTbase directly on private wikis too [mediawiki-config] - 10https://gerrit.wikimedia.org/r/200107 (owner: 10Jforrester) [17:00:05] gehel: Dear anthropoid, the time has come. Please deploy Wikidata Query Service weekly deploy (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170828T1700). [17:00:05] SMalyshev and gehel: A patch you scheduled for Wikidata Query Service weekly deploy is about to be deployed. Please be available during the process. [17:00:29] (03PS5) 10Alexandros Kosiaris: Upgrade to kubernetes 1.7.4 [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/373554 (https://phabricator.wikimedia.org/T170119) [17:07:21] !log gehel@tin Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided) [17:07:28] !log gehel@tin Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided) [17:07:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:07:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:08:37] (03CR) 10Samtar: [C: 031] Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374358 (https://phabricator.wikimedia.org/T174357) (owner: 10Urbanecm) [17:08:46] (03CR) 10Gehel: [C: 032] wdqs - send logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/371939 (https://phabricator.wikimedia.org/T172710) (owner: 10Gehel) [17:08:51] (03PS13) 10Gehel: wdqs - send logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/371939 (https://phabricator.wikimedia.org/T172710) [17:09:18] 10Operations, 10netops: DDOS_PROTOCOL_VIOLATION_SET: Protocol Rejectv6:aggregate is violated - https://phabricator.wikimedia.org/T174364#3559390 (10ayounsi) [17:09:54] !log gehel@tin Finished deploy [wdqs/wdqs@90f4e2d]: (no justification provided) (duration: 02m 27s) [17:10:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:10:35] (03PS6) 10Alexandros Kosiaris: Upgrade to kubernetes 1.7.4 [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/373554 (https://phabricator.wikimedia.org/T170119) [17:10:39] 10Operations, 10Phabricator, 10Traffic, 10Zero: Missing IP addresses for Maroc Telecom - https://phabricator.wikimedia.org/T174342#3559407 (10Aklapper) Thanks for reporting this. **Phab conf:** Regarding the manual IP list for Phab itself in https://gerrit.wikimedia.org/r/#/c/368775/ , based on a few file... [17:11:07] !log pushing "aggregate defaults discard" to cr2-knams - T174364 [17:11:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:11:19] T174364: DDOS_PROTOCOL_VIOLATION_SET: Protocol Rejectv6:aggregate is violated - https://phabricator.wikimedia.org/T174364 [17:12:38] PROBLEM - puppet last run on wdqs2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:12:39] PROBLEM - puppet last run on wdqs2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:12:48] PROBLEM - puppet last run on wdqs2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:13:06] ^ puppet dependency issue, this is me, fix coming up [17:14:13] (03PS1) 10Gehel: wdqs - fix broken dependency [puppet] - 10https://gerrit.wikimedia.org/r/374359 [17:15:04] (03CR) 10Gehel: [C: 032] wdqs - fix broken dependency [puppet] - 10https://gerrit.wikimedia.org/r/374359 (owner: 10Gehel) [17:15:15] (03CR) 10Smalyshev: Enable access to arbitrary namespaces for WDQS (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/373404 (https://phabricator.wikimedia.org/T157676) (owner: 10Smalyshev) [17:16:02] (03PS2) 10Smalyshev: Enable access to arbitrary namespaces for WDQS [puppet] - 10https://gerrit.wikimedia.org/r/373404 (https://phabricator.wikimedia.org/T157676) [17:16:04] (03PS1) 10Elukey: profile::pmacct: pin librdkafka to stretch version [puppet] - 10https://gerrit.wikimedia.org/r/374360 (https://phabricator.wikimedia.org/T173489) [17:18:38] (03PS2) 10Elukey: profile::pmacct: pin librdkafka to stretch version [puppet] - 10https://gerrit.wikimedia.org/r/374360 (https://phabricator.wikimedia.org/T173489) [17:18:39] RECOVERY - puppet last run on wdqs2002 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [17:18:39] RECOVERY - puppet last run on wdqs2001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [17:18:49] RECOVERY - puppet last run on wdqs2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:19:40] !log restarting wdqs-blazegraph and wdqs-updater for config change [17:19:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:20:52] (03PS3) 10Gehel: Enable access to arbitrary namespaces for WDQS [puppet] - 10https://gerrit.wikimedia.org/r/373404 (https://phabricator.wikimedia.org/T157676) (owner: 10Smalyshev) [17:21:18] (03CR) 10Gehel: [C: 032] Enable access to arbitrary namespaces for WDQS [puppet] - 10https://gerrit.wikimedia.org/r/373404 (https://phabricator.wikimedia.org/T157676) (owner: 10Smalyshev) [17:23:00] (03PS7) 10Alexandros Kosiaris: Upgrade to kubernetes 1.7.4 [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/373554 (https://phabricator.wikimedia.org/T170119) [17:28:59] (03PS1) 10Gehel: wdqs - use logstach UDP instead of logstash TCP as a logback appender [puppet] - 10https://gerrit.wikimedia.org/r/374362 (https://phabricator.wikimedia.org/T172710) [17:29:10] (03PS2) 10Gehel: wdqs - use logstach UDP instead of logstash TCP as a logback appender [puppet] - 10https://gerrit.wikimedia.org/r/374362 (https://phabricator.wikimedia.org/T172710) [17:29:50] (03CR) 10Gehel: [C: 032] wdqs - use logstach UDP instead of logstash TCP as a logback appender [puppet] - 10https://gerrit.wikimedia.org/r/374362 (https://phabricator.wikimedia.org/T172710) (owner: 10Gehel) [17:31:05] !log restarting wdqs-blazegraph and wdqs-updater for config change [17:31:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:42:52] (03PS2) 10Niharika29: Enable wgEchoPerUserBlacklist at all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373133 (https://phabricator.wikimedia.org/T173838) (owner: 10Urbanecm) [17:48:42] 10Operations, 10Traffic, 10Community-Liaisons (Jul-Sep 2017), 10Patch-For-Review, 10User-Johan: Communicate dropping IE8-on-XP support (a security change) to affected editors and other community members - https://phabricator.wikimedia.org/T163251#3191313 (10leila) I see that the Arabic text in the banner... [17:48:51] 10Operations, 10ops-codfw, 10netops: Power alarm flap on asw-d-codfw:et-7/0/52 channel 3 - https://phabricator.wikimedia.org/T174366#3559481 (10ayounsi) [17:51:45] (03PS1) 10Herron: icinga: add -u option to check_nrpe commands [puppet] - 10https://gerrit.wikimedia.org/r/374368 (https://phabricator.wikimedia.org/T172131) [17:56:45] !log T169939: Decommission Cassandra: restbase2005-b.codfw.wmnet [17:56:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:56:59] T169939: End of August milestone: Cassandra 3 cluster in production - https://phabricator.wikimedia.org/T169939 [17:57:48] 10Operations, 10ops-eqiad, 10Discovery, 10Wikidata, and 3 others: rack/setup/install wdqs100[45].eqiad.wmnet - https://phabricator.wikimedia.org/T171210#3559508 (10RobH) a:05Cmjohnson>03RobH [17:59:43] !log pushing "aggregate defaults discard" to *ams - T174364 [17:59:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:59:54] T174364: DDOS_PROTOCOL_VIOLATION_SET: Protocol Rejectv6:aggregate is violated - https://phabricator.wikimedia.org/T174364 [18:00:04] addshore, hashar, anomie, RainbowSprinkles, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: Respected human, time to deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170828T1800). Please do the needful. [18:00:05] TheresNoTime, AaronSchulz, davidwbarratt, Dmaza, Jdlrobson, Urbanecm, MaxSem, and Niharika: A patch you scheduled for Morning SWAT (Max 8 patches) is about to be deployed. Please be available during the process. [18:00:15] I'm here [18:00:18] Who's the swatter? [18:00:19] here! [18:00:19] o/ [18:02:28] Nobody's SWATting? [18:03:08] 10Operations, 10ops-eqiad, 10Cloud-Services, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3559555 (10madhuvishy) Update: We are still blocked on talking to HP Support about the disk shelves. [18:03:30] I can SWAT. [18:04:14] The extension distributor one is kinda silly for a SWAT deploy. It can wait until tomorrow when it'll go with the train [18:04:36] sure, either way [18:04:49] (03PS4) 10Niharika29: Upload wikipedia-wordmark-zh-c.svg [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373948 (https://phabricator.wikimedia.org/T174192) (owner: 10Samtar) [18:06:30] (03CR) 10Niharika29: [C: 032] "SWAT." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373948 (https://phabricator.wikimedia.org/T174192) (owner: 10Samtar) [18:07:12] 10Operations, 10ops-eqiad, 10Discovery, 10Wikidata, and 3 others: rack/setup/install wdqs100[45].eqiad.wmnet - https://phabricator.wikimedia.org/T171210#3559557 (10RobH) [18:07:27] (03CR) 10Niharika29: [C: 032] "SWAT." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373133 (https://phabricator.wikimedia.org/T173838) (owner: 10Urbanecm) [18:08:10] (im here) [18:08:15] (03Merged) 10jenkins-bot: Upload wikipedia-wordmark-zh-c.svg [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373948 (https://phabricator.wikimedia.org/T174192) (owner: 10Samtar) [18:08:17] Nikerabbit: let me know when you're done [18:08:24] (03CR) 10jenkins-bot: Upload wikipedia-wordmark-zh-c.svg [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373948 (https://phabricator.wikimedia.org/T174192) (owner: 10Samtar) [18:08:30] *Niharika [18:08:37] tab completed too fast ;) [18:08:42] Will do! [18:09:04] (03PS2) 10Madhuvishy: wmcs: generate /etc/dbusers.yaml with ordered_yaml() [puppet] - 10https://gerrit.wikimedia.org/r/372876 (owner: 10BryanDavis) [18:09:10] 10Operations, 10Ops-Access-Requests: Requesting access to restricted hosts for dbarratt - https://phabricator.wikimedia.org/T173779#3539582 (10herron) This request was approved at todays operations meeting. Will follow up with a patch for shell access shortly! [18:09:32] (03PS1) 10MarcoAurelio: Update es.wiktionary logo from SVG version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374370 [18:09:57] TheresNoTime: https://gerrit.wikimedia.org/r/#/c/373948/4 is on mwdebug1002. Please check. [18:10:36] (03PS3) 10Niharika29: Enable wgEchoPerUserBlacklist at all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373133 (https://phabricator.wikimedia.org/T173838) (owner: 10Urbanecm) [18:11:45] Niharika: all good :) [18:11:46] (03CR) 10Madhuvishy: [C: 032] wmcs: generate /etc/dbusers.yaml with ordered_yaml() [puppet] - 10https://gerrit.wikimedia.org/r/372876 (owner: 10BryanDavis) [18:12:02] TheresNoTime: Alright. Syncing it out then. [18:14:17] !log niharika29@tin Synchronized static/images/mobile/copyright/: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 44s) [18:14:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:15:16] (03CR) 10Niharika29: [C: 032] Enable wgEchoPerUserBlacklist at all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373133 (https://phabricator.wikimedia.org/T173838) (owner: 10Urbanecm) [18:15:26] !log niharika29@tin Synchronized wmf-config/InitialiseSettings.php: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 45s) [18:15:30] TheresNoTime: Done. [18:15:32] ^ first SWAT'd patch, sorta sadly proud :') thanks for obliging Niharika, and apologies for the ExtensionDistributor one RainbowSprinkles [18:15:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:15:59] TheresNoTime: No worries, just kinda a trivial thing to SWAT when it goes out tomorrow anyway :) [18:16:03] (03PS2) 10Niharika29: Fix incorrect Special:Userlogin name in Popups blacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373696 (https://phabricator.wikimedia.org/T170169) (owner: 10Pmiazga) [18:16:36] (03PS5) 10Niharika29: pagePreviews: remove invalidated popup sampling rate variables [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373171 (https://phabricator.wikimedia.org/T172291) (owner: 10Niedzielski) [18:17:28] jdlrobson: https://gerrit.wikimedia.org/r/#/c/373920/ gives a merge conflict for rebasing. [18:17:43] (03Merged) 10jenkins-bot: Enable wgEchoPerUserBlacklist at all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373133 (https://phabricator.wikimedia.org/T173838) (owner: 10Urbanecm) [18:17:48] Niharika: will take a look [18:17:53] (03CR) 10jenkins-bot: Enable wgEchoPerUserBlacklist at all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373133 (https://phabricator.wikimedia.org/T173838) (owner: 10Urbanecm) [18:18:01] (03PS1) 10Rush: mirror openstack refactor values up to changeset 11816fd1b2 [labs/private] - 10https://gerrit.wikimedia.org/r/374372 [18:18:35] davidwbarratt: DMaza Your Echo patch is on mwdebug1002. [18:19:00] davidwbarratt: Can you check? I'm here if something's confusing about the process. :) [18:19:09] Niharika: done. weird gerrit couldnt handle that [18:19:12] (03PS3) 10Jdlrobson: Enable an A/B test for page previews on EN and DE wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373920 (https://phabricator.wikimedia.org/T172291) [18:19:27] Niharika woot! it looks good to me! [18:19:51] jdlrobson: RainbowSprinkles: There are increasingly patches like that. Simple rebases that gerrit doesn't seem to be able to handle. Happened with Roan the other day and me last week. [18:19:55] Niharika for the record I checked on English Wikipedia [18:20:16] davidwbarratt: Alright then! [18:20:39] (03CR) 10Rush: [V: 032 C: 032] mirror openstack refactor values up to changeset 11816fd1b2 [labs/private] - 10https://gerrit.wikimedia.org/r/374372 (owner: 10Rush) [18:20:39] DMaza ¿bueno? [18:20:52] * AaronSchulz proceeds [18:21:15] davidwbarratt, i'm not sure how to check [18:21:16] ah, already being merged, nice [18:21:29] * AaronSchulz will just monitor then [18:21:37] DMaza enable the extenion in your browser and select mwdebug1002 [18:21:43] !log niharika29@tin Synchronized wmf-config/InitialiseSettings.php: Enable wgEchoPerUserBlacklist at all wikis T173838 (duration: 00m 43s) [18:21:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:21:56] T173838: Enable EchoPerUserBlacklist on all Wikimedia wikis with Echo enabled - https://phabricator.wikimedia.org/T173838 [18:22:00] (03PS3) 10Niharika29: Fix incorrect Special:Userlogin name in Popups blacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373696 (https://phabricator.wikimedia.org/T170169) (owner: 10Pmiazga) [18:22:10] (03CR) 10Niharika29: [C: 032] pagePreviews: remove invalidated popup sampling rate variables [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373171 (https://phabricator.wikimedia.org/T172291) (owner: 10Niedzielski) [18:22:14] davidwbarratt, then what? [18:22:21] (03PS6) 10Niharika29: pagePreviews: remove invalidated popup sampling rate variables [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373171 (https://phabricator.wikimedia.org/T172291) (owner: 10Niedzielski) [18:22:22] DMaza and then go to any wikimedia site and go to the echo preference [18:22:34] DMaza errr.. Preferences -> Notifications [18:22:48] davidwbarratt: I synced it now so it's live everywhere. [18:22:56] Niharika doh! [18:23:00] Niharika well good! [18:23:23] davidwbarratt, got it.. looks good here [18:25:07] (03CR) 10Niharika29: [C: 032] Fix incorrect Special:Userlogin name in Popups blacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373696 (https://phabricator.wikimedia.org/T170169) (owner: 10Pmiazga) [18:28:17] zeljkof: Why is wmf.15 on detached HEAD? [18:28:42] Niharika: he is probably not around any longer [18:28:55] greg-g: Okay, what do you recommend I do? [18:29:25] RainbowSprinkles: ^ [18:32:15] twentyafterfour: ^^ [18:32:16] One of Aarons commits... [18:32:26] is merged into the branch [18:32:30] but it's ontop of security patches [18:32:44] !log ayounsi@tin Started deploy [librenms/librenms@8c9da11]: (no justification provided) [18:32:48] !log ayounsi@tin Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 05s) [18:32:49] And another isn't merged [18:32:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:32:55] greg-g: I think it's just a rebase needed [18:33:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:33:09] fyi im on tornado and flash flood watch (just in case i disappear without explanation) [18:33:12] * greg-g nods [18:33:19] !log upgrading librenms to 1.31 [18:33:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:33:33] Reedy: yeah [18:33:38] (03PS7) 10Niharika29: pagePreviews: remove invalidated popup sampling rate variables [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373171 (https://phabricator.wikimedia.org/T172291) (owner: 10Niedzielski) [18:34:09] jdlrobson: https://gerrit.wikimedia.org/r/#/c/373696/3 is on mwdebug1002. [18:34:16] k testing [18:34:34] 10Operations, 10Discovery, 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Search (Current work): rack/setup/install wdqs100[45].eqiad.wmnet - https://phabricator.wikimedia.org/T171210#3559597 (10RobH) a:05RobH>03Gehel [18:34:35] (03CR) 10Niharika29: [C: 032] pagePreviews: remove invalidated popup sampling rate variables [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373171 (https://phabricator.wikimedia.org/T172291) (owner: 10Niedzielski) [18:34:39] greg-g: Niharika fixed [18:34:39] Niharika: I was deploying a core patch during EU SWAT today [18:34:47] I think Aarons change needs deploying still [18:34:51] 10Operations, 10Discovery, 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Search (Current work): rack/setup/install wdqs100[45].eqiad.wmnet - https://phabricator.wikimedia.org/T171210#3457615 (10RobH) I've assigned this to @gehel as both hosts are now online and ready for service implementation. [18:35:00] Your branch and 'origin/wmf/1.30.0-wmf.15' have diverged, [18:35:00] and have 4 and 2 different commits each, respectively. [18:35:03] I do that rarely, maybe I did something wrong [18:35:15] PROBLEM - puppet last run on cp3034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:35:43] Probably sync-dir .15 for consistency before doing anything else [18:35:46] Niharika: looks good to go [18:36:00] Reedy: Okay. Will sync the directory first. [18:36:04] (03CR) 10RobH: [C: 031] Add shell account dbarratt and add to group restricted. [puppet] - 10https://gerrit.wikimedia.org/r/374374 (https://phabricator.wikimedia.org/T173779) (owner: 10Herron) [18:36:26] PROBLEM - LibreNMS HTTPS on netmon1002 is CRITICAL: HTTP CRITICAL: HTTP/1.0 500 Internal Server Error - 286 bytes in 0.026 second response time [18:36:45] Reedy: which rebase command did you use? [18:36:47] (03CR) 10Bartosz Dziewoński: [C: 031] Enable jQuery 3 on nlwiki, svwiki, plwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373987 (https://phabricator.wikimedia.org/T124742) (owner: 10Krinkle) [18:36:50] (03CR) 10Herron: [C: 032] Add shell account dbarratt and add to group restricted. [puppet] - 10https://gerrit.wikimedia.org/r/374374 (https://phabricator.wikimedia.org/T173779) (owner: 10Herron) [18:36:56] (03PS2) 10Herron: Add shell account dbarratt and add to group restricted. [puppet] - 10https://gerrit.wikimedia.org/r/374374 (https://phabricator.wikimedia.org/T173779) [18:37:22] Reedy: For future, what did you do to fix it? [18:37:40] (03Merged) 10jenkins-bot: pagePreviews: remove invalidated popup sampling rate variables [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373171 (https://phabricator.wikimedia.org/T172291) (owner: 10Niedzielski) [18:37:44] !log niharika29@tin Synchronized wmf-config/InitialiseSettings.php: Fix incorrect Special:UserLogin in Popups blacklist https://gerrit.wikimedia.org/r/#/c/373696/ (duration: 00m 43s) [18:37:47] I wasn't sure if checkout + rebase would mess with the security patches [18:37:49] (03CR) 10jenkins-bot: pagePreviews: remove invalidated popup sampling rate variables [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373171 (https://phabricator.wikimedia.org/T172291) (owner: 10Niedzielski) [18:37:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:38:40] git fetch --all; git rebase wmf/1.30.0-wmf.15; git checkout wmf/1.30.0-wmf.15; git pull; git rebase [18:38:50] 10Operations, 10Phabricator, 10Traffic, 10Zero: Missing IP addresses for Maroc Telecom - https://phabricator.wikimedia.org/T174342#3559611 (10Dispenser) @Aklapper The user loads 403 Phabricator, pulls down the notification bar with Wi-Fi disabled, Maroc Telecom, and disables/re-enables Mobile Data a few ti... [18:39:41] Reedy: ok, so the sec. patches work doing that. Good to know. [18:39:54] Reedy: `git pull -r` [18:39:57] Less characters :p [18:40:00] heh [18:40:53] AaronSchulz: RainbowSprinkles (I think) made a git config change a while ago to stop git pull just nuking security patches [18:41:02] I did [18:41:07] pull rebases by default [18:41:10] that one [18:41:30] And if there are conflicts? [18:41:44] rm -rf and cry [18:41:49] :P [18:41:58] * Niharika documents that [18:42:08] "Why are you crying?" [18:42:12] "Reedy told me I should" [18:42:35] AFAIK, it's very rare that you actually get any conflicts [18:42:40] RainbowSprinkles: * Fewer, BTW. :-) [18:43:07] Syncing the wmf15 directory is going real slow..... [18:43:12] Ewwwwww [18:43:13] NO [18:43:19] Don't sync-file a whole php-* directory [18:43:22] !log aaron@tin Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: 2d83569397 - Fix old regression in HTMLCacheUpdate de-duplication (duration: 00m 44s) [18:43:22] It lints everything [18:43:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:43:35] RainbowSprinkles: Whoops, okay. Aborted. [18:43:44] If you need to sync that much, just do a full scap [18:43:59] * AaronSchulz wonders why there was no lock conflict...meh [18:44:00] RainbowSprinkles: I was going by what Reedy said above "Probably sync-dir .15 for consistency before doing anything else" [18:44:05] (best case scenario scap is down to about 15 minutes, fwiw) [18:44:08] 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Requesting access to restricted hosts for dbarratt - https://phabricator.wikimedia.org/T173779#3559639 (10herron) 05Open>03Resolved a:03herron Shell account `dbarratt` has been created and added to group `restricted` Notice: /Stage[main]/Admin... [18:44:19] Niharika: Can't trust that Reedy fellow [18:44:22] Very shady [18:44:24] :p [18:44:30] :P [18:44:36] There's reasons bd808 refers to me as a Chaos Monkey [18:45:01] jdlrobson: https://gerrit.wikimedia.org/r/#/c/373171/ is on mwdebug1002. [18:45:08] cool on it [18:45:17] Reedy: He didn't give you a sticker for it yet? :P [18:45:42] (03PS4) 10Niharika29: Enable an A/B test for page previews on EN and DE wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373920 (https://phabricator.wikimedia.org/T172291) (owner: 10Jdlrobson) [18:45:57] How many times must one rebase before gerrit lets one merge a change? :| [18:46:37] or,...how many rebases does it take to get the center of a merge? [18:46:43] (03CR) 10Niharika29: [C: 032] "SWAT." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373920 (https://phabricator.wikimedia.org/T172291) (owner: 10Jdlrobson) [18:46:47] rebasecpetion [18:46:57] Niharika: Basically, don't rebase anything until you're ready to C+2 [18:47:17] Niharika: you can sync that one [18:47:17] And if someone else merges something, shout at them [18:47:42] !log ayounsi@tin Started deploy [librenms/librenms@8c9da11]: (no justification provided) [18:47:44] !log ayounsi@tin Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 02s) [18:47:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:48:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:48:14] Gerrit could merge more stuff without rebases if people weren't so obsessive about no merges in their git logs :p [18:48:42] !log niharika29@tin Synchronized wmf-config/InitialiseSettings.php: pagePreviews: remove invalidated popup sampling rate variables https://gerrit.wikimedia.org/r/#/c/373171/7 (duration: 00m 43s) [18:48:45] !log restarted pdfrender instances in eqiad (T159922) [18:48:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:49:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:49:02] T159922: pdfrender fails to serve requests since Mar 8 00:30:32 UTC on scb1003 - https://phabricator.wikimedia.org/T159922 [18:49:09] 10Operations, 10Phabricator, 10Traffic, 10Zero: Missing IP addresses for Maroc Telecom - https://phabricator.wikimedia.org/T174342#3559649 (10Dispenser) I found those ranges in a GeoIP database for Morocco back in June (Z567#10320). We didn't include them for fear of blocking Orange Morocco (the only non-... [18:49:33] PROBLEM - Check systemd state on mw1259 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [18:49:39] (03Merged) 10jenkins-bot: Enable an A/B test for page previews on EN and DE wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373920 (https://phabricator.wikimedia.org/T172291) (owner: 10Jdlrobson) [18:49:51] (03CR) 10jenkins-bot: Enable an A/B test for page previews on EN and DE wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373920 (https://phabricator.wikimedia.org/T172291) (owner: 10Jdlrobson) [18:50:13] PROBLEM - pdfrender on scb1001 is CRITICAL: connect to address 10.64.0.16 and port 5252: Connection refused [18:50:55] (03CR) 10Niharika29: Enable an A/B test for page previews on EN and DE wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373920 (https://phabricator.wikimedia.org/T172291) (owner: 10Jdlrobson) [18:51:04] jdlrobson: https://gerrit.wikimedia.org/r/#/c/373920/4 [18:51:09] I left a comment. [18:51:26] (03PS2) 10Niharika29: Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374358 (https://phabricator.wikimedia.org/T174357) (owner: 10Urbanecm) [18:51:31] (03CR) 10Niharika29: [C: 032] Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374358 (https://phabricator.wikimedia.org/T174357) (owner: 10Urbanecm) [18:52:06] ack Niharika you are right [18:52:20] TBH MaxSem caught it not me. :P [18:52:29] fixing.. this really needs to go out today :-S [18:53:05] (03PS1) 10Jdlrobson: Add missing wg Prefix [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374378 [18:53:08] thanks for catching that MaxSem [18:53:12] that saved me some testing time :) [18:53:15] ^ Niharika [18:53:59] AaronSchulz: Did you deploy your change? [18:54:07] yep [18:54:11] Gotcha. [18:54:17] 10Operations, 10Ops-Access-Requests, 10Analytics, 10Research, and 2 others: NDA, MOU and LDAP (analytics cluster) for Shilad Sen - https://phabricator.wikimedia.org/T171988#3482327 (10Ottomata) Ya, pretty sure this will need `analytics-privatedata-users`. I'm on clinic duty now, this has already been appr... [18:54:34] (03Merged) 10jenkins-bot: Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374358 (https://phabricator.wikimedia.org/T174357) (owner: 10Urbanecm) [18:55:43] (03PS2) 10Niharika29: Add missing wg Prefix [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374378 (owner: 10Jdlrobson) [18:55:49] (03CR) 10Niharika29: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374378 (owner: 10Jdlrobson) [18:56:22] (03PS2) 10Mforns: Add QuickSurvey schemas to EventLogging white-list [puppet] - 10https://gerrit.wikimedia.org/r/368769 (https://phabricator.wikimedia.org/T172112) [18:56:31] (03PS1) 10Ottomata: Add shiladsen to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/374379 (https://phabricator.wikimedia.org/T171988) [18:56:39] (03CR) 10jerkins-bot: [V: 04-1] Add QuickSurvey schemas to EventLogging white-list [puppet] - 10https://gerrit.wikimedia.org/r/368769 (https://phabricator.wikimedia.org/T172112) (owner: 10Mforns) [18:56:45] 10Operations, 10Electron-PDFs, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking), 10Services (blocked): pdfrender fails to serve requests since Mar 8 00:30:32 UTC on scb1003 - https://phabricator.wikimedia.org/T159922#3559658 (10GWicke) [18:57:00] (03CR) 10Ottomata: [C: 032] Add shiladsen to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/374379 (https://phabricator.wikimedia.org/T171988) (owner: 10Ottomata) [18:57:31] Niharika, did you deploy my change? [18:57:34] RECOVERY - LibreNMS HTTPS on netmon1002 is OK: HTTP OK: HTTP/1.1 200 OK - 8519 bytes in 0.031 second response time [18:57:51] Urbanecm: Not yet, I'm waiting on jdlrobson's fix to be merged. [18:58:14] Ok [18:59:07] jdlrobson: https://gerrit.wikimedia.org/r/#/c/373920/4 and Urbanecm your change are both on mwdebug1002. [18:59:16] Niharika: on it [18:59:44] Niharika, my change is working [18:59:59] MaxSem: Your change too. [19:01:17] Niharika, confirmed [19:01:28] Niharika: on 1002 ? [19:01:35] 10Operations, 10DBA, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3559677 (10KTC) [19:01:43] jdlrobson: Yeah. [19:02:45] jdlrobson: Something wrong? [19:02:59] Niharika: yeh the new patch doesn't look synced? [19:03:09] (https://gerrit.wikimedia.org/r/374378) [19:03:09] !log niharika29@tin Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: Fix exception on some combination of quotes T174060 (duration: 00m 44s) [19:03:13] jdlrobson: Not synced yet, just on mwdebug1002. [19:03:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:03:22] T174060: Triple apostrophe fails - https://phabricator.wikimedia.org/T174060 [19:03:24] yeh but im not seeing evidence of it working on mwdebug1002 [19:03:29] the other part is [19:03:35] (03PS32) 10Ottomata: Increase max kafka message size for changeprop and kafka main [puppet] - 10https://gerrit.wikimedia.org/r/372179 (owner: 10Ppchelko) [19:03:40] (03CR) 10Ottomata: [V: 032 C: 032] Increase max kafka message size for changeprop and kafka main [puppet] - 10https://gerrit.wikimedia.org/r/372179 (owner: 10Ppchelko) [19:03:44] jdlrobson: I can confirm using the log that it's there. Maybe caching? [19:03:50] (03PS1) 10Jforrester: RCFilters: Enable on watchlist for Beta Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374381 [19:03:52] (03PS1) 10Jforrester: RCFilters: Enable on watchlist for all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374382 [19:03:54] (03PS1) 10Jforrester: Cleanup: Removed wgEnableRcFiltersBetaFeature setting for Beta Cluster, true everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374383 [19:04:04] (03PS2) 10Jforrester: RCFilters: Enable on watchlist for Beta Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374381 [19:04:22] (03CR) 10Jforrester: [C: 04-2] "Not until next week." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374382 (owner: 10Jforrester) [19:04:28] Niharika: and you are 100% sure? I've tried clearing cache but nowt [19:04:32] (03PS1) 10Reedy: Add electcomwiki to private_wikis [puppet] - 10https://gerrit.wikimedia.org/r/374384 (https://phabricator.wikimedia.org/T174370) [19:04:33] RECOVERY - puppet last run on cp3034 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:05:06] jdlrobson: It's pulled into wmf-config/ and I did a scap pull on mwdebug1002. So yep, it's there. [19:05:22] 10Operations, 10OCG-General, 10Reading-Community-Engagement, 10Epic, and 3 others: [EPIC] (Proposal) Replicate core OCG features and sunset OCG service - https://phabricator.wikimedia.org/T150871#3559699 (10GWicke) @ovasileva, thank you for the update. Does this mean that OCG will be switched off by the en... [19:05:44] (03PS1) 10Reedy: Add electcom.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/374385 (https://phabricator.wikimedia.org/T174370) [19:06:05] Niharika: what's the value of wgPopupsEnabled ? [19:06:26] sorry $wgUsePopups [19:06:40] and $wgPopupsEventLogging [19:06:46] herron umm, so bast1001.wikimedia.org keeps asking for a password? [19:07:15] davidwbarratt, you're not providing a key? [19:07:25] MaxSem I am [19:07:33] jdlrobson, on which wiki? [19:07:42] MaxSem: enwiki [19:07:43] davidwbarratt it looks like the account isn't present there yet... [19:07:44] MaxSem basically this config https://wikitech.wikimedia.org/wiki/Production_shell_access#Standard_config [19:07:50] herron oh ok [19:07:57] puppet *just* created it! [19:08:16] herron BOOM! it works now! [19:08:20] awesome! [19:08:25] wmgUsePopups' => [ [19:08:26] 'default' => false, [19:08:26] 'sewikimedia' => true, // T68374 [19:08:26] // T136602, T162162, T162672: Make Page Previews enabled by default for the [19:08:26] // stage 0 and stage 1 wikis. [19:08:26] 'pp_stage0' => true, [19:08:26] 'pp_stage1' => true, [19:08:26] T68374: Enable Hovercards on se.wikimedia.org (Swedish chapter wiki) - https://phabricator.wikimedia.org/T68374 [19:08:26] T162672: Deploy page previews to 90% of users on all wikis but English and German - https://phabricator.wikimedia.org/T162672 [19:08:26] T136602: Graduate the Page Previews beta feature on stage 0 wikis - https://phabricator.wikimedia.org/T136602 [19:08:26] T162162: Deploy page previews to Hungarian and Hebrew wikipedias - https://phabricator.wikimedia.org/T162162 [19:08:26] ], [19:08:29] Whoops. [19:08:34] Niharika: i mean what happens when you check the value? [19:08:50] jdlrobson, Notice: Undefined variable: wgUsePopups [19:08:51] i think it's gonna be false, because i think pp_stage1 is not set up correctly [19:08:53] ak [19:09:02] !log rolling restart and rebalances of main-* kafka clusters for https://gerrit.wikimedia.org/r/#/c/372179/ [19:09:07] (03PS1) 10Reedy: Add electcom.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/374389 (https://phabricator.wikimedia.org/T174370) [19:09:08] ^ so that's needed as well for the fundraising A/B test [19:09:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:09:14] (03PS1) 10Jdlrobson: Enable Popups on en and de wiki for A/B test [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374390 (https://phabricator.wikimedia.org/T172291) [19:09:19] ^ that rather [19:10:28] (03PS2) 10Niharika29: Enable Popups on en and de wiki for A/B test [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374390 (https://phabricator.wikimedia.org/T172291) (owner: 10Jdlrobson) [19:11:13] (03CR) 10Niharika29: [C: 032] Enable Popups on en and de wiki for A/B test [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374390 (https://phabricator.wikimedia.org/T172291) (owner: 10Jdlrobson) [19:13:20] jdlrobson: Check now. [19:13:23] Niharika: on it [19:13:33] w00 [19:13:34] testing [19:14:23] RECOVERY - pdfrender on scb1001 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.003 second response time [19:14:35] !log restarting eventbus in codfw to verify https://gerrit.wikimedia.org/r/#/c/372179/ [19:14:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:15:38] !log scb1001: restarted pdfrender service [19:15:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:16:09] (03CR) 10jenkins-bot: Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374358 (https://phabricator.wikimedia.org/T174357) (owner: 10Urbanecm) [19:16:11] (03CR) 10jenkins-bot: Add missing wg Prefix [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374378 (owner: 10Jdlrobson) [19:16:13] (03CR) 10jenkins-bot: Enable Popups on en and de wiki for A/B test [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374390 (https://phabricator.wikimedia.org/T172291) (owner: 10Jdlrobson) [19:17:10] 10Operations, 10monitoring, 10netops, 10Patch-For-Review, 10User-fgiunchedi: Evaluate LibreNMS' Graphite backend - https://phabricator.wikimedia.org/T171167#3559847 (10ayounsi) LibreNMS upgraded to 1.31. https://github.com/librenms/librenms/releases/tag/1.31 [19:17:20] jdlrobson: Tested? [19:17:28] Niharika: still testing [19:20:51] What's with my change Niharika? [19:21:16] Niharika: looks like you can sync [19:21:30] jdlrobson: Unfortunately I pulled both your and jdlrobson's patch at the same time so they'll be synced together. Doing that now. [19:23:25] !log niharika29@tin Synchronized wmf-config/InitialiseSettings.php: Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki T174357; Enable popups on en and de wiki for A/B test T172291 (duration: 00m 43s) [19:23:26] jdlrobson: Urbanecm: Your changes are now live. Sorry Urbanecm for the delay. [19:23:34] Niharika: thanks testing again [19:23:35] Thank you [19:23:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:23:38] T174357: Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki - https://phabricator.wikimedia.org/T174357 [19:23:38] T172291: Launch page previews A/B test on enwiki and dewiki - https://phabricator.wikimedia.org/T172291 [19:24:37] !log restarting eventbus in eqiad to apply https://gerrit.wikimedia.org/r/#/c/372179/ [19:24:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:34:31] thanks Niharika just finished [19:34:34] stressful swat deploy :) [19:34:39] all is looking good [19:38:17] jouncebot: next [19:38:18] In 0 hour(s) and 21 minute(s): Services – Parsoid / OCG / Citoid / Mobileapps / ORES / … (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170828T2000) [19:41:34] Nothing for ORES, thanks! [19:47:44] (03Abandoned) 10Reedy: Rename -labs to -beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/320425 (https://phabricator.wikimedia.org/T150268) (owner: 10Reedy) [19:50:13] (03PS1) 10Ppchelko: Enable JobQueueEventBus on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374399 [19:51:39] (03CR) 10jerkins-bot: [V: 04-1] Enable JobQueueEventBus on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374399 (owner: 10Ppchelko) [19:53:54] PROBLEM - cassandra-c CQL 10.64.0.116:9042 on restbase1010 is CRITICAL: connect to address 10.64.0.116 and port 9042: Connection refused [19:55:08] (03PS2) 10Ppchelko: Enable JobQueueEventBus on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374399 [20:00:04] gwicke, cscott, arlolra, subbu, bearND, halfak, and Amir1: Respected human, time to deploy Services – Parsoid / OCG / Citoid / Mobileapps / ORES / … (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170828T2000). Please do the needful. [20:00:17] Nothing for ORES today :) [20:01:17] 10Operations, 10ORES, 10Scoring-platform-team, 10Patch-For-Review, and 2 others: Stress/capacity test new ores* cluster - https://phabricator.wikimedia.org/T169246#3560090 (10Halfak) New test today. Moral of the story is **TOO MANY FILE HANDLES**. https://grafana.wikimedia.org/dashboard/db/ores?orgId=1&... [20:03:57] I've got a Striker update to push if there is space in this services deploy window [20:07:10] 10Operations, 10ops-eqiad, 10Cloud-Services, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3560132 (10RobH) On labstore1006 when loading the HP raid utility via bios, it gives the following error: error: no such device: EMBEDDED250. It doe... [20:07:42] !log bd808@tin Started deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes) [20:07:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:08:07] !log bd808@tin Finished deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes) (duration: 00m 26s) [20:08:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:16:06] !log upgrading librenms to 1.31.01 [20:16:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:16:38] !log ayounsi@tin Started deploy [librenms/librenms@5fea59c]: (no justification provided) [20:16:44] !log ayounsi@tin Finished deploy [librenms/librenms@5fea59c]: (no justification provided) (duration: 00m 05s) [20:16:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:16:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:23:47] 10Operations, 10Services (doing): Disk errors: restbase1010.eqiad.wmnet - https://phabricator.wikimedia.org/T174392#3560205 (10Eevans) [20:23:58] 10Operations, 10Services (doing): Disk errors: restbase1010.eqiad.wmnet - https://phabricator.wikimedia.org/T174392#3560222 (10Eevans) p:05Triage>03High [20:26:57] anyone know specifically what the following disk errors mean? https://phabricator.wikimedia.org/P5934 [20:27:56] urandom: not a expert but it looks like write and ??save?? Errors [20:28:25] yeah, definitely [20:28:26] I forget the laymans term for flush i cant remember if thats save or load [20:30:41] 10Operations, 10ops-eqiad, 10Cloud-Services, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3560242 (10Cmjohnson) I started a ticket with HP. Your case was successfully submitted. Please note your Case ID: 5322481808 for future reference. [20:31:16] Sorry i couldnt decode it more urandom :/ [20:31:46] i'm pretty sure it means, 'replace your busted drive, dude' [20:35:19] urandom: +1 [20:35:20] :D [20:35:28] which host? [20:35:51] volans: restbase1010 [20:36:38] interestingly is not detected by the controller though [20:36:43] Lol [20:39:38] 10Operations, 10ops-eqiad, 10DC-Ops, 10cloud-services-team (Kanban): labvirt1015 crashes - https://phabricator.wikimedia.org/T171473#3560268 (10Cmjohnson) So far the error at least in the h/w log on the server has not returned.....keeping this open to monitor. [20:40:43] 10Operations, 10ops-eqiad, 10Discovery-Search (Current work): Degraded RAID on logstash1006 - https://phabricator.wikimedia.org/T173679#3560271 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson @gehel no, resolving....thx [20:42:08] 10Operations, 10Ops-Access-Requests, 10Analytics, 10Research, and 2 others: NDA, MOU and LDAP (analytics cluster) for Shilad Sen - https://phabricator.wikimedia.org/T171988#3560278 (10Shilad) 05Open>03Resolved Everything looks good now! Thanks for your quick help, @Ottomata! I'm going to close this tic... [20:42:53] i wonder if there is any chance it could limp along for a few more hours [20:43:37] those are JBOD right? so if it fails more work for you? :D [20:43:52] volans: umm, raid-0 [20:44:03] PROBLEM - Check Varnish expiry mailbox lag on cp1074 is CRITICAL: CRITICAL: expiry mailbox lag is 2031996 [20:45:07] volans: many eggs, one basket [20:45:22] perfect for a disk failure! :D [20:45:50] indeed [20:45:59] optimized, even [20:46:35] isn't the idea of raid-0 that the cluster duplicates the data, instead of the servers. so just depool the server? [20:46:43] PROBLEM - Check Varnish expiry mailbox lag on cp1099 is CRITICAL: CRITICAL: expiry mailbox lag is 2035058 [20:47:28] ebernhardson: there are 3 "servers" on each host, and each has some share of the data (or a replica of it) [20:47:49] :( [20:48:04] so if you just mark the whole thing a loss, which you can, you have a bunch of data to repair [20:48:17] so...doable but not ideal [20:48:26] better if you can gracefully exit the node [20:48:31] herron so it looks like when I query the database it is missing the cu_changes table, do I not have access to that? [20:48:39] and better if you don't have a several TB blast radius [20:50:05] * volans staring at the config wondering why is this way... [20:52:13] RECOVERY - cassandra-c CQL 10.64.0.116:9042 on restbase1010 is OK: TCP OK - 0.000 second response time on 10.64.0.116 port 9042 [20:52:30] !log T169939: Decommissioning restbase1010-c.eqiad.wmnet [20:52:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:52:42] T169939: End of August milestone: Cassandra 3 cluster in production - https://phabricator.wikimedia.org/T169939 [20:52:45] * urandom crosses his fingers [20:53:07] so, we have a physical raid controller with 4 disks, we configured the hardware raid controller to just give us the disks as a JBOD, then we did an MD raid0 on top of them (together with other partitions) and use the software raid0 as a physical volume for LVM... [20:53:30] !log pushing "aggregate defaults discard" to all the cr* routers - T174364 [20:53:35] volans: aye [20:53:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:53:41] T174364: DDOS_PROTOCOL_VIOLATION_SET: Protocol Rejectv6:aggregate is violated - https://phabricator.wikimedia.org/T174364 [20:53:46] urandom: any good reason why? :D [20:53:57] volans: which part :) [20:54:02] all of it! :D [20:54:44] so...i can't tell you why software raid for machines that have hardware controllers, that predates me [20:55:01] and if i recall, that was ops' call [20:55:13] i.e. not a wacky request from services :) [20:55:20] yeah I guess everything comes dowm from that decision ;) [20:55:35] (03PS6) 10Reedy: Same namespace for global mail blacklist as for global spam blacklist. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/368770 (owner: 10Steinsplitter) [20:56:32] the raid-0 predates running multiple instances on a single host, or even node densities this high [20:57:28] and JBOD was kinda broken in Cassandra until more recently [20:57:52] anyway I quickly looked at the smart data of the disks, and all looks good at first sight [20:57:59] (03CR) 10Reedy: [C: 032] Same namespace for global mail blacklist as for global spam blacklist. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/368770 (owner: 10Steinsplitter) [20:59:28] (03Merged) 10jenkins-bot: Same namespace for global mail blacklist as for global spam blacklist. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/368770 (owner: 10Steinsplitter) [20:59:38] (03CR) 10jenkins-bot: Same namespace for global mail blacklist as for global spam blacklist. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/368770 (owner: 10Steinsplitter) [21:00:04] dapatrick, bawolff, and Reedy: Dear anthropoid, the time has come. Please deploy Weekly Security deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170828T2100). [21:00:27] urandom: btw you'll have the same problem to replace it, given that you'll loose the raid0 [21:00:59] volans: so, we're in the process of carving out a total of six nodes to create a parallel cluster [21:01:15] i was going to take another node from this rack, (and i wasn't quite there yet) [21:01:21] 10Operations, 10netops: DDOS_PROTOCOL_VIOLATION_SET: Protocol Rejectv6:aggregate is violated - https://phabricator.wikimedia.org/T174364#3560364 (10ayounsi) 05Open>03Resolved [21:01:23] herron if it helps it looks like passwords are not in there as well [21:01:24] !log reedy@tin Synchronized wmf-config/CommonSettings.php: Move email blacklist to [[meta:Email_blacklist]] (duration: 00m 45s) [21:01:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:01:53] (03PS8) 10Reedy: Run Lilypond from Firejail [mediawiki-config] - 10https://gerrit.wikimedia.org/r/370358 (https://phabricator.wikimedia.org/T172582) (owner: 10Ebe123) [21:02:02] but if i can get this out cleanly, then there will no need to replace it (in this cluster), we can fix the disk/raid and provision it in the new cluster [21:02:14] 10Operations, 10Services (doing): Disk errors: restbase1010.eqiad.wmnet - https://phabricator.wikimedia.org/T174392#3560205 (10Volans) Adding #ops-eqiad, looks like we'll probably end up replacing the disk [21:02:17] (03CR) 10Reedy: [C: 032] Run Lilypond from Firejail [mediawiki-config] - 10https://gerrit.wikimedia.org/r/370358 (https://phabricator.wikimedia.org/T172582) (owner: 10Ebe123) [21:02:31] urandom: ack [21:02:42] (03PS1) 10RobH: further tweaking of kafka-jumbo.cfg [puppet] - 10https://gerrit.wikimedia.org/r/374419 [21:03:32] (03CR) 10RobH: [C: 032] further tweaking of kafka-jumbo.cfg [puppet] - 10https://gerrit.wikimedia.org/r/374419 (owner: 10RobH) [21:03:34] PROBLEM - Router interfaces on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, interfaces up: 212, down: 1, dormant: 0, excluded: 0, unused: 0 [21:03:38] 10Operations, 10ops-eqiad, 10Services (doing): Disk errors: restbase1010.eqiad.wmnet - https://phabricator.wikimedia.org/T174392#3560367 (10Volans) [21:03:43] PROBLEM - Router interfaces on cr2-esams is CRITICAL: CRITICAL: host 91.198.174.244, interfaces up: 57, down: 1, dormant: 0, excluded: 0, unused: 0 [21:03:44] (03Merged) 10jenkins-bot: Run Lilypond from Firejail [mediawiki-config] - 10https://gerrit.wikimedia.org/r/370358 (https://phabricator.wikimedia.org/T172582) (owner: 10Ebe123) [21:05:03] PROBLEM - Check Varnish expiry mailbox lag on cp1072 is CRITICAL: CRITICAL: expiry mailbox lag is 2050538 [21:06:15] (03CR) 10jenkins-bot: Run Lilypond from Firejail [mediawiki-config] - 10https://gerrit.wikimedia.org/r/370358 (https://phabricator.wikimedia.org/T172582) (owner: 10Ebe123) [21:06:23] !log reedy@tin Synchronized wmf-config/CommonSettings.php: Run Score binaries in firejail T172582 (duration: 00m 43s) [21:06:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:07:37] 10Operations, 10MediaWiki-extensions-Score, 10Security: Run score binaries in a firejail - https://phabricator.wikimedia.org/T172582#3560372 (10Reedy) 05Open>03Resolved a:03Reedy https://gerrit.wikimedia.org/r/#/c/370358/ has been deployed [21:07:42] 10Operations, 10MediaWiki-extensions-Score, 10Security: Run score binaries in a firejail - https://phabricator.wikimedia.org/T172582#3560375 (10Reedy) [21:12:04] 10Operations, 10ops-eqiad, 10Cloud-Services, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3560381 (10madhuvishy) @Cmjohnson Thank you! [21:14:03] RECOVERY - Check Varnish expiry mailbox lag on cp1074 is OK: OK: expiry mailbox lag is 0 [21:17:09] (03CR) 10Mattflaschen: [C: 032] RCFilters: Enable on watchlist for Beta Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374381 (owner: 10Jforrester) [21:17:35] 10Operations, 10MediaWiki-extensions-Score, 10Security: Run score binaries in a firejail - https://phabricator.wikimedia.org/T172582#3560390 (10Reedy) And tested with score code from https://en.wikipedia.org/wiki/Symphony_No._9_(Beethoven) on a user page, including making modifications and seeing images were... [21:19:57] (03PS3) 10Mattflaschen: RCFilters: Enable on watchlist for Beta Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374381 (owner: 10Jforrester) [21:21:23] (03CR) 10Mattflaschen: [C: 032] RCFilters: Enable on watchlist for Beta Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374381 (owner: 10Jforrester) [21:22:35] (03Merged) 10jenkins-bot: RCFilters: Enable on watchlist for Beta Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374381 (owner: 10Jforrester) [21:22:45] (03CR) 10jenkins-bot: RCFilters: Enable on watchlist for Beta Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374381 (owner: 10Jforrester) [21:26:30] 10Operations, 10netops: Tracking task for network syslog messages - https://phabricator.wikimedia.org/T174397#3560405 (10ayounsi) [21:26:34] (03PS1) 10Rush: openstack: remove legacy firewall rules for controller [puppet] - 10https://gerrit.wikimedia.org/r/374424 (https://phabricator.wikimedia.org/T171494) [21:26:45] 10Operations, 10netops: Tracking task for network syslog messages - https://phabricator.wikimedia.org/T174397#3560424 (10ayounsi) [21:29:09] (03CR) 10Eevans: "So this is ~50G for the data raid-1? If so, that seems to be about 2x what we're currently using ((8G commitlog + ~500G of saved caches) " [puppet] - 10https://gerrit.wikimedia.org/r/373863 (https://phabricator.wikimedia.org/T169939) (owner: 10Filippo Giunchedi) [21:31:09] 10Operations, 10MediaWiki-extensions-Score, 10Security: Run score binaries in a firejail - https://phabricator.wikimedia.org/T172582#3560441 (10Reedy) 05Resolved>03Open Seems ABC might be broken (per @Ebe123 on IRC) ``` Processing `.../file.ly' Parsing... .../file.ly:1:17: error: syntax error, unexpecte... [21:33:17] (03PS19) 10MarcoAurelio: [WIP DNM] Create computed list of wikis that can use SecurePoll [mediawiki-config] - 10https://gerrit.wikimedia.org/r/371926 [21:33:19] (03PS1) 10Reedy: Disable firejail profile for wgScoreAbc2Ly [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374426 (https://phabricator.wikimedia.org/T172582) [21:33:42] (03PS2) 10Madhuvishy: firstboot: Prevent non-root logins while NFS mounts aren't available [puppet] - 10https://gerrit.wikimedia.org/r/368223 (https://phabricator.wikimedia.org/T171508) [21:34:08] (03PS20) 10MarcoAurelio: [WIP DNM] Create computed list of wikis that can use SecurePoll [mediawiki-config] - 10https://gerrit.wikimedia.org/r/371926 (https://phabricator.wikimedia.org/T174398) [21:34:15] (03CR) 10Reedy: [C: 032] Disable firejail profile for wgScoreAbc2Ly [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374426 (https://phabricator.wikimedia.org/T172582) (owner: 10Reedy) [21:35:17] (03PS1) 10Rush: openstack: remove redis replication rule [puppet] - 10https://gerrit.wikimedia.org/r/374427 (https://phabricator.wikimedia.org/T171494) [21:36:20] (03Merged) 10jenkins-bot: Disable firejail profile for wgScoreAbc2Ly [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374426 (https://phabricator.wikimedia.org/T172582) (owner: 10Reedy) [21:36:22] (03CR) 10jerkins-bot: [V: 04-1] [WIP DNM] Create computed list of wikis that can use SecurePoll [mediawiki-config] - 10https://gerrit.wikimedia.org/r/371926 (https://phabricator.wikimedia.org/T174398) (owner: 10MarcoAurelio) [21:36:30] (03CR) 10jenkins-bot: Disable firejail profile for wgScoreAbc2Ly [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374426 (https://phabricator.wikimedia.org/T172582) (owner: 10Reedy) [21:36:38] argh I'ma kill that bot [21:37:58] !log reedy@tin Synchronized wmf-config/CommonSettings.php: Disable wgScoreAbc2Ly firejail T172582 (duration: 00m 44s) [21:38:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:38:10] T172582: Run score binaries in a firejail - https://phabricator.wikimedia.org/T172582 [21:41:25] (03PS21) 10MarcoAurelio: [WIP DNM] Create computed list of wikis that can use SecurePoll [mediawiki-config] - 10https://gerrit.wikimedia.org/r/371926 (https://phabricator.wikimedia.org/T174398) [21:42:58] (03CR) 10jerkins-bot: [V: 04-1] [WIP DNM] Create computed list of wikis that can use SecurePoll [mediawiki-config] - 10https://gerrit.wikimedia.org/r/371926 (https://phabricator.wikimedia.org/T174398) (owner: 10MarcoAurelio) [21:43:04] (03PS2) 10Krinkle: Enable jQuery 3 on nlwiki, svwiki, plwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373987 (https://phabricator.wikimedia.org/T124742) [21:44:07] Reedy: If you're done before 22:00 UTC, let me know, otherwise I'll add ^ to SWAT. [21:44:26] Krinkle: I'm not deploying anything else [21:44:36] col [21:44:37] cool [21:44:43] I can't get the firejail profile change deployed if I get it fixed ;) [21:44:45] (03CR) 10Krinkle: [C: 032] Enable jQuery 3 on nlwiki, svwiki, plwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373987 (https://phabricator.wikimedia.org/T124742) (owner: 10Krinkle) [21:45:13] Aye, sorry. [21:46:05] (03CR) 10GWicke: [C: 031] Enable JobQueueEventBus on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374399 (owner: 10Ppchelko) [21:46:09] Too bad :( [21:46:18] (03Merged) 10jenkins-bot: Enable jQuery 3 on nlwiki, svwiki, plwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373987 (https://phabricator.wikimedia.org/T124742) (owner: 10Krinkle) [21:46:27] (03CR) 10jenkins-bot: Enable jQuery 3 on nlwiki, svwiki, plwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373987 (https://phabricator.wikimedia.org/T124742) (owner: 10Krinkle) [21:47:01] (03PS22) 10MarcoAurelio: [WIP DNM] Create computed list of wikis that can use SecurePoll [mediawiki-config] - 10https://gerrit.wikimedia.org/r/371926 (https://phabricator.wikimedia.org/T174398) [21:47:27] Reedy: file.ly (untracked file) [21:47:39] bblack: better, at least => https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?orgId=1&from=now-24h&to=now&panelId=6&fullscreen [21:47:43] silly thing writing relatively [21:47:48] gon [21:47:48] e [21:48:47] AaronSchulz: nice :) Both of them helped, right? [21:48:52] !log krinkle@tin Synchronized wmf-config/InitialiseSettings.php: I1c9c9c0e - Enable jQuery 3 on nlwiki, svwiki, plwiki (duration: 00m 44s) [21:48:59] 100K>60K, then 60K>~40K [21:49:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:49:07] Krinkle: yes, though I wonder what else can be done [21:49:23] AaronSchulz: https://gerrit.wikimedia.org/r/#/c/295027/ [21:49:27] (03CR) 10jerkins-bot: [V: 04-1] [WIP DNM] Create computed list of wikis that can use SecurePoll [mediawiki-config] - 10https://gerrit.wikimedia.org/r/371926 (https://phabricator.wikimedia.org/T174398) (owner: 10MarcoAurelio) [21:49:28] these were just fixes to old issues, so there was still an increase lately [21:49:44] (03CR) 10Rush: firstboot: Prevent non-root logins while NFS mounts aren't available (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/368223 (https://phabricator.wikimedia.org/T171508) (owner: 10Madhuvishy) [21:50:23] (03CR) 10Rush: firstboot: Prevent non-root logins while NFS mounts aren't available (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/368223 (https://phabricator.wikimedia.org/T171508) (owner: 10Madhuvishy) [21:50:54] (03PS23) 10MarcoAurelio: [WIP DNM] Create computed list of wikis that can use SecurePoll [mediawiki-config] - 10https://gerrit.wikimedia.org/r/371926 (https://phabricator.wikimedia.org/T174398) [21:57:13] (03PS3) 10Madhuvishy: firstboot: Prevent non-root logins while NFS mounts aren't available [puppet] - 10https://gerrit.wikimedia.org/r/368223 (https://phabricator.wikimedia.org/T171508) [21:58:22] Krinkle: meh, you could just do backoff throttling [21:58:49] rather than hoping that parse time and DB lag will happen to also be a good varnish throttle (though it may indeed work atm) [21:59:01] (03CR) 10Madhuvishy: [C: 032] firstboot: Prevent non-root logins while NFS mounts aren't available [puppet] - 10https://gerrit.wikimedia.org/r/368223 (https://phabricator.wikimedia.org/T171508) (owner: 10Madhuvishy) [22:07:05] (03PS1) 10Reedy: Revert "Disable firejail profile for wgScoreAbc2Ly" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374429 [22:07:09] (03CR) 10Reedy: [C: 04-1] Revert "Disable firejail profile for wgScoreAbc2Ly" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374429 (owner: 10Reedy) [22:07:26] (03CR) 10Reedy: [C: 04-2] "DNM!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374429 (owner: 10Reedy) [22:09:46] 10Operations, 10MediaWiki-extensions-Score, 10Patch-For-Review, 10Security: Run score binaries in a firejail - https://phabricator.wikimedia.org/T172582#3560587 (10Reedy) Seems to work outside MW though... ``` reedy@tin:/srv/mediawiki-staging$ cat /tmp/file.abc X:1 T:The Legacy Jid M:6/8 L:1/8 R:jig K:G G... [22:14:33] RECOVERY - Router interfaces on cr2-eqiad is OK: OK: host 208.80.154.197, interfaces up: 214, down: 0, dormant: 0, excluded: 0, unused: 0 [22:14:43] RECOVERY - Router interfaces on cr2-esams is OK: OK: host 91.198.174.244, interfaces up: 59, down: 0, dormant: 0, excluded: 0, unused: 0 [22:23:31] (03PS1) 10Reedy: Add --quiet to abc2ly firejail [puppet] - 10https://gerrit.wikimedia.org/r/374430 [22:25:43] PROBLEM - cassandra-c CQL 10.64.0.116:9042 on restbase1010 is CRITICAL: connect to address 10.64.0.116 and port 9042: Connection refused [22:25:54] PROBLEM - cassandra-c SSL 10.64.0.116:7001 on restbase1010 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused [22:30:31] 10Operations, 10Research, 10The-Wikipedia-Library, 10Traffic, and 6 others: Set an explicit "Origin When Cross-Origin" referer policy via the meta referrer tag - https://phabricator.wikimedia.org/T87276#3560638 (10Tgr) Chrome seems to claim our referrer policy is `no-referrer-when-downgrade`. (Chrome 60.0.... [22:42:18] (03PS1) 10Ayounsi: Icinga: Add basic monitoring for routers' active RE [puppet] - 10https://gerrit.wikimedia.org/r/374435 (https://phabricator.wikimedia.org/T174397) [22:44:54] needing to chase https://phabricator.wikimedia.org/T173374 up - we've still got an undeletable file on Commons at https://commons.wikimedia.org/wiki/File:Literature_II,_Harutyun_Surkhatian.djvu [22:45:15] think it might fix itself if I overwrite it with another djvu file ? [22:46:09] !log T169939: Decommissioning restbase1010-b.eqiad.wmnet [22:46:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:46:23] T169939: End of August milestone: Cassandra 3 cluster in production - https://phabricator.wikimedia.org/T169939 [22:46:23] (03PS3) 10Ottomata: [WIP] Initial commit of certpy [software/certpy] - 10https://gerrit.wikimedia.org/r/359960 (https://phabricator.wikimedia.org/T166167) [22:47:23] ACKNOWLEDGEMENT - cassandra-c CQL 10.64.0.116:9042 on restbase1010 is CRITICAL: connect to address 10.64.0.116 and port 9042: Connection refused eevans Decommissioning (T169939) [22:47:23] ACKNOWLEDGEMENT - cassandra-c SSL 10.64.0.116:7001 on restbase1010 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused eevans Decommissioning (T169939) [22:47:42] (03CR) 10Ottomata: "Still not ready for review" [software/certpy] - 10https://gerrit.wikimedia.org/r/359960 (https://phabricator.wikimedia.org/T166167) (owner: 10Ottomata) [22:59:39] 10Operations, 10ops-eqiad, 10Cloud-Services, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3560733 (10RobH) I also checked other systems iwth HP raid controllers. So these showing the raid1 as sdd is a problem, it should show as sda. Othe... [23:00:04] addshore, hashar, anomie, RainbowSprinkles, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: Respected human, time to deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170828T2300). Please do the needful. [23:00:05] Smalyshev and ebernhardson: A patch you scheduled for Evening SWAT (Max 8 patches) is about to be deployed. Please be available during the process. [23:00:21] here [23:00:26] \o [23:00:31] i guess i can ship these [23:00:33] 10Operations, 10MediaWiki-extensions-Score, 10Patch-For-Review, 10Security: Run score binaries in a firejail - https://phabricator.wikimedia.org/T172582#3560735 (10Reedy) ```lang=php throw new MWException( var_export( $cmd, true ) . "\n" . var_export( $code, true ) . "\n" . var_export( file_get_... [23:00:56] coolio [23:02:12] (03CR) 10EBernhardson: [C: 032] Add list for wikis that would have categories dumped into RDF [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373167 (https://phabricator.wikimedia.org/T173892) (owner: 10Smalyshev) [23:03:43] (03Merged) 10jenkins-bot: Add list for wikis that would have categories dumped into RDF [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373167 (https://phabricator.wikimedia.org/T173892) (owner: 10Smalyshev) [23:05:03] RECOVERY - Check Varnish expiry mailbox lag on cp1072 is OK: OK: expiry mailbox lag is 0 [23:06:08] (03CR) 10jenkins-bot: Add list for wikis that would have categories dumped into RDF [mediawiki-config] - 10https://gerrit.wikimedia.org/r/373167 (https://phabricator.wikimedia.org/T173892) (owner: 10Smalyshev) [23:06:33] SMalyshev: looks safe enough to just ship yours [23:07:04] !log ebernhardson@tin Synchronized dblists/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s) [23:07:11] ebernhardson: should be safe, it just adds a list. if you can deploy it on terbium, I can check it works [23:07:13] though I don't see any reason why it wouldn't [23:07:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:07:15] T173892: Setup dump for categories RDF representation - https://phabricator.wikimedia.org/T173892 [23:07:46] ebernhardson: yep, works, thanks! [23:08:34] !log ebernhardson@tin Synchronized docroot/noc/conf/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s) [23:08:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:13:34] !log ebernhardson@tin Synchronized php-1.30.0-wmf.15/extensions/WikimediaEvents/: Turn off two cirrus AB tests T171742 T171214 (duration: 00m 44s) [23:13:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:13:49] T171742: Search Relevance: MVP test (turn it off) - https://phabricator.wikimedia.org/T171742 [23:13:49] T171214: Interleaved results A/B test: turn off test - https://phabricator.wikimedia.org/T171214 [23:14:32] (03Abandoned) 10Reedy: Add --quiet to abc2ly firejail [puppet] - 10https://gerrit.wikimedia.org/r/374430 (owner: 10Reedy) [23:16:28] swat complete [23:18:33] (03CR) 10Reedy: [C: 032] Revert "Disable firejail profile for wgScoreAbc2Ly" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374429 (owner: 10Reedy) [23:19:17] 10Operations, 10MediaWiki-extensions-Score, 10Patch-For-Review, 10Security: Run score binaries in a firejail - https://phabricator.wikimedia.org/T172582#3560790 (10Reedy) ``` firejail --help | grep output --output=logfile - stdout logging and log rotation. Copy stdout and stderr ``` I guess that'll ex... [23:19:58] (03Merged) 10jenkins-bot: Revert "Disable firejail profile for wgScoreAbc2Ly" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374429 (owner: 10Reedy) [23:20:11] (03CR) 10jenkins-bot: Revert "Disable firejail profile for wgScoreAbc2Ly" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/374429 (owner: 10Reedy) [23:22:20] 10Operations, 10Research, 10The-Wikipedia-Library, 10Traffic, and 6 others: Set an explicit "Origin When Cross-Origin" referer policy via the meta referrer tag - https://phabricator.wikimedia.org/T87276#3560795 (10DarTar) @Tgr I can't speak for these services, it sounds likely they only check headers and n... [23:22:30] !log reedy@tin Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: Fix escaping (duration: 00m 43s) [23:22:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:23:34] !log reedy@tin Synchronized wmf-config/CommonSettings.php: abc2ly in firejail (duration: 00m 43s) [23:23:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:24:43] (03PS5) 10Smalyshev: Add RDF dumps for categories [puppet] - 10https://gerrit.wikimedia.org/r/373354 (https://phabricator.wikimedia.org/T173892) [23:26:17] 10Operations, 10Cassandra, 10Epic, 10Goal, and 2 others: End of August milestone: Cassandra 3 cluster in production - https://phabricator.wikimedia.org/T169939#3560812 (10Eevans) [23:27:29] (03PS1) 10Thcipriani: Mask jobchron and jobrunner in non-active DC [puppet] - 10https://gerrit.wikimedia.org/r/374438 (https://phabricator.wikimedia.org/T167104) [23:27:43] !log reedy@tin Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 42s) [23:27:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:31:05] !log reedy@tin Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 43s) [23:31:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:31:40] 10Operations, 10MediaWiki-extensions-Score, 10Patch-For-Review, 10Security: Run score binaries in a firejail - https://phabricator.wikimedia.org/T172582#3560818 (10Reedy) Yup, bingo [23:37:25] !log reedy@tin Synchronized php-1.30.0-wmf.15/extensions/Score/: consistency (duration: 00m 43s) [23:37:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:38:08] 10Operations, 10MediaWiki-extensions-Score, 10Patch-For-Review, 10Security: Run score binaries in a firejail - https://phabricator.wikimedia.org/T172582#3560823 (10Reedy) 05Open>03Resolved (Y) [23:39:46] 10Operations, 10MediaWiki-extensions-Score, 10Patch-For-Review, 10Security: Run score binaries in a firejail - https://phabricator.wikimedia.org/T172582#3560828 (10Reedy) [23:49:18] (03CR) 10Thcipriani: [C: 04-1] "A few inline comments, but I think that this patches covers everything that needs to be done in puppet for scholarships in production." (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/326461 (https://phabricator.wikimedia.org/T129134) (owner: 10Niharika29)