[02:43:53] (03PS1) 10Madhuvishy: dumps_public: Cleanup the public_server profile [puppet] - 10https://gerrit.wikimedia.org/r/401419 [02:46:47] (03CR) 10Madhuvishy: [C: 032] dumps_public: Cleanup the public_server profile [puppet] - 10https://gerrit.wikimedia.org/r/401419 (owner: 10Madhuvishy) [02:50:33] (03PS1) 10Madhuvishy: dumps: Set up NFS on the dumps distribution servers [puppet] - 10https://gerrit.wikimedia.org/r/401420 (https://phabricator.wikimedia.org/T181431) [02:56:13] (03CR) 10Madhuvishy: [C: 032] dumps: Set up NFS on the dumps distribution servers [puppet] - 10https://gerrit.wikimedia.org/r/401420 (https://phabricator.wikimedia.org/T181431) (owner: 10Madhuvishy) [06:21:24] 10Operations, 10ops-codfw, 10DBA: db2054: Disk with predictive failure - https://phabricator.wikimedia.org/T183887#3866270 (10Marostegui) [06:21:44] 10Operations, 10ops-codfw, 10DBA: db2054: Disk with predictive failure - https://phabricator.wikimedia.org/T183887#3866270 (10Marostegui) p:05Triage>03Normal [06:23:50] (03PS1) 10Marostegui: db-eqiad.php: Depool db1110 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401431 [06:25:42] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1110 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401431 (owner: 10Marostegui) [06:27:11] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1110 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401431 (owner: 10Marostegui) [06:27:23] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1110 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401431 (owner: 10Marostegui) [06:28:37] !log marostegui@tin Synchronized wmf-config/db-eqiad.php: Depool db1110 to reimport dewiki.langlinks on dbstore1002 (duration: 00m 50s) [06:28:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:47:06] !log Stop db1110 and dbstore1002.s5 replication in sync [06:47:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:47:34] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1110" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401432 [06:49:20] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1110" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401432 (owner: 10Marostegui) [06:50:46] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1110" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401432 (owner: 10Marostegui) [06:50:56] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1110" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401432 (owner: 10Marostegui) [06:51:58] !log marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 52s) [06:52:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:02:34] 10Operations, 10MediaWiki-Vagrant, 10MediaWiki-extensions-Scribunto: php-luasandbox in Wikimedia's Stretch apt repo depends on php5 - https://phabricator.wikimedia.org/T183888#3866300 (10bd808) p:05Triage>03Normal [07:06:42] (03PS1) 10Marostegui: db1070.yaml: Update new socket location [puppet] - 10https://gerrit.wikimedia.org/r/401433 (https://phabricator.wikimedia.org/T177208) [07:07:30] (03CR) 10Marostegui: [C: 04-2] "Do not merge until db1070 puppet is stopped on the s8 failover date" [puppet] - 10https://gerrit.wikimedia.org/r/401433 (https://phabricator.wikimedia.org/T177208) (owner: 10Marostegui) [07:09:12] (03PS1) 10Marostegui: db-eqiad.php: Set s5 on read_only [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401434 (https://phabricator.wikimedia.org/T177208) [07:09:30] (03CR) 10Marostegui: [C: 04-2] "Do not submit until we are on the failover day/time" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401434 (https://phabricator.wikimedia.org/T177208) (owner: 10Marostegui) [07:13:22] (03PS1) 10Marostegui: db-eqiad.php: Point wikidatawiki to s8 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401436 (https://phabricator.wikimedia.org/T177208) [07:13:30] (03CR) 10Marostegui: [C: 04-2] "Do not submit until we are on the failover day/time" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401436 (https://phabricator.wikimedia.org/T177208) (owner: 10Marostegui) [07:16:52] 10Operations, 10MediaWiki-Vagrant, 10MediaWiki-extensions-Scribunto: php-luasandbox in Wikimedia's Stretch apt repo depends on php5 - https://phabricator.wikimedia.org/T183888#3866325 (10Legoktm) The packaging was probably based on the PHP5 version, since it also depends upon a non-existent `phpapi-` package... [07:56:39] !log Deploy schema change on dbstore1001.s7 - T174569 [07:56:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:56:51] T174569: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569 [07:59:45] (03PS1) 10Marostegui: s7: Add labsdb1009,10 and 11 to s7 [software] - 10https://gerrit.wikimedia.org/r/401437 [08:00:54] (03CR) 10Marostegui: [C: 032] s7: Add labsdb1009,10 and 11 to s7 [software] - 10https://gerrit.wikimedia.org/r/401437 (owner: 10Marostegui) [08:01:53] (03Merged) 10jenkins-bot: s7: Add labsdb1009,10 and 11 to s7 [software] - 10https://gerrit.wikimedia.org/r/401437 (owner: 10Marostegui) [08:06:05] !log Deploy alter table on db1039 (already depooled) - T174569 [08:06:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:06:16] T174569: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569 [08:19:48] <_joe_> !log restarting hhvm on mw1313, concurrency HPHP::VariableUnserializer::unserializeVariant [08:19:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:23:14] !log restart druid coordinators on druid* to pick up new jvm settings [08:23:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:26:24] <_joe_> !log restarting hhvm on mw1317, multiple threads stuck in HPHP::jit::enterTCImpl [08:26:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:29:04] <_joe_> !log restarting hhvm on mw1280,1282 for the same reasons [08:29:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:36:54] <_joe_> !log likewise for mw1285,mw1235,mw1232 [08:37:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:39:19] (03CR) 10Jcrespo: [C: 04-2] "Not sure..." (033 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399792 (https://phabricator.wikimedia.org/T134476) (owner: 10Jcrespo) [08:41:50] 10Operations, 10ops-codfw, 10DBA: pc2005 crashed: CPU2 internal error - https://phabricator.wikimedia.org/T183750#3866368 (10jcrespo) I think these servers are leased CC @RobH [08:46:42] (03CR) 10Muehlenhoff: mediawiki: Ensure Python 3 is available for Pygments (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/400458 (https://phabricator.wikimedia.org/T182851) (owner: 10Legoktm) [08:50:07] 10Operations, 10Cleanup, 10Continuous-Integration-Config, 10Gerrit, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3866385 (10hashar) [08:51:03] 10Operations, 10Cleanup, 10Continuous-Integration-Config, 10Gerrit, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3866385 (10hashar) [08:52:10] 10Operations, 10Cleanup, 10Continuous-Integration-Config, 10Gerrit, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3866385 (10Legoktm) Err, Extension:Collection is definitely still used and deployed. Only the OCG service was sunset. [08:52:20] 10Operations, 10Cleanup, 10Continuous-Integration-Config, 10Gerrit, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3866398 (10hashar) [08:52:53] <_joe_> !log restarting also mw1226-8, mw1223, mw1201,mw1203, mw1205-7 [08:53:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:53:40] 10Operations, 10Cleanup, 10Continuous-Integration-Config, 10Gerrit, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3866385 (10hashar) Argh I copy pasted too many of them. So I guess we keep `Collection` but can archive the `mediawiki/extensions/... [08:56:06] 10Operations, 10Cleanup, 10Continuous-Integration-Config, 10Gerrit, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3866423 (10Legoktm) I think so :) [08:56:09] (03CR) 10Filippo Giunchedi: "See inline" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/399826 (https://phabricator.wikimedia.org/T182215) (owner: 10Dzahn) [08:57:14] 10Operations, 10Cleanup, 10Continuous-Integration-Config, 10Gerrit, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3866424 (10MarcoAurelio) [08:57:26] 10Operations, 10Cleanup, 10Continuous-Integration-Config, 10Gerrit, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3866385 (10MarcoAurelio) Task description editted. [09:00:54] (03CR) 10Muehlenhoff: "One comment, but ttyS[01]-115200 are fine." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/399826 (https://phabricator.wikimedia.org/T182215) (owner: 10Dzahn) [09:03:22] 10Operations: Migrate racktables to servermon - https://phabricator.wikimedia.org/T88424#3866435 (10akosiaris) 05Open>03declined Yeah, it has. The hwdoc component of servermon is nowhere near as feature full as netbox and we haven't really invested in it in years. [09:03:24] 10Operations, 10Tracking: Hardware Automation Workflow - Overall Tracking - https://phabricator.wikimedia.org/T116063#3866437 (10akosiaris) [09:05:32] (03CR) 10Muehlenhoff: "@Chad: This changes sudo privileges, so needs an access request Phab ticket for approval in Ops meeting, can you please create one?" [puppet] - 10https://gerrit.wikimedia.org/r/399123 (owner: 10Chad) [09:20:04] (03CR) 10Jcrespo: [C: 031] db1070.yaml: Update new socket location [puppet] - 10https://gerrit.wikimedia.org/r/401433 (https://phabricator.wikimedia.org/T177208) (owner: 10Marostegui) [09:21:02] (03CR) 10Jcrespo: [C: 031] "This is ok, but should we add a more specific text? A link to a wiki?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401434 (https://phabricator.wikimedia.org/T177208) (owner: 10Marostegui) [09:21:31] (03CR) 10Jcrespo: [C: 031] db-eqiad.php: Point wikidatawiki to s8 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401436 (https://phabricator.wikimedia.org/T177208) (owner: 10Marostegui) [09:22:07] jouncebot: next [09:22:08] In 4 hour(s) and 37 minute(s): European Mid-day SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180102T1400) [09:22:52] (03CR) 10Jcrespo: db-eqiad.php: Point wikidatawiki to s8 (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401436 (https://phabricator.wikimedia.org/T177208) (owner: 10Marostegui) [09:24:26] no_justification: wondering if I could get https://gerrit.wikimedia.org/r/#/c/401375/ merged before the train? So we don't have to change it again afterwards. Thanks. [09:28:22] !log reboot ms-be1033 - T183724 [09:28:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:28:32] T183724: Degraded RAID on ms-be1033 - https://phabricator.wikimedia.org/T183724 [09:30:25] 10Operations, 10ops-eqiad: Degraded RAID on ms-be1033 - https://phabricator.wikimedia.org/T183724#3866480 (10fgiunchedi) a:03Cmjohnson @Cmjohnson `sdk` failed here, please replace, thanks! [09:36:39] 10Operations, 10ops-eqiad, 10User-Elukey, 10User-Joe: Decommission mw1180-1200 - https://phabricator.wikimedia.org/T183895#3866486 (10Joe) p:05Triage>03Normal [09:37:13] <_joe_> !log setting appservers in the mw1180-1200 range to pooled=inactive, T183895 [09:37:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:37:26] T183895: Decommission mw1180-1200 - https://phabricator.wikimedia.org/T183895 [09:38:56] nice _joe_, I need mw118[0-9] to be decommed to allow the last row-c servers to be racked :D [09:38:58] (03CR) 10Filippo Giunchedi: [C: 031] thumbor: use the canonical definition of logstash host [puppet] - 10https://gerrit.wikimedia.org/r/399652 (https://phabricator.wikimedia.org/T182304) (owner: 10Gehel) [09:40:33] (03CR) 10Filippo Giunchedi: [C: 031] graphite: cleanup configparser_format a little bit [puppet] - 10https://gerrit.wikimedia.org/r/359451 (owner: 10Faidon Liambotis) [09:42:37] 10Operations, 10ops-eqiad: Degraded RAID on ms-be1033 - https://phabricator.wikimedia.org/T183896#3866502 (10ops-monitoring-bot) [09:43:28] !log oblivian@puppetmaster1001 conftool action : set/pooled=inactive; selector: cluster=appserver,name=mw1(1.*|200).eqiad.wmnet [09:43:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:43:40] <_joe_> elukey: yeah removing those now [09:44:07] <_joe_> !log setting api_appservers in the mw1180-1200 range to pooled=inactive, T183895 [09:44:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:44:17] T183895: Decommission mw1180-1200 - https://phabricator.wikimedia.org/T183895 [09:44:35] 10Operations, 10ops-eqiad: Degraded RAID on ms-be1033 - https://phabricator.wikimedia.org/T183896#3866528 (10fgiunchedi) [09:44:37] 10Operations, 10ops-eqiad: Degraded RAID on ms-be1033 - https://phabricator.wikimedia.org/T183724#3866530 (10fgiunchedi) [09:45:10] !log oblivian@puppetmaster1001 conftool action : set/pooled=inactive; selector: cluster=api_appserver,name=mw1(1.*|200).eqiad.wmnet [09:45:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:45:28] (03PS2) 10Marostegui: db-eqiad.php: Point wikidatawiki to s8 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401436 (https://phabricator.wikimedia.org/T177208) [09:45:39] 10Operations, 10ops-eqiad: Degraded RAID on ms-be1033 - https://phabricator.wikimedia.org/T183896#3866502 (10fgiunchedi) Merged with duplicate, likely related to the fact that upon reboot the controller saw the disk as "OK" and then I marked it manually as failed. Actually attempting to write to the disk resul... [09:47:33] (03PS2) 10Marostegui: db-eqiad.php: Set s5 on read_only [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401434 (https://phabricator.wikimedia.org/T177208) [09:47:45] (03CR) 10Marostegui: [C: 04-2] "> This is ok, but should we add a more specific text? A link to a" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401434 (https://phabricator.wikimedia.org/T177208) (owner: 10Marostegui) [09:54:14] (03CR) 10Jcrespo: db-eqiad.php: Point wikidatawiki to s8 (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401436 (https://phabricator.wikimedia.org/T177208) (owner: 10Marostegui) [09:57:25] (03PS3) 10Marostegui: db-eqiad.php: Point wikidatawiki to s8 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401436 (https://phabricator.wikimedia.org/T177208) [09:57:27] (03CR) 10Marostegui: [C: 04-2] db-eqiad.php: Point wikidatawiki to s8 (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401436 (https://phabricator.wikimedia.org/T177208) (owner: 10Marostegui) [09:58:12] (03CR) 10Alexandros Kosiaris: [C: 031] tor: add an additional relay instance [puppet] - 10https://gerrit.wikimedia.org/r/399972 (owner: 10Faidon Liambotis) [10:03:21] 10Operations, 10ops-eqiad, 10User-Elukey, 10User-Joe: Decommission mw1180-1200 - https://phabricator.wikimedia.org/T183895#3866568 (10Joe) [10:05:17] 10Operations, 10Traffic, 10media-storage: Swift invalid range requests causing 501s - https://phabricator.wikimedia.org/T183902#3866584 (10fgiunchedi) [10:06:57] (03CR) 10Alexandros Kosiaris: [C: 032] "https://puppet-compiler.wmflabs.org/compiler02/9485/sca1003.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/399969 (owner: 10Dzahn) [10:07:01] (03PS2) 10Alexandros Kosiaris: zotero: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/399969 (owner: 10Dzahn) [10:07:03] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] zotero: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/399969 (owner: 10Dzahn) [10:08:45] (03CR) 10Alexandros Kosiaris: [C: 04-1] tcpircbot: convert role to profile (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/400250 (owner: 10Dzahn) [10:10:30] (03PS1) 10Giuseppe Lavagetto: appservers: move mw1180-1188 to role::spare::system [puppet] - 10https://gerrit.wikimedia.org/r/401479 (https://phabricator.wikimedia.org/T183895) [10:10:32] (03PS1) 10Giuseppe Lavagetto: hieradata: remove old leftovers [puppet] - 10https://gerrit.wikimedia.org/r/401480 [10:12:30] 10Operations, 10Traffic, 10media-storage: Swift invalid range requests causing 501s - https://phabricator.wikimedia.org/T183902#3866605 (10fgiunchedi) [10:16:29] (03CR) 10Alexandros Kosiaris: [C: 04-1] rancid: convert role to profile (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/399968 (owner: 10Dzahn) [10:20:03] (03CR) 10Muehlenhoff: [C: 031] appservers: move mw1180-1188 to role::spare::system [puppet] - 10https://gerrit.wikimedia.org/r/401479 (https://phabricator.wikimedia.org/T183895) (owner: 10Giuseppe Lavagetto) [10:20:06] (03CR) 10Alexandros Kosiaris: [C: 04-1] librenms: convert role to profile, variables to params (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/399966 (owner: 10Dzahn) [10:20:53] (03CR) 10Legoktm: mediawiki: Ensure Python 3 is available for Pygments (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/400458 (https://phabricator.wikimedia.org/T182851) (owner: 10Legoktm) [10:23:06] (03CR) 10Elukey: [C: 031] appservers: move mw1180-1188 to role::spare::system [puppet] - 10https://gerrit.wikimedia.org/r/401479 (https://phabricator.wikimedia.org/T183895) (owner: 10Giuseppe Lavagetto) [10:25:29] (03CR) 10Muehlenhoff: mediawiki: Ensure Python 3 is available for Pygments (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/400458 (https://phabricator.wikimedia.org/T182851) (owner: 10Legoktm) [10:26:54] (03PS1) 10Elukey: profile::kafka::monitoring: blacklist unwanted JMX Mbeans [puppet] - 10https://gerrit.wikimedia.org/r/401481 [10:27:41] (03CR) 10Elukey: [C: 032] profile::kafka::monitoring: blacklist unwanted JMX Mbeans [puppet] - 10https://gerrit.wikimedia.org/r/401481 (owner: 10Elukey) [10:30:22] (03PS1) 10Giuseppe Lavagetto: api: move mw1189-1200 to role::spare::system [puppet] - 10https://gerrit.wikimedia.org/r/401482 (https://phabricator.wikimedia.org/T183895) [10:32:48] (03CR) 10Elukey: [C: 031] api: move mw1189-1200 to role::spare::system [puppet] - 10https://gerrit.wikimedia.org/r/401482 (https://phabricator.wikimedia.org/T183895) (owner: 10Giuseppe Lavagetto) [10:44:09] (03PS1) 10Muehlenhoff: Add docker to account check consistency check whitelist [puppet] - 10https://gerrit.wikimedia.org/r/401485 [10:45:40] (03PS2) 10Muehlenhoff: Add docker to group membership consistency check whitelist [puppet] - 10https://gerrit.wikimedia.org/r/401485 [11:06:59] <_joe_> uh where is wikibugs? [11:11:59] coffee break? [11:15:11] _joe_ tools i think. [11:15:48] https://www.mediawiki.org/wiki/Wikibugs [11:17:14] <_joe_> paladox: yeah I don't really want to look into that myself right now [11:17:17] <_joe_> :) [11:17:24] ok. [11:17:57] <_joe_> I am doing something else completely, but if no one bothers I'll go take a look. I suspect that page is very, very outdated [11:25:03] (03CR) 10Giuseppe Lavagetto: [C: 032] "https://puppet-compiler.wmflabs.org/compiler02/9489/ this should work as expected in production." [puppet] - 10https://gerrit.wikimedia.org/r/394043 (owner: 10Giuseppe Lavagetto) [11:27:01] !log mobrovac@tin Started deploy [citoid/deploy@ee0bdf4]: Update to service template v0.5.4 - T151394 [11:27:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:27:12] T151394: Update Citoid to service-template-node v0.5.4 - https://phabricator.wikimedia.org/T151394 [11:28:06] _joe_: kunal / sam are good ones to ping for wikibugs iirc [11:28:27] <_joe_> p858snake: thanks :) [11:29:14] Seems kunal is already on it from the !log's in -cloud [11:29:33] <_joe_> yes [11:30:01] <_joe_> eddiegp: hi! while we're here - I'll find time to look again at your patches ASAP [11:30:07] <_joe_> the apache ones [11:30:37] _joe_: I've still not managed to find some time to test the biggest one of it, so no hurry ;) [11:30:46] !log jynus@tin Synchronized wmf-config/db-codfw.php: Repool db1055 & db1056 as x1 replicas (duration: 00m 50s) [11:30:48] <_joe_> ok cool :P [11:30:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:31:20] !log mobrovac@tin Finished deploy [citoid/deploy@ee0bdf4]: Update to service template v0.5.4 - T151394 (duration: 04m 19s) [11:31:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:31:36] (03PS2) 10Jcrespo: maridb: Prevent db1055 and db1056 from accidentally reimaging [puppet] - 10https://gerrit.wikimedia.org/r/401488 (https://phabricator.wikimedia.org/T183469) [11:32:42] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1055 & db1056 as x1 replicas (duration: 00m 50s) [11:32:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:34:05] (03PS1) 10Filippo Giunchedi: hieradata: partial eqiad SMART metrics rollout [puppet] - 10https://gerrit.wikimedia.org/r/401491 (https://phabricator.wikimedia.org/T86552) [11:35:23] (03PS1) 10Alexandros Kosiaris: WIP: Add all ops members to docker group [puppet] - 10https://gerrit.wikimedia.org/r/401492 [11:37:40] <_joe_> akosiaris: oh docker root without sudo? [11:37:44] (03PS1) 10Jcrespo: Revert "mariadb: Repool db1055 & db1056 as x1 replicas" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401493 [11:37:47] (03CR) 10Jcrespo: [V: 032 C: 032] Revert "mariadb: Repool db1055 & db1056 as x1 replicas" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401493 (owner: 10Jcrespo) [11:37:48] <_joe_> just on boron/ci, right? [11:37:58] (03CR) 10jenkins-bot: Revert "mariadb: Repool db1055 & db1056 as x1 replicas" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401493 (owner: 10Jcrespo) [11:38:26] _joe_: not even boron yet [11:38:29] just ci [11:39:10] so, it turns out that scap detected a 50% error rate [11:39:17] but continued deploying [11:39:29] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Revert: Repool db1055 & db1056 as x1 replicas (duration: 00m 51s) [11:39:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:40:27] does anyone know where scap logs live? [11:40:51] do scap deploy-log [11:41:01] in the repo of the software being deployed [11:41:18] otherwise you will have to go throught the pain of parsing json visually [11:41:41] but to answer the question, /scap/logs/ [11:42:01] it is empty [11:42:31] tin:/srv/mediawiki-staging/scap/log is empty to me [11:42:48] !log installing ncurses security updates [11:42:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:43:41] and untouched since 31 Jan 2017 [11:43:54] I am guessing only logstash then ? [11:44:17] but deployment errors should be somewhere [11:44:50] localy? [11:45:34] oh there are... for all services [11:45:45] mediawiki-staging seems to be an exception though [11:46:04] btw https://wikitech.wikimedia.org/wiki/Scap3#Structured_Logging [11:46:28] no, that is ok [11:46:35] I am worried about the lack of logs [11:46:59] <_joe_> jynus: 50% error rate on what? [11:47:15] _joe_: I am trying to check the logs to see why [11:47:15] <_joe_> akosiaris: mediawiki's scap uses completely separate code patterns than scap3 [11:47:31] or what [11:47:47] _joe_: come again ? [11:48:28] <_joe_> akosiaris: 'scap sync-file' uses different tasks than the ones used by 'scap deploy' [11:48:42] <_joe_> that means also it doesn't have logs locally, maybe [11:48:58] yeah I guess that's true [11:50:02] <_joe_> jynus: I doubt that's any log different than stdout for scap sync-file [11:50:22] _joe_: that is the problem I do not have stdour [11:50:27] I need that [11:50:40] <_joe_> what did you see on stdout? [11:50:45] <_joe_> it must have printed something [11:50:51] something something 50% error rate [11:50:58] I am trying to read it [11:51:01] from the logs [11:51:07] :-) [11:51:22] <_joe_> can you paste something something? [11:51:28] no [11:51:34] as I said, I do not have it [11:51:41] I am trying to search it on the logs [11:51:49] maybe I missread it [11:52:07] can you see why I need the logs ? XD [11:53:00] scap for mediawiki seems to have some debugging enabled and it spits 800 lines of debuggging on scap [11:53:15] so it gets lots on the buffer, you understand now ? :-D [11:53:51] ok, I finally found it [11:54:00] (03CR) 10Filippo Giunchedi: "PCC https://puppet-compiler.wmflabs.org/compiler02/9492/" [puppet] - 10https://gerrit.wikimedia.org/r/401491 (https://phabricator.wikimedia.org/T86552) (owner: 10Filippo Giunchedi) [11:54:03] Check 'Logstash Error rate for mw1276.eqiad.wmnet' failed: ERROR: 50% OVER_THRESHOLD (Avg. Error rate: Before: 0.03, After: 2.00, Threshold: 1.00) [11:54:57] <_joe_> jynus: it's telling you that the error rate after deploying the change was higher than before [11:54:59] so, mw1276, questions- what that happened, is it my fault? is it something else unrelated? [11:55:18] <_joe_> the deploy should've aborted unless you confirmed to go on [11:55:19] should scap abort after the rated is detected as high? is that a bug? [11:55:32] _joe_: my point entirely is it didn't abort [11:55:39] <_joe_> so either a bug or scap sync-file doesn't abort, I dunno [11:55:46] <_joe_> worth opening a bug about it [11:56:06] thanks, now I was able to explain myself, sorry for the missunderstanding [12:02:51] (03PS2) 10Alexandros Kosiaris: WIP: Add all ops members to docker group [puppet] - 10https://gerrit.wikimedia.org/r/401492 [12:10:45] (03PS2) 10Giuseppe Lavagetto: profile::mediawiki::jobrunner: restrict firewall rules [puppet] - 10https://gerrit.wikimedia.org/r/376024 [12:11:29] !log add missing mysql grants to db1055 and db1056 [12:11:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:17:05] (03PS1) 10Jcrespo: Revert "Revert "mariadb: Repool db1055 & db1056 as x1 replicas"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401496 [12:17:21] (03CR) 10Jcrespo: [C: 032] "Missing grants fixed" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401496 (owner: 10Jcrespo) [12:20:27] (03Merged) 10jenkins-bot: Revert "Revert "mariadb: Repool db1055 & db1056 as x1 replicas"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401496 (owner: 10Jcrespo) [12:20:40] (03CR) 10jenkins-bot: Revert "Revert "mariadb: Repool db1055 & db1056 as x1 replicas"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401496 (owner: 10Jcrespo) [12:21:43] !log empty ganeti1008 for kernel downgrade. T181121 [12:21:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:21:55] T181121: Hardware errors on ganeti1005- ganeti1008 - https://phabricator.wikimedia.org/T181121 [12:22:56] (03CR) 10Muehlenhoff: [C: 031] profile::mediawiki::jobrunner: restrict firewall rules [puppet] - 10https://gerrit.wikimedia.org/r/376024 (owner: 10Giuseppe Lavagetto) [12:24:23] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1055 & db1056 as x1 replicas (2nd try) (duration: 00m 51s) [12:24:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:25:36] !log jynus@tin Synchronized wmf-config/db-codfw.php: Repool db1055 & db1056 as x1 replicas (duration: 00m 51s) [12:25:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:34:49] 10Operations, 10ops-eqiad, 10DBA, 10hardware-requests, 10Patch-For-Review: Decommission db1045 - https://phabricator.wikimedia.org/T174806#3573553 (10MoritzMuehlenhoff) This host still shows up in puppetdb, i.e. misses the deactivate step (e.g. visible in https://servermon.wikimedia.org/hosts/) [12:35:49] 10Operations, 10ops-eqiad, 10Analytics, 10hardware-requests: Decommission db104[67] - https://phabricator.wikimedia.org/T181784#3802296 (10MoritzMuehlenhoff) These hosts still shows up in puppetdb, i.e. misses the deactivate step (e.g. visible in https://servermon.wikimedia.org/hosts/) [12:36:26] 10Operations, 10ops-eqiad, 10DBA, 10hardware-requests: Decommission db1049 - https://phabricator.wikimedia.org/T175264#3588137 (10MoritzMuehlenhoff) This host still shows up in puppetdb, i.e. misses the deactivate step (e.g. visible in https://servermon.wikimedia.org/hosts/) [12:39:17] 10Operations, 10ops-codfw, 10Analytics, 10DC-Ops: Decomission eventlog2001 - https://phabricator.wikimedia.org/T182397#3866939 (10MoritzMuehlenhoff) 05Resolved>03Open This host still shows up in puppetdb, i.e. misses the deactivate step (e.g. visible in https://servermon.wikimedia.org/hosts/) [13:00:26] 10Operations, 10Domains, 10Research, 10Traffic: Create subdomain for Research landing page - https://phabricator.wikimedia.org/T183916#3866972 (10bmansurov) p:05Triage>03Normal [13:00:33] !log installing openssl updates on remaining mw* hosts in eqiad [13:00:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:03:05] 10Operations, 10Domains, 10Research, 10Traffic: Create subdomain for Research landing page - https://phabricator.wikimedia.org/T183916#3866972 (10Krenair) Why is it not simply a redirect to a page on meta? [13:13:49] (03PS8) 10Rush: openstack: whitelist kernel versions for compute [puppet] - 10https://gerrit.wikimedia.org/r/399243 [13:15:54] (03PS9) 10Rush: openstack: whitelist kernel versions for compute [puppet] - 10https://gerrit.wikimedia.org/r/399243 [13:19:42] (03CR) 10Muehlenhoff: "Ubuntu bumps the name of the kernel package with every release, we should not hardcode a specific (and also outdated) kernel release, othe" [puppet] - 10https://gerrit.wikimedia.org/r/399243 (owner: 10Rush) [13:22:07] (03CR) 10Rush: "Needing to change this intentionally is the idea we are going for." [puppet] - 10https://gerrit.wikimedia.org/r/399243 (owner: 10Rush) [13:23:55] (03PS4) 10Rush: openstack: only run rabbitmq cleanup on active control node [puppet] - 10https://gerrit.wikimedia.org/r/398900 (https://phabricator.wikimedia.org/T183144) [13:24:32] 10Operations, 10ops-eqiad, 10Analytics, 10User-Elukey: Check analytics1037 power supply status - https://phabricator.wikimedia.org/T179192#3867003 (10elukey) Tried to check the ipmi command that the icinga check calls: ``` elukey@analytics1037:/var/log$ sudo ipmimonitoring -v | grep -i power 83 | PS Red... [13:32:27] (03CR) 10Jcrespo: [C: 032] maridb: Prevent db1055 and db1056 from accidentally reimaging [puppet] - 10https://gerrit.wikimedia.org/r/401488 (https://phabricator.wikimedia.org/T183469) (owner: 10Jcrespo) [13:35:39] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey, 10User-Joe: rack and setup mw1307-1348 - https://phabricator.wikimedia.org/T165519#3867018 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['mw1335.eqiad.wmnet'] ``` The log can be... [13:36:08] 10Operations, 10monitoring: remove cloud VPS project 'ganglia' - https://phabricator.wikimedia.org/T183917#3867019 (10Dzahn) p:05Triage>03Normal [13:39:30] (03PS1) 10Marostegui: db-eqiad.php: Depool db1098:3317 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401499 (https://phabricator.wikimedia.org/T174569) [13:39:57] 10Operations, 10Domains, 10Research, 10Traffic: Create subdomain for Research landing page - https://phabricator.wikimedia.org/T183916#3866972 (10Dzahn) Is the request to host the actual HTML files on Wikimedia infrastructure? That would be good, but i agree a wiki page would be even better (and save all t... [13:41:42] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1098:3317 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401499 (https://phabricator.wikimedia.org/T174569) (owner: 10Marostegui) [13:41:43] !log enable live traffic for new appservers mw1329->mw1333 (T165519) [13:41:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:41:53] T165519: rack and setup mw1307-1348 - https://phabricator.wikimedia.org/T165519 [13:42:23] !log elukey@puppetmaster1001 conftool action : set/pooled=yes; selector: name=mw1329.*.eqiad.wmnet [13:42:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:43:09] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1098:3317 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401499 (https://phabricator.wikimedia.org/T174569) (owner: 10Marostegui) [13:43:25] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1098:3317 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401499 (https://phabricator.wikimedia.org/T174569) (owner: 10Marostegui) [13:44:20] !log marostegui@tin Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 - T174569 (duration: 00m 51s) [13:44:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:44:30] T174569: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569 [13:45:13] !log elukey@puppetmaster1001 conftool action : set/pooled=yes; selector: name=mw1330.*.eqiad.wmnet [13:45:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:45:34] !log Deploy alter table db1098:3317 - T174569 [13:45:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:47:18] !log elukey@puppetmaster1001 conftool action : set/pooled=yes; selector: name=mw1331.*.eqiad.wmnet [13:47:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:48:59] !log elukey@puppetmaster1001 conftool action : set/pooled=yes; selector: name=mw1332.*.eqiad.wmnet [13:49:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:49:52] !log elukey@puppetmaster1001 conftool action : set/pooled=yes; selector: name=mw1333.*.eqiad.wmnet [13:50:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:50:18] all right all of these are running with confctl weight 10 [13:50:47] logs looks good, I'll raise them to 30 later on if all the metrics will look good [13:52:25] jouncebot: next [13:52:26] In 0 hour(s) and 7 minute(s): European Mid-day SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180102T1400) [13:57:35] Why is there no morning swat today (only EU & evening)? [13:57:53] did not even notice... [13:58:17] maybe because it's early morning after a looong weekend? ;) [13:59:10] Already asked in -releng earlier, because EU already hit the limit of 8 patches and evening is middle of the night for me. Wasn't sure if the missing morning window is on purpose. [13:59:59] I think it's only hashar and me in -releng at the moment, the schedule is created by greg-g, and he is asleep now [14:00:05] addshore, hashar, anomie, no_justification, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: #bothumor I � Unicode. All rise for European Mid-day SWAT(Max 8 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180102T1400). [14:00:05] Urbanecm and Hauskatze: A patch you scheduled for European Mid-day SWAT(Max 8 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [14:00:10] meow [14:00:19] I can SWAT today [14:00:33] !log installing further openssl updates [14:00:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:01:07] Urbanecm: around for SWAT? [14:02:04] Hauskatze: please stand by, if Urbanecm is not around, your patches are first to deploy [14:02:11] zeljkof: if he's not around you can start with me [14:02:13] I'm here :) [14:02:17] okay [14:02:46] * Urbanecm is ready for deploying process zeljkof [14:02:53] Urbanecm: ok, starting with the first patch, can you test it when it's at mwdebug? [14:03:31] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400093 (https://phabricator.wikimedia.org/T183655) (owner: 10Urbanecm) [14:03:33] zeljkof, they should be testable (all of them) [14:03:51] Urbanecm: ok, will ping you then as each of them is at mwdebug1002 [14:04:09] ok [14:04:11] Hauskatze: there is a chance I will not have the time to deploy you patches today [14:04:57] (03Merged) 10jenkins-bot: Create rollbacker user group for ruwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400093 (https://phabricator.wikimedia.org/T183655) (owner: 10Urbanecm) [14:04:58] zeljkof: well, okay I guess [14:05:33] Hauskatze: I'll do my best, but I don't think I was ever able to deploy all 8 patches in an hour [14:05:37] although as eddiegp said, it is quite unfortunate that we exit a code freeze with just two swat windows limited to each patches each [14:05:45] *eight patches each [14:06:00] true, not sure why is taht [14:06:01] that [14:07:00] !log removed 2FA for Martin_Urbanec [14:07:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:07:17] Urbanecm: 400093 is at mwdebug1002 [14:07:26] (03CR) 10jenkins-bot: Create rollbacker user group for ruwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400093 (https://phabricator.wikimedia.org/T183655) (owner: 10Urbanecm) [14:07:35] (03PS2) 10Zfilipin: Add suppressredirect to autoreview/editor at ruwikt [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400409 (https://phabricator.wikimedia.org/T183719) (owner: 10Urbanecm) [14:07:37] foks, thanks a lot! [14:07:47] Urbanecm, Sorry for the delay. :( [14:08:09] zeljkof, going to test [14:08:53] zeljkof, working, please deploy to the whole universe [14:09:00] (03CR) 10Zfilipin: [C: 031] Add suppressredirect to autoreview/editor at ruwikt [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400409 (https://phabricator.wikimedia.org/T183719) (owner: 10Urbanecm) [14:09:06] Urbanecm: deploying [14:09:37] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400409 (https://phabricator.wikimedia.org/T183719) (owner: 10Urbanecm) [14:09:47] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey, 10User-Joe: rack and setup mw1307-1348 - https://phabricator.wikimedia.org/T165519#3867097 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['mw1335.eqiad.wmnet'] ``` Of which those **FAILED**: ``` ['mw1335.eqiad.wmnet'] ``` [14:10:27] (03PS1) 10Filippo Giunchedi: mtail: stop sending metrics to graphite [puppet] - 10https://gerrit.wikimedia.org/r/401502 [14:11:01] (03Merged) 10jenkins-bot: Add suppressredirect to autoreview/editor at ruwikt [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400409 (https://phabricator.wikimedia.org/T183719) (owner: 10Urbanecm) [14:11:11] (03CR) 10jenkins-bot: Add suppressredirect to autoreview/editor at ruwikt [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400409 (https://phabricator.wikimedia.org/T183719) (owner: 10Urbanecm) [14:12:20] !log zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:400093|Create rollbacker user group for ruwiktionary (T183655)]] (duration: 00m 52s) [14:12:32] * foks Urbanecm, to save Phabricator-spam - thanks for the info :) Good to know the scope of that script for future [14:12:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:12:33] T183655: Create rollbacker group for ruwiktionary - https://phabricator.wikimedia.org/T183655 [14:13:04] ... I don't know why that formatted that way :/ [14:13:12] foks, you're wlecome. Asked at phab, replied at phab :D [14:13:15] *welcome [14:13:19] (03CR) 10Marostegui: "I would prefer to avoid using es1011 as it is the master, I would suggest es1015 instead" [puppet] - 10https://gerrit.wikimedia.org/r/401491 (https://phabricator.wikimedia.org/T86552) (owner: 10Filippo Giunchedi) [14:13:29] Urbanecm: 400093 deployed [14:13:35] zeljkof, thanks [14:14:14] (03PS7) 10Zfilipin: Switch Wikipedias from $wgLogoHD to direct using of a SVG [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399805 (https://phabricator.wikimedia.org/T178942) (owner: 10Urbanecm) [14:15:06] Urbanecm: 400409 is at mwdebug1002 [14:15:21] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey, 10User-Joe: rack and setup mw1307-1348 - https://phabricator.wikimedia.org/T165519#3867108 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['mw1335.eqiad.wmnet'] ``` The log can be... [14:15:46] zeljkof, works, please deploy [14:15:57] Urbanecm: deploying [14:17:02] !log zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:400409|Add suppressredirect to autoreview/editor at ruwikt (T183719)]] (duration: 00m 51s) [14:17:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:17:14] T183719: Add suppressredirect permission to groups (?) on Russian Wiktionary. - https://phabricator.wikimedia.org/T183719 [14:17:15] (03PS2) 10Giuseppe Lavagetto: utils/git-setup: add installation of the post-commit hook [puppet] - 10https://gerrit.wikimedia.org/r/381936 [14:17:19] Urbanecm: 400409 is deployed [14:17:29] ack [14:18:38] (03CR) 10Giuseppe Lavagetto: [C: 032] utils/git-setup: add installation of the post-commit hook [puppet] - 10https://gerrit.wikimedia.org/r/381936 (owner: 10Giuseppe Lavagetto) [14:19:08] (03Abandoned) 10Giuseppe Lavagetto: parsoid: test commmit for T149432 [puppet] - 10https://gerrit.wikimedia.org/r/370168 (owner: 10Giuseppe Lavagetto) [14:19:26] (03PS2) 10Filippo Giunchedi: hieradata: partial eqiad SMART metrics rollout [puppet] - 10https://gerrit.wikimedia.org/r/401491 (https://phabricator.wikimedia.org/T86552) [14:19:40] zeljkof, what's next patch? [14:19:59] Urbanecm: I am going in calendar order, reviewing 399805 [14:20:11] Ok [14:20:24] (03CR) 10Filippo Giunchedi: "> I would prefer to avoid using es1011 as it is the master, I would" [puppet] - 10https://gerrit.wikimedia.org/r/401491 (https://phabricator.wikimedia.org/T86552) (owner: 10Filippo Giunchedi) [14:22:03] (03PS1) 10Filippo Giunchedi: secrets: create digicert 2017 empty certs [labs/private] - 10https://gerrit.wikimedia.org/r/401504 [14:22:43] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399805 (https://phabricator.wikimedia.org/T178942) (owner: 10Urbanecm) [14:23:27] (03CR) 10Filippo Giunchedi: [V: 032 C: 032] secrets: create digicert 2017 empty certs [labs/private] - 10https://gerrit.wikimedia.org/r/401504 (owner: 10Filippo Giunchedi) [14:24:10] (03Merged) 10jenkins-bot: Switch Wikipedias from $wgLogoHD to direct using of a SVG [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399805 (https://phabricator.wikimedia.org/T178942) (owner: 10Urbanecm) [14:25:33] (03CR) 10Filippo Giunchedi: [C: 032] "PCC https://puppet-compiler.wmflabs.org/compiler02/9498/" [puppet] - 10https://gerrit.wikimedia.org/r/401502 (owner: 10Filippo Giunchedi) [14:25:45] (03PS2) 10Filippo Giunchedi: mtail: stop sending metrics to graphite [puppet] - 10https://gerrit.wikimedia.org/r/401502 [14:26:00] (03PS5) 10Zfilipin: Update chrwiki logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399806 (https://phabricator.wikimedia.org/T180553) (owner: 10Urbanecm) [14:27:01] Urbanecm: 399805 is at mwdebug1002 [14:27:17] (03CR) 10jenkins-bot: Switch Wikipedias from $wgLogoHD to direct using of a SVG [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399805 (https://phabricator.wikimedia.org/T178942) (owner: 10Urbanecm) [14:27:43] testing [14:27:46] (03CR) 10Rush: [C: 032] openstack: only run rabbitmq cleanup on active control node [puppet] - 10https://gerrit.wikimedia.org/r/398900 (https://phabricator.wikimedia.org/T183144) (owner: 10Rush) [14:27:50] (03PS5) 10Rush: openstack: only run rabbitmq cleanup on active control node [puppet] - 10https://gerrit.wikimedia.org/r/398900 (https://phabricator.wikimedia.org/T183144) [14:28:00] (03PS1) 10Andrew Bogott: Horizon puppet tab: disable role filters for now [puppet] - 10https://gerrit.wikimedia.org/r/401506 (https://phabricator.wikimedia.org/T181551) [14:28:28] (03CR) 10jerkins-bot: [V: 04-1] Horizon puppet tab: disable role filters for now [puppet] - 10https://gerrit.wikimedia.org/r/401506 (https://phabricator.wikimedia.org/T181551) (owner: 10Andrew Bogott) [14:28:56] <_joe_> jynus: ok to merge your change? [14:29:25] oh, yes, I didn't? [14:29:37] (03CR) 10Zfilipin: [C: 031] Update chrwiki logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399806 (https://phabricator.wikimedia.org/T180553) (owner: 10Urbanecm) [14:29:44] (I caught both but am waiting now for a sec) [14:29:44] apparently I left it on yes/no screen [14:29:45] sorry [14:30:01] zeljkof, working, you can deploy [14:30:12] Urbanecm: deploying [14:30:31] (03PS2) 10Andrew Bogott: Horizon puppet tab: disable role filters for now [puppet] - 10https://gerrit.wikimedia.org/r/401506 (https://phabricator.wikimedia.org/T181551) [14:31:04] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey, 10User-Joe: rack and setup mw1307-1348 - https://phabricator.wikimedia.org/T165519#3867157 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['mw1335.eqiad.wmnet'] ``` The log can be... [14:32:36] !log zfilipin@tin Synchronized static/images/project-logos/: SWAT: [[gerrit:399805|Switch Wikipedias from $wgLogoHD to direct using of a SVG (T178942)]] (duration: 01m 59s) [14:32:47] (03PS3) 10Andrew Bogott: Horizon puppet tab: disable role filters for now [puppet] - 10https://gerrit.wikimedia.org/r/401506 (https://phabricator.wikimedia.org/T181551) [14:32:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:32:49] T178942: Switch existing wikis from high density logos to SVG - https://phabricator.wikimedia.org/T178942 [14:33:30] !log zfilipin@tin scap failed: average error rate on 6/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) [14:33:38] (03CR) 10Andrew Bogott: [C: 032] Horizon puppet tab: disable role filters for now [puppet] - 10https://gerrit.wikimedia.org/r/401506 (https://phabricator.wikimedia.org/T181551) (owner: 10Andrew Bogott) [14:33:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:33:45] Urbanecm: uh oh [14:33:49] zeljkof, what's happening? [14:34:00] "scap failed: average error rate on 6/11 canaries increased by 10x" [14:34:21] Do you know what that mean? [14:34:33] something broken :) [14:34:49] not sure what yet, something in Skin.php [14:35:01] Notice: Array to string conversion in /srv/mediawiki/php-1.31.0-wmf.12/includes/skins/Skin.php on line 937 [14:35:15] what logstash is saying? [14:35:29] Hmm, that's not good... paladox, can you help? [14:35:46] Warning: strpos() expects parameter 1 to be string, array given in /srv/mediawiki/php-1.31.0-wmf.12/includes/OutputPage.php on line 3800 [14:36:09] jynus: should I revert last patch? [14:36:09] (03PS1) 10Rush: rabbitmq: drain_queue is defined dupe [puppet] - 10https://gerrit.wikimedia.org/r/401508 [14:36:11] 3,570 errors in the last 3 minutes [14:36:24] I would , then investigate later [14:36:31] zeljkof, I think we should revert and investigate later in the phab task [14:36:41] (03PS2) 10Rush: rabbitmq: drain_queue is defined dupe [puppet] - 10https://gerrit.wikimedia.org/r/401508 [14:36:42] jynus, Urbanecm: reverting 399805 [14:36:46] ack [14:36:50] 5000 errors/minutes is a lot [14:36:52] (03PS1) 10Zfilipin: Revert "Switch Wikipedias from $wgLogoHD to direct using of a SVG" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401509 [14:37:02] Urbanecm i just got back, but yeh revert. [14:37:14] that was fixed when i last tested the change. [14:37:15] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401509 (owner: 10Zfilipin) [14:37:17] (03CR) 10Rush: [C: 032] rabbitmq: drain_queue is defined dupe [puppet] - 10https://gerrit.wikimedia.org/r/401508 (owner: 10Rush) [14:37:20] Not sure how the error got back. [14:37:35] Could a task be created please? [14:37:54] paladox, I can only create "SVG usage in wgLogo is not working", I prefer if you can create it [14:37:54] paladox, Urbanecm: could you please create the task? [14:38:50] (03Merged) 10jenkins-bot: Revert "Switch Wikipedias from $wgLogoHD to direct using of a SVG" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401509 (owner: 10Zfilipin) [14:39:00] (03CR) 10jenkins-bot: Revert "Switch Wikipedias from $wgLogoHD to direct using of a SVG" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401509 (owner: 10Zfilipin) [14:39:36] paladox, I've commented about reverting in T178942 [14:39:37] T178942: Switch existing wikis from high density logos to SVG - https://phabricator.wikimedia.org/T178942 [14:39:52] 10Operations, 10Patch-For-Review: Tracking and Reducing cron-spam from root@ - https://phabricator.wikimedia.org/T132324#3867174 (10chasemp) [14:39:54] 10Operations, 10cloud-services-team, 10Patch-For-Review: labcontrol1002 Error: unable to connect to node rabbit@labcontrol1002: nodedown - https://phabricator.wikimedia.org/T183144#3867172 (10chasemp) 05Open>03Resolved ```Notice: /Stage[main]/Rabbitmq::Cleanup/Cron[drain and log rabbit notifications.erro... [14:41:11] zeljkof, please skip 399806. Can't be deployed now. I'll abandon it in a moment [14:41:17] !log zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:401509|Revert "Switch Wikipedias from $wgLogoHD to direct using of a SVG" (T178942)]] (duration: 00m 51s) [14:41:21] Urbanecm: ok [14:41:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:41:50] Urbanecm https://phabricator.wikimedia.org/T183919 [14:42:00] (03CR) 10Rush: [C: 032] openstack: whitelist kernel versions for compute [puppet] - 10https://gerrit.wikimedia.org/r/399243 (owner: 10Rush) [14:42:04] (03PS10) 10Rush: openstack: whitelist kernel versions for compute [puppet] - 10https://gerrit.wikimedia.org/r/399243 [14:42:06] (03Abandoned) 10Urbanecm: Update chrwiki logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399806 (https://phabricator.wikimedia.org/T180553) (owner: 10Urbanecm) [14:42:40] paladox, thanks [14:42:41] !log zfilipin@tin Synchronized static/images/project-logos/: SWAT: [[gerrit:401509|Revert "Switch Wikipedias from $wgLogoHD to direct using of a SVG" (T178942)]] (duration: 00m 51s) [14:42:47] your welcome. [14:42:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:42:51] (03CR) 10Rush: [C: 032] "talked this through briefly with moritz on irc and I think we are gtg on this for the moment, coming kernel upgrades will suss out the act" [puppet] - 10https://gerrit.wikimedia.org/r/399243 (owner: 10Rush) [14:43:23] (03PS2) 10Zfilipin: Enable mapframe on lvwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400096 (https://phabricator.wikimedia.org/T183661) (owner: 10Urbanecm) [14:43:35] zeljkof, I just marked the errors in the calendar [14:44:07] Urbanecm: thanks [14:44:11] yw [14:44:54] Urbanecm: 399805 is reverted by 401509, all deployed, errors dropping [14:45:10] sorry to insist, but for trivial changes/reverts, we should not doubt- we revert even if later happens the errors are unrelated [14:45:24] worse case scenario, we commit again [14:45:51] jynus, this was not a trivial change [14:45:53] jynus: agreed, it's all in git, easier to redeploy than to debug a hairy bug [14:46:00] oh, it wasn't? [14:46:26] it changed a lot of logos, to svg [14:46:30] Urbanecm: then it should NOT be on swat! [14:46:40] Urbanecm oh i see what i did that was wrong. [14:46:43] ;-D [14:47:01] zeljkof: exactly my thought [14:47:14] jynus, where it should be then? ;) [14:47:32] I think this was appropriate for swat, a big (in number of files) change, but not something complicated, I think, just a bunch of logos [14:47:55] then that is trivial to me [14:48:04] Urbanecm: you can always request a deploy window for big changes [14:48:23] trivial: anything that can be easily reverted with no consequences [14:48:31] but updating logos, even a lot of them is good for swat, I think [14:48:31] ideally, all patches should be trivial [14:49:18] jynus, ok, if that's the definition of trivial, it was trivial [14:49:20] updating logos is trivial [14:49:32] I agree [14:49:41] worst case is caching issues which are solved with purgeList.php [14:49:44] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400096 (https://phabricator.wikimedia.org/T183661) (owner: 10Urbanecm) [14:49:54] so back to the original question of "revert first, ask questions later" :-) [14:50:07] jynus, that's true anyway, isn't it? [14:50:21] Hauskatze, paladox said he know the issue. But IMHO we should not discuss the issue here but continue with SWAT. We have 10 mins left [14:51:11] (03Merged) 10jenkins-bot: Enable mapframe on lvwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400096 (https://phabricator.wikimedia.org/T183661) (owner: 10Urbanecm) [14:51:23] (03CR) 10jenkins-bot: Enable mapframe on lvwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400096 (https://phabricator.wikimedia.org/T183661) (owner: 10Urbanecm) [14:52:00] 10Operations, 10Cloud-VPS, 10cloud-services-team: labstore Tools and Misc share very full - https://phabricator.wikimedia.org/T183920#3867239 (10chasemp) [14:52:17] (03PS4) 10Andrew Bogott: puppet enc getter: only allow acccess to VMs and puppetmasters [puppet] - 10https://gerrit.wikimedia.org/r/399901 (https://phabricator.wikimedia.org/T169086) [14:52:21] Urbanecm i've updated the task with the correct configuration now. [14:52:36] Urbanecm: 400096 is at mwdebug1002 [14:52:40] zeljkof, will test [14:52:44] paladox, thx, will look at it later [14:52:45] 10Operations, 10Cloud-VPS, 10cloud-services-team: labstore Tools and Misc share very full - https://phabricator.wikimedia.org/T183920#3867250 (10chasemp) p:05Triage>03High [14:52:49] Hauskatze: sorry, there will be no time for your patches today [14:53:05] zeljkof: don't worry, I was falling asleep anyway [14:53:10] (03CR) 10Andrew Bogott: [C: 032] puppet enc getter: only allow acccess to VMs and puppetmasters [puppet] - 10https://gerrit.wikimedia.org/r/399901 (https://phabricator.wikimedia.org/T169086) (owner: 10Andrew Bogott) [14:53:24] zeljkof, working, please deploy [14:53:28] I need another espresso [14:54:04] Urbanecm: deploying [14:54:17] ack [14:54:50] !log zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:400096|Enable mapframe on lvwiki (T183661)]] (duration: 00m 51s) [14:55:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:55:01] T183661: Enable on Latvian Wikipedia - https://phabricator.wikimedia.org/T183661 [14:55:10] Urbanecm: 400096 is deployed [14:55:51] zeljkof, thanks! [14:56:21] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400579 (https://phabricator.wikimedia.org/T178750) (owner: 10Urbanecm) [14:56:28] (03CR) 10Zfilipin: Set 'watchcreations' preference to true by default on Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400579 (https://phabricator.wikimedia.org/T178750) (owner: 10Urbanecm) [14:56:32] (03PS2) 10Zfilipin: Set 'watchcreations' preference to true by default on Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400579 (https://phabricator.wikimedia.org/T178750) (owner: 10Urbanecm) [14:56:32] Just as a side note, if other swatters share the impression of what zeljkof said earlier "I don't think I was ever able to deploy all 8 patches in an hour" you should consider setting a lower limit to avoid false expectations. :) [14:56:48] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400579 (https://phabricator.wikimedia.org/T178750) (owner: 10Urbanecm) [14:57:02] eddiegp: it might be just me being slow [14:57:22] or increase swat window duration to 1'30 hours [14:57:39] Anyway, SWAT are normally a lot shorter. [14:57:59] (03CR) 10Ottomata: [C: 032] Replace libmysqlclient-dev with default-libmysqlclient-dev [puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/394818 (https://phabricator.wikimedia.org/T51652) (owner: 10Gergő Tisza) [14:58:02] indeed, they ain't that full nor we usually run into errors [14:58:19] (03Merged) 10jenkins-bot: Set 'watchcreations' preference to true by default on Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400579 (https://phabricator.wikimedia.org/T178750) (owner: 10Urbanecm) [14:58:31] (03CR) 10jenkins-bot: Set 'watchcreations' preference to true by default on Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400579 (https://phabricator.wikimedia.org/T178750) (owner: 10Urbanecm) [14:58:44] (03PS1) 10Andrew Bogott: Revert "puppet enc getter: only allow acccess to VMs and puppetmasters" [puppet] - 10https://gerrit.wikimedia.org/r/401511 [14:58:54] Urbanecm: any order files in 400579 should be deployed? common then initialise? vice-versa? any order? [14:59:33] IS.php, then CS.php (otherwise CS.php will reffer to nonexistent variable) [14:59:49] Urbanecm: ok [15:00:02] Urbanecm: 400579 is at mwdebug1002 [15:00:17] testing [15:01:57] (03PS1) 10Andrew Bogott: cloud puppetmasters: fix an erb issue where we need an array vs. a string [puppet] - 10https://gerrit.wikimedia.org/r/401512 (https://phabricator.wikimedia.org/T169086) [15:02:10] zeljkof, works, please deploy [15:02:18] Urbanecm: deploying [15:02:36] (03PS1) 10Jcrespo: mariadb: Increase db1055 and db1056 x1 weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401513 (https://phabricator.wikimedia.org/T183469) [15:02:39] ack [15:03:26] !log zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:400579|Set watchcreations preference to true by default on Commons (T178750)]] (duration: 00m 51s) [15:03:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:03:37] T178750: "Add pages I create and files I upload to my watchlist" should be checked by default on Commons - https://phabricator.wikimedia.org/T178750 [15:04:11] (03PS2) 10Rush: dumps: add wikidata-primary-sources-tool mount [puppet] - 10https://gerrit.wikimedia.org/r/399223 (https://phabricator.wikimedia.org/T183229) [15:04:27] !log zfilipin@tin Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:400579|Set watchcreations preference to true by default on Commons (T178750)]] (duration: 00m 51s) [15:04:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:04:54] Urbanecm: 400579 deployed, please check and thanks for deploying with #releng ;) [15:05:02] !log EU SWAT finished [15:05:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:05:15] (03CR) 10Lydia Pintscher: [C: 031] "Let's please get this in and enable fine grained usage tracking in Lua wiki by wiki. Better safe than sorry here." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/398823 (https://phabricator.wikimedia.org/T172914) (owner: 10Ladsgroup) [15:05:19] (03CR) 10Rush: [C: 032] dumps: add wikidata-primary-sources-tool mount [puppet] - 10https://gerrit.wikimedia.org/r/399223 (https://phabricator.wikimedia.org/T183229) (owner: 10Rush) [15:05:58] (03PS2) 10Volans: ClusterShell backend: fix execute() return code [software/cumin] - 10https://gerrit.wikimedia.org/r/399829 [15:06:00] (03PS3) 10Volans: PuppetDB backend: add support for API v4 [software/cumin] - 10https://gerrit.wikimedia.org/r/399821 (https://phabricator.wikimedia.org/T182575) [15:06:28] Works, thanks for deploying for me! [15:07:01] (03PS1) 10Filippo Giunchedi: prometheus: add mtail to varnish-text job [puppet] - 10https://gerrit.wikimedia.org/r/401516 (https://phabricator.wikimedia.org/T177199) [15:08:29] (03PS3) 10Volans: ClusterShell backend: fix execute() return code [software/cumin] - 10https://gerrit.wikimedia.org/r/399829 [15:08:31] (03PS4) 10Volans: PuppetDB backend: add support for API v4 [software/cumin] - 10https://gerrit.wikimedia.org/r/399821 (https://phabricator.wikimedia.org/T182575) [15:08:59] 10Operations, 10Internet-Archive, 10Offline-Working-Group: Create backups of Wikimedia content in diverse geographic places - https://phabricator.wikimedia.org/T156544#3867284 (10Ottomata) I've heard rumors (just rumors!) about backup improvements becoming a more prioritized project in the coming year, soooo... [15:09:16] (03CR) 10Volans: "@gehel: thanks for the review, addressed comments." (032 comments) [software/cumin] - 10https://gerrit.wikimedia.org/r/399829 (owner: 10Volans) [15:09:55] 10Operations, 10ops-eqiad, 10Analytics-Kanban: dbstore1002 possibly MEMORY issues - https://phabricator.wikimedia.org/T183771#3867285 (10Ottomata) > Alternatively, not purchase anything- being fully aware that if it fails, there is no backup service available Most of this data is now available for SQL query... [15:10:14] (03CR) 10Ema: [C: 031] prometheus: add mtail to varnish-text job [puppet] - 10https://gerrit.wikimedia.org/r/401516 (https://phabricator.wikimedia.org/T177199) (owner: 10Filippo Giunchedi) [15:10:29] (03PS2) 10Andrew Bogott: cloud puppetmasters: fix an erb issue with the big puppetmaster list [puppet] - 10https://gerrit.wikimedia.org/r/401512 (https://phabricator.wikimedia.org/T169086) [15:11:08] (03CR) 10jerkins-bot: [V: 04-1] cloud puppetmasters: fix an erb issue with the big puppetmaster list [puppet] - 10https://gerrit.wikimedia.org/r/401512 (https://phabricator.wikimedia.org/T169086) (owner: 10Andrew Bogott) [15:11:51] (03CR) 10Ottomata: [C: 031] "Don't have much context, but I am fine with testing SMART metrics monitoring on any analytics boxes." [puppet] - 10https://gerrit.wikimedia.org/r/401491 (https://phabricator.wikimedia.org/T86552) (owner: 10Filippo Giunchedi) [15:12:14] (03PS1) 10Rush: labstore: correct yaml from 399223 [puppet] - 10https://gerrit.wikimedia.org/r/401517 (https://phabricator.wikimedia.org/T183229) [15:12:42] (03CR) 10Rush: [C: 032] labstore: correct yaml from 399223 [puppet] - 10https://gerrit.wikimedia.org/r/401517 (https://phabricator.wikimedia.org/T183229) (owner: 10Rush) [15:13:03] (03PS2) 10Filippo Giunchedi: prometheus: add mtail to varnish-text job [puppet] - 10https://gerrit.wikimedia.org/r/401516 (https://phabricator.wikimedia.org/T177199) [15:14:08] (03CR) 10Filippo Giunchedi: [C: 032] prometheus: add mtail to varnish-text job [puppet] - 10https://gerrit.wikimedia.org/r/401516 (https://phabricator.wikimedia.org/T177199) (owner: 10Filippo Giunchedi) [15:14:20] (03PS2) 10Jcrespo: mariadb: Increase db1055 and db1056 x1 weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401513 (https://phabricator.wikimedia.org/T183469) [15:15:31] (03PS3) 10Andrew Bogott: cloud puppetmasters: fix an erb issue with the big puppetmaster list [puppet] - 10https://gerrit.wikimedia.org/r/401512 (https://phabricator.wikimedia.org/T169086) [15:15:33] (03CR) 10Jcrespo: [C: 031] hieradata: partial eqiad SMART metrics rollout [puppet] - 10https://gerrit.wikimedia.org/r/401491 (https://phabricator.wikimedia.org/T86552) (owner: 10Filippo Giunchedi) [15:15:54] (03CR) 10jerkins-bot: [V: 04-1] cloud puppetmasters: fix an erb issue with the big puppetmaster list [puppet] - 10https://gerrit.wikimedia.org/r/401512 (https://phabricator.wikimedia.org/T169086) (owner: 10Andrew Bogott) [15:16:36] (03CR) 10Jcrespo: [C: 032] mariadb: Increase db1055 and db1056 x1 weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401513 (https://phabricator.wikimedia.org/T183469) (owner: 10Jcrespo) [15:16:50] !log boot ganeti1008 with older 4.4 kernel and migrate multiple VMs to it. T181121 [15:17:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:17:01] T181121: Hardware errors on ganeti1005- ganeti1008 - https://phabricator.wikimedia.org/T181121 [15:17:02] (03PS4) 10Andrew Bogott: cloud puppetmasters: fix an erb issue with the big puppetmaster list [puppet] - 10https://gerrit.wikimedia.org/r/401512 (https://phabricator.wikimedia.org/T169086) [15:17:58] (03CR) 10Andrew Bogott: [C: 032] cloud puppetmasters: fix an erb issue with the big puppetmaster list [puppet] - 10https://gerrit.wikimedia.org/r/401512 (https://phabricator.wikimedia.org/T169086) (owner: 10Andrew Bogott) [15:18:09] (03Merged) 10jenkins-bot: mariadb: Increase db1055 and db1056 x1 weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401513 (https://phabricator.wikimedia.org/T183469) (owner: 10Jcrespo) [15:18:19] (03CR) 10Ottomata: [C: 032] Add and link readme pages for analytics datasets [puppet] - 10https://gerrit.wikimedia.org/r/395917 (https://phabricator.wikimedia.org/T167033) (owner: 10Milimetric) [15:18:21] (03CR) 10jenkins-bot: mariadb: Increase db1055 and db1056 x1 weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401513 (https://phabricator.wikimedia.org/T183469) (owner: 10Jcrespo) [15:18:24] (03PS3) 10Ottomata: Add and link readme pages for analytics datasets [puppet] - 10https://gerrit.wikimedia.org/r/395917 (https://phabricator.wikimedia.org/T167033) (owner: 10Milimetric) [15:18:26] (03CR) 10Ottomata: [V: 032 C: 032] Add and link readme pages for analytics datasets [puppet] - 10https://gerrit.wikimedia.org/r/395917 (https://phabricator.wikimedia.org/T167033) (owner: 10Milimetric) [15:21:03] (03CR) 10Ottomata: "librdkafka and Java use different names for the ciphers. For librdkafka, we need to use:" [puppet] - 10https://gerrit.wikimedia.org/r/399700 (https://phabricator.wikimedia.org/T167304) (owner: 10Ottomata) [15:21:28] 10Operations, 10Cloud-VPS, 10cloud-services-team: labstore Tools and Misc share very full - https://phabricator.wikimedia.org/T183920#3867302 (10chasemp) running as of now > labstore1004:~# find /srv/tools -type f -size +100M -printf "%p %k KB\n" &> /root/tools_large_files_01022018.txt [15:30:00] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Increase db1055 & db1056 x1 weight (duration: 00m 50s) [15:30:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:30:31] (03PS1) 10Urbanecm: Move wiktionary HD logo to wiktionaries [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401519 (https://phabricator.wikimedia.org/T183922) [15:31:40] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey, 10User-Joe: rack and setup mw1307-1348 - https://phabricator.wikimedia.org/T165519#3867349 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['mw1335.eqiad.wmnet'] ``` and were **ALL** successful. [15:32:24] (03PS3) 10Filippo Giunchedi: mtail: stop sending metrics to graphite [puppet] - 10https://gerrit.wikimedia.org/r/401502 [15:33:11] !log elukey@puppetmaster1001 conftool action : set/pooled=yes; selector: name=mw1335.*.eqiad.wmnet [15:33:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:35:43] (03CR) 10Marostegui: [C: 031] hieradata: partial eqiad SMART metrics rollout [puppet] - 10https://gerrit.wikimedia.org/r/401491 (https://phabricator.wikimedia.org/T86552) (owner: 10Filippo Giunchedi) [15:42:21] (03PS2) 10Urbanecm: Move wiktionary HD logo to wiktionaries [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401519 (https://phabricator.wikimedia.org/T183922) [15:43:06] 10Operations, 10ops-codfw, 10DBA: db2054: Disk with predictive failure - https://phabricator.wikimedia.org/T183887#3867388 (10Papaul) Dear Mr Papaul Tshibamba, Thank you for contacting Hewlett Packard Enterprise for your service request. This email confirms your request for service and the details are below... [15:46:45] 10Operations, 10ops-eqiad, 10Analytics-Kanban: dbstore1002 possibly MEMORY issues - https://phabricator.wikimedia.org/T183771#3867410 (10Ottomata) DOH Ignore ^ I for some reason was thinking yall were talking about eventlogging, not MW analytics slave dbs. Carry on! [15:49:26] (03PS1) 10Urbanecm: Switch Wikipedias from $wgLogoHD to direct using of a SVG [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401523 [15:49:57] (03PS2) 10Urbanecm: Switch Wikipedias from $wgLogoHD to direct using of a SVG [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401523 (https://phabricator.wikimedia.org/T178942) [15:50:00] 10Operations, 10ops-codfw, 10DBA: pc2005 crashed: CPU2 internal error - https://phabricator.wikimedia.org/T183750#3867417 (10RobH) Lease versus purchase has no change in warranty support, just in our tracking of hardware. This should be able to be processed as a normal under warranty server. (Leasing just... [15:52:16] (03PS1) 10Jcrespo: mariadb: Depool db1029 from x1 in preparation for decommission [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401525 (https://phabricator.wikimedia.org/T183469) [15:53:37] !log elukey@puppetmaster1001 conftool action : set/weight=20; selector: name=mw13(29|3[0-3]).*.eqiad.wmnet [15:53:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:54:22] (03CR) 10Paladox: "LGTM :)." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401523 (https://phabricator.wikimedia.org/T178942) (owner: 10Urbanecm) [15:54:26] (03PS1) 10Ema: varnish: add varnishmtail instance for varnish backends [puppet] - 10https://gerrit.wikimedia.org/r/401526 (https://phabricator.wikimedia.org/T177199) [15:54:37] (03CR) 10Paladox: [C: 031] Switch Wikipedias from $wgLogoHD to direct using of a SVG [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401523 (https://phabricator.wikimedia.org/T178942) (owner: 10Urbanecm) [15:55:21] !log installing openssl updates on restbase* hosts [15:55:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:57:09] (03PS1) 10Jcrespo: mariadb: Remove db1029 from dblists [software] - 10https://gerrit.wikimedia.org/r/401527 (https://phabricator.wikimedia.org/T183469) [16:00:26] (03PS1) 10Jcrespo: mariadb: Decommission db1029, former x1 replica [puppet] - 10https://gerrit.wikimedia.org/r/401529 (https://phabricator.wikimedia.org/T183469) [16:01:22] 10Operations, 10DBA, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#3867470 (10jcrespo) [16:02:00] 10Operations, 10ops-eqiad, 10Analytics, 10User-Elukey: Check analytics1037 power supply status - https://phabricator.wikimedia.org/T179192#3867475 (10RobH) It seems odd that the harware says its fine, but the software check doesn't. I'd rather we not close the task if its showing the alarm, but leave it o... [16:04:24] (03CR) 10Halfak: [C: 031] ores: install myspell-ca [puppet] - 10https://gerrit.wikimedia.org/r/400227 (https://phabricator.wikimedia.org/T182612) (owner: 10Ladsgroup) [16:07:29] 10Operations, 10ops-eqiad, 10Analytics, 10User-Elukey: Check analytics1037 power supply status - https://phabricator.wikimedia.org/T179192#3867494 (10elukey) 05Open>03stalled [16:07:36] (03CR) 10Jcrespo: [C: 032] mariadb: Depool db1029 from x1 in preparation for decommission [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401525 (https://phabricator.wikimedia.org/T183469) (owner: 10Jcrespo) [16:07:46] (03Abandoned) 10Andrew Bogott: Revert "puppet enc getter: only allow acccess to VMs and puppetmasters" [puppet] - 10https://gerrit.wikimedia.org/r/401511 (owner: 10Andrew Bogott) [16:09:22] (03CR) 10Jcrespo: "Am I missing something?" [puppet] - 10https://gerrit.wikimedia.org/r/401529 (https://phabricator.wikimedia.org/T183469) (owner: 10Jcrespo) [16:09:49] (03Merged) 10jenkins-bot: mariadb: Depool db1029 from x1 in preparation for decommission [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401525 (https://phabricator.wikimedia.org/T183469) (owner: 10Jcrespo) [16:13:45] 10Operations, 10Cloud-VPS, 10cloud-services-team: labstore Tools and Misc share very full - https://phabricator.wikimedia.org/T183920#3867540 (10chasemp) >>! In T183920#3867302, @chasemp wrote: > running as of now > >> labstore1004:~# find /srv/tools -type f -size +100M -printf "%p %k KB\n" &> /root/tools_l... [16:14:53] 10Operations, 10Analytics-Cluster, 10Analytics-Kanban, 10Traffic, 10User-Elukey: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#3867544 (10Ottomata) > Is restricting protocol to TLSv1.2 explicitly worth any gain on top of the above, given librdkafka has no config for i... [16:15:40] (03PS2) 10Ema: varnish: add varnishmtail instance for varnish backends [puppet] - 10https://gerrit.wikimedia.org/r/401526 (https://phabricator.wikimedia.org/T177199) [16:16:03] (03CR) 10jerkins-bot: [V: 04-1] varnish: add varnishmtail instance for varnish backends [puppet] - 10https://gerrit.wikimedia.org/r/401526 (https://phabricator.wikimedia.org/T177199) (owner: 10Ema) [16:16:20] (03PS2) 10Alexandros Kosiaris: ores: install myspell-ca [puppet] - 10https://gerrit.wikimedia.org/r/400227 (https://phabricator.wikimedia.org/T182612) (owner: 10Ladsgroup) [16:16:24] (03CR) 10Alexandros Kosiaris: [C: 032] ores: install myspell-ca [puppet] - 10https://gerrit.wikimedia.org/r/400227 (https://phabricator.wikimedia.org/T182612) (owner: 10Ladsgroup) [16:16:26] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] ores: install myspell-ca [puppet] - 10https://gerrit.wikimedia.org/r/400227 (https://phabricator.wikimedia.org/T182612) (owner: 10Ladsgroup) [16:20:00] (03PS1) 10Ema: mtail::program: allow to specify destination directory [puppet] - 10https://gerrit.wikimedia.org/r/401533 [16:22:13] (03CR) 10Thcipriani: "Looks good to me." (031 comment) [dumps/scap] - 10https://gerrit.wikimedia.org/r/400598 (owner: 10ArielGlenn) [16:24:49] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1029 (duration: 00m 51s) [16:24:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:26:07] (03PS3) 10Filippo Giunchedi: varnish: add varnishmtail instance for varnish backends [puppet] - 10https://gerrit.wikimedia.org/r/401526 (https://phabricator.wikimedia.org/T177199) (owner: 10Ema) [16:26:09] (03PS1) 10Filippo Giunchedi: mtail: add program to count varnish backend metrics [puppet] - 10https://gerrit.wikimedia.org/r/401535 (https://phabricator.wikimedia.org/T177199) [16:27:35] (03PS4) 10Ema: varnish: add varnishmtail instance for varnish backends [puppet] - 10https://gerrit.wikimedia.org/r/401526 (https://phabricator.wikimedia.org/T177199) [16:27:37] (03CR) 10jenkins-bot: mariadb: Depool db1029 from x1 in preparation for decommission [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401525 (https://phabricator.wikimedia.org/T183469) (owner: 10Jcrespo) [16:29:40] 10Operations, 10Cloud-VPS, 10cloud-services-team: labstore Tools and Misc share very full - https://phabricator.wikimedia.org/T183920#3867614 (10chasemp) > cat tools_large_files_01022018.txt | sort -h -k 2 [16:31:08] (03PS2) 10Filippo Giunchedi: mtail: add program to count varnish backend metrics [puppet] - 10https://gerrit.wikimedia.org/r/401535 (https://phabricator.wikimedia.org/T177199) [16:37:30] (03PS2) 10Ema: mtail::program: allow to specify destination directory [puppet] - 10https://gerrit.wikimedia.org/r/401533 [16:44:41] (03PS3) 10Ema: mtail::program: allow to specify destination directory [puppet] - 10https://gerrit.wikimedia.org/r/401533 [16:44:49] 10Operations, 10Traffic, 10Goal, 10User-fgiunchedi: Limit http methods reported by varnishmtail - https://phabricator.wikimedia.org/T183926#3867684 (10fgiunchedi) p:05Triage>03Normal [16:47:50] 10Operations, 10Developer-Relations: Bring discourse.mediawiki.org to production - https://phabricator.wikimedia.org/T180853#3867718 (10Qgil) [16:47:54] (03CR) 10Filippo Giunchedi: [C: 031] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/401533 (owner: 10Ema) [16:48:38] !log elukey@puppetmaster1001 conftool action : set/weight=30; selector: name=mw13(29|3[0-3]).*.eqiad.wmnet [16:48:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:51:34] !log restarted exim and spamd services on fermium, mx1001 and mx2001 for openssl update [16:51:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:52:26] (03CR) 10Ema: [C: 032] mtail::program: allow to specify destination directory [puppet] - 10https://gerrit.wikimedia.org/r/401533 (owner: 10Ema) [16:53:05] !log add missing mysql grants to db1097:s4 [16:53:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:54:02] (03PS5) 10Filippo Giunchedi: varnish: add varnishmtail instance for varnish backends [puppet] - 10https://gerrit.wikimedia.org/r/401526 (https://phabricator.wikimedia.org/T177199) (owner: 10Ema) [16:54:15] (03CR) 10jerkins-bot: [V: 04-1] varnish: add varnishmtail instance for varnish backends [puppet] - 10https://gerrit.wikimedia.org/r/401526 (https://phabricator.wikimedia.org/T177199) (owner: 10Ema) [16:54:49] (03PS3) 10Filippo Giunchedi: mtail: add program to count varnish backend metrics [puppet] - 10https://gerrit.wikimedia.org/r/401535 (https://phabricator.wikimedia.org/T177199) [16:55:24] (03PS6) 10Ema: varnish: add varnishmtail instance for varnish backends [puppet] - 10https://gerrit.wikimedia.org/r/401526 (https://phabricator.wikimedia.org/T177199) [16:56:34] Hi no_justification! Give me a ping when you make the branch and before you do the train so that I can make a MCR revert patch for the branch! :) [16:57:23] I'll start the branching now :) [16:57:33] (03PS2) 10Thcipriani: Revert "l10nupdate: don't run over holidays" [puppet] - 10https://gerrit.wikimedia.org/r/399848 (owner: 10Volans) [16:58:55] thcipriani: want me to merge it? ^^^ [16:59:00] 10Operations, 10HHVM: Potential shard confussion on loadbalancer checking s1 lag on x1 hosts? Or just config outdated/fail? - https://phabricator.wikimedia.org/T183925#3867796 (10jcrespo) These are all occurrences on db1055, all from mw1191 https://logstash.wikimedia.org/goto/914fce6861a9d0d52db4afa97384af41 ,... [16:59:17] 10Operations, 10HHVM, 10Wikimedia-log-errors: Potential shard confussion on loadbalancer checking s1 lag on x1 hosts? Or just config outdated/fail? - https://phabricator.wikimedia.org/T183925#3867798 (10jcrespo) [16:59:27] volans: yes please, I posted it for puppet swat :) [17:00:04] godog, moritzm, and _joe_: That opportune time is upon us again. Time for a Puppet SWAT(Max 8 patches) deploy. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180102T1700). [17:00:04] thcipriani: A patch you scheduled for Puppet SWAT(Max 8 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [17:00:25] thcipriani: ack, merging [17:00:26] * godog looking at puppet swat [17:00:33] I see volans has volunteered! thanks [17:00:54] 10Operations, 10HHVM, 10Wikimedia-log-errors: Potential shard confussion on loadbalancer checking s1 lag on x1 hosts? Or just config outdated/fail? - https://phabricator.wikimedia.org/T183925#3867662 (10jcrespo) 05Open>03Invalid Effectively, mw1191 has outdated code, and those queries are only coming fro... [17:01:07] (03CR) 10Volans: [C: 032] Revert "l10nupdate: don't run over holidays" [puppet] - 10https://gerrit.wikimedia.org/r/399848 (owner: 10Volans) [17:01:09] !log add missing mysql grants to db1103:s4 [17:01:12] godog: yeah it's the revert of the one I did before the holidays, I'll take care [17:01:17] thanks for checking ;) [17:01:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:01:25] volans: thank you! [17:03:19] thcipriani: puppet run on tin, crontab entry back there [17:03:22] 10Operations, 10ops-eqiad: mw1191 ipmi-sel cpu errors - https://phabricator.wikimedia.org/T179640#3731680 (10jcrespo) This ticket confused me: T183925 We should stop apache/HHVM to prevent logging errors to production kibana. [17:04:06] Puppet SWAT completed [17:04:14] volans: great! thanks again! [17:04:19] yw! [17:04:42] nuria_: Heyas, I'm working with chris on the two jupyter notebook systems [17:04:58] these are direct replacements for notebook100[12]? [17:05:18] I'm wondering if they have the same networking requirements or if anything is changing =] [17:07:53] nothing should have changed afaik [17:08:26] ottomata -^ ? [17:09:30] (03PS4) 10Ema: mtail: add program to count varnish backend metrics [puppet] - 10https://gerrit.wikimedia.org/r/401535 (https://phabricator.wikimedia.org/T177199) (owner: 10Filippo Giunchedi) [17:11:52] 10Operations, 10ops-eqiad, 10DBA: Degraded RAID on db1001 - https://phabricator.wikimedia.org/T183708#3867855 (10Cmjohnson) Disk Swapped [17:11:58] 10Operations, 10ops-eqiad, 10Analytics: rack/setup/install noteboot100[34] - https://phabricator.wikimedia.org/T183935#3867856 (10RobH) p:05Triage>03Normal [17:12:53] 10Operations, 10ops-eqiad, 10DBA: Degraded RAID on db1001 - https://phabricator.wikimedia.org/T183708#3867877 (10Marostegui) Thanks! ``` root@db1001:~# megacli -pdrbld -showprog -physdrv\[32:1\] -aALL Rebuild Progress on Device at Enclosure 32, Slot 1 Completed 3% in 1 Minutes. ``` [17:12:54] (03CR) 10Filippo Giunchedi: [C: 031] varnish: add varnishmtail instance for varnish backends [puppet] - 10https://gerrit.wikimedia.org/r/401526 (https://phabricator.wikimedia.org/T177199) (owner: 10Ema) [17:14:12] elukey / ottomata cool https://phabricator.wikimedia.org/T183935 [17:14:22] they are onsite and cmjohnson1 is going to get them setup [17:14:50] i made two assumptions: 1) notebook100[12] are going away when these come online so sharing a row with 1 or 2 is non issue [17:15:05] but notebook100[34] should be in different rows, much like notebook100[12] were [17:15:09] (in analytics vlan) [17:15:56] and 2) set them up identical in vlan and os (analytics vlan and jessie) [17:16:25] addshore: We should remove "Wikidata" from make-wmf-branch, right? [17:16:31] (it fails, obviously) [17:16:39] yes [17:16:42] 10Operations, 10ops-eqiad, 10Analytics: rack/setup/install noteboot100[34] - https://phabricator.wikimedia.org/T183935#3867886 (10RobH) [17:16:45] i think i might have a patch up for ot [17:16:46] it [17:16:50] elukey: i just also listed either you or otto to hand off these systems to when done [17:16:56] no_justification: https://gerrit.wikimedia.org/r/#/c/394288/ [17:16:57] let me know if that isnt right! [17:17:18] just removed the DNM and rebased it [17:17:29] :) [17:17:41] *looks for other wikidata build killing related patches in his list* [17:17:51] +2'd [17:18:39] (03PS5) 10Filippo Giunchedi: mtail: add program to count varnish backend metrics [puppet] - 10https://gerrit.wikimedia.org/r/401535 (https://phabricator.wikimedia.org/T177199) [17:18:47] I have a couple of patches switching some of the extensions that were in the build to use the json entry point :) [17:19:10] (03PS6) 10Ema: mtail: add program to count varnish backend metrics [puppet] - 10https://gerrit.wikimedia.org/r/401535 (https://phabricator.wikimedia.org/T177199) (owner: 10Filippo Giunchedi) [17:21:44] robh: yep I think everything looks good, thanks! [17:22:01] 10Operations, 10ORES, 10Graphite, 10Scoring-platform-team (Current), 10User-fgiunchedi: Regularly purge old ores graphite metrics - https://phabricator.wikimedia.org/T169969#3867919 (10Halfak) @fgiunchedi, can you help me figure out what our next step should be here? [17:22:05] 10Operations, 10ops-eqiad: rack/setup/install labvirt102[12] - https://phabricator.wikimedia.org/T183937#3867914 (10RobH) p:05Triage>03Normal [17:22:05] awesome =] [17:22:51] thanks robh! [17:22:51] looks good [17:23:16] HMmmmmm elukey i wonder if we should install the new notebook servers as stretch now [17:23:23] they are just analytics clients, and we run stat1005 stretch already [17:23:57] !log demon@tin Pruned MediaWiki: 1.31.0-wmf.10 (duration: 01m 29s) [17:24:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:25:41] 10Operations, 10ops-eqiad, 10Analytics-Kanban: dbstore1002 possibly MEMORY issues - https://phabricator.wikimedia.org/T183771#3867954 (10elukey) I've discussed this task with my team and a couple of things came up: 1) The host is two months from being OOW, so getting a replacement if it breaks might become... [17:26:28] ottomata: we could yes, good point [17:26:54] but it is something that we can easily do when the rack/setup/deploy task is created (it will take no time for wmf-auto-reimage) [17:28:13] !log demon@tin Pruned MediaWiki: 1.31.0-wmf.11 (duration: 01m 24s) [17:28:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:30:24] elukey: this one? https://phabricator.wikimedia.org/T183935#3867886 [17:30:31] i'll edit task [17:30:31] :) [17:30:53] 10Operations, 10ops-eqiad, 10Analytics: rack/setup/install noteboot100[34] - https://phabricator.wikimedia.org/T183935#3867970 (10Ottomata) Let's do Stretch. [17:30:58] 10Operations, 10ops-eqiad, 10Analytics: rack/setup/install noteboot100[34] - https://phabricator.wikimedia.org/T183935#3867971 (10Ottomata) [17:31:11] ok! [17:31:51] (03PS3) 10Andrew Bogott: Puppetmaster web frontend: support specifying different certs for a hostname [puppet] - 10https://gerrit.wikimedia.org/r/399459 (https://phabricator.wikimedia.org/T183414) [17:32:26] <_joe_> incoming [17:32:59] (03PS1) 10Giuseppe Lavagetto: site.pp: convert dns recursors to single role [puppet] - 10https://gerrit.wikimedia.org/r/401547 [17:33:01] (03PS1) 10Giuseppe Lavagetto: bastionhost: add role for caching PoPs [puppet] - 10https://gerrit.wikimedia.org/r/401548 [17:33:03] (03PS1) 10Giuseppe Lavagetto: site.pp: use role keyword for striker::web only on californium [puppet] - 10https://gerrit.wikimedia.org/r/401549 [17:33:05] (03PS1) 10Giuseppe Lavagetto: cache: add ipsec to basic roles [puppet] - 10https://gerrit.wikimedia.org/r/401550 [17:33:07] (03PS1) 10Giuseppe Lavagetto: site.pp: simplify role() keyword call for cache::canary [puppet] - 10https://gerrit.wikimedia.org/r/401551 [17:33:09] (03PS1) 10Giuseppe Lavagetto: site.pp: one role for dbstore2001.codfw.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/401552 [17:33:11] (03PS1) 10Giuseppe Lavagetto: monitoring: create role::alerting_host [puppet] - 10https://gerrit.wikimedia.org/r/401553 [17:33:13] (03PS1) 10Giuseppe Lavagetto: eventlogging: create compound role, consolidate hiera [puppet] - 10https://gerrit.wikimedia.org/r/401554 [17:33:15] (03PS1) 10Giuseppe Lavagetto: labtestservices2001: one role() call [puppet] - 10https://gerrit.wikimedia.org/r/401555 [17:34:36] addshore: Branches created, not checked out to tin yet though [17:34:40] (so ideal time for a merge :)) [17:35:24] (03CR) 10Ottomata: [C: 032] eventlogging: create compound role, consolidate hiera (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/401554 (owner: 10Giuseppe Lavagetto) [17:35:29] (03CR) 10Ottomata: [C: 031] "1 nit, don't really care." [puppet] - 10https://gerrit.wikimedia.org/r/401554 (owner: 10Giuseppe Lavagetto) [17:35:58] (03CR) 10Jcrespo: "This is going to be refactored for the backup migration goal anyway." [puppet] - 10https://gerrit.wikimedia.org/r/401552 (owner: 10Giuseppe Lavagetto) [17:37:06] (03CR) 10Andrew Bogott: [C: 032] Puppetmaster web frontend: support specifying different certs for a hostname [puppet] - 10https://gerrit.wikimedia.org/r/399459 (https://phabricator.wikimedia.org/T183414) (owner: 10Andrew Bogott) [17:38:17] <_joe_> ottomata: thanks for looking, I'm trying to go quick & dirty to be able to cut out the role hiera backend completely, and simplify our hiera lookup system in production [17:40:44] _joe_: +1 proceed! :) [17:41:14] we're still holding out on refactoring that single EL box + role for k8 one day... [17:41:25] (03PS1) 10Cmjohnson: Adding user groovier temporary access to stat1006, researchers eventlog data per T181952 [puppet] - 10https://gerrit.wikimedia.org/r/401558 [17:41:49] (03CR) 10jerkins-bot: [V: 04-1] Adding user groovier temporary access to stat1006, researchers eventlog data per T181952 [puppet] - 10https://gerrit.wikimedia.org/r/401558 (owner: 10Cmjohnson) [17:42:52] no_justification: okay! Give me a few moments :) [17:43:10] (03PS1) 10Chad: group0 to wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401559 [17:43:12] (03CR) 10Chad: [C: 04-2] group0 to wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401559 (owner: 10Chad) [17:43:49] (03PS14) 10Andrew Bogott: wmcs puppet: Support instance agents using the 'puppet' master hostname [puppet] - 10https://gerrit.wikimedia.org/r/398323 (https://phabricator.wikimedia.org/T183414) [17:44:51] (03CR) 10Andrew Bogott: [C: 032] wmcs puppet: Support instance agents using the 'puppet' master hostname [puppet] - 10https://gerrit.wikimedia.org/r/398323 (https://phabricator.wikimedia.org/T183414) (owner: 10Andrew Bogott) [17:50:37] (03PS2) 10Cmjohnson: Adding user groovier temporary access to stat1006, researchers eventlog data per T181952 [puppet] - 10https://gerrit.wikimedia.org/r/401558 [17:51:03] (03CR) 10jerkins-bot: [V: 04-1] Adding user groovier temporary access to stat1006, researchers eventlog data per T181952 [puppet] - 10https://gerrit.wikimedia.org/r/401558 (owner: 10Cmjohnson) [17:51:55] (03PS7) 10Filippo Giunchedi: mtail: add program to count varnish backend metrics [puppet] - 10https://gerrit.wikimedia.org/r/401535 (https://phabricator.wikimedia.org/T177199) [17:52:20] (03PS3) 10Smalyshev: Lower refresh interval for Wikidata to 5s [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399466 (https://phabricator.wikimedia.org/T183053) [17:54:22] (03PS4) 10Smalyshev: Lower ElasticSearch index refresh interval for Wikidata to 5s [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399466 (https://phabricator.wikimedia.org/T183053) [17:54:58] (03PS3) 10Cmjohnson: Add usr:groovier temp access stat1006 &eventlog data bug# T181952 [puppet] - 10https://gerrit.wikimedia.org/r/401558 [17:55:18] (03CR) 10jerkins-bot: [V: 04-1] Add usr:groovier temp access stat1006 &eventlog data bug# T181952 [puppet] - 10https://gerrit.wikimedia.org/r/401558 (owner: 10Cmjohnson) [17:55:23] no_justification: https://gerrit.wikimedia.org/r/#/c/401562/ [17:55:24] (03PS8) 10Filippo Giunchedi: mtail: add program to count varnish backend metrics [puppet] - 10https://gerrit.wikimedia.org/r/401535 (https://phabricator.wikimedia.org/T177199) [17:55:55] (03PS4) 10Cmjohnson: Add usr:groovier temp access stat1006 &eventlog data bug# T181952 [puppet] - 10https://gerrit.wikimedia.org/r/401558 [18:00:05] cscott, arlolra, subbu, halfak, and Amir1: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Services – Graphoid / Parsoid / Citoid / ORES deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180102T1800). [18:00:05] No GERRIT patches in the queue for this window AFAICS. [18:00:24] We’re doing a minor ORES service deployment. [18:00:46] !log awight@tin Started deploy [ores/deploy@eb0f776]: Update ORES service to eb0f776: T182614 [18:00:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:00:57] T182614: Investigate why ORES logs are being written to syslog despite explicit logging config. Fix. - https://phabricator.wikimedia.org/T182614 [18:04:19] (03CR) 10EBernhardson: [C: 031] Lower ElasticSearch index refresh interval for Wikidata to 5s [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399466 (https://phabricator.wikimedia.org/T183053) (owner: 10Smalyshev) [18:04:58] someone mentioned it, I forget how, but somehow the "morning swat" (aka 11am pacific, 19:00 UTC) was gone from today: I've re-added it. Enjoy your swats. [18:06:17] mooeypoo: https://phabricator.wikimedia.org/T183910 - "Exclude selected" in "Namespaces" doesn't work when using "Saved filters" in RecentChanges [18:06:45] (03PS5) 10Cmjohnson: Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) [18:07:10] (03CR) 10jerkins-bot: [V: 04-1] Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:09:03] !log arlolra@tin Started deploy [parsoid/deploy@4d55952]: Updating Parsoid to 28d7734 [18:09:06] (03PS6) 10Cmjohnson: Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) [18:09:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:09:31] (03CR) 10jerkins-bot: [V: 04-1] Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:10:29] !log rebooting multatuli for kernel test [18:10:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:11:12] cmjohnson1: when adding a new user, you need to first create it.. merge just the user, and in a second patch add it to the group [18:11:24] or jerkins-bot will keep failing [18:12:36] i "use" it to just create users without giving them access yet, so access request is simpler change [18:12:36] (03PS7) 10Cmjohnson: Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) [18:13:00] (03CR) 10jerkins-bot: [V: 04-1] Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:13:47] greg-g: Awesome, thanks! :) [18:14:07] it doesnt get that a "non-existing" user is added to a group if it happens in a single change [18:15:20] jouncebot: refresh [18:15:22] I refreshed my knowledge about deployments. [18:15:29] jouncebot: next [18:15:29] In 0 hour(s) and 44 minute(s): Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180102T1900) [18:15:48] (03PS3) 10EddieGP: Restrict sending mails to new users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/397768 (https://phabricator.wikimedia.org/T182541) [18:18:25] (03PS1) 10Cmjohnson: mend [puppet] - 10https://gerrit.wikimedia.org/r/401569 [18:19:01] (03Abandoned) 10Cmjohnson: mend [puppet] - 10https://gerrit.wikimedia.org/r/401569 (owner: 10Cmjohnson) [18:20:40] !log awight@tin Finished deploy [ores/deploy@eb0f776]: Update ORES service to eb0f776: T182614 (duration: 19m 55s) [18:20:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:20:54] T182614: Investigate why ORES logs are being written to syslog despite explicit logging config. Fix. - https://phabricator.wikimedia.org/T182614 [18:21:01] !log arlolra@tin Finished deploy [parsoid/deploy@4d55952]: Updating Parsoid to 28d7734 (duration: 11m 57s) [18:21:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:22:39] (03PS8) 10Cmjohnson: Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) [18:23:09] (03CR) 10jerkins-bot: [V: 04-1] Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:23:19] ^ \o/ [18:23:24] (03PS9) 10Cmjohnson: Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) [18:23:46] (03CR) 10jerkins-bot: [V: 04-1] Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:25:56] 10Operations, 10ORES, 10Scoring-platform-team (Current): Investigate why ORES logs are being written to syslog despite explicit logging config. Fix. - https://phabricator.wikimedia.org/T182614#3868235 (10awight) A little additional excitement... Now that we're seeing all the logs, some previously hidden er... [18:26:45] 10Operations, 10Cloud-VPS, 10monitoring: remove cloud VPS project 'ganglia' - https://phabricator.wikimedia.org/T183917#3868240 (10Dzahn) [18:29:30] (03PS10) 10RobH: Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:29:53] (03CR) 10jerkins-bot: [V: 04-1] Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:31:40] (03PS11) 10RobH: Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:32:11] (03CR) 10jerkins-bot: [V: 04-1] Adding Groovier to shell access [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:32:37] this commit message failures are fucking driving me crazy [18:32:39] atf [18:32:41] wtf [18:32:44] now it hates all blank lines... [18:33:05] cmjohnson1: i hate it [18:34:18] (03PS12) 10RobH: adding new shell user groovier new shell user and addition to the researchers group, all approvals are on the linked task. Bug: T181952 Change-Id: I0a951849841ded1c80815af1e5a9b4edf584712b [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:34:48] (03CR) 10jerkins-bot: [V: 04-1] adding new shell user groovier new shell user and addition to the researchers group, all approvals are on the linked task. Bug: T181952 Change-Id: I0a951849841ded1c80815af1e5a9b4edf584712b [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:36:15] 10Operations, 10Cloud-VPS, 10monitoring: remove cloud VPS project 'ganglia' - https://phabricator.wikimedia.org/T183917#3867019 (10Paladox) I think this project has been deleted as it is not showing up here https://tools.wmflabs.org/openstack-browser/project/ganglia [18:37:19] !log T183053 update index.refresh_interval for wikidatawiki_{content,general} on eqiad to 5s [18:37:26] (03CR) 10RobH: [V: 031 C: 031] "commit message keeps giving us trouble, but the patch is fine. I'm not sure why the commit message failures happen for everything (includ" [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:37:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:37:30] T183053: New Wikidata items appear in search with a delay - https://phabricator.wikimedia.org/T183053 [18:38:05] 10Operations, 10Analytics-Cluster, 10Analytics-Kanban, 10Traffic, 10User-Elukey: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#3868327 (10MoritzMuehlenhoff) >>! In T182993#3867544, @Ottomata wrote: > K! Kafka [[ https://github.com/apache/kafka/blob/trunk/clients/src/... [18:38:28] 10Operations, 10Cloud-VPS, 10monitoring: remove cloud VPS project 'ganglia' - https://phabricator.wikimedia.org/T183917#3868330 (10Dzahn) a:05Dzahn>03Andrew Thanks for the link Paladox, i wasn't aware of that search on tools.wmflabs. I guess i will just deleted the wiki pages? @Andrew i was going to a... [18:40:39] 10Operations, 10Cloud-VPS, 10monitoring: remove cloud VPS project 'ganglia' - https://phabricator.wikimedia.org/T183917#3868349 (10Dzahn) I just added {{historical}} on the Nova_resource wiki pages as is suggested on T183873. [18:40:49] !log demon@tin Started scap: wmf.15 bootstrap [18:40:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:43:13] (03CR) 10Cmjohnson: [C: 032] adding new shell user groovier new shell user and addition to the researchers group, all approvals are on the linked task. Bug: T181952 Chan [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:43:16] (03CR) 10Cmjohnson: [V: 032 C: 032] adding new shell user groovier new shell user and addition to the researchers group, all approvals are on the linked task. Bug: T181952 Chan [puppet] - 10https://gerrit.wikimedia.org/r/401558 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:45:03] (03PS1) 10Cmjohnson: Revert "adding new shell user groovier new shell user and addition to the researchers group, all approvals are on the linked task. Bug: T181952 Change-Id: I0a951849841ded1c80815af1e5a9b4edf584712b" [puppet] - 10https://gerrit.wikimedia.org/r/401579 (https://phabricator.wikimedia.org/T181952) [18:45:17] (03PS1) 10Ottomata: Mirror Kafka 1.0 from confluent in stretch and use 1.0 protocol version [puppet] - 10https://gerrit.wikimedia.org/r/401580 [18:45:29] (03CR) 10jerkins-bot: [V: 04-1] Revert "adding new shell user groovier new shell user and addition to the researchers group, all approvals are on the linked task. Bug: T181952 Change-Id: I0a951849841ded1c80815af1e5a9b4edf584712b" [puppet] - 10https://gerrit.wikimedia.org/r/401579 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:45:57] (03PS2) 10Ottomata: Mirror Kafka 1.0 from confluent in stretch and use 1.0 protocol version [puppet] - 10https://gerrit.wikimedia.org/r/401580 [18:46:05] !log started linter-reparse script on terbium to reprocess itwiki pages (safe to kill -9 the script at any point) [18:46:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:46:50] (03CR) 10Ottomata: "Has been tested in labs." [puppet] - 10https://gerrit.wikimedia.org/r/401580 (owner: 10Ottomata) [18:46:52] (03CR) 10Ottomata: [C: 032] Mirror Kafka 1.0 from confluent in stretch and use 1.0 protocol version [puppet] - 10https://gerrit.wikimedia.org/r/401580 (owner: 10Ottomata) [18:47:19] robh ok to merge your groovier shell user change? [18:47:33] it was chris's change [18:47:36] and now he is reverting it [18:47:38] so uhh [18:47:42] oh puppet-merge has your name on it [18:47:49] ahh, i had a commit message edit [18:47:51] ok cmjohnson1 you can puppet-merge my kafka chnage when you are ready [18:47:55] thanks [18:47:58] so yeah if your change can wait [18:47:58] just let me know when you've done [18:48:02] best to have his revert go live for merge [18:48:05] k [18:51:19] (03PS2) 10RobH: Revert "adding new shell user groovier new shell user and addition to the researchers group, all approvals are on the linked task. Bug: T181952 Change-Id: I0a951849841ded1c80815af1e5a9b4edf584712b" [puppet] - 10https://gerrit.wikimedia.org/r/401579 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:51:42] (03CR) 10jerkins-bot: [V: 04-1] Revert "adding new shell user groovier new shell user and addition to the researchers group, all approvals are on the linked task. Bug: T181952 Change-Id: I0a951849841ded1c80815af1e5a9b4edf584712b" [puppet] - 10https://gerrit.wikimedia.org/r/401579 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:52:11] (03CR) 10RobH: [V: 032 C: 032] Revert "adding new shell user groovier new shell user and addition to the researchers group, all approvals are on the linked task. Bug: T181 [puppet] - 10https://gerrit.wikimedia.org/r/401579 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:52:13] (03CR) 10Cmjohnson: [C: 032] Revert "adding new shell user groovier new shell user and addition to the researchers group, all approvals are on the linked task. Bug: T181 [puppet] - 10https://gerrit.wikimedia.org/r/401579 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [18:52:21] the commit message issue is just a missing new line [18:52:34] before the Bug: line [18:52:36] (03PS2) 10Andrew Bogott: labnet dnsmasq: use upstream dns servers [puppet] - 10https://gerrit.wikimedia.org/r/398510 (https://phabricator.wikimedia.org/T181375) [18:53:27] (03PS3) 10Andrew Bogott: labnet dnsmasq: use upstream dns servers [puppet] - 10https://gerrit.wikimedia.org/r/398510 (https://phabricator.wikimedia.org/T181375) [18:53:59] 10Operations, 10Cloud-VPS, 10cloud-services-team: tools.iabot is using 1.3T of 8T available tools nfs storage - https://phabricator.wikimedia.org/T183953#3868405 (10madhuvishy) p:05Triage>03High [18:54:39] (03CR) 10Andrew Bogott: [C: 032] labnet dnsmasq: use upstream dns servers [puppet] - 10https://gerrit.wikimedia.org/r/398510 (https://phabricator.wikimedia.org/T181375) (owner: 10Andrew Bogott) [18:54:41] ottomata: done [18:54:58] yay [18:55:15] mutante: so the issue there is we HAD a blank line [18:55:19] and it bitched about it being a blank line [18:55:19] heh [18:55:25] 10Operations, 10Cloud-VPS, 10cloud-services-team: tools.iabot is using 1.3T of 8T available tools nfs storage - https://phabricator.wikimedia.org/T183953#3868432 (10Cyberpower678) Interesting. I'm not even sure what files are being created to cause this problem as files that are created are deleted afterwards. [18:55:30] see history on https://gerrit.wikimedia.org/r/#/c/401558/ [18:55:48] but oh well, chris will have to make a new one now so we'll see how it handles the commit message [18:56:15] 10Operations, 10Cloud-VPS, 10cloud-services-team: templatetiger is using 827G of 8T available tools nfs storage - https://phabricator.wikimedia.org/T183954#3868439 (10madhuvishy) p:05Triage>03High [18:56:43] 10Operations, 10Cloud-VPS, 10cloud-services-team: 2018-01-02: labstore Tools and Misc share very full - https://phabricator.wikimedia.org/T183920#3868461 (10bd808) [18:56:55] danke [18:58:36] hmm. yea. let's see on the new one [18:58:38] https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines [18:58:59] (03CR) 10Krinkle: [C: 04-1] "Blocking per https://phabricator.wikimedia.org/T178942#3868489" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401523 (https://phabricator.wikimedia.org/T178942) (owner: 10Urbanecm) [18:59:57] 10Operations, 10Cloud-VPS, 10cloud-services-team: tools.iabot is using 1.3T of 8T available tools nfs storage - https://phabricator.wikimedia.org/T183953#3868604 (10madhuvishy) @Cyberpower678 See Chase's comments on the parent task for more info T183920. [19:00:04] addshore, hashar, anomie, no_justification, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: Time to snap out of that daydream and deploy Morning SWAT (Max 8 patches). Get on with it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180102T1900). [19:00:04] eddiegp: A patch you scheduled for Morning SWAT (Max 8 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [19:01:12] * eddiegp is here [19:01:34] Huh, I thought we didn't have morning SWATs on Tuesdays [19:01:36] ( greg-g ? ) [19:02:39] Uhmm, and I didn't know that and complained that there's only EU & Evening for today, which lead to greg-g adding the morning window :D [19:02:43] 10Operations, 10Ops-Access-Requests, 10AICaptcha, 10WMF-NDA-Requests, 10Patch-For-Review: Requesting access to EventLogging data for Vinitha - https://phabricator.wikimedia.org/T181952#3868617 (10RobH) Please disregard the patchset merge and revert above, it has nothing to do with any actions on the task... [19:03:01] 10Operations, 10Cloud-VPS, 10cloud-services-team: wikidumpparse is using 1.2TB of 5T available NFS misc storage - https://phabricator.wikimedia.org/T183970#3868620 (10madhuvishy) p:05Triage>03High [19:03:01] yeah, that ^, did we remove them due to proximity to the train? [19:03:12] * greg-g is having brain boot up after vacation issues [19:03:31] no joke having that long off work is messing with my thought process as well! [19:03:54] Yes, we did [19:03:58] (remove them) [19:04:04] Yes you did [19:04:21] (03PS1) 10Cmjohnson: adding new shell user groovier new shell user and addition to the researchers group, all approvals are on the linked task. [puppet] - 10https://gerrit.wikimedia.org/r/401584 (https://phabricator.wikimedia.org/T181952) [19:04:45] (03CR) 10jerkins-bot: [V: 04-1] adding new shell user groovier new shell user and addition to the researchers group, all approvals are on the linked task. [puppet] - 10https://gerrit.wikimedia.org/r/401584 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [19:05:19] In my defense, the box on wikitech:Deployments still says "SWAT deploys happen thrice daily" ;) [19:05:34] 10Operations, 10Cloud-VPS, 10cloud-services-team: dumps project is using 2T of 5T available NFS misc storage - https://phabricator.wikimedia.org/T183971#3868651 (10madhuvishy) p:05Triage>03High [19:05:56] Yeah, they do on normal days. A while ago Greg and team decided to remove the morning one on Tuesdays (but leave it on other days) [19:06:09] (03PS2) 10Cmjohnson: adding new shell user groovier [puppet] - 10https://gerrit.wikimedia.org/r/401584 (https://phabricator.wikimedia.org/T181952) [19:06:20] Should I move my patch back to Thursday and remove the window again? [19:06:20] 10Operations, 10Ops-Access-Requests, 10Release-Engineering-Team: Allow "releasers-mediawiki" sudo rights to manage Jenkins - https://phabricator.wikimedia.org/T183972#3868664 (10demon) [19:06:25] * greg-g fixes calendar [19:06:30] sorry eddiegp :) [19:06:33] (03CR) 10jerkins-bot: [V: 04-1] adding new shell user groovier [puppet] - 10https://gerrit.wikimedia.org/r/401584 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [19:06:43] (03PS2) 10Chad: Nightly server: let MW releasers manage Jenkins [puppet] - 10https://gerrit.wikimedia.org/r/399123 (https://phabricator.wikimedia.org/T183972) [19:07:15] mutante: take a look at https://gerrit.wikimedia.org/r/#/c/401584/2 [19:07:20] its failing for commit message stuff [19:07:28] first it says needs a linke, now says unexpected blank line [19:07:51] cmjohnson1: id stop changing it for right now [19:07:54] Yeah you can't have a blank line between Bug: Tnnnn and Change-Id [19:07:55] because what you have looks right to me [19:08:03] Just remove that blank line and you'll be OK [19:08:18] ahhh [19:08:18] (03PS3) 10Dzahn: adding new shell user groovier [puppet] - 10https://gerrit.wikimedia.org/r/401584 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [19:08:36] It's a silly rule but it's because of a limitation in the way the bots deal with Bug: Tnnnn, they can't pick it up if it's not in the bottom section [19:08:44] ... [19:08:48] Yeah... thank sgit [19:08:51] *thanks git [19:09:06] fixed [19:09:16] Commit metadata is at the bottom, and if you put that blank line there it doesn't count as commit metadata, just free text [19:09:29] it needs the line before Bug: but not after [19:09:31] You could always add support for that in its-phabricator [19:09:34] eddiegp: I just removed your thing, was going to let you add it to the swat window you want (sorry about the confusion, blame the new year!) [19:09:38] so i wonder if htats something his editor did or what [19:10:25] Hmm he did introduce it, not sure why [19:10:52] I'm not aware of editors causing it, mostly people writing it themselves and forgetting, but in this case the Bug: line was already there and he only added the blank line [19:10:53] well, that falls in line with my previous experiences with git, in looking at my patch history [19:10:57] 10Operations, 10Analytics, 10ChangeProp, 10EventBus, and 4 others: Migrate htmlCacheUpdate job to Kafka - https://phabricator.wikimedia.org/T182023#3868701 (10Pchelolo) I've looked over all the logs and graphs we've acquired over the past several weeks and found no indications of issues. This makes me beli... [19:11:03] i didnt introcuce any lines between bug and change id so meh [19:11:37] I'm actually kind of happy that we have Jenkins checking for this now, previously you'd be able to merge commits like that and they'd never get reported on Phabricator [19:12:08] At least now we have something preventing that from happening [19:12:59] (03CR) 10RobH: [C: 031] adding new shell user groovier [puppet] - 10https://gerrit.wikimedia.org/r/401584 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [19:13:18] (03CR) 10Cmjohnson: [C: 032] adding new shell user groovier [puppet] - 10https://gerrit.wikimedia.org/r/401584 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [19:14:13] the commit-msg hook used to be kinda buggy and introduce said newlines for you. I think that's fixed in newer versions [19:14:27] greg-g: That's fine, I was already moving my patch, you gave me a chance to test the new two-column edit-conflict ;) [19:14:36] So if you had: "Foo\n\nBug: T123" it would add a newline before Change-Id: I..... [19:14:38] T123: Turn on "diffusion.allow-http-auth" - https://phabricator.wikimedia.org/T123 [19:14:39] eddiegp: you're welcome :P [19:14:43] Oh shut up stashbot [19:15:44] !log demon@tin Finished scap: wmf.15 bootstrap (duration: 34m 55s) [19:15:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:16:02] no_justification: im cooking dinner, gimmie a ping before you drive the train :) [19:18:14] (03PS1) 10Cmjohnson: Adding user groovier to stat1006 [puppet] - 10https://gerrit.wikimedia.org/r/401586 (https://phabricator.wikimedia.org/T181952) [19:19:44] (03CR) 10RobH: [C: 031] Adding user groovier to stat1006 [puppet] - 10https://gerrit.wikimedia.org/r/401586 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [19:23:49] (03PS2) 10Cmjohnson: Adding user groovier to stat1006 [puppet] - 10https://gerrit.wikimedia.org/r/401586 (https://phabricator.wikimedia.org/T181952) [19:25:48] addshore: I did testwiki only for bootstrapping, will start the rest of group0 in ~35m [19:25:55] (time to walk the dog now) [19:26:03] (03CR) 10Cmjohnson: [C: 032] Adding user groovier to stat1006 [puppet] - 10https://gerrit.wikimedia.org/r/401586 (https://phabricator.wikimedia.org/T181952) (owner: 10Cmjohnson) [19:26:21] (03PS3) 10Cmjohnson: Adding user groovier to stat1006 [puppet] - 10https://gerrit.wikimedia.org/r/401586 (https://phabricator.wikimedia.org/T181952) [19:28:26] (03CR) 10Cmjohnson: [C: 031] remove uranium.wikimedia.org, v4 + v6 [dns] - 10https://gerrit.wikimedia.org/r/399125 (https://phabricator.wikimedia.org/T183209) (owner: 10Dzahn) [19:28:57] (03CR) 10Cmjohnson: [C: 031] decom: remove uranium from site,DHCP,netboot [puppet] - 10https://gerrit.wikimedia.org/r/399684 (https://phabricator.wikimedia.org/T183209) (owner: 10Dzahn) [19:31:36] (03PS1) 10Andrew Bogott: bootstrap: use generic 'puppet' puppetmaster name on first boot [puppet] - 10https://gerrit.wikimedia.org/r/401589 (https://phabricator.wikimedia.org/T181375) [19:31:57] (03PS2) 10Dzahn: ntp: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400247 [19:32:19] mutante: o/ regarding https://phabricator.wikimedia.org/T183916, I'll reply properly to why we're not going with a Wikipage, but for now I had a couple of questions for you if you don't mind. [19:32:30] (03CR) 10Dzahn: [C: 032] "http://puppet-compiler.wmflabs.org/9507/" [puppet] - 10https://gerrit.wikimedia.org/r/400247 (owner: 10Dzahn) [19:32:33] (03PS1) 10Urbanecm: Enable wgKartographerStaticMapframe for all projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401590 (https://phabricator.wikimedia.org/T183981) [19:32:58] mutante: Can you share any links to how to get these static files to productions? Any step-by-step instructions? [19:33:17] mutante: Also, can the code live on github, or do I need to port it to gerrit? [19:33:56] (03PS2) 10Andrew Bogott: bootstrap: use generic 'puppet' puppetmaster name on first boot [puppet] - 10https://gerrit.wikimedia.org/r/401589 (https://phabricator.wikimedia.org/T181375) [19:35:42] (03CR) 10Andrew Bogott: [C: 032] bootstrap: use generic 'puppet' puppetmaster name on first boot [puppet] - 10https://gerrit.wikimedia.org/r/401589 (https://phabricator.wikimedia.org/T181375) (owner: 10Andrew Bogott) [19:37:12] bmansurov: i dont think we have the step-by-step instructions yet, but the process is: copy one of the the existing "class profile::microsites::*" classes and add that on role(webserver_misc_static). the puppet class will then git::clone from the content repo. that should be on Gerrit, yes. please move it there. then if we set it to "ensure => latest" puppet will pull new changes on each [19:37:18] run and non-ops people can have +2 in the content repo, so they dont need ops/puppet changes for just content changes [19:37:46] it's the simple way to deploy without the full deployment system. good enough for simple static sites [19:38:02] mutante: thanks! [19:38:14] maybe you can just take care of the content repo, like request one [19:38:17] and i can do the puppet part [19:38:43] mutante: that would be awesome, I'll create a request for a new gerrit repo. [19:38:53] sounds good [19:39:25] (03PS1) 10Urbanecm: Update logo for chrwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401593 (https://phabricator.wikimedia.org/T180553) [19:42:05] mutante: done! Can you also ping me on your puppet patch? I'd appreciate it. [19:42:31] (03PS2) 10Urbanecm: Enable wgKartographerStaticMapframe for lvwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401590 (https://phabricator.wikimedia.org/T183981) [19:42:48] 10Operations, 10Cloud-VPS, 10cloud-services-team: dumps project is using 2T of 5T available NFS misc storage - https://phabricator.wikimedia.org/T183971#3868886 (10madhuvishy) [19:43:05] bmansurov: yes, in a little while, multi-tasking a little [19:43:20] sounds good, no rush [19:43:24] 10Operations, 10Cloud-VPS, 10cloud-services-team: 2018-01-02: labstore Tools and Misc share very full - https://phabricator.wikimedia.org/T183920#3868889 (10madhuvishy) [19:49:24] (03PS2) 10Urbanecm: Update logo for chrwiki, add the HD version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401593 (https://phabricator.wikimedia.org/T180553) [19:49:41] (03PS1) 10Cmjohnson: fixing indentation issue for groovier [puppet] - 10https://gerrit.wikimedia.org/r/401596 [19:50:54] (03CR) 10jerkins-bot: [V: 04-1] Update logo for chrwiki, add the HD version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401593 (https://phabricator.wikimedia.org/T180553) (owner: 10Urbanecm) [19:50:57] (03CR) 10Cmjohnson: [C: 032] fixing indentation issue for groovier [puppet] - 10https://gerrit.wikimedia.org/r/401596 (owner: 10Cmjohnson) [19:51:02] (03PS2) 10Cmjohnson: fixing indentation issue for groovier [puppet] - 10https://gerrit.wikimedia.org/r/401596 [19:51:05] (03CR) 10Cmjohnson: [V: 032 C: 032] fixing indentation issue for groovier [puppet] - 10https://gerrit.wikimedia.org/r/401596 (owner: 10Cmjohnson) [19:54:56] bblack: yt? [19:55:04] bblack: or wait maybe there but on vacation [19:55:57] 10Operations, 10Ops-Access-Requests, 10AICaptcha, 10WMF-NDA-Requests, 10Patch-For-Review: Requesting access to EventLogging data for Vinitha - https://phabricator.wikimedia.org/T181952#3868958 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson Your user exists on stat1006 now and expires on 31/3/2018 /... [20:00:04] no_justification: That opportune time is upon us again. Time for a MediaWiki train deploy. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180102T2000). [20:00:04] No GERRIT patches in the queue for this window AFAICS. [20:00:53] bmansurov: what project/repo name did you request? [20:01:04] mutante: research/landing-page [20:02:40] (03PS1) 10Dzahn: microsites: create research.wikimedia.org static page [puppet] - 10https://gerrit.wikimedia.org/r/401597 (https://phabricator.wikimedia.org/T183916) [20:03:19] (03CR) 10jerkins-bot: [V: 04-1] microsites: create research.wikimedia.org static page [puppet] - 10https://gerrit.wikimedia.org/r/401597 (https://phabricator.wikimedia.org/T183916) (owner: 10Dzahn) [20:05:12] (03PS1) 10Ottomata: Use key from apt-key list --with-colons for confluent 4.0 reprepro updates [puppet] - 10https://gerrit.wikimedia.org/r/401598 [20:06:12] (03CR) 10Ottomata: [C: 032] Use key from apt-key list --with-colons for confluent 4.0 reprepro updates [puppet] - 10https://gerrit.wikimedia.org/r/401598 (owner: 10Ottomata) [20:06:41] the -1 can not be fixed currently, in that specific case [20:06:44] (03PS2) 10Dzahn: microsites: create research.wikimedia.org static page [puppet] - 10https://gerrit.wikimedia.org/r/401597 (https://phabricator.wikimedia.org/T183916) [20:07:03] bmansurov: ^ that's it basically [20:07:09] plus DNS [20:07:11] (03CR) 10jerkins-bot: [V: 04-1] microsites: create research.wikimedia.org static page [puppet] - 10https://gerrit.wikimedia.org/r/401597 (https://phabricator.wikimedia.org/T183916) (owner: 10Dzahn) [20:07:34] the -1 is for including ::apache defines but that will be replaced all at once [20:07:45] mutante: so once the gerrit repo is created, the above patch pull it to production? [20:07:58] mutante: what do we need to do in regards to DNS? [20:08:44] bmansurov: yes, puppet will git clone. we need to add the research name and point it to the caching cluster [20:09:13] mutante: do I need to create another task for that or is https://phabricator.wikimedia.org/T183916 enough? [20:09:36] 1 ticket is enough, it's just 2 gerrit changes because it's another repo. coming up [20:09:47] mutante: ok, got it [20:11:23] (03PS1) 10Dzahn: add research.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/401603 (https://phabricator.wikimedia.org/T183916) [20:14:19] (03PS1) 10Jgreen: add mx records for civicrm.wikimedia.org pointing to production mx's [dns] - 10https://gerrit.wikimedia.org/r/401604 [20:14:52] mutante: besides security review, do I need any other team's approval to get the code to production? [20:15:17] bmansurov: ops and security is all, afaict [20:15:30] mutante: ok, thanks! [20:15:37] yw [20:17:56] mutante: oh one more thing ;) are webrequest logs enabled for this subdomain automatically, or should I request to enable it too? [20:18:20] (03PS1) 10Ottomata: Revert back to correct confluent VerifyRelease key in updates [puppet] - 10https://gerrit.wikimedia.org/r/401606 [20:19:06] bmansurov: well, there is the CustomLog line in https://gerrit.wikimedia.org/r/#/c/401597/1/modules/profile/templates/research/apache-research.wikimedia.org.erb but there is nothing that lets you see that in a web interface [20:20:07] mutante: but I can get the data via hive right? [20:20:25] bmansurov: no, i don't think hive has any of that [20:20:29] well, i dont know [20:20:50] but i dont think so [20:20:57] hmmm [20:20:59] mutante: ok, i'll ask around. thanks [20:21:01] what cache cluster serves this [20:21:02] ? [20:21:03] misc, right? [20:21:05] "misc" [20:21:20] ya, it'llb e in the webrequest_source='misc' hive partition in the webrequest table [20:21:22] bmansurov: ^ [20:21:27] it's not actually in existence yet, but it would be misc, yea [20:21:33] ah :) [20:21:37] ottomata, cool! thanks [20:23:26] (03CR) 10Ottomata: [C: 032] Revert back to correct confluent VerifyRelease key in updates [puppet] - 10https://gerrit.wikimedia.org/r/401606 (owner: 10Ottomata) [20:33:20] o/ [20:34:55] (03CR) 10Dzahn: [C: 031] "would save a lot of iptables lines and monitoring hosts already alllowed everything by default" [puppet] - 10https://gerrit.wikimedia.org/r/376024 (owner: 10Giuseppe Lavagetto) [20:35:31] (03PS2) 10Dzahn: decom: remove uranium from site,DHCP,netboot [puppet] - 10https://gerrit.wikimedia.org/r/399684 (https://phabricator.wikimedia.org/T183209) [20:43:03] addshore: Ready? [20:43:16] no_justification: just thinking, would it be possible to create some sort of .15-mcr branch ? [20:43:17] no_justification: yup! [20:43:23] * addshore opens some tabs [20:43:56] Um, a branch...? [20:43:58] For....? [20:44:50] so that we could try the mcr changes on just a test wiki at some point this week, instead of on all of group 0 [20:45:04] Ah. Maybe [20:45:07] avoid having to try and write in a switch [20:45:09] Let's get through this first :) [20:45:11] yup ;) [20:45:19] (03CR) 10Chad: [C: 032] group0 to wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401559 (owner: 10Chad) [20:45:26] sorry, I was writing that message before you asked if I was ready ;) [20:46:50] (03Merged) 10jenkins-bot: group0 to wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401559 (owner: 10Chad) [20:47:19] (03CR) 10jenkins-bot: group0 to wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401559 (owner: 10Chad) [20:48:33] !log restarting kafka-jumbo brokers for version 1.0 upgrade [20:48:35] (03PS3) 10Dzahn: librenms: convert role to profile, variables to params [puppet] - 10https://gerrit.wikimedia.org/r/399966 [20:48:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:49:00] no_justification: can you deploy https://gerrit.wikimedia.org/r/#/c/401610/ ? [20:49:25] Can it wait until after I do group0? [20:49:51] preferably before :) [20:49:56] Mehhhhhh, ok [20:50:22] Also: https://gerrit.wikimedia.org/r/#/q/I24a3380650febd09a410a717425747ac5a60d162 is ugly :( [20:50:26] https://gerrit.wikimedia.org/r/#/c/396546/ is the proper fix, but no one got around to merging that one yet [20:51:22] (03CR) 10Debt: [C: 031] Enable wgKartographerStaticMapframe for lvwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401590 (https://phabricator.wikimedia.org/T183981) (owner: 10Urbanecm) [20:55:20] (03PS1) 10Smalyshev: Add configuration deboosting scientific articles [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401612 (https://phabricator.wikimedia.org/T183510) [20:56:26] (03CR) 10Sjoerddebruin: [C: 031] "Consensus exists for this change." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401612 (https://phabricator.wikimedia.org/T183510) (owner: 10Smalyshev) [20:58:16] (03CR) 10Dzahn: "needs https://phabricator.wikimedia.org/T183982" [puppet] - 10https://gerrit.wikimedia.org/r/401597 (https://phabricator.wikimedia.org/T183916) (owner: 10Dzahn) [20:59:36] !log demon@tin Synchronized php-1.31.0-wmf.15/includes/Setup.php: Aaron made me do it (duration: 01m 04s) [20:59:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:59:53] AaronSchulz: ^ [21:00:54] !log demon@tin rebuilt and synchronized wikiversions files: group0 to wmf.15 [21:01:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:01:49] i see no explosions [21:02:41] * no_justification hides in bunker anyway [21:03:43] I see 6 "Fatal error: request has exceeded memory limit in /srv/mediawiki/php-1.31.0-wmf.12/includes/libs/rdbms/database/Database.php on line 1504" for group0 [21:03:56] starting at 20:58:30 [21:04:30] (03CR) 10Dzahn: [C: 031] "http://puppet-compiler.wmflabs.org/9508/" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/399966 (owner: 10Dzahn) [21:05:14] that was .12 anyway... [21:07:14] (03PS2) 10Dzahn: rancid: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/399968 [21:07:33] (03CR) 10Dzahn: rancid: convert role to profile (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/399968 (owner: 10Dzahn) [21:11:18] addshore: Sadly, with all the noise I basically disregard DB errors :( [21:12:36] (03PS3) 10Dzahn: tcpircbot: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400250 [21:12:41] (03CR) 10Dzahn: tcpircbot: convert role to profile (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/400250 (owner: 10Dzahn) [21:21:08] addshore: Oh, are the "WikibaseQuality\ConstraintReport\ConstraintCheck\Helper\ConstraintParameterException" exceptions known? [21:23:08] (03PS3) 10Zoranzoki21: Update logo for chrwiki, add the HD version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401593 (https://phabricator.wikimedia.org/T180553) (owner: 10Urbanecm) [21:40:52] no_justification: if they mention sparql I think I just filed a bug in the last hour about them [21:42:05] I tagged it with wikimedia-errors (only on a phone right now) [21:43:12] Mmk [21:43:38] And yeah, was sparql [21:44:03] 10Operations, 10Cloud-VPS, 10cloud-services-team: templatetiger is using 827G of 8T available tools nfs storage - https://phabricator.wikimedia.org/T183954#3869509 (10Kolossos) Done. [21:57:22] (03PS20) 10Chad: Add wikidata and mediawiki.org to $wgLocalVirtualHosts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/392999 (https://phabricator.wikimedia.org/T117302) (owner: 10TerraCodes) [21:58:14] (03CR) 10EBernhardson: [C: 04-1] Add configuration deboosting scientific articles (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401612 (https://phabricator.wikimedia.org/T183510) (owner: 10Smalyshev) [21:58:18] (03PS1) 10Ottomata: Dont' expand wildcards in kafka acls command [puppet] - 10https://gerrit.wikimedia.org/r/401621 (https://phabricator.wikimedia.org/T167304) [22:00:32] (03CR) 10Ottomata: [C: 032] Dont' expand wildcards in kafka acls command [puppet] - 10https://gerrit.wikimedia.org/r/401621 (https://phabricator.wikimedia.org/T167304) (owner: 10Ottomata) [22:00:55] (03PS34) 10TerraCodes: $wmf* -> $wmg* [mediawiki-config] - 10https://gerrit.wikimedia.org/r/392184 (https://phabricator.wikimedia.org/T45956) [22:04:26] !log upgrading trusty puppet agents to puppet 4 [22:04:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:10:28] no_justification: thanks [22:11:17] Question: could the CAS thing be an ERROR or does it need to be an exception? [22:15:42] * addshore heads to bed [22:34:59] (03CR) 10Dzahn: "http://puppet-compiler.wmflabs.org/9509/netmon1002.wikimedia.org/" [puppet] - 10https://gerrit.wikimedia.org/r/399968 (owner: 10Dzahn) [22:35:05] (03CR) 10Dzahn: [C: 031] rancid: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/399968 (owner: 10Dzahn) [22:37:37] (03CR) 10Dzahn: [C: 031] "http://puppet-compiler.wmflabs.org/9510/rutherfordium.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/400245 (owner: 10Dzahn) [22:38:40] 10Operations, 10Puppet: Puppet - Error: /Stage[main]/Apparmor/Service[apparmor]: Provider init is not functional on this host - https://phabricator.wikimedia.org/T184017#3869845 (10herron) p:05Triage>03Normal [22:39:15] 10Operations, 10Puppet: Trusty puppet 4 approach - https://phabricator.wikimedia.org/T182894#3869859 (10herron) Thanks @MoritzMuehlenhoff! New packages have been built following this guidance, uploaded to apt.wikimedia.org and trusty hosts have been upgraded to the puppet 4 agent today. Once T184017 is resol... [22:42:34] (03CR) 10Dzahn: "http://puppet-compiler.wmflabs.org/9512/einsteinium.wikimedia.org/" [puppet] - 10https://gerrit.wikimedia.org/r/400250 (owner: 10Dzahn) [23:13:47] 10Operations, 10ops-codfw: mw2251 failed memory dimm - https://phabricator.wikimedia.org/T181263#3869970 (10Papaul) Dear Tshibamba, Papaul, Your dispatch shipped on 1/2/2018 2:55 PM Dispatch Number: 342210423 Work Order Number: SR958796798 [23:16:32] (03PS2) 10Smalyshev: Add configuration deboosting scientific articles [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401612 (https://phabricator.wikimedia.org/T183510) [23:16:55] (03CR) 10Smalyshev: Add configuration deboosting scientific articles (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/401612 (https://phabricator.wikimedia.org/T183510) (owner: 10Smalyshev) [23:39:42] (03PS1) 10Andrew Bogott: bootstrapvz: remove ldap setup from firstboot script [puppet] - 10https://gerrit.wikimedia.org/r/401633 (https://phabricator.wikimedia.org/T181375)