[00:10:46] 10Operations, 10TechCom-RFC, 10Traffic, 10Patch-For-Review, and 3 others: Harmonise the identification of requests across our stack - https://phabricator.wikimedia.org/T201409 (10Tgr) One issue brought up in {T113817} was that we might want to avoid connecting too many things for privacy reasons (specifica... [01:20:38] 10Operations, 10Performance-Team, 10Traffic, 10Wikimedia-General-or-Unknown, and 2 others: Search engines continue to link to JS-redirect destination after Wikipedia copyright protest - https://phabricator.wikimedia.org/T199252 (10Tbayer) >>! In T199252#4595021, @Imarlier wrote: > @Nemo_bis Regarding your... [02:37:27] !log l10nupdate@deploy1001 scap sync-l10n completed (1.32.0-wmf.22) (duration: 16m 02s) [02:37:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:48:16] !log l10nupdate@deploy1001 ResourceLoader cache refresh completed at Mon Sep 24 02:48:16 UTC 2018 (duration 10m 49s) [02:48:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:29:15] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 922.35 seconds [03:29:50] Q: For deploying, we need to use deploy1001.codfw.wmnet during DC switch, right? [03:38:30] deploy2001. Ah. [03:40:08] Nope. Big fat warning there. I'll wait someone to reply. [03:51:05] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 279.63 seconds [04:07:04] kart_: based on https://wikitech.wikimedia.org/wiki/Server_Admin_Log everyone has been deploying from 1001/eqiad [04:07:24] legoktm: just saw that. Thanks. [04:10:54] !log kartik@deploy1001 Started deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) [04:11:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:11:08] T202716: CX2: Infoboxes missing in source article - https://phabricator.wikimedia.org/T202716 [04:11:09] T203947: Implement a caching mechanism for API requests - https://phabricator.wikimedia.org/T203947 [04:11:10] T203780: Rewrite the section-wrap logic - https://phabricator.wikimedia.org/T203780 [04:11:10] T203551: Api requests are not cached - https://phabricator.wikimedia.org/T203551 [04:15:13] !log kartik@deploy1001 Finished deploy [cxserver/deploy@3e2d668]: Update cxserver to d913793 (T203551, T203780, T202716, T203947) (duration: 04m 20s) [04:15:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:56:15] PROBLEM - Nginx local proxy to apache on mw2198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:57:14] RECOVERY - Nginx local proxy to apache on mw2198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 0.176 second response time [05:21:41] (03PS1) 10Marostegui: db-codfw.php: Depool db2088:3311 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462338 [05:23:47] (03CR) 10Marostegui: [C: 032] db-codfw.php: Depool db2088:3311 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462338 (owner: 10Marostegui) [05:25:36] (03Merged) 10jenkins-bot: db-codfw.php: Depool db2088:3311 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462338 (owner: 10Marostegui) [05:27:12] !log marostegui@deploy1001 Synchronized wmf-config/db-codfw.php: Depool db2088:3311 (duration: 00m 50s) [05:27:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:28:41] !log Deploy schema change on db2088:3311 - T203709 [05:28:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:28:48] T203709: Schema change for adding indexes of ct_tag_id - https://phabricator.wikimedia.org/T203709 [05:35:01] (03CR) 10jenkins-bot: db-codfw.php: Depool db2088:3311 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462338 (owner: 10Marostegui) [05:42:05] (03PS1) 10Marostegui: Revert "db-codfw.php: Depool db2088:3311" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462339 [05:43:05] RECOVERY - Check systemd state on ms-be1037 is OK: OK - running: The system is fully operational [05:45:37] (03CR) 10Marostegui: [C: 032] Revert "db-codfw.php: Depool db2088:3311" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462339 (owner: 10Marostegui) [05:47:11] (03Merged) 10jenkins-bot: Revert "db-codfw.php: Depool db2088:3311" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462339 (owner: 10Marostegui) [05:47:33] (03PS1) 10Mholloway: Remove unused 'style' var from Kartotherian module [puppet] - 10https://gerrit.wikimedia.org/r/462340 (https://phabricator.wikimedia.org/T195328) [05:48:21] !log marostegui@deploy1001 Synchronized wmf-config/db-codfw.php: Repool db2088:3311 (duration: 00m 50s) [05:48:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:50:36] (03PS4) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert login.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461866 (https://phabricator.wikimedia.org/T196968) [05:53:46] 10Operations, 10Maps, 10Maps-Sprint, 10Reading-Infrastructure-Team-Backlog, 10Patch-For-Review: migrate maps servers to stretch with the current style - https://phabricator.wikimedia.org/T198622 (10Mholloway) a:03Gehel [05:54:09] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki::web::prod_sites: convert login.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461866 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [05:56:34] (03CR) 10jenkins-bot: Revert "db-codfw.php: Depool db2088:3311" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462339 (owner: 10Marostegui) [06:06:27] (03CR) 10Giuseppe Lavagetto: "https://puppet-compiler.wmflabs.org/compiler1002/12542/mw1261.eqiad.wmnet/ no real changes; this should be GTG." [puppet] - 10https://gerrit.wikimedia.org/r/461867 (owner: 10Giuseppe Lavagetto) [06:28:35] PROBLEM - puppet last run on mw1314 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:29:55] PROBLEM - puppet last run on cp1090 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/varnishmtail-backend/varnishbackend.mtail] [06:33:34] PROBLEM - puppet last run on bast3002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/profile.d/bash_autologout.sh] [06:41:10] 10Operations, 10ops-codfw: MCE errors on mw2181 / temperature warnings - https://phabricator.wikimedia.org/T205240 (10MoritzMuehlenhoff) [06:41:29] ACKNOWLEDGEMENT - Memory correctable errors -EDAC- on mw2181 is CRITICAL: 8.001 ge 4 Muehlenhoff T205240 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=mw2181&var-datasource=codfw%2520prometheus%252Fops [06:43:54] RECOVERY - puppet last run on mw1314 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:48:44] RECOVERY - puppet last run on bast3002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:51:28] 10Operations, 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash, 10Release-Engineering-Team (Watching / External): logstash-beta.wmflab throws multiple "Error: Could not locate that visualization" - https://phabricator.wikimedia.org/T204845 (10fgiunchedi) a:05fgiunchedi>03None Thanks @greg that's ind... [06:52:28] (03PS1) 10Marostegui: db-codfw.php: Depool db2062 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462344 [06:52:42] 10Operations, 10TechCom-RFC, 10Traffic, 10Patch-For-Review, and 3 others: Harmonise the identification of requests across our stack - https://phabricator.wikimedia.org/T201409 (10Joe) As long as a request ID doesn't get associated with PII-worthy data, I don't see it being a privacy issue. Since it's a per... [06:54:45] !log volans@deploy1001 Started deploy [netbox/deploy@5e70423]: Cherry pick of custom fields fix [06:54:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:54:51] !log volans@deploy1001 Finished deploy [netbox/deploy@5e70423]: Cherry pick of custom fields fix (duration: 00m 05s) [06:54:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:55:24] (03CR) 10Marostegui: [C: 032] db-codfw.php: Depool db2062 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462344 (owner: 10Marostegui) [06:57:07] !log repair sdn on ms-be1041 - T199198 [06:57:09] (03Merged) 10jenkins-bot: db-codfw.php: Depool db2062 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462344 (owner: 10Marostegui) [06:57:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:57:16] T199198: Some swift filesystems reporting negative disk usage - https://phabricator.wikimedia.org/T199198 [06:57:47] !log repair sde on ms-be2041 - T199198 [06:57:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:58:31] !log marostegui@deploy1001 Synchronized wmf-config/db-codfw.php: Depool db2062 (duration: 00m 50s) [06:58:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:00:25] RECOVERY - puppet last run on cp1090 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [07:00:44] PROBLEM - swift-object-server on ms-be2041 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-server [07:00:45] PROBLEM - swift-object-auditor on ms-be2041 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [07:00:45] PROBLEM - swift-object-updater on ms-be2041 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-updater [07:00:45] PROBLEM - swift-object-replicator on ms-be2041 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-replicator [07:00:53] (03CR) 10Urbanecm: [C: 031] "Ahh, missed there's no + in front of "wikitech" (perhaps it should be?)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461240 (owner: 10Gergő Tisza) [07:02:56] (03CR) 10jenkins-bot: db-codfw.php: Depool db2062 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462344 (owner: 10Marostegui) [07:04:20] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki::web::vhost: Allow if using canonical name or not [puppet] - 10https://gerrit.wikimedia.org/r/461867 (owner: 10Giuseppe Lavagetto) [07:04:35] (03PS4) 10Giuseppe Lavagetto: mediawiki::web::vhost: Allow if using canonical name or not [puppet] - 10https://gerrit.wikimedia.org/r/461867 [07:05:42] !log wiping netbox DB to re-import it cleanly from racktables - T199083 [07:05:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:05:49] T199083: Migrate the hardware inventory from Racktables to Netbox - https://phabricator.wikimedia.org/T199083 [07:08:45] RECOVERY - Filesystem available is greater than filesystem size on ms-be2041 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=ms-be2041&var-datasource=codfw%2520prometheus%252Fops [07:09:01] (03PS1) 10Smalyshev: Add phrase rescoring to config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462347 (https://phabricator.wikimedia.org/T163642) [07:09:05] !log installing libarchive-zip-perl security updates [07:09:08] (03Abandoned) 10Volans: Custom fields: add label field [software/netbox] - 10https://gerrit.wikimedia.org/r/462273 (https://phabricator.wikimedia.org/T199083) (owner: 10Volans) [07:09:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:10:38] (03CR) 10jerkins-bot: [V: 04-1] Add phrase rescoring to config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462347 (https://phabricator.wikimedia.org/T163642) (owner: 10Smalyshev) [07:13:47] (03PS5) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert foundation.w.o to use vhost [puppet] - 10https://gerrit.wikimedia.org/r/461394 (https://phabricator.wikimedia.org/T196968) [07:13:49] (03PS6) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert simple wikis in remnant.conf [puppet] - 10https://gerrit.wikimedia.org/r/452323 (https://phabricator.wikimedia.org/T196968) [07:13:51] (03PS4) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert usability wiki [puppet] - 10https://gerrit.wikimedia.org/r/452635 (https://phabricator.wikimedia.org/T196968) [07:13:53] (03PS4) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: migrate wikispecies [puppet] - 10https://gerrit.wikimedia.org/r/452636 (https://phabricator.wikimedia.org/T196968) [07:13:55] (03PS3) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert commons.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461395 (https://phabricator.wikimedia.org/T196968) [07:13:57] (03PS3) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert meta.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461396 (https://phabricator.wikimedia.org/T196968) [07:13:59] (03PS3) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikisource.org [puppet] - 10https://gerrit.wikimedia.org/r/461397 (https://phabricator.wikimedia.org/T196968) [07:14:01] (03PS2) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikivoyage.org [puppet] - 10https://gerrit.wikimedia.org/r/461976 (https://phabricator.wikimedia.org/T196968) [07:15:54] (03PS2) 10Smalyshev: Add phrase rescoring to config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462347 (https://phabricator.wikimedia.org/T163642) [07:17:01] (03PS6) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert foundation.w.o to use vhost [puppet] - 10https://gerrit.wikimedia.org/r/461394 (https://phabricator.wikimedia.org/T196968) [07:17:03] (03PS7) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert simple wikis in remnant.conf [puppet] - 10https://gerrit.wikimedia.org/r/452323 (https://phabricator.wikimedia.org/T196968) [07:17:05] (03PS5) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert usability wiki [puppet] - 10https://gerrit.wikimedia.org/r/452635 (https://phabricator.wikimedia.org/T196968) [07:17:07] (03PS5) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: migrate wikispecies [puppet] - 10https://gerrit.wikimedia.org/r/452636 (https://phabricator.wikimedia.org/T196968) [07:17:09] <_joe_> sorry for the noise [07:17:10] (03PS4) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert commons.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461395 (https://phabricator.wikimedia.org/T196968) [07:17:12] (03PS4) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert meta.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461396 (https://phabricator.wikimedia.org/T196968) [07:17:14] (03PS4) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikisource.org [puppet] - 10https://gerrit.wikimedia.org/r/461397 (https://phabricator.wikimedia.org/T196968) [07:17:16] (03PS3) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikivoyage.org [puppet] - 10https://gerrit.wikimedia.org/r/461976 (https://phabricator.wikimedia.org/T196968) [07:17:47] (03PS1) 10Smalyshev: Enable phrase search config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462351 (https://phabricator.wikimedia.org/T163642) [07:18:16] (03PS2) 10Elukey: Guard analytics cron jobs to ease the hw refresh of an1003 [puppet] - 10https://gerrit.wikimedia.org/r/461988 (https://phabricator.wikimedia.org/T203635) [07:23:49] (03CR) 10jerkins-bot: [V: 04-1] Enable phrase search config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462351 (https://phabricator.wikimedia.org/T163642) (owner: 10Smalyshev) [07:25:10] (03PS1) 10Marostegui: Revert "db-codfw.php: Depool db2062" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462391 [07:25:49] (03CR) 10Giuseppe Lavagetto: [C: 031] "Diffs in the vhost:" [puppet] - 10https://gerrit.wikimedia.org/r/461394 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [07:25:57] !log manually running l10n update (T205238) [07:26:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:26:25] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [07:27:32] (03CR) 10Elukey: [C: 032] Guard analytics cron jobs to ease the hw refresh of an1003 [puppet] - 10https://gerrit.wikimedia.org/r/461988 (https://phabricator.wikimedia.org/T203635) (owner: 10Elukey) [07:28:24] (03CR) 10Marostegui: [C: 032] Revert "db-codfw.php: Depool db2062" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462391 (owner: 10Marostegui) [07:30:08] (03Merged) 10jenkins-bot: Revert "db-codfw.php: Depool db2062" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462391 (owner: 10Marostegui) [07:31:08] !log marostegui@deploy1001 Synchronized wmf-config/db-codfw.php: Repool db2062 (duration: 00m 46s) [07:31:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:35:58] (03CR) 10Muehlenhoff: [C: 031] "Looks good to me" [puppet] - 10https://gerrit.wikimedia.org/r/461394 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [07:39:35] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [07:42:47] (03CR) 10jenkins-bot: Revert "db-codfw.php: Depool db2062" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462391 (owner: 10Marostegui) [07:44:28] !log bawolff@deploy1001 scap sync-l10n completed (1.32.0-wmf.22) (duration: 07m 44s) [07:44:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:55:18] !log l10nupdate@deploy1001 ResourceLoader cache refresh completed at Mon Sep 24 07:55:18 UTC 2018 (duration 10m 50s) [07:55:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:01:25] RECOVERY - Filesystem available is greater than filesystem size on ms-be1041 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=ms-be1041&var-datasource=eqiad%2520prometheus%252Fops [08:01:50] (03CR) 10Muehlenhoff: [C: 031] "Looks good to me" [puppet] - 10https://gerrit.wikimedia.org/r/452323 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [08:13:16] (03CR) 10Muehlenhoff: [C: 031] "Looks good to me" [puppet] - 10https://gerrit.wikimedia.org/r/452635 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [08:14:40] (03PS1) 10Ladsgroup: ores: install hunspell-gl on ores nodes [puppet] - 10https://gerrit.wikimedia.org/r/462403 (https://phabricator.wikimedia.org/T201142) [08:15:00] 10Operations, 10Maps-Sprint, 10Maps (Tilerator): Log slow queries on postgresql / maps - https://phabricator.wikimedia.org/T204106 (10Mholloway) [08:15:17] 10Operations, 10Maps-Sprint, 10Discovery-Search (Current work), 10Maps (Tilerator): Log slow queries on postgresql / maps - https://phabricator.wikimedia.org/T204106 (10Mholloway) [08:19:24] (03CR) 10Muehlenhoff: [C: 031] "Looks good to me" [puppet] - 10https://gerrit.wikimedia.org/r/452636 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [08:22:41] (03CR) 10Filippo Giunchedi: "Which files were unreadable by icinga out of the box on /etc/nagios ? I'm asking because between /etc/nagios and /etc/icinga there's a who" [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) (owner: 10Cwhite) [08:31:07] (03CR) 10Filippo Giunchedi: [C: 032] Upgrade to 2.2 [debs/python-thumbor-wikimedia] - 10https://gerrit.wikimedia.org/r/461793 (https://phabricator.wikimedia.org/T20871) (owner: 10Gilles) [08:36:10] (03CR) 10Muehlenhoff: [C: 031] "Looks good to me" [puppet] - 10https://gerrit.wikimedia.org/r/461395 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [08:46:38] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki::web::prod_sites: convert foundation.w.o to use vhost [puppet] - 10https://gerrit.wikimedia.org/r/461394 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [08:46:50] (03PS7) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert foundation.w.o to use vhost [puppet] - 10https://gerrit.wikimedia.org/r/461394 (https://phabricator.wikimedia.org/T196968) [08:48:54] (03CR) 10Muehlenhoff: "The old conf had the rewrite rule for ShortUrl, we'll need to also set short_urls for the new scheme" [puppet] - 10https://gerrit.wikimedia.org/r/461396 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [08:50:48] (03CR) 10DCausse: [C: 031] Add phrase rescoring to config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462347 (https://phabricator.wikimedia.org/T163642) (owner: 10Smalyshev) [08:57:47] (03PS1) 10Jcrespo: mariadb: Setup 2 hosts for api on s5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462407 [09:00:04] (03CR) 10Marostegui: [C: 031] mariadb: Setup 2 hosts for api on s5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462407 (owner: 10Jcrespo) [09:03:52] !log stop and upgrade db1067 (s1 eqiad master)- it may create some temporary lag on s1 [09:03:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:04:50] 10Operations, 10ops-eqiad, 10DC-Ops: db1069 has errored disk in slot 7 - https://phabricator.wikimedia.org/T205253 (10Banyek) p:05Triage>03Normal [09:05:52] (03CR) 10Muehlenhoff: [C: 04-1] mediawiki::web::prod_sites: convert wikisource.org (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/461397 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [09:06:24] 10Operations, 10ops-eqiad, 10DC-Ops: db1069 has errored disk in slot 7 - https://phabricator.wikimedia.org/T205253 (10Banyek) Here's the important part ```PD: 1 Information Enclosure Device ID: 32 Slot Number: 7 Drive's position: DiskGroup: 0, Span: 3, Arm: 1 Enclosure position: 1 Device Id: 7 WWN: 5000C500... [09:07:17] 10Puppet, 10Cloud-Services, 10cloud-services-team (Kanban): Ban spam arriving to my tools email - https://phabricator.wikimedia.org/T202558 (10MarcoAurelio) [09:07:56] (03CR) 10Lucas Werkmeister (WMDE): "Not sure if we’ll also need a lexeme-truthy dump in the future…" [puppet] - 10https://gerrit.wikimedia.org/r/461862 (https://phabricator.wikimedia.org/T202830) (owner: 10Smalyshev) [09:09:06] 10Operations, 10ops-eqiad, 10DC-Ops: db1069 has errored disk in slot 7 - https://phabricator.wikimedia.org/T205253 (10Banyek) [09:12:24] ACKNOWLEDGEMENT - Device not healthy -SMART- on db1069 is CRITICAL: cluster=mysql device=megaraid,7 instance=db1069:9100 job=node site=eqiad Banyek Ticket T204462 is created https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=db1069&var-datasource=eqiad%2520prometheus%252Fops [09:16:03] 08Warning Alert for device asw-esams.mgmt.esams.wmnet - Sensor over limit [09:19:18] (03CR) 10Giuseppe Lavagetto: [C: 032] "https://puppet-compiler.wmflabs.org/compiler1002/12560/mw1261.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/452323 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [09:19:39] (03PS8) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert simple wikis in remnant.conf [puppet] - 10https://gerrit.wikimedia.org/r/452323 (https://phabricator.wikimedia.org/T196968) [09:23:23] (03CR) 10Muehlenhoff: mediawiki::web::prod_sites: convert wikivoyage.org (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/461976 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [09:30:06] !log upgrade / roll restart thumbor in eqiad / codfw - T20871 T198370 [09:30:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:30:19] T20871: Include at least some EXIF metadata in resized pictures - https://phabricator.wikimedia.org/T20871 [09:30:20] T198370: Transparent background renders as white in PNG thumbnails - https://phabricator.wikimedia.org/T198370 [09:41:17] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki::web::prod_sites: convert usability wiki [puppet] - 10https://gerrit.wikimedia.org/r/452635 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [09:41:34] (03PS6) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert usability wiki [puppet] - 10https://gerrit.wikimedia.org/r/452635 (https://phabricator.wikimedia.org/T196968) [09:42:05] !log stop and upgrade db1066 (s2 eqiad master)- it may create some temporary lag on s2 [09:42:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:42:35] RECOVERY - swift-object-server on ms-be2041 is OK: PROCS OK: 101 processes with regex args ^/usr/bin/python /usr/bin/swift-object-server [09:42:35] RECOVERY - swift-object-auditor on ms-be2041 is OK: PROCS OK: 3 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [09:42:44] RECOVERY - swift-object-updater on ms-be2041 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-object-updater [09:42:44] RECOVERY - swift-object-replicator on ms-be2041 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-object-replicator [09:57:06] (03CR) 10Muehlenhoff: mediawiki: move php to a profile, use the php class (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/453093 (https://phabricator.wikimedia.org/T201140) (owner: 10Giuseppe Lavagetto) [09:59:18] (03PS6) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: migrate wikispecies [puppet] - 10https://gerrit.wikimedia.org/r/452636 (https://phabricator.wikimedia.org/T196968) [10:02:30] 10Operations, 10ops-eqiad: Heating alerts / memory errors on mw1254 - https://phabricator.wikimedia.org/T204491 (10MoritzMuehlenhoff) 05Open>03Resolved a:03MoritzMuehlenhoff This error hasn't resurfaced, I'm closing the task. [10:04:42] 10Operations, 10ops-eqiad: Heating alerts on kafka1014 - https://phabricator.wikimedia.org/T204479 (10MoritzMuehlenhoff) a:03elukey [10:05:50] 10Operations, 10Puppet: Why doesn't profile::mediawiki::nutcracker create /var/run/nutcracker/ ? - https://phabricator.wikimedia.org/T204450 (10MoritzMuehlenhoff) The .sock file is created via systemd-tmpfiles, which is only read during boot, the socket will be created with the next restart [10:07:35] 10Operations, 10ops-codfw: ms-be2030 spontaneous reboot - https://phabricator.wikimedia.org/T204567 (10MoritzMuehlenhoff) p:05Triage>03Normal a:03Papaul [10:07:58] 10Operations, 10ops-codfw: MCE errors on mw2181 / temperature warnings - https://phabricator.wikimedia.org/T205240 (10MoritzMuehlenhoff) p:05Triage>03Normal [10:08:06] 10Operations, 10ops-eqiad: decommission thulium.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T203520 (10MoritzMuehlenhoff) p:05Triage>03Normal [10:10:56] 10Operations, 10Traffic: Update certspotter - https://phabricator.wikimedia.org/T204993 (10MoritzMuehlenhoff) Adding the Debian maintainer :-) This seems fixed in 0.9-1 so updating stretch-backports to 0.9 could fix this. [10:11:01] 10Operations, 10Traffic: Update certspotter - https://phabricator.wikimedia.org/T204993 (10MoritzMuehlenhoff) p:05Triage>03Normal [10:15:00] (03CR) 10ArielGlenn: [C: 031] "I double checked all the config settings and changes in the new catalog as compared to current production setup on a dumps host. All good." [puppet] - 10https://gerrit.wikimedia.org/r/453093 (https://phabricator.wikimedia.org/T201140) (owner: 10Giuseppe Lavagetto) [10:16:04] 08̶W̶a̶r̶n̶i̶n̶g Device asw-esams.mgmt.esams.wmnet recovered from Sensor over limit [10:18:17] <_joe_> apergos: thanks for the careful review :) [10:18:40] you're welcome. paranoia means that even if they are out to get you they don't get away with it as often :-P [10:19:55] <_joe_> more than paranoia, it's a big change with so many details one can miss it's good to have someone else do a thorough check [10:20:10] for sure [10:20:11] (03PS2) 10Arturo Borrero Gonzalez: cloud: update MediaWiki-Vagrant container start actions [puppet] - 10https://gerrit.wikimedia.org/r/462000 (owner: 10BryanDavis) [10:20:55] PROBLEM - HHVM rendering on mw2190 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:54] RECOVERY - HHVM rendering on mw2190 is OK: HTTP OK: HTTP/1.1 200 OK - 80620 bytes in 0.253 second response time [10:21:59] (03CR) 10Arturo Borrero Gonzalez: [C: 032] cloud: update MediaWiki-Vagrant container start actions [puppet] - 10https://gerrit.wikimedia.org/r/462000 (owner: 10BryanDavis) [10:22:19] (03CR) 10Giuseppe Lavagetto: "https://puppet-compiler.wmflabs.org/compiler1002/12566/mw1261.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/452636 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [10:22:32] (03PS7) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: migrate wikispecies [puppet] - 10https://gerrit.wikimedia.org/r/452636 (https://phabricator.wikimedia.org/T196968) [10:27:15] PROBLEM - MediaWiki memcached error rate on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [5000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [10:27:23] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki::web::prod_sites: migrate wikispecies [puppet] - 10https://gerrit.wikimedia.org/r/452636 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [10:28:11] (03CR) 10ArielGlenn: "This looks ok just by visual inspection. What does the mw-vagrant test environment show? :-) (You might need to add some lexemes by hand " (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/461862 (https://phabricator.wikimedia.org/T202830) (owner: 10Smalyshev) [10:29:24] RECOVERY - MediaWiki memcached error rate on graphite1001 is OK: OK: Less than 40.00% above the threshold [1000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [10:30:07] jan_drewniak: It is that lovely time of the day again! You are hereby commanded to deploy Wikimedia Portals Update. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180924T1030). [10:31:27] (03PS5) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert commons.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461395 (https://phabricator.wikimedia.org/T196968) [10:35:16] 10Operations, 10ops-eqiad: Heating alerts on kafka1014 - https://phabricator.wikimedia.org/T204479 (10elukey) @Cmjohnson it would be better to stop the host only for the time needed, so I can stop it before you are ready to apply the paste. Lemme know 10 mins beforehand and I'll shut it down. Thanks! [10:35:16] (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462416 (https://phabricator.wikimedia.org/T128546) [10:35:27] (03CR) 10jerkins-bot: [V: 04-1] Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462416 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:36:45] (03Abandoned) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462416 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:37:11] (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462417 (https://phabricator.wikimedia.org/T128546) [10:38:33] (03CR) 10Jdrewniak: [C: 032] Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462417 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:39:46] (03Merged) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462417 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:40:21] (03CR) 10Giuseppe Lavagetto: [C: 031] "https://puppet-compiler.wmflabs.org/compiler1002/12567/mw1261.eqiad.wmnet/ we forgot to expand includes here but I feel quite confident an" [puppet] - 10https://gerrit.wikimedia.org/r/461395 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [10:43:29] !log jdrewniak@deploy1001 Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:462417|Bumping portals to master (T128546)]] (duration: 00m 50s) [10:43:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:43:38] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [10:44:19] !log jdrewniak@deploy1001 Synchronized portals: Wikimedia Portals Update: [[gerrit:462417|Bumping portals to master (T128546)]] (duration: 00m 49s) [10:44:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:50:32] (03CR) 10Giuseppe Lavagetto: "> Patch Set 4:" [puppet] - 10https://gerrit.wikimedia.org/r/461396 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [10:50:58] (03CR) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikisource.org (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/461397 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [10:51:30] (03CR) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462417 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:55:16] (03PS2) 10Sbisson: Labs: rename wp10 to articlequality [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461734 (https://phabricator.wikimedia.org/T203080) [10:57:00] (03CR) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikivoyage.org (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/461976 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [11:00:04] addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: How many deployers does it take to do European Mid-day SWAT(Max 6 patches) deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180924T1100). [11:00:04] No GERRIT patches in the queue for this window AFAICS. [11:00:35] o/ [11:00:38] no patches, no swat [11:04:07] (03PS6) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert commons.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461395 (https://phabricator.wikimedia.org/T196968) [11:04:07] (03PS5) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert meta.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461396 (https://phabricator.wikimedia.org/T196968) [11:04:09] (03PS5) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikisource.org [puppet] - 10https://gerrit.wikimedia.org/r/461397 (https://phabricator.wikimedia.org/T196968) [11:04:11] (03PS4) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikivoyage.org [puppet] - 10https://gerrit.wikimedia.org/r/461976 (https://phabricator.wikimedia.org/T196968) [11:04:13] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikiversity.org [puppet] - 10https://gerrit.wikimedia.org/r/462424 (https://phabricator.wikimedia.org/T196968) [11:04:15] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert mediawiki.org [puppet] - 10https://gerrit.wikimedia.org/r/462425 (https://phabricator.wikimedia.org/T196968) [11:16:08] 10Operations, 10Release-Engineering-Team, 10Scap: mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild - https://phabricator.wikimedia.org/T203625 (10MoritzMuehlenhoff) Compared to the rest, mwdebug* are VMs, how large is the difference to the other servers you were seeing? [11:21:30] !log installing texlive-bin security updates [11:21:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:23:35] PROBLEM - HHVM rendering on mw2253 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:24:35] RECOVERY - HHVM rendering on mw2253 is OK: HTTP OK: HTTP/1.1 200 OK - 80621 bytes in 0.389 second response time [11:28:15] PROBLEM - puppet last run on mw2247 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[set debconf flag seen for wireshark-common/install-setuid] [11:29:55] PROBLEM - puppet last run on mw2229 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[set debconf flag seen for wireshark-common/install-setuid] [11:30:14] PROBLEM - puppet last run on mw1314 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[set debconf flag seen for wireshark-common/install-setuid] [11:30:15] PROBLEM - puppet last run on mw1283 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[set debconf flag seen for wireshark-common/install-setuid] [11:30:25] PROBLEM - puppet last run on mw2197 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[set debconf flag seen for wireshark-common/install-setuid] [11:30:45] PROBLEM - puppet last run on mw2241 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[set debconf flag seen for wireshark-common/install-setuid] [11:30:45] PROBLEM - puppet last run on mw2217 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[set debconf flag seen for wireshark-common/install-setuid] [11:30:46] PROBLEM - puppet last run on mw2166 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[set debconf flag seen for wireshark-common/install-setuid] [11:31:05] PROBLEM - puppet last run on mw2179 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[set debconf flag seen for wireshark-common/install-setuid] [11:32:15] PROBLEM - puppet last run on mw1303 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[set debconf flag seen for wireshark-common/install-setuid] [11:32:49] moritzm: is that you? ^^^ [11:33:21] checking [11:34:52] that's just puppet noise, the texlive updates are fairly big and take some time to get fully installed and during that time deb debconf db is locked for the puppet-triggered "debconf seen" run [11:35:03] should all recover soonish [11:35:07] ack [11:36:36] (03CR) 10Elukey: "Comparing 1:1 with current commons vhost and pcc one, I can't account the following bits:" [puppet] - 10https://gerrit.wikimedia.org/r/461395 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [11:37:15] RECOVERY - puppet last run on mw1303 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:39:36] !log reboot an-master100[1,2] as part of the pre-checks before the hadoop master daemons swap - T203635 [11:39:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:39:45] T203635: Replace the Analytics HDFS/Yarn masters (hardware refresh) - https://phabricator.wikimedia.org/T203635 [11:55:15] RECOVERY - puppet last run on mw2229 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [11:55:45] RECOVERY - puppet last run on mw2197 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:56:05] RECOVERY - puppet last run on mw2241 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:56:05] RECOVERY - puppet last run on mw2217 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:56:14] RECOVERY - puppet last run on mw2166 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:56:25] RECOVERY - puppet last run on mw2179 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:58:35] RECOVERY - puppet last run on mw2247 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [12:00:35] RECOVERY - puppet last run on mw1314 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [12:00:45] RECOVERY - puppet last run on mw1283 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [12:03:58] (03CR) 10Muehlenhoff: mediawiki::web::prod_sites: convert wikisource.org (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/461397 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [12:06:13] (03CR) 10Muehlenhoff: [C: 031] "Yeah, these are part of the vhosts standardisation" [puppet] - 10https://gerrit.wikimedia.org/r/461395 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [12:09:57] 10Operations, 10Citoid, 10Patch-For-Review, 10Services (watching), 10VisualEditor (Current work): Transition citoid to use Zotero's translation-server-v2 - https://phabricator.wikimedia.org/T197242 (10Mvolz) [12:10:26] (03CR) 10Muehlenhoff: [C: 031] "Looks good to me" [puppet] - 10https://gerrit.wikimedia.org/r/461396 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [12:10:49] 10Operations, 10Citoid, 10Services, 10Patch-For-Review, and 3 others: Deploy translation-server-v2 - https://phabricator.wikimedia.org/T201611 (10Mvolz) [12:11:51] 10Operations, 10Citoid, 10Patch-For-Review, 10Services (watching), 10VisualEditor (Current work): Transition citoid to use Zotero's translation-server-v2 - https://phabricator.wikimedia.org/T197242 (10Mvolz) >>! In T197242#4587749, @Sebastian_Berlin-WMSE wrote: > I noticed that a [[ https://gerrit.wikime... [12:15:50] (03CR) 10Banyek: [C: 031] mariadb: Setup 2 hosts for api on s5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462407 (owner: 10Jcrespo) [12:16:47] (03CR) 10Banyek: [C: 031] Enable cumin2001 as mysql maintenance client [puppet] - 10https://gerrit.wikimedia.org/r/460323 (https://phabricator.wikimedia.org/T177385) (owner: 10Muehlenhoff) [12:16:52] 10Operations, 10Mail: Mail relays needed for VMs in eqiad1 - https://phabricator.wikimedia.org/T205158 (10Krenair) [12:16:57] 10Operations, 10Mail: Mail relays needed for VMs in eqiad1 - https://phabricator.wikimedia.org/T205158 (10Krenair) Marking this as a blocker for eqiad1-r usage based on the comments at https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/462012/ [12:17:28] 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install mwmaint1002.eqiad.wmnet - https://phabricator.wikimedia.org/T201343 (10MoritzMuehlenhoff) Disk space on the root partition of mwmaint1002 is depleted, which results in failing puppet runs [12:17:52] jouncebot: next [12:17:52] In 4 hour(s) and 42 minute(s): Wikidata Query Service weekly deploy (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180924T1700) [12:28:00] 10Operations, 10Mail: Mail relays needed for VMs in eqiad1 - https://phabricator.wikimedia.org/T205158 (10Krenair) Am I the only one missing here why we can't just fix the firewall rule for the MX servers to allow the new range and do T41785 later? This doesn't seem in-scope for the eqiad1 migration. [12:30:34] (03CR) 10Muehlenhoff: [C: 031] "Looks good" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462278 (https://phabricator.wikimedia.org/T201346) (owner: 10Volans) [12:35:30] (03PS3) 10ArielGlenn: use iohandlers for recompressxml input and output [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/441485 [12:35:31] (03PS2) 10ArielGlenn: option to skip siteinfo header, mw footer for recompresing files [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/442774 [12:35:33] (03PS2) 10ArielGlenn: options for writeuptopageid to skip writing header or footer [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/442775 [12:36:26] (03CR) 10Muehlenhoff: [C: 031] Add cumin1001 IPs and PTRs [dns] - 10https://gerrit.wikimedia.org/r/462274 (https://phabricator.wikimedia.org/T201346) (owner: 10Volans) [12:45:25] (03PS2) 10ArielGlenn: Link to the wikitech page [dumps] - 10https://gerrit.wikimedia.org/r/347906 (owner: 10Awight) [12:46:02] (03PS1) 10Muehlenhoff: Remove account expiry date [puppet] - 10https://gerrit.wikimedia.org/r/462451 [12:46:58] (03CR) 10ArielGlenn: [C: 032] Link to the wikitech page [dumps] - 10https://gerrit.wikimedia.org/r/347906 (owner: 10Awight) [12:47:16] (03CR) 10Alex Monk: [C: 04-1] "doesn't do anything" [debs/prometheus-openstack-exporter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/461382 (owner: 10Arturo Borrero Gonzalez) [12:51:15] PROBLEM - MediaWiki memcached error rate on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [5000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [12:53:25] RECOVERY - MediaWiki memcached error rate on graphite1001 is OK: OK: Less than 40.00% above the threshold [1000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [12:56:17] (03PS2) 10Muehlenhoff: Remove account expiry date [puppet] - 10https://gerrit.wikimedia.org/r/462451 [12:57:36] (03CR) 10Muehlenhoff: [C: 032] Remove account expiry date [puppet] - 10https://gerrit.wikimedia.org/r/462451 (owner: 10Muehlenhoff) [12:59:55] PROBLEM - MediaWiki memcached error rate on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [5000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [13:00:11] (03PS1) 10Arturo Borrero Gonzalez: cloudvps: add prometheus-openstack-exporter [puppet] - 10https://gerrit.wikimedia.org/r/462455 (https://phabricator.wikimedia.org/T203177) [13:01:06] (03CR) 10jerkins-bot: [V: 04-1] cloudvps: add prometheus-openstack-exporter [puppet] - 10https://gerrit.wikimedia.org/r/462455 (https://phabricator.wikimedia.org/T203177) (owner: 10Arturo Borrero Gonzalez) [13:03:05] !log start isolating maps1004 for reimage to stretch - T195285 [13:03:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:04:24] RECOVERY - MediaWiki memcached error rate on graphite1001 is OK: OK: Less than 40.00% above the threshold [1000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [13:09:14] (03PS2) 10Sbisson: Rename wp10 to articlequality [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461715 (https://phabricator.wikimedia.org/T203080) [13:10:12] (03PS2) 10Arturo Borrero Gonzalez: cloudvps: add prometheus-openstack-exporter [puppet] - 10https://gerrit.wikimedia.org/r/462455 (https://phabricator.wikimedia.org/T203177) [13:10:37] (03CR) 10jerkins-bot: [V: 04-1] cloudvps: add prometheus-openstack-exporter [puppet] - 10https://gerrit.wikimedia.org/r/462455 (https://phabricator.wikimedia.org/T203177) (owner: 10Arturo Borrero Gonzalez) [13:12:53] !log rebooting rdb1003 for kernel security update [13:12:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:19:57] jouncebot: next [13:19:57] In 3 hour(s) and 40 minute(s): Wikidata Query Service weekly deploy (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180924T1700) [13:23:52] !log rebooting rdb1004 for kernel security update [13:23:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:25:44] (03PS2) 10Volans: cumin: installation of cumin1001 [puppet] - 10https://gerrit.wikimedia.org/r/462278 (https://phabricator.wikimedia.org/T201346) [13:26:12] (03CR) 10Volans: "done" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462278 (https://phabricator.wikimedia.org/T201346) (owner: 10Volans) [13:28:26] (03CR) 10Muehlenhoff: [C: 031] cumin: installation of cumin1001 [puppet] - 10https://gerrit.wikimedia.org/r/462278 (https://phabricator.wikimedia.org/T201346) (owner: 10Volans) [13:31:04] PROBLEM - High CPU load on API appserver on mw2138 is CRITICAL: CRITICAL - load average: 64.56, 35.81, 21.17 [13:32:29] (03CR) 10Jcrespo: [C: 032] mariadb: Setup 2 hosts for api on s5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462407 (owner: 10Jcrespo) [13:34:25] !log jynus@deploy1001 Synchronized wmf-config/db-codfw.php: Setup to db hosts for api-s5 (duration: 00m 50s) [13:34:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:34:31] 10Operations, 10Citoid, 10Patch-For-Review, 10Services (watching), 10VisualEditor (Current work): Transition citoid to use Zotero's translation-server-v2 - https://phabricator.wikimedia.org/T197242 (10Sebastian_Berlin-WMSE) [13:35:34] RECOVERY - High CPU load on API appserver on mw2138 is OK: OK - load average: 26.17, 31.42, 23.10 [13:37:10] (03PS1) 10Marostegui: db-codfw.php: Depool db2089:3315 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462467 [13:37:30] (03CR) 10Niedzielski: [C: 031] Increase sampling ratio for ReadingDepth [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462042 (https://phabricator.wikimedia.org/T205176) (owner: 10HaeB) [13:40:10] (03CR) 10Marostegui: [C: 032] db-codfw.php: Depool db2089:3315 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462467 (owner: 10Marostegui) [13:41:45] (03Merged) 10jenkins-bot: db-codfw.php: Depool db2089:3315 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462467 (owner: 10Marostegui) [13:43:12] !log marostegui@deploy1001 Synchronized wmf-config/db-codfw.php: Depool db2089 (duration: 00m 50s) [13:43:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:44:37] (03CR) 10Ottomata: [C: 031] "For when the time comes :)" [puppet] - 10https://gerrit.wikimedia.org/r/461979 (https://phabricator.wikimedia.org/T203635) (owner: 10Elukey) [13:45:19] (03CR) 10jenkins-bot: mariadb: Setup 2 hosts for api on s5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462407 (owner: 10Jcrespo) [13:45:21] (03CR) 10jenkins-bot: db-codfw.php: Depool db2089:3315 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462467 (owner: 10Marostegui) [13:45:55] (03PS1) 10Marostegui: Revert "db-codfw.php: Depool db2089:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462470 [13:49:15] 10Operations, 10Wikimedia-Logstash, 10User-fgiunchedi, 10User-herron: Logstash hardware expansion - https://phabricator.wikimedia.org/T203169 (10fgiunchedi) >>! In T203169#4599357, @herron wrote: > Technically we could explore multiple ES instances per-host to support mixed disk server configuration, but I... [13:50:20] (03CR) 10Marostegui: [C: 032] Revert "db-codfw.php: Depool db2089:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462470 (owner: 10Marostegui) [13:51:24] (03PS1) 10Herron: tools mail: add spamhaus rbl check [puppet] - 10https://gerrit.wikimedia.org/r/462472 (https://phabricator.wikimedia.org/T202558) [13:51:38] 10Operations, 10Mail: Mail relays needed for VMs in eqiad1 - https://phabricator.wikimedia.org/T205158 (10Andrew) It worked previously by accident (because eqiad vms are in 10.0.0.0/8 which is also the internal production range). Moving VMs out of that range is a feature rather than a bug (it gets us better s... [13:51:49] (03Merged) 10jenkins-bot: Revert "db-codfw.php: Depool db2089:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462470 (owner: 10Marostegui) [13:52:56] !log marostegui@deploy1001 Synchronized wmf-config/db-codfw.php: Repool db2089 (duration: 00m 49s) [13:53:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:53:21] 10Puppet, 10Cloud-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Ban spam arriving to my tools email - https://phabricator.wikimedia.org/T202558 (10herron) >>! In T202558#4558069, @GTirloni wrote: > I also took a sample of originating IP address (all from ChinaNet) and checked them against [... [13:58:37] ACKNOWLEDGEMENT - MegaRAID on rdb1004 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T205284 [13:58:41] 10Operations, 10ops-eqiad: Degraded RAID on rdb1004 - https://phabricator.wikimedia.org/T205284 (10ops-monitoring-bot) [13:59:48] (03CR) 10jenkins-bot: Revert "db-codfw.php: Depool db2089:3315" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462470 (owner: 10Marostegui) [13:59:53] !log stop and upgrade es1015 (es2 eqiad master)- it may create some temporary lag on es2 [13:59:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:01:20] 10Puppet, 10Cloud-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Ban spam arriving to my tools email - https://phabricator.wikimedia.org/T202558 (10Krenair) >>! In T202558#4609349, @GTirloni wrote: > I don't have a full understanding of all the implications. Would this be too radical? I unde... [14:08:32] 10Operations: Degraded RAID on rdb1004 - https://phabricator.wikimedia.org/T205287 (10MoritzMuehlenhoff) [14:08:59] ACKNOWLEDGEMENT - MegaRAID on rdb1004 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) Muehlenhoff T205287 [14:12:07] (03PS2) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert mediawiki.org [puppet] - 10https://gerrit.wikimedia.org/r/462425 (https://phabricator.wikimedia.org/T196968) [14:12:09] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wiktionary.org [puppet] - 10https://gerrit.wikimedia.org/r/462477 (https://phabricator.wikimedia.org/T196968) [14:12:11] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikiquote.org [puppet] - 10https://gerrit.wikimedia.org/r/462478 (https://phabricator.wikimedia.org/T196968) [14:12:13] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert donate.w.o [puppet] - 10https://gerrit.wikimedia.org/r/462479 (https://phabricator.wikimedia.org/T196968) [14:12:15] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikinews.org [puppet] - 10https://gerrit.wikimedia.org/r/462480 (https://phabricator.wikimedia.org/T196968) [14:18:43] !log stop and upgrade es1017 (es3 eqiad master)- it may create some temporary lag on es3 [14:18:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:19:45] (03PS31) 10Gehel: convert role::logstash::elasticsearch to profiles [puppet] - 10https://gerrit.wikimedia.org/r/441894 (https://phabricator.wikimedia.org/T198351) (owner: 10EBernhardson) [14:20:03] !log rebooting rdb1005 for kernel security update [14:20:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:28:06] !log upgrade gdnsd 2.99.42 -> 2.99.1729 on authdns1001 [14:28:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:28:54] !log rebooting rdb1006 for kernel security update [14:28:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:30:45] !log upgrade gdnsd 2.99.42 -> 2.99.1729 on authdns2001 [14:30:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:34:33] (03PS2) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikiquote.org [puppet] - 10https://gerrit.wikimedia.org/r/462478 (https://phabricator.wikimedia.org/T196968) [14:34:35] (03PS2) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert donate.w.o [puppet] - 10https://gerrit.wikimedia.org/r/462479 (https://phabricator.wikimedia.org/T196968) [14:34:37] (03PS2) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikinews.org [puppet] - 10https://gerrit.wikimedia.org/r/462480 (https://phabricator.wikimedia.org/T196968) [14:34:39] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikisource.org [puppet] - 10https://gerrit.wikimedia.org/r/462486 (https://phabricator.wikimedia.org/T196968) [14:34:41] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikibooks.org [puppet] - 10https://gerrit.wikimedia.org/r/462487 (https://phabricator.wikimedia.org/T196968) [14:51:56] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert vote.w.o [puppet] - 10https://gerrit.wikimedia.org/r/462492 (https://phabricator.wikimedia.org/T196968) [14:51:59] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert test.wikidata.org [puppet] - 10https://gerrit.wikimedia.org/r/462493 (https://phabricator.wikimedia.org/T196968) [14:51:59] !log rebooting rdb1007 for kernel security update [14:52:01] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikidata.org [puppet] - 10https://gerrit.wikimedia.org/r/462494 (https://phabricator.wikimedia.org/T196968) [14:52:03] (03PS1) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert wikipedia.org [puppet] - 10https://gerrit.wikimedia.org/r/462495 (https://phabricator.wikimedia.org/T196968) [14:52:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:52:06] <_joe_> I promise I'm done sending changes [14:53:12] I'll pick up the review fun tomorrow, then :-) [14:53:55] (03CR) 10Giuseppe Lavagetto: [C: 032] "> Patch Set 6:" [puppet] - 10https://gerrit.wikimedia.org/r/461395 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [14:54:06] (03PS7) 10Giuseppe Lavagetto: mediawiki::web::prod_sites: convert commons.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461395 (https://phabricator.wikimedia.org/T196968) [14:55:52] (03CR) 10Cwhite: "> Which files were unreadable by icinga out of the box on /etc/nagios" [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) (owner: 10Cwhite) [14:57:06] (03PS3) 10Cwhite: monitoring: set mode on host and service configs [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) [15:01:45] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:02:09] 10Operations, 10DBA, 10Research, 10Services (designing): Storage of data for recommendation API - https://phabricator.wikimedia.org/T203039 (10bmansurov) @jcrespo anything else blocking us from importing data to the database? Any documentation on connecting to the database from the services? @Pchelolo whe... [15:02:21] 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install mwmaint1002.eqiad.wmnet - https://phabricator.wikimedia.org/T201343 (10Dzahn) the new root partition is smaller than before on mwmaint1001 and also terbium. So copying home dirs from there would not work anymore, not enough space. Fixed by de... [15:04:01] !log rebooting rdb1008 for kernel security update [15:04:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:07:05] PROBLEM - Filesystem available is greater than filesystem size on ms-be2043 is CRITICAL: cluster=swift device=/dev/sdm1 fstype=xfs instance=ms-be2043:9100 job=node mountpoint=/srv/swift-storage/sdm1 site=codfw https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=ms-be2043&var-datasource=codfw%2520prometheus%252Fops [15:07:20] (03PS1) 10Jcrespo: mariadb: Depool db1064 for maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462499 [15:11:24] 10Operations, 10Cloud-Services, 10Mail: Create a Cloud VPS SMTP smarthost - https://phabricator.wikimedia.org/T41785 (10herron) Instances `smtp-out1001` and `smtp-out1002` have been created in `project-smtp` and floating IPs assigned. Do DNS entries for these floating IPs (to be seen by downstream mail syst... [15:12:07] 10Operations, 10SRE-Access-Requests, 10Discovery-Search (Current work), 10Patch-For-Review: add onimisionipe to maps-admin - https://phabricator.wikimedia.org/T204960 (10EBjune) Approved, thanks! [15:12:51] 10Operations, 10SRE-Access-Requests, 10Discovery-Search (Current work), 10Patch-For-Review: add onimisionipe to maps-admin - https://phabricator.wikimedia.org/T204960 (10EBjune) a:05EBjune>03RobH [15:13:20] 10Operations, 10DBA, 10Research, 10Services (designing): Storage of data for recommendation API - https://phabricator.wikimedia.org/T203039 (10jcrespo) > anything else blocking us from importing data to the database? There is no formal request yet, you need to create a ticket to #DBAs to ask to create a... [15:14:39] (03CR) 10Jcrespo: [C: 032] mariadb: Depool db1064 for maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462499 (owner: 10Jcrespo) [15:16:36] ^ godog (ms-be2043) [15:16:41] (03Merged) 10jenkins-bot: mariadb: Depool db1064 for maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462499 (owner: 10Jcrespo) [15:17:44] (03PS1) 10Jcrespo: Revert "mariadb: Depool db1064 for maintenance" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462501 [15:18:01] 10Operations, 10Cloud-Services, 10Mail: Create a Cloud VPS SMTP smarthost - https://phabricator.wikimedia.org/T41785 (10Krenair) We could modify the wmflabs.org zone itself to call them mx01.wmflabs.org and mx02.wmflabs.org or something if we don't think these should keep the project-smtp reference. It proba... [15:18:41] !log jynus@deploy1001 Synchronized wmf-config/db-eqiad.php: depool db1064 (duration: 00m 50s) [15:18:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:19:32] (03PS1) 10Alexandros Kosiaris: Add nodejs10-devel docker production image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/462503 (https://phabricator.wikimedia.org/T201611) [15:20:00] 10Operations, 10Discovery-Search, 10Elasticsearch, 10SRE-Access-Requests, 10Patch-For-Review: add onimisionipe to restricted group - https://phabricator.wikimedia.org/T204980 (10EBjune) Approved, thank you @RobH [15:20:14] 10Operations, 10Discovery-Search, 10Elasticsearch, 10SRE-Access-Requests, 10Patch-For-Review: add onimisionipe to restricted group - https://phabricator.wikimedia.org/T204980 (10EBjune) a:05EBjune>03RobH [15:21:25] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:23:52] !log stop and upgrade db1064 (x1) [15:23:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:24:33] (03CR) 10Bstorm: "The only worry I'd have here is a log exploding in size with warnings, lol, vs. actually blocking them." [puppet] - 10https://gerrit.wikimedia.org/r/462472 (https://phabricator.wikimedia.org/T202558) (owner: 10Herron) [15:24:38] (03CR) 10Alexandros Kosiaris: [C: 032] ores: install hunspell-gl on ores nodes [puppet] - 10https://gerrit.wikimedia.org/r/462403 (https://phabricator.wikimedia.org/T201142) (owner: 10Ladsgroup) [15:24:44] (03PS2) 10Alexandros Kosiaris: ores: install hunspell-gl on ores nodes [puppet] - 10https://gerrit.wikimedia.org/r/462403 (https://phabricator.wikimedia.org/T201142) (owner: 10Ladsgroup) [15:25:03] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] ores: install hunspell-gl on ores nodes [puppet] - 10https://gerrit.wikimedia.org/r/462403 (https://phabricator.wikimedia.org/T201142) (owner: 10Ladsgroup) [15:25:39] (03CR) 10jenkins-bot: mariadb: Depool db1064 for maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462499 (owner: 10Jcrespo) [15:26:55] (03CR) 10Bstorm: [C: 031] tools mail: add spamhaus rbl check [puppet] - 10https://gerrit.wikimedia.org/r/462472 (https://phabricator.wikimedia.org/T202558) (owner: 10Herron) [15:29:56] 10Puppet, 10Cloud-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Ban spam arriving to my tools email - https://phabricator.wikimedia.org/T202558 (10Bstorm) Thanks @herron and @GTirloni! The spamhaus list certainly can't hurt. I don't think we get so much email as to run afoul of their poli... [15:33:24] (03CR) 10Herron: "> The only worry I'd have here is a log exploding in size with" [puppet] - 10https://gerrit.wikimedia.org/r/462472 (https://phabricator.wikimedia.org/T202558) (owner: 10Herron) [15:34:24] 10Operations, 10Gadgets, 10MediaWiki-Cache, 10Performance-Team, and 2 others: Mcrouter periodically reports soft TKOs for mc[1,2]035 leading to MW Memcached exceptions - https://phabricator.wikimedia.org/T203786 (10Joe) @elukey I'm not so sure about the mcrouter failure behaviour. We need to check the actu... [15:35:07] (03PS2) 10Alexandros Kosiaris: Add nodejs10-devel docker production image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/462503 (https://phabricator.wikimedia.org/T201611) [15:35:48] 10Operations, 10Cloud-Services, 10Mail: Create a Cloud VPS SMTP smarthost - https://phabricator.wikimedia.org/T41785 (10herron) >>! In T41785#4611050, @Krenair wrote: > We could modify the wmflabs.org zone itself to call them mx01.wmflabs.org and mx02.wmflabs.org or something if we don't think these should k... [15:39:35] (03PS1) 10Bmansurov: Increase Schema:CitationUsagePageLoad population size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462507 (https://phabricator.wikimedia.org/T191086) [15:43:39] (03CR) 10Catrope: [C: 032] Labs: rename wp10 to articlequality [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461734 (https://phabricator.wikimedia.org/T203080) (owner: 10Sbisson) [15:44:31] (03CR) 10Filippo Giunchedi: [C: 031] monitoring: set mode on host and service configs [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) (owner: 10Cwhite) [15:44:53] (03Merged) 10jenkins-bot: Labs: rename wp10 to articlequality [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461734 (https://phabricator.wikimedia.org/T203080) (owner: 10Sbisson) [15:46:18] (03PS2) 10Bmansurov: Increase Schema:CitationUsagePageLoad sampling rate [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462507 (https://phabricator.wikimedia.org/T191086) [15:46:21] (03CR) 10Jcrespo: [C: 032] Revert "mariadb: Depool db1064 for maintenance" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462501 (owner: 10Jcrespo) [15:46:50] !log Deploy schema change on s8 eqiad master with replication - might generate lag - T204006 [15:46:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:46:58] T204006: Execute the schema change for Partial Blocks - https://phabricator.wikimedia.org/T204006 [15:47:32] 10Operations, 10Cloud-Services, 10Mail: Create a Cloud VPS SMTP smarthost - https://phabricator.wikimedia.org/T41785 (10Krenair) >>! In T41785#4611107, @herron wrote: >>>! In T41785#4611050, @Krenair wrote: >> We could modify the wmflabs.org zone itself to call them mx01.wmflabs.org and mx02.wmflabs.org or s... [15:47:34] (03Merged) 10jenkins-bot: Revert "mariadb: Depool db1064 for maintenance" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462501 (owner: 10Jcrespo) [15:48:46] I have your revert here, marostegui [15:49:00] which revert? [15:49:00] should I wait, should I deploy, should I leave it to you? [15:49:07] on deploy1001 [15:49:19] db2089? [15:49:33] not yourse [15:49:36] I had not rebase [15:49:38] d [15:49:43] it is Bmansurov [15:49:45] 's [15:49:46] Ah :-) [15:50:07] it is labs only apparently, do you know his irc nick to confirm the rebase? [15:50:34] stephanebisson ^? [15:50:50] rebase on deployment server? [15:51:04] jynus: as per the contact list either baha or bmansurov [15:51:18] has someone merged without deploying? [15:51:31] yeah d86649e5bcf3be1c3e [15:51:37] but it may be labs only [15:51:47] there is a +2 from Catrope [15:51:49] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] Add nodejs10-devel docker production image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/462503 (https://phabricator.wikimedia.org/T201611) (owner: 10Alexandros Kosiaris) [15:51:57] ˜/wikibugs 17:43> (CR) Catrope: [C: 2] Labs: rename wp10 to articlequality [mediawiki-config] - https://gerrit.wikimedia.org/r/461734 (https://phabricator.wikimedia.org/T203080) (owner: Sbisson) [15:51:57] RoanKattouw [15:52:08] !log rebooting rdb1009/rdb1010 for kernel security update [15:52:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:52:17] I am going to deploy my change [15:52:22] which is on another file [15:52:36] and most likely no need to deploy anything [15:52:48] should be safe to ignore InitialiseSettings-labs changes in prod [15:53:13] Yes that was me; it was a labs-only change so I assumed it'd be OK. Didn't realize you wre in the middle of something here, apologies if it caused issues [15:53:14] well, I just saw one commit extra [15:53:27] didn't look further [15:53:38] it'll be okay but IIRC you're expected to update the repo in prod at least so it doesn't alarm the next person [15:53:41] rebase should be enough [15:53:44] ^that [15:54:03] (03CR) 10jenkins-bot: Labs: rename wp10 to articlequality [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461734 (https://phabricator.wikimedia.org/T203080) (owner: 10Sbisson) [15:54:03] basicaly because one doesn't want to deploy code that doesn't know by accident [15:54:05] (03CR) 10jenkins-bot: Revert "mariadb: Depool db1064 for maintenance" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462501 (owner: 10Jcrespo) [15:54:06] used to do no-op syncs of those -labs files in prod for the sake of following the whole process, though it didn't really do much [15:54:35] PROBLEM - cassandra CQL 10.64.48.154:9042 on maps1004 is CRITICAL: connect to address 10.64.48.154 and port 9042: Connection refused [15:54:57] 10Puppet, 10Cloud-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Ban spam arriving to my tools email - https://phabricator.wikimedia.org/T202558 (10herron) >>! In T202558#4611075, @Bstorm wrote: > Thanks @herron and @GTirloni! The spamhaus list certainly can't hurt. I don't think we get so... [15:55:12] !log jynus@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1064 (duration: 00m 50s) [15:55:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:55:19] mw2227.codfw.wmnet' failed: ERROR: 75% OVER_THRESHOLD (Avg. Error rate: Before: 0.08, After: 4.00, Threshold: 1.00) [15:55:38] FYI^ [15:56:44] wikidata seems a bit overloaded [15:57:03] probably for the reasons discussed at -databases [15:57:05] (03Abandoned) 10Gergő Tisza: Temporarily prevent users from accessing Special:RenderBook/test [mediawiki-config] - 10https://gerrit.wikimedia.org/r/377929 (https://phabricator.wikimedia.org/T175868) (owner: 10Gergő Tisza) [15:58:07] (03PS2) 10Bstorm: tools mail: add spamhaus rbl check [puppet] - 10https://gerrit.wikimedia.org/r/462472 (https://phabricator.wikimedia.org/T202558) (owner: 10Herron) [15:59:53] (03PS1) 10Mathew.onipe: Add elasticsearch_operations [cookbooks] - 10https://gerrit.wikimedia.org/r/462514 (https://phabricator.wikimedia.org/T202885) [16:01:55] (03CR) 10Bstorm: [C: 032] tools mail: add spamhaus rbl check [puppet] - 10https://gerrit.wikimedia.org/r/462472 (https://phabricator.wikimedia.org/T202558) (owner: 10Herron) [16:05:21] 10Operations, 10Citoid, 10Services, 10Patch-For-Review, and 3 others: Deploy translation-server-v2 - https://phabricator.wikimedia.org/T201611 (10akosiaris) >>! In T201611#4606202, @thcipriani wrote: >>>! In T201611#4606083, @akosiaris wrote: >> @thcipriani, @dduvall, nodejs 10 image built and uploaded. It... [16:11:45] PROBLEM - Varnish traffic drop between 30min ago and now at eqiad on einsteinium is CRITICAL: 56.62 le 60 https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [16:14:04] RECOVERY - Varnish traffic drop between 30min ago and now at eqiad on einsteinium is OK: (C)60 le (W)70 le 76.91 https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [16:15:13] 10Operations, 10Discovery-Search, 10Elasticsearch, 10SRE-Access-Requests, 10Patch-For-Review: add onimisionipe to restricted group - https://phabricator.wikimedia.org/T204980 (10Gehel) Approved in SRE meeting [16:15:17] 10Operations, 10SRE-Access-Requests, 10Discovery-Search (Current work), 10Patch-For-Review: add onimisionipe to maps-admin - https://phabricator.wikimedia.org/T204960 (10Gehel) Approved in SRE meeting [16:23:15] 10Operations, 10DBA, 10Research, 10Services (designing): Storage of data for recommendation API - https://phabricator.wikimedia.org/T203039 (10Pchelolo) > @Pchelolo where would database settings live? Would it be the service codebase itself or do we have a separate repository for that? Usually the source... [16:28:00] gehel: is maps1004 a known issue? [16:28:37] urandom: it gets reimaged to stretch [16:28:42] urandom: yeah, I'm getting it ready for reimage [16:29:01] silencing it! [16:29:20] ok, I did see your log message fwiw, but the 3 hour gap made me think I should ask anyway [16:31:13] urandom: always good to ask! And my bad for not silencing it before starting. [16:31:32] * gehel was slightly surprised by cassandra shutting down on decomission [16:31:43] * gehel was expecting only data to move around [16:34:25] well, the successful completion of a decommission means that node is no longer a member of the cluster, it basically ceases to exist. [16:36:01] 10Puppet, 10Cloud-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Ban spam arriving to my tools email - https://phabricator.wikimedia.org/T202558 (10Bstorm) Noticed on deploying this that the mail queue was clogged up with frozen messages rejected from qq.com servers. I cleaned them with: `e... [16:36:32] in that respect, element of least surprise is probably that it stop accepting connections [16:45:19] 10Operations, 10Citoid, 10Services, 10Patch-For-Review, and 3 others: Deploy translation-server-v2 - https://phabricator.wikimedia.org/T201611 (10akosiaris) @Mvolz, SRE has a question about this migration. Assuming this gets deployed successfully next quarter, this will allow us to migrate off the current... [16:48:05] PROBLEM - MediaWiki memcached error rate on graphite1001 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [5000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [16:49:56] (03CR) 10EBernhardson: "ran puppet compiler: https://puppet-compiler.wmflabs.org/compiler1002/12573" [puppet] - 10https://gerrit.wikimedia.org/r/441894 (https://phabricator.wikimedia.org/T198351) (owner: 10EBernhardson) [16:54:34] RECOVERY - MediaWiki memcached error rate on graphite1001 is OK: OK: Less than 40.00% above the threshold [1000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [16:58:55] !log gehel@deploy1001 Started deploy [wdqs/wdqs@195ea0e]: new version of wdqs GUI and updater (wdqs1009 only) [16:59:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:59:19] (03PS2) 10Dzahn: Add Matt to restricted group [puppet] - 10https://gerrit.wikimedia.org/r/461693 (https://phabricator.wikimedia.org/T204980) (owner: 10Mathew.onipe) [16:59:26] !log gehel@deploy1001 Finished deploy [wdqs/wdqs@195ea0e]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 31s) [16:59:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:00:04] gehel: (Dis)respected human, time to deploy Wikidata Query Service weekly deploy (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180924T1700). Please do the needful. [17:00:14] (03PS3) 10Dzahn: admins: Add Matt Onipe to restricted group [puppet] - 10https://gerrit.wikimedia.org/r/461693 (https://phabricator.wikimedia.org/T204980) (owner: 10Mathew.onipe) [17:00:47] (03CR) 10Dzahn: [C: 032] "approved in SRE meeting" [puppet] - 10https://gerrit.wikimedia.org/r/461693 (https://phabricator.wikimedia.org/T204980) (owner: 10Mathew.onipe) [17:01:08] mutante: ^ thanks! (cc onimisionipe) [17:02:14] !log gehel@deploy1001 Started deploy [wdqs/wdqs@195ea0e]: new version of wdqs GUI and updater [17:02:15] (03PS2) 10Dzahn: admins: Add Matt Onipe to maps-admin [puppet] - 10https://gerrit.wikimedia.org/r/461642 (https://phabricator.wikimedia.org/T204960) (owner: 10Mathew.onipe) [17:02:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:02:49] 10Puppet, 10Cloud-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Ban spam arriving to my tools email - https://phabricator.wikimedia.org/T202558 (10Bstorm) We are now generating additional frozen messages due to the reply this sends, lol. It's interesting. [17:03:33] (03PS3) 10Dzahn: admins: Add Matt Onipe to maps-admin [puppet] - 10https://gerrit.wikimedia.org/r/461642 (https://phabricator.wikimedia.org/T204960) (owner: 10Mathew.onipe) [17:03:46] 10Puppet, 10Cloud-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Ban spam arriving to my tools email - https://phabricator.wikimedia.org/T202558 (10Bstorm) It's not a huge load, though, to be clear. [17:04:11] (03CR) 10Dzahn: [C: 032] "approved in SRE meeting" [puppet] - 10https://gerrit.wikimedia.org/r/461642 (https://phabricator.wikimedia.org/T204960) (owner: 10Mathew.onipe) [17:04:58] moritzm: hi, around? [17:05:47] gehel: you're welcome, confirming on mwmaint1002 and maps1001 [17:06:03] mutante: Thanks! [17:06:06] onimisionipe: ^ you can try ssh to "maps1001.eqiad.wmnet" now, it should work [17:06:06] I can't change content model in https://ja.wikipedia.org/wiki/%E3%83%A2%E3%82%B8%E3%83%A5%E3%83%BC%E3%83%AB:Microsoft_Windows_10 ; action log is created. [17:06:11] the other ones after puppet ran [17:06:35] someone check out this ? I haven't prod logstash access . [17:07:27] s/check out/check/ [17:07:48] I saw T205299 [17:07:49] T205299: It can not move from Module to Article. - https://phabricator.wikimedia.org/T205299 [17:07:52] But it's a little hard to understand [17:08:01] onimisionipe: i should use codfw as example.. currently. so "mwmaint2001.codfw.wmnet" is the mediawiki maintenance host and shoudl work now [17:08:02] mutante: will do so now [17:08:15] onimisionipe: and for maps.. maps2001.codfw.wmnet [17:08:35] Reedy: https://phabricator.wikimedia.org/T205299 ; it is related with content model, [17:09:14] the page content model is "Scribunto". ; Scribunto content model is not allowed in ns0. [17:09:17] 10Operations, 10SRE-Access-Requests, 10Discovery-Search (Current work), 10Patch-For-Review: add onimisionipe to maps-admin - https://phabricator.wikimedia.org/T204960 (10Dzahn) [17:10:16] 10Operations, 10SRE-Access-Requests, 10Discovery-Search (Current work), 10Patch-For-Review: add onimisionipe to maps-admin - https://phabricator.wikimedia.org/T204960 (10Dzahn) merged and ran puppet on maps1001.eqiad.wmnet and maps2001.codfw.wmnet. i saw Matt's user got created. all other maps hosts will f... [17:10:16] for this, I tried to change content model in https://ja.wikipedia.org/wiki/Special:ChangeContentModel , but content model is not changed ( but log is created ) [17:10:46] tried mwmaint and maps... i can confirm it works.. [17:11:52] onimisionipe: :) great! i would say you can call your own ticket resolved then [17:12:15] PROBLEM - MediaWiki memcached error rate on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [5000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [17:12:57] 10Operations, 10Discovery-Search, 10Elasticsearch, 10SRE-Access-Requests, 10Patch-For-Review: add onimisionipe to restricted group - https://phabricator.wikimedia.org/T204980 (10Dzahn) merged and saw your user has been created on mwmaint2001.codfw.wmnet so that should also resolve the ticket, right [17:13:21] 10Operations, 10Discovery-Search, 10Elasticsearch, 10SRE-Access-Requests, 10Patch-For-Review: add onimisionipe to restricted group - https://phabricator.wikimedia.org/T204980 (10Dzahn) [17:13:40] rxy: page table says it's scribunto, revisions say it's wikitext [17:13:49] ugh [17:14:33] so.. [17:15:30] !log gehel@deploy1001 Finished deploy [wdqs/wdqs@195ea0e]: new version of wdqs GUI and updater (duration: 13m 15s) [17:15:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:16:44] RECOVERY - MediaWiki memcached error rate on graphite1001 is OK: OK: Less than 40.00% above the threshold [1000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [17:17:09] rxy: fixed [17:17:18] oh, thanks :) [17:17:25] 10Puppet, 10Cloud-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Ban spam arriving to my tools email - https://phabricator.wikimedia.org/T202558 (10herron) >>! In T202558#4611572, @Bstorm wrote: > We are now generating additional frozen messages due to the reply this sends, lol. It's intere... [17:17:51] SMalyshev: ^^ deployment complete, tests are green [17:18:53] 10Operations, 10Discovery-Search, 10Elasticsearch, 10SRE-Access-Requests, 10Patch-For-Review: add onimisionipe to restricted group - https://phabricator.wikimedia.org/T204980 (10Mathew.onipe) 05Open>03Resolved [17:19:05] 10Operations, 10SRE-Access-Requests, 10Discovery-Search (Current work), 10Patch-For-Review: add onimisionipe to maps-admin - https://phabricator.wikimedia.org/T204960 (10Mathew.onipe) 05Open>03Resolved [17:25:24] PROBLEM - MediaWiki memcached error rate on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [5000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [17:27:34] RECOVERY - MediaWiki memcached error rate on graphite1001 is OK: OK: Less than 40.00% above the threshold [1000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [17:27:47] (03PS1) 10Effie Mouzeli: Added zone wikisource.gr [dns] - 10https://gerrit.wikimedia.org/r/462532 (https://phabricator.wikimedia.org/T205077) [17:28:31] 10Operations, 10Release-Engineering-Team, 10Scap: mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild - https://phabricator.wikimedia.org/T203625 (10Legoktm) IIRC it was about 3-4 minutes. But that was also with HHVM, it's probably worth checking again with PHP 7 to see if... [17:30:57] (03CR) 10Dzahn: [C: 031] "yes, symlinked to wikisource.org and WHOIS shows this is registered with MarkMonitor and points to our DNS servers" [dns] - 10https://gerrit.wikimedia.org/r/462532 (https://phabricator.wikimedia.org/T205077) (owner: 10Effie Mouzeli) [17:34:17] (03PS2) 10Smalyshev: Enable phrase search config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462351 (https://phabricator.wikimedia.org/T163642) [17:34:36] (03CR) 10Dzahn: [C: 031] "for some reason this already works now.. how is that ?:)" [dns] - 10https://gerrit.wikimedia.org/r/462532 (https://phabricator.wikimedia.org/T205077) (owner: 10Effie Mouzeli) [17:35:37] (03PS1) 10Pmiazga: Increate ReadingDepthSamplingRate to 0.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462535 (https://phabricator.wikimedia.org/T205176) [17:36:57] (03CR) 10jerkins-bot: [V: 04-1] Increate ReadingDepthSamplingRate to 0.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462535 (https://phabricator.wikimedia.org/T205176) (owner: 10Pmiazga) [17:39:38] !log powerdown and move bast4002 (not in prod) [17:39:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:40:56] 10Operations, 10MediaWiki-ResourceLoader, 10Performance-Team, 10Traffic: Investigate source of 404 Not Found responses from load.php - https://phabricator.wikimedia.org/T202479 (10Krinkle) >>! In T202479#4580307, @ema wrote: >>>! In T202479#4575922, @Krinkle wrote: >> 2. Hostnames we route to text-lb that... [17:42:53] 10Operations, 10netops, 10Patch-For-Review: Evaluate NetBox as a Racktables replacement & IPAM - https://phabricator.wikimedia.org/T170144 (10Volans) [17:44:23] (03PS2) 10Pmiazga: Increate ReadingDepthSamplingRate to 0.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462535 (https://phabricator.wikimedia.org/T205176) [17:44:29] 10Operations, 10Goal, 10Patch-For-Review: Migrate the hardware inventory from Racktables to Netbox - https://phabricator.wikimedia.org/T199083 (10Volans) Data from Racktables has been exported and imported into Netbox, starting the migration week in which we'll apply changes to both tools. At the end of the... [17:44:40] gehel: did you do the Updater restart? [17:44:51] 10Operations, 10Goal, 10Patch-For-Review: Migrate the hardware inventory from Racktables to Netbox - https://phabricator.wikimedia.org/T199083 (10Volans) [17:44:56] I think deploy doesn't restart updater [17:58:07] (03CR) 10Effie Mouzeli: [C: 032] "It is not on our DNS yet:)" [dns] - 10https://gerrit.wikimedia.org/r/462532 (https://phabricator.wikimedia.org/T205077) (owner: 10Effie Mouzeli) [17:59:34] SMalyshev: Damn, only restarted on test node [17:59:53] !log restarting updater on wdqs* [17:59:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:00:01] SMalyshev: done! [18:00:01] gehel: yeah probably needs restart everywhere. if you want that log throttling actually enabled :) [18:00:05] Deploy window Morning SWAT (Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180924T1800) [18:00:05] Jayprakash12345, bmansurov, tgr, and framawiki: A patch you scheduled for Morning SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [18:00:12] here [18:00:16] o/ [18:00:41] yo [18:01:00] tgr: Jayprakash12345 here as well? [18:01:07] o/ [18:01:10] yeah [18:03:55] greg-g: Are you going to handle SWAT? [18:03:58] yup [18:04:06] Jayprakash12345: ready? [18:04:16] Yes, I am ready [18:04:28] (03CR) 10Greg Grossmeier: [C: 032] Enable Extension:NewUserMessage on kn.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462045 (https://phabricator.wikimedia.org/T204405) (owner: 10Jayprakash12345) [18:04:39] (03CR) 10Greg Grossmeier: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462045 (https://phabricator.wikimedia.org/T204405) (owner: 10Jayprakash12345) [18:06:18] (03Merged) 10jenkins-bot: Enable Extension:NewUserMessage on kn.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462045 (https://phabricator.wikimedia.org/T204405) (owner: 10Jayprakash12345) [18:07:32] (03PS1) 10Ayounsi: Smokeping: replace bast4002 with bast4001 [puppet] - 10https://gerrit.wikimedia.org/r/462547 [18:07:34] (03PS2) 10Framawiki: Remove mhs.ox.ac.uk from $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/459365 (https://phabricator.wikimedia.org/T203904) [18:07:37] Jayprakash12345: please check (mw2017 in the extension) [18:07:50] greg-g: Ok [18:08:11] "works" OK or "on it" OK? :) [18:08:32] (03CR) 10Ayounsi: [C: 032] Smokeping: replace bast4002 with bast4001 [puppet] - 10https://gerrit.wikimedia.org/r/462547 (owner: 10Ayounsi) [18:08:34] (03PS2) 10Framawiki: Create Cookbook NS in bnwikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/458870 (https://phabricator.wikimedia.org/T203534) [18:10:14] Jayprakash12345: you good? https://wikitech.wikimedia.org/wiki/X-Wikimedia-Debug#Staging_changes [18:10:28] (03PS2) 10Framawiki: Set wgRestrictDisplayTitle = false for pflwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462011 (https://phabricator.wikimedia.org/T205055) [18:10:51] greg-g: Yeah, Please run StatshBot [18:11:27] (03PS3) 10Framawiki: Add NS 110 to wgNamespacesToBeSearchedDefault on frwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462271 (https://phabricator.wikimedia.org/T205198) [18:11:46] 10Operations: Update wikimedia apt repo to include debs for shiny-server - https://phabricator.wikimedia.org/T106435 (10mpopov) For archive happiness, this was done in T164603 :) [18:12:10] 10Operations, 10Product-Analytics: Upload shiny-server .deb to our Jessie apt repository - https://phabricator.wikimedia.org/T168967 (10mpopov) [18:13:00] greg-g: I have checked at mw2017. Please go ahead [18:13:24] Jayprakash12345: thanks, deploying [18:13:25] 10Operations, 10Puppet, 10Beta-Cluster-Infrastructure, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q1): exported puppet resources are not queryable: cannot create grafana graphs of EventLogging running in beta cluster - https://phabricator.wikimedia.org/T204088 (10Jdlrobson) Any luck @Ottoma... [18:14:02] !log gjg@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT [config] {{gerrit|462045}} Enable Extension:NewUserMessage on kn.wikisource ({{phab|T204405}}) (duration: 00m 50s) [18:14:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:14:10] T204405: Enable Extension:NewUserMessage on kn.wikisource - https://phabricator.wikimedia.org/T204405 [18:14:31] (03CR) 10jenkins-bot: Enable Extension:NewUserMessage on kn.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462045 (https://phabricator.wikimedia.org/T204405) (owner: 10Jayprakash12345) [18:14:36] (03PS3) 10Greg Grossmeier: Increase Schema:CitationUsagePageLoad sampling rate [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462507 (https://phabricator.wikimedia.org/T191086) (owner: 10Bmansurov) [18:14:43] (03CR) 10Greg Grossmeier: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462507 (https://phabricator.wikimedia.org/T191086) (owner: 10Bmansurov) [18:15:09] Jayprakash12345: you're good, have a nice day :) [18:15:14] bmansurov: ready? [18:15:18] greg-g: yes [18:15:35] greg-g: Nice to meet you. Good Night :) [18:16:48] (03Merged) 10jenkins-bot: Increase Schema:CitationUsagePageLoad sampling rate [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462507 (https://phabricator.wikimedia.org/T191086) (owner: 10Bmansurov) [18:17:25] bmansurov: alrighty, please check [18:17:36] greg-g: which server? [18:17:50] mw2017 [18:18:27] greg-g: checking [18:18:53] greg-g: it's working. please deploy everywhere [18:19:29] cool [18:19:50] (03PS33) 10EBernhardson: prometheus/elasticsearch support multiple exporters per host [puppet] - 10https://gerrit.wikimedia.org/r/441321 (https://phabricator.wikimedia.org/T198351) [18:20:21] syncing [18:20:59] !log gjg@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT [config] {{gerrit|462507}} Increase Schema:CitationUsagePageLoad population size ({{phab|T191086}}) (duration: 00m 50s) [18:21:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:21:07] T191086: Instrument and collect data via CitationUsage schema - https://phabricator.wikimedia.org/T191086 [18:21:09] bmansurov: ^ out [18:21:14] greg-g: thank you! [18:21:17] tgr: ready? [18:21:33] (03PS3) 10Greg Grossmeier: Allow wikitech bureaucrats to promote to interface-admin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461240 (owner: 10Gergő Tisza) [18:22:19] (03CR) 10Greg Grossmeier: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461240 (owner: 10Gergő Tisza) [18:24:53] (03Merged) 10jenkins-bot: Allow wikitech bureaucrats to promote to interface-admin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461240 (owner: 10Gergő Tisza) [18:25:07] tgr: hello? :) about to put on debug server [18:25:17] (mw2017) [18:26:45] greg-g: sorry, got distracted [18:26:47] it's there [18:27:00] (03PS3) 10Framawiki: Remove mhs.ox.ac.uk from $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/459365 (https://phabricator.wikimedia.org/T203904) [18:27:33] (03PS3) 10Framawiki: Create Cookbook NS in bnwikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/458870 (https://phabricator.wikimedia.org/T203534) [18:27:56] I assume I can just push this tgr? :) [18:28:04] (03PS3) 10Framawiki: Set wgRestrictDisplayTitle = false for pflwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462011 (https://phabricator.wikimedia.org/T205055) [18:28:05] (03PS4) 10Framawiki: Add NS 110 to wgNamespacesToBeSearchedDefault on frwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462271 (https://phabricator.wikimedia.org/T205198) [18:28:07] hm, it does not seem to work [18:28:51] (03CR) 10jenkins-bot: Increase Schema:CitationUsagePageLoad sampling rate [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462507 (https://phabricator.wikimedia.org/T191086) (owner: 10Bmansurov) [18:28:53] (03CR) 10jenkins-bot: Allow wikitech bureaucrats to promote to interface-admin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461240 (owner: 10Gergő Tisza) [18:29:45] greg-g: are you sure it's on mw2017? [18:31:40] greg-g: Woah, you're SWATing? Are we sure it's safe to let managers touch these things? ;-) [18:32:13] tgr: just grep'd and found it [18:32:55] I just pulled it on mwdebug2001, which supposedly is mw2017 in the extension dropdown [18:33:12] James_F: usually not :) [18:33:25] * James_F grins. [18:34:17] tgr: have you tried again? [18:34:44] yeah, also with curl [18:34:48] does not work [18:34:51] huh [18:34:58] revert try again later? [18:35:07] wikitech has its own deploy cycle, right? does that extend to config? [18:35:47] tgr: I just touch'd it, try again? [18:36:07] yeah, that's separate, g'damn, this won't effect that [18:36:45] just deploy and hope it eventually gets there? the patch is super trivial [18:36:48] sync'ing it to get it out of the way so we can move on [18:36:51] tgr: yup [18:36:55] syncing will get it to wikitech [18:37:01] Just can't do the debug testing [18:37:05] got it [18:37:15] !log gjg@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [config] {{gerrit|461240}} Allow wikitech bureaucrats to promote to interface-admin, but uh, only wikitech (duration: 00m 50s) [18:37:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:37:22] ok, there ^ [18:37:26] yeah, it works now [18:37:36] framawiki: ready? any special order? [18:37:45] tgr: stupid wikitech [18:37:47] thanks! [18:38:11] hello greg-g, no order and it doesn't matter if we don't do everything [18:38:21] (03CR) 10Awight: [C: 031] "Hard cutover looks good to me, there are no MediaWiki consumers of the data yet." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461715 (https://phabricator.wikimedia.org/T203080) (owner: 10Sbisson) [18:39:03] (03PS3) 10Greg Grossmeier: Throttle for October 11 event [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462014 (https://phabricator.wikimedia.org/T204829) (owner: 10Framawiki) [18:39:22] (03CR) 10Greg Grossmeier: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462014 (https://phabricator.wikimedia.org/T204829) (owner: 10Framawiki) [18:41:14] (03Merged) 10jenkins-bot: Throttle for October 11 event [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462014 (https://phabricator.wikimedia.org/T204829) (owner: 10Framawiki) [18:41:43] (03PS3) 10Sbisson: Rename wp10 to articlequality [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461715 (https://phabricator.wikimedia.org/T203080) [18:43:30] (03CR) 10jenkins-bot: Throttle for October 11 event [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462014 (https://phabricator.wikimedia.org/T204829) (owner: 10Framawiki) [18:44:25] !log gjg@deploy1001 Synchronized wmf-config/throttle.php: [config] {{gerrit|462014}} New users on IP for edit-a-thon (October 11) ({{phab|T204829}}) (duration: 00m 49s) [18:44:26] (03CR) 10Greg Grossmeier: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462271 (https://phabricator.wikimedia.org/T205198) (owner: 10Framawiki) [18:44:31] (03PS42) 10EBernhardson: Split instance define out of elasticsearch class [puppet] - 10https://gerrit.wikimedia.org/r/441338 (https://phabricator.wikimedia.org/T198351) [18:44:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:44:33] (03PS69) 10EBernhardson: Allow multiple elasticsearch instances per host [puppet] - 10https://gerrit.wikimedia.org/r/440049 (https://phabricator.wikimedia.org/T198351) [18:44:33] T204829: New users on IP for edit-a-thon (October 11, Texas A&M Univ-Corpus Christi library) - https://phabricator.wikimedia.org/T204829 [18:45:41] (03Merged) 10jenkins-bot: Add NS 110 to wgNamespacesToBeSearchedDefault on frwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462271 (https://phabricator.wikimedia.org/T205198) (owner: 10Framawiki) [18:45:56] (03CR) 10jerkins-bot: [V: 04-1] Allow multiple elasticsearch instances per host [puppet] - 10https://gerrit.wikimedia.org/r/440049 (https://phabricator.wikimedia.org/T198351) (owner: 10EBernhardson) [18:46:25] framawiki: the NS 110 is on mw2017 [18:46:33] greg-g: please skip https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/458870/ for now [18:46:35] will look [18:47:15] (03CR) 10Framawiki: [C: 04-1] "Need to check something" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/458870 (https://phabricator.wikimedia.org/T203534) (owner: 10Framawiki) [18:47:44] framawiki: ack re that one ^ [18:48:24] greg-g: NS 110 ok [18:48:42] cool, going everywhere [18:49:26] !log gjg@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [config] {{gerrit|462271}} Search by default on the Reconstruction namespace at frwikt ({{phab|T205198}}) (duration: 00m 50s) [18:49:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:49:35] T205198: Search by default on the Reconstruction namespace at frwikt - https://phabricator.wikimedia.org/T205198 [18:49:40] (03CR) 10Greg Grossmeier: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462011 (https://phabricator.wikimedia.org/T205055) (owner: 10Framawiki) [18:52:37] (03PS4) 10Greg Grossmeier: Set wgRestrictDisplayTitle = false for pflwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462011 (https://phabricator.wikimedia.org/T205055) (owner: 10Framawiki) [18:52:48] (03CR) 10Greg Grossmeier: Set wgRestrictDisplayTitle = false for pflwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462011 (https://phabricator.wikimedia.org/T205055) (owner: 10Framawiki) [18:52:56] (03CR) 10Greg Grossmeier: [C: 032] "SWATx2" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462011 (https://phabricator.wikimedia.org/T205055) (owner: 10Framawiki) [18:53:07] grr [18:53:32] (my bad, thought it was rebase'd, but just didn't have the latest view in gerrit) [18:53:57] (03Merged) 10jenkins-bot: Set wgRestrictDisplayTitle = false for pflwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462011 (https://phabricator.wikimedia.org/T205055) (owner: 10Framawiki) [18:54:44] framawiki: on mw2017 [18:56:04] greg-g: that works [18:56:09] cool [18:57:25] !log gjg@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [config] {{gerrit|462011}} Set wgRestrictDisplayTitle = false for pflwiki ({{phab|T205055}}) (duration: 00m 48s) [18:57:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:57:33] T205055: Configuration change for pfl.wikipedia.org: Set wgRestrictDisplayTitle = false - https://phabricator.wikimedia.org/T205055 [18:57:53] (03CR) 10Greg Grossmeier: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/459365 (https://phabricator.wikimedia.org/T203904) (owner: 10Framawiki) [18:57:58] (03CR) 10jenkins-bot: Add NS 110 to wgNamespacesToBeSearchedDefault on frwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462271 (https://phabricator.wikimedia.org/T205198) (owner: 10Framawiki) [18:58:00] (03CR) 10jenkins-bot: Set wgRestrictDisplayTitle = false for pflwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462011 (https://phabricator.wikimedia.org/T205055) (owner: 10Framawiki) [18:58:11] (03PS4) 10Greg Grossmeier: Remove mhs.ox.ac.uk from $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/459365 (https://phabricator.wikimedia.org/T203904) (owner: 10Framawiki) [18:58:19] (03CR) 10Greg Grossmeier: Remove mhs.ox.ac.uk from $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/459365 (https://phabricator.wikimedia.org/T203904) (owner: 10Framawiki) [18:58:29] (03CR) 10Greg Grossmeier: [C: 032] "SWATx2 (rebase hell)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/459365 (https://phabricator.wikimedia.org/T203904) (owner: 10Framawiki) [18:59:48] (03PS2) 10Dzahn: trafficserver: replace mwmaint1001 with 1002 as noc.wm.org backend [puppet] - 10https://gerrit.wikimedia.org/r/461490 (https://phabricator.wikimedia.org/T201343) [19:00:27] (03Merged) 10jenkins-bot: Remove mhs.ox.ac.uk from $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/459365 (https://phabricator.wikimedia.org/T203904) (owner: 10Framawiki) [19:01:19] framawiki: I'll just push this wgcopydomains one [19:02:47] !log gjg@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT - [config] {{gerrit|459365}} Remove mhs.ox.ac.uk from $wgCopyUploadsDomains ({{phab|T203904}}) (duration: 00m 50s) [19:02:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:02:54] T203904: Remove mhs.ox.ac.uk from $wgCopyUploadsDomains - https://phabricator.wikimedia.org/T203904 [19:03:26] greg-g: practically at time, thanks ! [19:03:32] framawiki: all done [19:03:37] framawiki: thank you. [19:03:55] This completes SWAT. Back to your regularly scheduled programming already in progress. [19:05:52] (03CR) 10Dzahn: "re:" [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) (owner: 10Cwhite) [19:07:12] (03CR) 10Dzahn: [C: 031] monitoring: set mode on host and service configs [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) (owner: 10Cwhite) [19:08:14] PROBLEM - Check systemd state on elastic2022 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [19:08:15] PROBLEM - Check systemd state on elastic2033 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [19:08:24] PROBLEM - Check systemd state on elastic2020 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [19:08:25] PROBLEM - Check systemd state on elastic2016 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [19:08:29] looking [19:08:34] PROBLEM - Check systemd state on elastic2018 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [19:09:11] gehel: ^ Somehow these hosts have production-search-codfw and production-search-eqiad defined :S [19:09:16] (in systemd) [19:10:12] looks like no immediate problem, because puppet didn't install the appropriate stuff to actually start the eqiad service. But i'm not sure yet how the service even got there.. [19:12:31] (03CR) 10jenkins-bot: Remove mhs.ox.ac.uk from $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/459365 (https://phabricator.wikimedia.org/T203904) (owner: 10Framawiki) [19:13:29] ebernhardson: ouch, that's strange! [19:13:58] even stranger that it happens now [19:14:19] ya [19:14:25] 10Operations, 10monitoring, 10Patch-For-Review: upgrade icinga server to stretch and replace einsteinium - https://phabricator.wikimedia.org/T202782 (10Dzahn) The next issue on icinga1001 on stretch at least when using the traditional init script - Warning: Could not open object cache file '/var/cache/icing... [19:14:55] 10Operations, 10Citoid, 10Services, 10Patch-For-Review, and 3 others: Deploy translation-server-v2 - https://phabricator.wikimedia.org/T201611 (10Mvolz) >>! In T201611#4611416, @akosiaris wrote: > @Mvolz, SRE has a question about this migration. Assuming this gets deployed successfully next quarter, this w... [19:19:15] PROBLEM - Check systemd state on elastic2034 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [19:32:05] RECOVERY - Check systemd state on elastic2022 is OK: OK - running: The system is fully operational [19:35:28] (03PS4) 10Cwhite: monitoring: set mode on host and service configs [puppet] - 10https://gerrit.wikimedia.org/r/462024 (https://phabricator.wikimedia.org/T202782) [19:36:35] RECOVERY - Check systemd state on elastic2033 is OK: OK - running: The system is fully operational [19:36:45] RECOVERY - Check systemd state on elastic2020 is OK: OK - running: The system is fully operational [19:36:45] RECOVERY - Check systemd state on elastic2034 is OK: OK - running: The system is fully operational [19:36:50] !log resetting failed units on elasticsearch codfw [19:36:54] RECOVERY - Check systemd state on elastic2016 is OK: OK - running: The system is fully operational [19:36:54] RECOVERY - Check systemd state on elastic2018 is OK: OK - running: The system is fully operational [19:36:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:47:19] 10Operations, 10Cloud-Services, 10Mail: Create a Cloud VPS SMTP smarthost - https://phabricator.wikimedia.org/T41785 (10herron) Public DNS records have been created, but it looks like like reverse dns will have to wait until T199374 is resolved. This doesn't block the setup of the smarthosts themselves, but... [19:53:00] (03PS1) 10Smalyshev: Switch internal cluster to Kafka event source [puppet] - 10https://gerrit.wikimedia.org/r/462564 [20:00:04] cscott, arlolra, subbu, bearND, halfak, and Amir1: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) Services – Parsoid / Citoid / Mobileapps / ORES / … deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180924T2000). [20:00:10] 10Operations, 10Puppet, 10Analytics-Kanban, 10Beta-Cluster-Infrastructure, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q1): exported puppet resources are not queryable: cannot create grafana graphs of EventLogging running in beta cluster - https://phabricator.wikimedia.org/T204088 (10Ottoma... [20:00:20] no parsoid deploy today [20:00:24] (03PS1) 10Ottomata: Scrape Kafka jmx exporters in deployment-prep [puppet] - 10https://gerrit.wikimedia.org/r/462567 (https://phabricator.wikimedia.org/T204088) [20:02:00] 10Operations, 10Puppet, 10Analytics-Kanban, 10Beta-Cluster-Infrastructure, and 2 others: exported puppet resources are not queryable: cannot create grafana graphs of EventLogging running in beta cluster - https://phabricator.wikimedia.org/T204088 (10Ottomata) I think ^ is what is needed (sorry was at our o... [20:03:23] 10Operations, 10Wikimedia-Logstash, 10User-fgiunchedi, 10User-herron: Logstash hardware expansion - https://phabricator.wikimedia.org/T203169 (10herron) >>! In T203169#4610742, @fgiunchedi wrote: >>>! In T203169#4599357, @herron wrote: >> Technically we could explore multiple ES instances per-host to suppo... [20:17:02] (03PS1) 10Eevans: resbase: increase latency notification thresholds [puppet] - 10https://gerrit.wikimedia.org/r/462569 (https://phabricator.wikimedia.org/T197477) [20:18:47] (03CR) 10Volans: resbase: increase latency notification thresholds (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462569 (https://phabricator.wikimedia.org/T197477) (owner: 10Eevans) [20:18:54] urandom: sorry caught my eye ;0 [20:18:56] ;) [20:21:52] volans: \o/ [20:22:10] (03PS2) 10Eevans: restbase: increase latency notification thresholds [puppet] - 10https://gerrit.wikimedia.org/r/462569 (https://phabricator.wikimedia.org/T197477) [20:23:46] !log bsitzmann@deploy1001 Started deploy [mobileapps/deploy@4be131b]: Update mobileapps to badb463 (T187098 T195838) [20:23:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:23:56] T195838: Document the announcement endpoint config parameters - https://phabricator.wikimedia.org/T195838 [20:23:57] T187098: Extra span at beginning of mobile-sections-lead - https://phabricator.wikimedia.org/T187098 [20:24:05] urandom: but now that I look at the patch itself I've another comment :D [20:25:36] (03CR) 10Volans: "nitpick inline" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462569 (https://phabricator.wikimedia.org/T197477) (owner: 10Eevans) [20:25:39] sorry :) [20:26:55] PROBLEM - MediaWiki memcached error rate on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [5000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [20:27:05] (03PS2) 10Andrew Bogott: Horizon: remove region defs for some deleted projects [puppet] - 10https://gerrit.wikimedia.org/r/462194 [20:27:40] volans: no reason to be sorry, valid nits both [20:28:05] (03CR) 10Andrew Bogott: [C: 032] Horizon: remove region defs for some deleted projects [puppet] - 10https://gerrit.wikimedia.org/r/462194 (owner: 10Andrew Bogott) [20:28:17] !log bsitzmann@deploy1001 Finished deploy [mobileapps/deploy@4be131b]: Update mobileapps to badb463 (T187098 T195838) (duration: 04m 31s) [20:28:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:29:01] volans: indicative of being in too big a hurry, i guess [20:29:14] :) [20:29:14] RECOVERY - MediaWiki memcached error rate on graphite1001 is OK: OK: Less than 40.00% above the threshold [1000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [20:29:44] (03PS3) 10Eevans: restbase: increase latency notification thresholds [puppet] - 10https://gerrit.wikimedia.org/r/462569 (https://phabricator.wikimedia.org/T197477) [20:30:09] (03CR) 10Eevans: restbase: increase latency notification thresholds (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462569 (https://phabricator.wikimedia.org/T197477) (owner: 10Eevans) [20:36:03] (03CR) 10Ppchelko: [C: 031] "There's also a 99p alert for the same endpoint, but I've just searched through the history, and seems 99p never actually fires." [puppet] - 10https://gerrit.wikimedia.org/r/462569 (https://phabricator.wikimedia.org/T197477) (owner: 10Eevans) [20:36:27] (03CR) 10Ppchelko: [C: 04-1] restbase: increase latency notification thresholds [puppet] - 10https://gerrit.wikimedia.org/r/462569 (https://phabricator.wikimedia.org/T197477) (owner: 10Eevans) [20:37:50] (03CR) 10Ppchelko: [C: 04-1] restbase: increase latency notification thresholds (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462569 (https://phabricator.wikimedia.org/T197477) (owner: 10Eevans) [20:38:55] PROBLEM - Varnish traffic drop between 30min ago and now at eqiad on einsteinium is CRITICAL: 58.54 le 60 https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [20:40:05] RECOVERY - Varnish traffic drop between 30min ago and now at eqiad on einsteinium is OK: (C)60 le (W)70 le 77.86 https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [20:40:46] (03CR) 10Ppchelko: [C: 04-1] restbase: increase latency notification thresholds (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462569 (https://phabricator.wikimedia.org/T197477) (owner: 10Eevans) [20:41:52] 10Operations, 10Cloud-Services, 10Mail: Create a Cloud VPS SMTP smarthost - https://phabricator.wikimedia.org/T41785 (10herron) Two new systems have been set up with `role::mail::mx`. A cursory test of outbound mail routing through a smarthost in `project-smtp` works, as long as the recipient does not have... [20:43:59] 10Operations, 10Product-Analytics: Upload shiny-server .deb to our Jessie apt repository - https://phabricator.wikimedia.org/T168967 (10mpopov) [20:46:03] (03CR) 10Eevans: restbase: increase latency notification thresholds (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462569 (https://phabricator.wikimedia.org/T197477) (owner: 10Eevans) [20:47:45] 10Operations, 10netops: analytics1-a VLAN has no DNS for gateway addresses to match other analytics VLANs - https://phabricator.wikimedia.org/T205340 (10chasemp) p:05Triage>03Low [20:47:53] (03PS1) 10Jdlrobson: Remove dead config relating to wgRelatedArticlesEnabledBucketSize [mediawiki-config] - 10https://gerrit.wikimedia.org/r/462573 (https://phabricator.wikimedia.org/T202306) [20:57:20] !log Started fulltext reindex for wikidatawiki [20:57:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:59:22] (03CR) 10Dzahn: [C: 031] mediawiki::web::prod_sites: convert meta.w.o [puppet] - 10https://gerrit.wikimedia.org/r/461396 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:00:04] bawolff and Reedy: Dear deployers, time to do the Weekly Security deployment window deploy. Dont look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180924T2100). [21:00:31] (03CR) 10Dzahn: [C: 031] "existing config is just like the default template plus "uploads are offsite" and "Firefox OS stuff", changing the default also makes sense" [puppet] - 10https://gerrit.wikimedia.org/r/461396 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:01:36] (03PS1) 10Faidon Liambotis: Remove 56.15.185.in-addr.arpa, moved to labs-ns0/1 [dns] - 10https://gerrit.wikimedia.org/r/462574 (https://phabricator.wikimedia.org/T199374) [21:01:42] (03CR) 10Dzahn: [C: 031] "not deleting the meta.wikimedia.org.conf.erb in this case though?" [puppet] - 10https://gerrit.wikimedia.org/r/461396 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:04:41] (03CR) 10Dzahn: [C: 04-1] "what Moritz said, the "uploads are offsite" part is missing. as i saw you adding it as an additional rule in the meta.wikimedia.org change" [puppet] - 10https://gerrit.wikimedia.org/r/461397 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:08:08] (03PS2) 10Imarlier: wmf-config: remove oversampling for Asian countries [mediawiki-config] - 10https://gerrit.wikimedia.org/r/460566 (https://phabricator.wikimedia.org/T204365) [21:08:28] (03CR) 10Imarlier: [C: 032] wmf-config: remove oversampling for Asian countries [mediawiki-config] - 10https://gerrit.wikimedia.org/r/460566 (https://phabricator.wikimedia.org/T204365) (owner: 10Imarlier) [21:13:15] (03Merged) 10jenkins-bot: wmf-config: remove oversampling for Asian countries [mediawiki-config] - 10https://gerrit.wikimedia.org/r/460566 (https://phabricator.wikimedia.org/T204365) (owner: 10Imarlier) [21:13:17] (03PS3) 10GTirloni: cloudvps: add prometheus-openstack-exporter [puppet] - 10https://gerrit.wikimedia.org/r/462455 (https://phabricator.wikimedia.org/T203177) (owner: 10Arturo Borrero Gonzalez) [21:17:39] (03PS4) 10GTirloni: cloudvps: add prometheus-openstack-exporter [puppet] - 10https://gerrit.wikimedia.org/r/462455 (https://phabricator.wikimedia.org/T203177) (owner: 10Arturo Borrero Gonzalez) [21:19:40] (03CR) 10jenkins-bot: wmf-config: remove oversampling for Asian countries [mediawiki-config] - 10https://gerrit.wikimedia.org/r/460566 (https://phabricator.wikimedia.org/T204365) (owner: 10Imarlier) [21:22:45] (03CR) 10Dzahn: [C: 04-1] mediawiki::web::prod_sites: convert wikisource.org (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/461397 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:23:03] !log imarlier@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [[gerrit:460566|Disable NavTiming oversampling from Asian countries, used during Singapore data center rollout (T204365)]] (duration: 00m 50s) [21:23:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:23:11] T204365: Stop oversampling Asian countries - https://phabricator.wikimedia.org/T204365 [21:23:50] (03CR) 10Dzahn: [C: 04-1] "use "upload_rewrite" parameter of mediawiki::web::vhost rather than manually adding it as an addition rewrite?" [puppet] - 10https://gerrit.wikimedia.org/r/461397 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:24:12] (03CR) 10Dzahn: mediawiki::web::prod_sites: convert meta.w.o (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/461396 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:25:32] (03Abandoned) 10Eevans: restbase: increase latency notification thresholds [puppet] - 10https://gerrit.wikimedia.org/r/462569 (https://phabricator.wikimedia.org/T197477) (owner: 10Eevans) [21:33:13] (03CR) 10Dzahn: mediawiki::web::prod_sites: convert wikiversity.org (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462424 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:35:49] (03CR) 10Dzahn: mediawiki::web::prod_sites: convert mediawiki.org (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462425 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:36:31] 10Operations, 10Cloud-VPS, 10DNS, 10Traffic: Inconsistent lists of labs-ns* nameservers - https://phabricator.wikimedia.org/T205344 (10Paladox) [21:36:55] (03CR) 10Dzahn: mediawiki::web::prod_sites: convert wiktionary.org (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462477 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:38:53] (03CR) 10Dzahn: [C: 031] mediawiki::web::prod_sites: convert donate.w.o [puppet] - 10https://gerrit.wikimedia.org/r/462479 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:38:56] 10Operations, 10DNS, 10Traffic, 10WMF-Communications, and 4 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776 (10Varnent) [21:39:39] (03PS1) 10Ottomata: Add analytics-search system user to analytics-search-users group [puppet] - 10https://gerrit.wikimedia.org/r/462580 (https://phabricator.wikimedia.org/T204415) [21:40:20] (03PS2) 10Ottomata: Add analytics-search system user to analytics-search-users group [puppet] - 10https://gerrit.wikimedia.org/r/462580 (https://phabricator.wikimedia.org/T204415) [21:40:29] (03PS3) 10Ottomata: Add analytics-search system user to analytics-search-users group [puppet] - 10https://gerrit.wikimedia.org/r/462580 (https://phabricator.wikimedia.org/T204415) [21:41:01] (03CR) 10Dzahn: mediawiki::web::prod_sites: convert wikinews.org (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462480 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:41:10] (03CR) 10Ottomata: [C: 032] Add analytics-search system user to analytics-search-users group [puppet] - 10https://gerrit.wikimedia.org/r/462580 (https://phabricator.wikimedia.org/T204415) (owner: 10Ottomata) [21:41:37] (03CR) 10Dzahn: [C: 04-1] "sort_urls = typo of short urls" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462480 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:42:15] (03CR) 10Dzahn: [C: 04-1] "sort_urls -> short_urls" [puppet] - 10https://gerrit.wikimedia.org/r/462486 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:42:41] (03CR) 10Dzahn: [C: 04-1] "sort_urls -> short_urls" [puppet] - 10https://gerrit.wikimedia.org/r/462487 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:44:00] (03CR) 10Dzahn: [C: 031] mediawiki::web::prod_sites: convert vote.w.o [puppet] - 10https://gerrit.wikimedia.org/r/462492 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:47:12] (03CR) 10Dzahn: mediawiki::web::prod_sites: convert test.wikidata.org (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462493 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:47:53] (03CR) 10Dzahn: [C: 04-1] "sort_urls -> short_urls" [puppet] - 10https://gerrit.wikimedia.org/r/462495 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [21:52:34] robh, mutante, moritzm: any chance we could get https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/461761/ merged today? (waiting period has passed without objetions https://phabricator.wikimedia.org/T204790 ) [21:57:34] welcome groceryheist! (^ is for him) [22:00:37] Hi [22:02:00] * bd808 hides if groceries [22:02:04] *his [22:02:17] lol [22:18:43] (03PS1) 10Andrew Bogott: Designate policy: allow project admins to create zones [puppet] - 10https://gerrit.wikimedia.org/r/462583 [22:19:27] (03PS1) 10Legoktm: mediawiki: Install php-gd for ZeroBanner [puppet] - 10https://gerrit.wikimedia.org/r/462584 [22:20:18] (03CR) 10Legoktm: "ZeroBanner uses the stil/gd-text library, which requires the gd extension to be installed." [puppet] - 10https://gerrit.wikimedia.org/r/462584 (owner: 10Legoktm) [22:20:48] (03CR) 10Andrew Bogott: [C: 032] Designate policy: allow project admins to create zones [puppet] - 10https://gerrit.wikimedia.org/r/462583 (owner: 10Andrew Bogott) [22:29:40] (03PS1) 10Andrew Bogott: Revert "Designate policy: allow project admins to create zones" [puppet] - 10https://gerrit.wikimedia.org/r/462586 [22:30:46] (03CR) 10Andrew Bogott: [C: 032] Revert "Designate policy: allow project admins to create zones" [puppet] - 10https://gerrit.wikimedia.org/r/462586 (owner: 10Andrew Bogott) [22:35:48] (03PS2) 10Faidon Liambotis: Remove 56.15.185.in-addr.arpa, moved to labs-ns0/1 [dns] - 10https://gerrit.wikimedia.org/r/462574 (https://phabricator.wikimedia.org/T199374) [22:38:06] (03CR) 10Faidon Liambotis: [C: 032] Remove 56.15.185.in-addr.arpa, moved to labs-ns0/1 [dns] - 10https://gerrit.wikimedia.org/r/462574 (https://phabricator.wikimedia.org/T199374) (owner: 10Faidon Liambotis) [22:39:42] jouncebot: now [22:39:42] For the next 0 hour(s) and 20 minute(s): Weekly Security deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180924T2100) [22:39:45] jouncebot: next [22:39:45] In 0 hour(s) and 20 minute(s): Evening SWAT (Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180924T2300) [22:40:25] (03PS2) 10Reedy: Remove DisableAccount from CommonSettings.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461731 (https://phabricator.wikimedia.org/T106067) [22:40:30] (03CR) 10Reedy: [C: 032] Remove DisableAccount from CommonSettings.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461731 (https://phabricator.wikimedia.org/T106067) (owner: 10Reedy) [22:41:49] (03Merged) 10jenkins-bot: Remove DisableAccount from CommonSettings.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461731 (https://phabricator.wikimedia.org/T106067) (owner: 10Reedy) [22:43:16] !log reedy@deploy1001 Synchronized wmf-config/CommonSettings.php: Bye bye DisableAccount T106067 (duration: 00m 51s) [22:43:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:43:24] T106067: Undeploy DisableAccount extension - https://phabricator.wikimedia.org/T106067 [22:44:16] (03CR) 10Jforrester: [C: 032] Remove DisableAccount from InitialiseSettings.php and extension-list [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461732 (https://phabricator.wikimedia.org/T106067) (owner: 10Reedy) [22:44:38] Needs rebasing first? [22:46:20] (03PS1) 10Ayounsi: Revert "Smokeping: replace bast4002 with bast4001" [puppet] - 10https://gerrit.wikimedia.org/r/462587 [22:46:36] (03CR) 10jenkins-bot: Remove DisableAccount from CommonSettings.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461731 (https://phabricator.wikimedia.org/T106067) (owner: 10Reedy) [22:46:43] (03PS2) 10Jforrester: Remove DisableAccount from InitialiseSettings.php and extension-list [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461732 (https://phabricator.wikimedia.org/T106067) (owner: 10Reedy) [22:46:53] (03CR) 10Jforrester: [C: 032] Remove DisableAccount from InitialiseSettings.php and extension-list [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461732 (https://phabricator.wikimedia.org/T106067) (owner: 10Reedy) [22:47:05] (03CR) 10Ayounsi: [C: 032] Revert "Smokeping: replace bast4002 with bast4001" [puppet] - 10https://gerrit.wikimedia.org/r/462587 (owner: 10Ayounsi) [22:47:08] Reedy: Also need to do T158594? [22:47:09] T158594: Once DisableAccount extension is removed: remove all users from the 'inactive' user group and remove it from private.dblist list of permissions - https://phabricator.wikimedia.org/T158594 [22:47:21] (03PS2) 10Ayounsi: Revert "Smokeping: replace bast4002 with bast4001" [puppet] - 10https://gerrit.wikimedia.org/r/462587 [22:47:45] James_F: I'm not sure... I re-enabled it on the wikis that I'd disabled it on before... And then ran it [22:48:11] We can find out easily enough [22:48:21] (03PS2) 10RobH: adding shell user nathante [puppet] - 10https://gerrit.wikimedia.org/r/461760 (https://phabricator.wikimedia.org/T204790) [22:48:40] (03Merged) 10jenkins-bot: Remove DisableAccount from InitialiseSettings.php and extension-list [mediawiki-config] - 10https://gerrit.wikimedia.org/r/461732 (https://phabricator.wikimedia.org/T106067) (owner: 10Reedy)