[00:00:14] PROBLEM - Maps - OSM synchronization lag - eqiad on einsteinium is CRITICAL: 1.728e+05 ge 1.728e+05 https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=11&fullscreen&orgId=1 [00:00:43] PROBLEM - Maps - OSM synchronization lag - codfw on einsteinium is CRITICAL: 1.728e+05 ge 1.728e+05 https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=12&fullscreen&orgId=1 [00:08:23] PROBLEM - puppet last run on druid1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:33:54] RECOVERY - puppet last run on druid1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:10:03] PROBLEM - DPKG on kubestage1002 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [01:54:43] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 2.918 second response time [01:58:13] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:08:24] !log repooling wdqs1003. It has caught up with others [02:08:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:22:54] RECOVERY - Maps - OSM synchronization lag - eqiad on einsteinium is OK: (C)1.728e+05 ge (W)9e+04 ge 8570 https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=11&fullscreen&orgId=1 [02:23:33] RECOVERY - Maps - OSM synchronization lag - codfw on einsteinium is OK: (C)1.728e+05 ge (W)9e+04 ge 8606 https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=12&fullscreen&orgId=1 [03:34:33] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 842.93 seconds [04:03:33] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 157.69 seconds [05:06:54] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 277 bytes in 6.948 second response time [05:10:23] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:40:43] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 9.417 second response time [05:44:03] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:10:50] (03PS7) 10Giuseppe Lavagetto: beta: start using set_handler instead of the proxy passes [puppet] - 10https://gerrit.wikimedia.org/r/469203 [06:10:54] (03CR) 10Giuseppe Lavagetto: [V: 032 C: 032] beta: start using set_handler instead of the proxy passes [puppet] - 10https://gerrit.wikimedia.org/r/469203 (owner: 10Giuseppe Lavagetto) [06:11:54] (03Abandoned) 10Giuseppe Lavagetto: mediawiki::web::beta_sites: enable serving content from php7-fpm [puppet] - 10https://gerrit.wikimedia.org/r/468990 (https://phabricator.wikimedia.org/T206338) (owner: 10Giuseppe Lavagetto) [06:12:28] (03Abandoned) 10Giuseppe Lavagetto: mediawiki::web::vhost: allow serving content from php7 [puppet] - 10https://gerrit.wikimedia.org/r/467878 (https://phabricator.wikimedia.org/T206338) (owner: 10Giuseppe Lavagetto) [06:13:48] (03Abandoned) 10Giuseppe Lavagetto: [PoC] How I'd like the mediawiki vhosts to be [puppet] - 10https://gerrit.wikimedia.org/r/468931 (owner: 10Giuseppe Lavagetto) [06:31:13] PROBLEM - puppet last run on stat1005 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/R/biocLite.R] [06:33:04] PROBLEM - puppet last run on aqs1005 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/gen_fingerprints] [06:46:40] (03PS3) 10Giuseppe Lavagetto: mediawiki: remove now unused common includes [puppet] - 10https://gerrit.wikimedia.org/r/468930 [06:56:43] RECOVERY - puppet last run on stat1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:06] (03CR) 10Giuseppe Lavagetto: [C: 032] "https://puppet-compiler.wmflabs.org/compiler1002/13231/" [puppet] - 10https://gerrit.wikimedia.org/r/468930 (owner: 10Giuseppe Lavagetto) [06:58:44] RECOVERY - puppet last run on aqs1005 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [07:30:27] 10Operations, 10Wikimedia-Site-requests, 10HHVM: Set hhvm.virtual_host[default][always_decode_post_data] = false - https://phabricator.wikimedia.org/T208191 (10Bawolff) [07:43:14] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 7.528 second response time [07:46:17] (03PS2) 10Muehlenhoff: Disable prometheus rsyncd module for now [puppet] - 10https://gerrit.wikimedia.org/r/469630 [07:46:34] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:43] (03CR) 10Muehlenhoff: [C: 032] Disable prometheus rsyncd module for now [puppet] - 10https://gerrit.wikimedia.org/r/469630 (owner: 10Muehlenhoff) [07:51:58] (03PS4) 10Muehlenhoff: Add initial profile for Kerberos client [puppet] - 10https://gerrit.wikimedia.org/r/469853 [07:52:21] 10Operations, 10Wikimedia-Site-requests, 10HHVM: Set hhvm.virtual_host[default][always_decode_post_data] = false - https://phabricator.wikimedia.org/T208191 (10Bawolff) [07:53:14] 10Operations, 10Wikimedia-Site-requests, 10HHVM: Set hhvm.virtual_host[default][always_decode_post_data] = false - https://phabricator.wikimedia.org/T208191 (10Bawolff) For context, my specific issue is that The api dispatch code basically works by doing `$foo = $_POST + $_GET; $action = $foo['action'];` and... [07:53:49] 10Operations, 10ops-codfw: Degraded RAID on ms-be2021 - https://phabricator.wikimedia.org/T208096 (10fgiunchedi) a:03Papaul Hi @Papaul, looks like this controller has a faulty battery, likely will need to have the battery replaced, what do you think? We have seen this issue before on a bunch of ms-be hosts i... [07:56:03] (03CR) 10Muehlenhoff: [C: 032] Add initial profile for Kerberos client [puppet] - 10https://gerrit.wikimedia.org/r/469853 (owner: 10Muehlenhoff) [08:00:04] gilles: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) Performance perception survey (QuickSurveys) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181029T0800). [08:00:29] indeed [08:00:40] (03PS2) 10Muehlenhoff: Convert udp2log::rsyncd to auto_ferm [puppet] - 10https://gerrit.wikimedia.org/r/469627 [08:00:54] !log Deploying time-sensitive backport to QuickSurveys [08:00:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:04:16] (03CR) 10Muehlenhoff: [C: 032] Convert udp2log::rsyncd to auto_ferm [puppet] - 10https://gerrit.wikimedia.org/r/469627 (owner: 10Muehlenhoff) [08:07:15] !log reformat ms-be1042 xfs filesystems - T199198 [08:07:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:07:19] T199198: Some swift filesystems reporting negative disk usage - https://phabricator.wikimedia.org/T199198 [08:07:32] (03CR) 10Gilles: [C: 032] Enable performance perception survey shuffling [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470065 (https://phabricator.wikimedia.org/T208088) (owner: 10Gilles) [08:08:39] (03PS2) 10Muehlenhoff: Use auto_ferm for profile::analytics::database::meta::backup_dest [puppet] - 10https://gerrit.wikimedia.org/r/469635 [08:08:42] (03Merged) 10jenkins-bot: Enable performance perception survey shuffling [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470065 (https://phabricator.wikimedia.org/T208088) (owner: 10Gilles) [08:09:02] jouncebot: now [08:09:02] For the next 0 hour(s) and 20 minute(s): Performance perception survey (QuickSurveys) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181029T0800) [08:09:07] jouncebot: next [08:09:07] In 2 hour(s) and 20 minute(s): Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181029T1030) [08:11:53] 10Operations, 10DBA, 10monitoring: Create a script to regenerate prometheus mysqld exporter listing that works with puppetdb - https://phabricator.wikimedia.org/T145072 (10jcrespo) I think this should be moved to zarcillo mariadb metadata database, and centralize there the active database control (substituti... [08:14:24] RECOVERY - Check systemd state on db1117 is OK: OK - running: The system is fully operational [08:15:22] 10Operations, 10Cloud-Services, 10Cloud-VPS: labs precise and jessie instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi Haven't seen this either lately, resolving. [08:17:00] jouncebot: refresh [08:17:01] I refreshed my knowledge about deployments. [08:17:02] jouncebot: next [08:17:02] In 0 hour(s) and 12 minute(s): Cleaning up Wikibase mw-config (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181029T0830) [08:19:39] (03CR) 10Filippo Giunchedi: [C: 031] Remove Diamond from LVSes [puppet] - 10https://gerrit.wikimedia.org/r/469594 (https://phabricator.wikimedia.org/T183454) (owner: 10Muehlenhoff) [08:19:48] (03CR) 10Filippo Giunchedi: [C: 031] Remove Pybal Diamond collector [puppet] - 10https://gerrit.wikimedia.org/r/469593 (https://phabricator.wikimedia.org/T183454) (owner: 10Muehlenhoff) [08:21:58] (03CR) 10jenkins-bot: Enable performance perception survey shuffling [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470065 (https://phabricator.wikimedia.org/T208088) (owner: 10Gilles) [08:27:25] (03PS1) 10Elukey: Allow tuning of the Yarn RM's zookeeper session timeout [puppet/cdh] - 10https://gerrit.wikimedia.org/r/470327 (https://phabricator.wikimedia.org/T206943) [08:27:46] (03CR) 10jerkins-bot: [V: 04-1] Allow tuning of the Yarn RM's zookeeper session timeout [puppet/cdh] - 10https://gerrit.wikimedia.org/r/470327 (https://phabricator.wikimedia.org/T206943) (owner: 10Elukey) [08:29:04] !log gilles@deploy1001 Started scap: T208088 Enable performance QuickSurvey shuffling [08:29:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:29:08] T208088: Add ability to randomize display order of answer options in QuickSurveys - https://phabricator.wikimedia.org/T208088 [08:30:04] addshore: It is that lovely time of the day again! You are hereby commanded to deploy Cleaning up Wikibase mw-config. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181029T0830). [08:30:15] gilles: let me know when you are done :) [08:30:30] addshore: sure thing, scap just started [08:30:51] thanks! [08:31:22] (03PS2) 10Elukey: Allow tuning of the Yarn RM's zookeeper session timeout [puppet/cdh] - 10https://gerrit.wikimedia.org/r/470327 (https://phabricator.wikimedia.org/T206943) [08:31:24] (03PS1) 10Elukey: Fix some linting warnings [puppet/cdh] - 10https://gerrit.wikimedia.org/r/470328 [08:32:20] (03PS3) 10Addshore: Wikibase, create and use wmgWikibaseMaxSerializedEntitySize [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470149 [08:32:24] (03CR) 10Addshore: [C: 032] Wikibase, create and use wmgWikibaseMaxSerializedEntitySize [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470149 (owner: 10Addshore) [08:32:37] gilles: aaah it s afull scap? [08:32:46] yeah, I'm adding new files [08:33:30] 10Operations, 10ops-eqiad, 10DBA: db1117 went away - https://phabricator.wikimedia.org/T208150 (10jcrespo) so the error happens because it is tried to be run manually, which it not a big deal if it errors out- just delete any file you may have added. I ran `systemctl disable is it https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/QuickSurveys/+/470257/ ? [08:33:36] 10Operations, 10Beta-Cluster-Infrastructure, 10DNS, 10Traffic, and 4 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10fgiunchedi) [08:33:52] I don't think that should need a full full scap, only a scap of the QuickSurveys directory [08:33:54] addshore: yes, and a config change [08:34:05] (03CR) 10Elukey: [C: 032] Allow tuning of the Yarn RM's zookeeper session timeout [puppet/cdh] - 10https://gerrit.wikimedia.org/r/470327 (https://phabricator.wikimedia.org/T206943) (owner: 10Elukey) [08:34:11] (03CR) 10Elukey: [C: 032] Fix some linting warnings [puppet/cdh] - 10https://gerrit.wikimedia.org/r/470328 (owner: 10Elukey) [08:34:18] i expect the full scap will end up taking ~40 mins? but the dir scap should only take a couple [08:34:22] ah, didn't know that was possible, didn't see that mentioned in the "how to deploy code" wiki page [08:34:44] 10Operations, 10ops-eqiad, 10DBA: db1117 went away - https://phabricator.wikimedia.org/T208150 (10jcrespo) [08:35:05] ah it's if you're "adding directories" duh... [08:35:09] sorry about that [08:35:14] :D [08:35:27] you should be able to stop the sync and just do a dir sync [08:35:33] cool [08:35:34] !log gilles@deploy1001 sync aborted: T208088 Enable performance QuickSurvey shuffling (duration: 06m 30s) [08:35:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:35:38] T208088: Add ability to randomize display order of answer options in QuickSurveys - https://phabricator.wikimedia.org/T208088 [08:35:54] !log gilles@deploy1001 Started scap: T208088 Enable performance QuickSurvey shuffling [08:35:54] !log gilles@deploy1001 sync aborted: T208088 Enable performance QuickSurvey shuffling (duration: 00m 00s) [08:35:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:36:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:37:03] i changed the heading of that section on the page to include the word "directories" [08:38:48] CDB stuff spewed some errors, but it kept going, is that expected? [08:38:50] !log gilles@deploy1001 Synchronized php-1.33.0-wmf.1/extensions/QuickSurveys: T208088 Add ability to shuffle answers display order (duration: 01m 51s) [08:38:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:39:12] gilles: when cancelling the previous one? [08:39:22] nope, when running the actual sync-file [08:40:02] paste them somewhere? [08:40:06] 10Operations: Access requests process: People sometimes specify hostnames instead of admin groups in access requests - https://phabricator.wikimedia.org/T207754 (10MoritzMuehlenhoff) Can you clarify the scope/intent of this task, it's not obvious to me. Is this about the fact that some access::group permissions... [08:40:43] addshore: https://phabricator.wikimedia.org/P7728 [08:40:46] the sync-file won't be syncing cdb files so should be nothing to worry about [08:41:24] should be fine, as the next scpa sync will re do all of those actions :) [08:42:33] !log gilles@deploy1001 Synchronized wmf-config/InitialiseSettings.php: T208088 Enable performance perception survey shuffling (duration: 00m 47s) [08:42:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:42:37] T208088: Add ability to randomize display order of answer options in QuickSurveys - https://phabricator.wikimedia.org/T208088 [08:42:58] (03CR) 10Addshore: [C: 032] Wikibase, create and use wmgWikibaseMaxSerializedEntitySize [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470149 (owner: 10Addshore) [08:43:36] addshore: I am done [08:43:44] gilles: amazing, thanks! [08:43:52] thanks for the help [08:43:58] np [08:44:18] (03Merged) 10jenkins-bot: Wikibase, create and use wmgWikibaseMaxSerializedEntitySize [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470149 (owner: 10Addshore) [08:44:51] (03PS1) 10Elukey: Rename Yarn zookeeper session timeout property to include "ms" [puppet/cdh] - 10https://gerrit.wikimedia.org/r/470333 [08:45:07] (03PS3) 10Addshore: Wikibase, Split specialSiteLinkGroups and manage from IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470150 [08:45:43] (03CR) 10jenkins-bot: Wikibase, create and use wmgWikibaseMaxSerializedEntitySize [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470149 (owner: 10Addshore) [08:46:17] (03CR) 10Elukey: [V: 032 C: 032] Rename Yarn zookeeper session timeout property to include "ms" [puppet/cdh] - 10https://gerrit.wikimedia.org/r/470333 (owner: 10Elukey) [08:49:29] meh, looks like because the cdb files were stopped part way through their regeneration scap doesn't like them now, so running a "scap sync-l10n" for the deployed branch... [08:49:58] !log addshore@deploy1001 sync-l10n aborted: (no justification provided) (duration: 01m 19s) [08:50:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:50:25] looks like that won't work either and I just have to do the full sync anyway... [08:50:32] !log addshore@deploy1001 Started scap: (no justification provided) [08:50:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:50:36] !log addshore@deploy1001 sync aborted: (no justification provided) (duration: 00m 03s) [08:50:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:50:43] (03PS1) 10Elukey: profile::hadoop::common: raise Yarn zookeeper timeout to 20s [puppet] - 10https://gerrit.wikimedia.org/r/470336 (https://phabricator.wikimedia.org/T206943) [08:51:00] (03PS1) 10Addshore: Revert "Wikibase, create and use wmgWikibaseMaxSerializedEntitySize" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470337 [08:51:04] (03CR) 10Addshore: [C: 032] Revert "Wikibase, create and use wmgWikibaseMaxSerializedEntitySize" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470337 (owner: 10Addshore) [08:52:07] (03Merged) 10jenkins-bot: Revert "Wikibase, create and use wmgWikibaseMaxSerializedEntitySize" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470337 (owner: 10Addshore) [08:52:35] !log addshore@deploy1001 Started scap: sync with no changes [08:52:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:52:42] 10Operations, 10ops-eqiad, 10DBA: db1117 went away - https://phabricator.wikimedia.org/T208150 (10Banyek) What cli command did you use for getting those logs? [08:55:14] (03CR) 10Elukey: [C: 032] "https://puppet-compiler.wmflabs.org/compiler1002/13235/" [puppet] - 10https://gerrit.wikimedia.org/r/470336 (https://phabricator.wikimedia.org/T206943) (owner: 10Elukey) [08:55:25] (03PS5) 10Giuseppe Lavagetto: mediawiki: add httpd class, alternative to mediawiki::web [puppet] - 10https://gerrit.wikimedia.org/r/467643 [08:55:27] (03PS5) 10Giuseppe Lavagetto: mediawiki::webserver: introduce profile, use it on mwdebug* [puppet] - 10https://gerrit.wikimedia.org/r/467644 [08:56:46] !log restart yarn on an-master100[1,2] to pick up new zookeeper timeout settings (10s -> 20s) - T206943 [08:56:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:56:50] T206943: JVM pauses cause Yarn master to failover - https://phabricator.wikimedia.org/T206943 [08:58:00] (03CR) 10Ema: [C: 031] Remove Pybal Diamond collector [puppet] - 10https://gerrit.wikimedia.org/r/469593 (https://phabricator.wikimedia.org/T183454) (owner: 10Muehlenhoff) [08:58:09] (03CR) 10Ema: [C: 031] Remove Diamond from LVSes [puppet] - 10https://gerrit.wikimedia.org/r/469594 (https://phabricator.wikimedia.org/T183454) (owner: 10Muehlenhoff) [08:59:47] 10Operations, 10User-jijiki: Redefine privileges and access for perf-roots group - https://phabricator.wikimedia.org/T207666 (10jijiki) p:05Triage>03Normal [09:02:02] 10Operations, 10monitoring, 10User-Joe, 10Wikimedia-Incident: Monitor redis memory/disk usage - https://phabricator.wikimedia.org/T110169 (10jijiki) Should we merge this with T148637 ? [09:03:09] (03CR) 10jenkins-bot: Revert "Wikibase, create and use wmgWikibaseMaxSerializedEntitySize" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470337 (owner: 10Addshore) [09:07:14] !log addshore@deploy1001 Finished scap: sync with no changes (duration: 14m 39s) [09:07:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:10:04] (03PS6) 10Giuseppe Lavagetto: mediawiki::webserver: introduce profile, use it on mwdebug* [puppet] - 10https://gerrit.wikimedia.org/r/467644 [09:12:24] (03PS1) 10Addshore: Revert "Revert "Wikibase, create and use wmgWikibaseMaxSerializedEntitySize"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470339 [09:12:29] (03CR) 10Addshore: [C: 032] Revert "Revert "Wikibase, create and use wmgWikibaseMaxSerializedEntitySize"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470339 (owner: 10Addshore) [09:14:18] (03Merged) 10jenkins-bot: Revert "Revert "Wikibase, create and use wmgWikibaseMaxSerializedEntitySize"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470339 (owner: 10Addshore) [09:14:40] 10Operations, 10Icinga, 10monitoring, 10Patch-For-Review: Concerns about icinga1001 check latency - https://phabricator.wikimedia.org/T208066 (10fgiunchedi) >>! In T208066#4697985, @Dzahn wrote: > other things i have tried but not puppetized so far: > > (point 5 from the tuning guide, Max Reaper Time.) >... [09:14:45] (03PS7) 10Giuseppe Lavagetto: mediawiki::webserver: introduce profile, use it on mwdebug* [puppet] - 10https://gerrit.wikimedia.org/r/467644 [09:17:34] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 3.985 second response time [09:17:52] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: PT1/2 [[gerrit:470339]] Introduce wmgWikibaseMaxSerializedEntitySize (duration: 00m 47s) [09:18:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:18:39] (03CR) 10jenkins-bot: Revert "Revert "Wikibase, create and use wmgWikibaseMaxSerializedEntitySize"" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470339 (owner: 10Addshore) [09:19:04] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: PT2/2 [[gerrit:470339]] Introduce wmgWikibaseMaxSerializedEntitySize (duration: 00m 46s) [09:19:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:19:08] (03PS1) 10Gilles: Double performance perception survey sampling on ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470340 (https://phabricator.wikimedia.org/T187299) [09:19:38] addshore: if you happen to finish your window early, let me know [09:21:04] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:17] gilles: acl [09:21:18] ack [09:23:32] (03PS4) 10Addshore: Wikibase, Split specialSiteLinkGroups and manage from IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470150 [09:23:35] (03CR) 10Addshore: [C: 032] Wikibase, Split specialSiteLinkGroups and manage from IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470150 (owner: 10Addshore) [09:24:50] (03Merged) 10jenkins-bot: Wikibase, Split specialSiteLinkGroups and manage from IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470150 (owner: 10Addshore) [09:29:00] (03PS8) 10Banyek: mariadb: table checker for monitoring data drift [puppet] - 10https://gerrit.wikimedia.org/r/469889 (https://phabricator.wikimedia.org/T207253) [09:29:37] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: PT1/2 [[gerrit:470150]] Wikibase, Split specialSiteLinkGroups and manage from IS.php (duration: 00m 47s) [09:29:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:30:09] !log resume cache hosts rolling reboots for kernel/microcode updates T203011 [09:30:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:30:41] (03PS6) 10Giuseppe Lavagetto: mediawiki: add httpd class, alternative to mediawiki::web [puppet] - 10https://gerrit.wikimedia.org/r/467643 [09:30:42] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: PT2/2 [[gerrit:470150]] Wikibase, Split specialSiteLinkGroups and manage from IS.php (duration: 00m 46s) [09:30:43] (03PS8) 10Giuseppe Lavagetto: mediawiki::webserver: introduce profile, use it on mwdebug* [puppet] - 10https://gerrit.wikimedia.org/r/467644 [09:30:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:31:01] (03PS3) 10Addshore: Wikibase, move wmgWBSiteLinkGroups to IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470151 [09:31:07] (03CR) 10Addshore: [C: 032] Wikibase, move wmgWBSiteLinkGroups to IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470151 (owner: 10Addshore) [09:32:04] jouncebot: now [09:32:04] For the next 0 hour(s) and 57 minute(s): Cleaning up Wikibase mw-config (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181029T0830) [09:32:48] (03Merged) 10jenkins-bot: Wikibase, move wmgWBSiteLinkGroups to IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470151 (owner: 10Addshore) [09:33:49] (03PS4) 10Addshore: Wikibase, kill $wmgWBSharedSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470152 [09:33:51] (03CR) 10Filippo Giunchedi: create rsyslog::ship_logfile - simplified logstash shipper via kafka (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/469945 (https://phabricator.wikimedia.org/T206454) (owner: 10Herron) [09:34:13] (03CR) 10Addshore: [C: 032] Wikibase, kill $wmgWBSharedSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470152 (owner: 10Addshore) [09:34:15] (03CR) 10jenkins-bot: Wikibase, Split specialSiteLinkGroups and manage from IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470150 (owner: 10Addshore) [09:34:17] (03CR) 10jenkins-bot: Wikibase, move wmgWBSiteLinkGroups to IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470151 (owner: 10Addshore) [09:34:26] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: PT1/2 [[gerrit:470151]] Wikibase, move wmgWBSiteLinkGroups to IS.php (duration: 00m 47s) [09:34:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:35:25] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: PT2/2 [[gerrit:470151]] Wikibase, move wmgWBSiteLinkGroups to IS.php (duration: 00m 46s) [09:35:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:35:48] (03Merged) 10jenkins-bot: Wikibase, kill $wmgWBSharedSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470152 (owner: 10Addshore) [09:37:14] (03PS3) 10Addshore: Wikibase, define $wgExtraNamespaces in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470153 [09:37:17] (03CR) 10Addshore: [C: 032] Wikibase, define $wgExtraNamespaces in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470153 (owner: 10Addshore) [09:37:33] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: [[gerrit:470152]] Wikibase, kill $wmgWBSharedSettings (duration: 00m 47s) [09:37:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:38:29] (03Merged) 10jenkins-bot: Wikibase, define $wgExtraNamespaces in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470153 (owner: 10Addshore) [09:40:04] (03PS3) 10Addshore: Wikibase, put all wgNamespaceAliases in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470154 [09:40:08] (03CR) 10Addshore: [C: 032] Wikibase, put all wgNamespaceAliases in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470154 (owner: 10Addshore) [09:40:40] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [[gerrit:470153]] PT1/2 Wikibase, define $wgExtraNamespaces in IS.php (duration: 00m 46s) [09:40:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:41:40] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: [[gerrit:470153]] PT2/2 Wikibase, define $wgExtraNamespaces in IS.php (duration: 00m 47s) [09:41:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:41:47] (03Merged) 10jenkins-bot: Wikibase, put all wgNamespaceAliases in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470154 (owner: 10Addshore) [09:43:11] !log addshore@deploy1001 sync-file aborted: [[gerrit:470154]] PT1/2 (duration: 00m 00s) [09:43:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:44:07] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [[gerrit:470154]] PT1/2 Wikibase, put all wgNamespaceAliases in IS.php (duration: 00m 47s) [09:44:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:44:15] (03PS3) 10Addshore: Wikibase, create and use wmgWikibaseClientInjectRecentChanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470155 [09:44:20] (03PS4) 10Addshore: Wikibase, create and use wmgWikibaseClientInjectRecentChanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470155 [09:44:25] (03CR) 10Addshore: [C: 032] Wikibase, create and use wmgWikibaseClientInjectRecentChanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470155 (owner: 10Addshore) [09:45:25] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: [[gerrit:470154]] PT2/2 Wikibase, put all wgNamespaceAliases in IS.php (duration: 00m 46s) [09:45:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:45:46] (03Merged) 10jenkins-bot: Wikibase, create and use wmgWikibaseClientInjectRecentChanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470155 (owner: 10Addshore) [09:49:12] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: pt1/3 [[gerrit:470155]] Wikibase, create and use wmgWikibaseClientInjectRecentChanges (duration: 00m 47s) [09:49:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:50:01] (03CR) 10jenkins-bot: Wikibase, kill $wmgWBSharedSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470152 (owner: 10Addshore) [09:50:03] (03CR) 10jenkins-bot: Wikibase, define $wgExtraNamespaces in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470153 (owner: 10Addshore) [09:50:05] (03CR) 10jenkins-bot: Wikibase, put all wgNamespaceAliases in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470154 (owner: 10Addshore) [09:50:07] (03CR) 10jenkins-bot: Wikibase, create and use wmgWikibaseClientInjectRecentChanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470155 (owner: 10Addshore) [09:50:10] (03PS3) 10Addshore: Wikibase, remove unused wmgWikibaseClientSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470156 [09:50:17] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: pt2/3 [[gerrit:470155]] Wikibase, create and use wmgWikibaseClientInjectRecentChanges (duration: 00m 47s) [09:50:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:50:42] (03CR) 10Addshore: [C: 032] Wikibase, remove unused wmgWikibaseClientSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470156 (owner: 10Addshore) [09:51:12] !log addshore@deploy1001 Synchronized wmf-config/Wikibase-production.php: pt3/3 [[gerrit:470155]] Wikibase, create and use wmgWikibaseClientInjectRecentChanges (duration: 00m 46s) [09:51:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:51:46] jouncebot: next [09:51:46] In 0 hour(s) and 38 minute(s): Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181029T1030) [09:51:59] (03Merged) 10jenkins-bot: Wikibase, remove unused wmgWikibaseClientSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470156 (owner: 10Addshore) [09:53:24] PROBLEM - High lag on wdqs1003 is CRITICAL: 3642 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [09:53:28] (03PS2) 10Elukey: profile::statistics::private: move geoip archive to another dir [puppet] - 10https://gerrit.wikimedia.org/r/469890 (https://phabricator.wikimedia.org/T208028) [09:54:09] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: PT1/2 [[gerrit:470156]] Wikibase, remove unused wmgWikibaseClientSettings (duration: 00m 47s) [09:54:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:54:34] (03PS3) 10Addshore: Wikibase, Remove unused wmgUseWikibaseQualityExternalValidation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470158 [09:54:37] (03CR) 10Addshore: [C: 032] Wikibase, Remove unused wmgUseWikibaseQualityExternalValidation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470158 (owner: 10Addshore) [09:55:08] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: PT2/2 [[gerrit:470156]] Wikibase, remove unused wmgWikibaseClientSettings (duration: 00m 47s) [09:55:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:55:52] (03Merged) 10jenkins-bot: Wikibase, Remove unused wmgUseWikibaseQualityExternalValidation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470158 (owner: 10Addshore) [09:56:12] (03PS2) 10Muehlenhoff: Remove Pybal Diamond collector [puppet] - 10https://gerrit.wikimedia.org/r/469593 (https://phabricator.wikimedia.org/T183454) [09:56:40] (03PS3) 10Addshore: Wikibase, add IS.php setting for each possible extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470159 [09:56:57] (03CR) 10Muehlenhoff: [C: 032] Remove Pybal Diamond collector [puppet] - 10https://gerrit.wikimedia.org/r/469593 (https://phabricator.wikimedia.org/T183454) (owner: 10Muehlenhoff) [09:57:14] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [[gerrit:470158]] Wikibase, Remove unused wmgUseWikibaseQualityExternalValidation (duration: 00m 47s) [09:57:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:57:36] (03CR) 10Addshore: [C: 032] Wikibase, add IS.php setting for each possible extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470159 (owner: 10Addshore) [09:58:45] (03Merged) 10jenkins-bot: Wikibase, add IS.php setting for each possible extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470159 (owner: 10Addshore) [10:00:47] (03PS3) 10Addshore: Wikibase.php, move a bunch of config into 'clean' area [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470160 [10:00:51] (03CR) 10Addshore: [C: 032] Wikibase.php, move a bunch of config into 'clean' area [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470160 (owner: 10Addshore) [10:01:23] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [[gerrit:470159]] PT1/2 Wikibase, add IS.php setting for each possible extension (duration: 00m 47s) [10:01:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:02:05] (03Merged) 10jenkins-bot: Wikibase.php, move a bunch of config into 'clean' area [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470160 (owner: 10Addshore) [10:02:24] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: [[gerrit:470159]] PT2/2 Wikibase, add IS.php setting for each possible extension (duration: 00m 47s) [10:02:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:03:08] (03PS1) 10Filippo Giunchedi: rsyslog: add prometheus-rsyslog-exporter support [puppet] - 10https://gerrit.wikimedia.org/r/470345 (https://phabricator.wikimedia.org/T205862) [10:03:10] (03PS1) 10Filippo Giunchedi: WIP: temporary workaround co-installability of two roles [puppet] - 10https://gerrit.wikimedia.org/r/470346 [10:03:20] !log starting to switch wdqs1003 and wdqs1006 - T207947 [10:03:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:03:24] T207947: Switch wdqs1003 with one of the internal wdqs cluster - https://phabricator.wikimedia.org/T207947 [10:03:28] (03PS3) 10Gehel: wdqs: remove wdqs1006 from internal cluster [puppet] - 10https://gerrit.wikimedia.org/r/469685 (https://phabricator.wikimedia.org/T207947) [10:03:43] PROBLEM - Check systemd state on an-coord1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [10:03:48] (03CR) 10jerkins-bot: [V: 04-1] rsyslog: add prometheus-rsyslog-exporter support [puppet] - 10https://gerrit.wikimedia.org/r/470345 (https://phabricator.wikimedia.org/T205862) (owner: 10Filippo Giunchedi) [10:03:52] (03PS3) 10Addshore: Wikibase, Create and use wmgWikibaseRepoStatementSections [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470161 [10:04:03] gilles: just a couple more, should have a little time left at the end of the slot for you [10:04:10] addshore: thanks :) [10:04:24] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: [[gerrit:470160]] Wikibase.php, move a bunch of config into clean area NOOP (duration: 00m 47s) [10:04:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:04:36] (03CR) 10Addshore: [C: 032] Wikibase, Create and use wmgWikibaseRepoStatementSections [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470161 (owner: 10Addshore) [10:05:35] (03CR) 10jenkins-bot: Wikibase, remove unused wmgWikibaseClientSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470156 (owner: 10Addshore) [10:05:37] (03CR) 10jenkins-bot: Wikibase, Remove unused wmgUseWikibaseQualityExternalValidation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470158 (owner: 10Addshore) [10:05:39] (03CR) 10jenkins-bot: Wikibase, add IS.php setting for each possible extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470159 (owner: 10Addshore) [10:05:41] (03CR) 10jenkins-bot: Wikibase.php, move a bunch of config into 'clean' area [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470160 (owner: 10Addshore) [10:05:48] (03Merged) 10jenkins-bot: Wikibase, Create and use wmgWikibaseRepoStatementSections [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470161 (owner: 10Addshore) [10:06:02] (03CR) 10jenkins-bot: Wikibase, Create and use wmgWikibaseRepoStatementSections [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470161 (owner: 10Addshore) [10:06:49] (03PS2) 10Gilles: Double performance perception survey sampling on ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470340 (https://phabricator.wikimedia.org/T187299) [10:07:11] on the last 3 syncs [10:07:15] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings.php: PT1/3 [[gerrit:470161]] Wikibase, Create and use wmgWikibaseRepoStatementSections (duration: 00m 50s) [10:07:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:07:54] (03PS2) 10Filippo Giunchedi: WIP: temporary workaround co-installability of two roles [puppet] - 10https://gerrit.wikimedia.org/r/470346 [10:07:56] (03PS2) 10Filippo Giunchedi: rsyslog: add prometheus-rsyslog-exporter support [puppet] - 10https://gerrit.wikimedia.org/r/470345 (https://phabricator.wikimedia.org/T205862) [10:08:07] (03CR) 10Gehel: [C: 032] wdqs: remove wdqs1006 from internal cluster [puppet] - 10https://gerrit.wikimedia.org/r/469685 (https://phabricator.wikimedia.org/T207947) (owner: 10Gehel) [10:08:15] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: PT2/3 [[gerrit:470161]] Wikibase, Create and use wmgWikibaseRepoStatementSections (duration: 00m 47s) [10:08:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:09:14] !log addshore@deploy1001 Synchronized wmf-config/Wikibase-production.php: PT3/3 [[gerrit:470161]] Wikibase, Create and use wmgWikibaseRepoStatementSections (duration: 00m 47s) [10:09:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:10:33] !log addshore@deploy1001 Synchronized wmf-config: final sync (duration: 00m 47s) [10:10:35] gilles: its all yours [10:10:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:10:38] jouncebot: next [10:10:38] In 0 hour(s) and 19 minute(s): Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181029T1030) [10:10:56] (03CR) 10Gilles: [C: 032] Double performance perception survey sampling on ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470340 (https://phabricator.wikimedia.org/T187299) (owner: 10Gilles) [10:12:01] !log Moving l10n_cache-ti.cdb files on deploy1001: sudo -u l10nupdate mv l10n_cache-ti.cdb.json l10n_cache-ti.cdb.json-back-T208196 && sudo -u l10nupdate mv l10n_cache-ti.cdb.MD5 l10n_cache-ti.cdb.MD5-back-T208196 # T208196 [10:12:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:12:08] T208196: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 [10:12:11] (03Merged) 10jenkins-bot: Double performance perception survey sampling on ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470340 (https://phabricator.wikimedia.org/T187299) (owner: 10Gilles) [10:13:06] (03PS3) 10Gehel: wdqs: add wdqs1006 to public cluster [puppet] - 10https://gerrit.wikimedia.org/r/469686 (https://phabricator.wikimedia.org/T207947) [10:13:22] (03PS7) 10Giuseppe Lavagetto: mediawiki: add httpd class, alternative to mediawiki::web [puppet] - 10https://gerrit.wikimedia.org/r/467643 [10:13:24] (03PS9) 10Giuseppe Lavagetto: mediawiki::webserver: introduce profile, use it on mwdebug* [puppet] - 10https://gerrit.wikimedia.org/r/467644 [10:13:27] (03PS1) 10Giuseppe Lavagetto: httpd: add httpd::env [puppet] - 10https://gerrit.wikimedia.org/r/470347 [10:14:39] (03CR) 10Gehel: [C: 032] wdqs: add wdqs1006 to public cluster [puppet] - 10https://gerrit.wikimedia.org/r/469686 (https://phabricator.wikimedia.org/T207947) (owner: 10Gehel) [10:15:59] !log gilles@deploy1001 Synchronized wmf-config/InitialiseSettings.php: T187299 Double sampling of performance perception survey on ruwiki (duration: 00m 47s) [10:16:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:18:50] (03PS3) 10Gehel: wdqs: remove wdqs1003 from public cluster [puppet] - 10https://gerrit.wikimedia.org/r/469687 (https://phabricator.wikimedia.org/T207947) [10:18:52] (03CR) 10Gehel: [C: 032] wdqs: remove wdqs1003 from public cluster [puppet] - 10https://gerrit.wikimedia.org/r/469687 (https://phabricator.wikimedia.org/T207947) (owner: 10Gehel) [10:19:34] PROBLEM - puppet last run on labtestservices2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:19:49] (03PS3) 10Gehel: wdqs: add wdqs1003 to internal cluster [puppet] - 10https://gerrit.wikimedia.org/r/469688 (https://phabricator.wikimedia.org/T207947) [10:20:59] (03CR) 10Gehel: [C: 032] wdqs: add wdqs1003 to internal cluster [puppet] - 10https://gerrit.wikimedia.org/r/469688 (https://phabricator.wikimedia.org/T207947) (owner: 10Gehel) [10:21:07] (03CR) 10jenkins-bot: Double performance perception survey sampling on ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470340 (https://phabricator.wikimedia.org/T187299) (owner: 10Gilles) [10:21:21] (03PS1) 10Addshore: Wikibase, move dispatching settings to IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470348 [10:21:21] !log Restore ti l10n files on deploy1001:/srv/mediawiki-staging/php-1.33.0-wmf.1/cache/l10n/upstream # T208196 [10:21:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:21:26] T208196: l10nupdate aborts due to l10n_cache-ti.cdb.json being truncated - https://phabricator.wikimedia.org/T208196 [10:21:31] hashar: fixed it? :) [10:22:55] !log switch wdqs1003 and wdqs1006 completed, wdqs1003 still depooled to catch up on update lag - T207947 [10:22:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:22:58] T207947: Switch wdqs1003 with one of the internal wdqs cluster - https://phabricator.wikimedia.org/T207947 [10:23:07] (03PS1) 10Addshore: Wikibase, Move $wgPropertySuggesterMinProbability to IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470349 [10:30:05] jan_drewniak: It is that lovely time of the day again! You are hereby commanded to deploy Wikimedia Portals Update. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181029T1030). [10:30:21] (03PS2) 10Muehlenhoff: Remove Diamond from LVSes [puppet] - 10https://gerrit.wikimedia.org/r/469594 (https://phabricator.wikimedia.org/T183454) [10:31:38] (03CR) 10Muehlenhoff: [C: 032] Remove Diamond from LVSes [puppet] - 10https://gerrit.wikimedia.org/r/469594 (https://phabricator.wikimedia.org/T183454) (owner: 10Muehlenhoff) [10:33:28] (03PS1) 10Addshore: Wikibase, Move property suggester settings to IS files [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470351 [10:34:24] PROBLEM - High lag on wdqs1003 is CRITICAL: 3918 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [10:35:35] ACKNOWLEDGEMENT - High lag on wdqs1003 is CRITICAL: 3978 ge 3600 Gehel depooled, should recover soon https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [10:35:44] PROBLEM - puppet last run on lvs2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:35:44] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:35:52] that's me, fixing [10:36:13] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:36:47] (03PS1) 10Muehlenhoff: Revert "Remove Diamond from LVSes" [puppet] - 10https://gerrit.wikimedia.org/r/470352 [10:37:31] (03CR) 10Muehlenhoff: [C: 032] Revert "Remove Diamond from LVSes" [puppet] - 10https://gerrit.wikimedia.org/r/470352 (owner: 10Muehlenhoff) [10:37:33] (03PS1) 10Ema: check_vcl_reload: no unknowns if reload-vcl still has to run [puppet] - 10https://gerrit.wikimedia.org/r/470353 (https://phabricator.wikimedia.org/T206950) [10:38:15] (03CR) 10Fdans: [C: 031] "There's a case to be made that maybe we don't need these files both in hdfs and in disk, but we don't have to decide that now. This patch " [puppet] - 10https://gerrit.wikimedia.org/r/469890 (https://phabricator.wikimedia.org/T208028) (owner: 10Elukey) [10:38:34] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:39:08] (03PS2) 10Ema: check_vcl_reload: no unknowns if reload-vcl still has to run [puppet] - 10https://gerrit.wikimedia.org/r/470353 (https://phabricator.wikimedia.org/T206950) [10:39:56] (03PS3) 10Elukey: profile::statistics::private: move geoip archive to another dir [puppet] - 10https://gerrit.wikimedia.org/r/469890 (https://phabricator.wikimedia.org/T208028) [10:40:44] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:41:14] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [10:41:54] PROBLEM - puppet last run on lvs4005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:42:23] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 6.179 second response time [10:43:03] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:43:43] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [10:44:00] (03CR) 10Elukey: [C: 032] profile::statistics::private: move geoip archive to another dir [puppet] - 10https://gerrit.wikimedia.org/r/469890 (https://phabricator.wikimedia.org/T208028) (owner: 10Elukey) [10:44:13] (03PS1) 10Addshore: Wikibase, move badge related config to IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470355 [10:44:37] (03PS2) 10Addshore: Wikibase, Move badge related config to IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470355 [10:44:54] RECOVERY - puppet last run on labtestservices2001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [10:45:34] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:40] (03CR) 10Arturo Borrero Gonzalez: "> Other than a bin/sbin typo, it seems okay so far. You can check" [puppet] - 10https://gerrit.wikimedia.org/r/469614 (https://phabricator.wikimedia.org/T207591) (owner: 10Arturo Borrero Gonzalez) [10:52:33] (03PS1) 10Elukey: profile::statistics::private: fix wrong file ensure [puppet] - 10https://gerrit.wikimedia.org/r/470357 (https://phabricator.wikimedia.org/T208028) [10:52:52] (03PS2) 10Rxy: Remove global action related permissions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/469864 (https://phabricator.wikimedia.org/T208035) [10:52:54] (03PS10) 10Giuseppe Lavagetto: mediawiki::webserver: introduce profile, use it on mwdebug* [puppet] - 10https://gerrit.wikimedia.org/r/467644 [10:53:07] (03CR) 10Elukey: [C: 032] profile::statistics::private: fix wrong file ensure [puppet] - 10https://gerrit.wikimedia.org/r/470357 (https://phabricator.wikimedia.org/T208028) (owner: 10Elukey) [10:53:45] PROBLEM - puppet last run on stat1007 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/geoip] [10:53:52] this is me --^ [10:57:16] jouncebot: next [10:57:16] In 0 hour(s) and 2 minute(s): European Mid-day SWAT(Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181029T1100) [10:58:53] RECOVERY - puppet last run on stat1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:00:04] addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: Dear deployers, time to do the European Mid-day SWAT(Max 6 patches) deploy. Dont look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20181029T1100). [11:00:04] rxy: A patch you scheduled for European Mid-day SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [11:00:10] o/ [11:00:56] (03PS1) 10Muehlenhoff: Remove absented Diamond collector for Pybal [puppet] - 10https://gerrit.wikimedia.org/r/470358 [11:01:45] o/ [11:02:09] hi [11:02:47] 10Operations: Access requests process: People sometimes specify hostnames instead of admin groups in access requests - https://phabricator.wikimedia.org/T207754 (10Krenair) >>! In T207754#4701935, @MoritzMuehlenhoff wrote: > Can you clarify the scope/intent of this task, it's not obvious to me. Change the acces... [11:04:28] rxy: sorry, Europe changed time zone over the weekend, and this event is locked to a US timezone, I forgot it's an hour earlier this week :) [11:05:15] (03CR) 10Ema: [C: 031] Remove absented Diamond collector for Pybal [puppet] - 10https://gerrit.wikimedia.org/r/470358 (owner: 10Muehlenhoff) [11:05:32] haha [11:05:43] 10Operations, 10monitoring, 10Patch-For-Review, 10User-fgiunchedi: Deprovision Diamond collectors no longer in use - https://phabricator.wikimedia.org/T183454 (10MoritzMuehlenhoff) [11:06:13] RECOVERY - puppet last run on lvs2006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:06:14] DST is confusable. ( JST is not effects DST ) [11:06:19] (03PS11) 10Giuseppe Lavagetto: mediawiki::webserver: introduce profile, use it on mwdebug* [puppet] - 10https://gerrit.wikimedia.org/r/467644 [11:07:31] zeljkof: anyway, can you SWAT the my patch? :) [11:07:44] rxy: sure, sorry, I'm reviewing it, forgot to say that [11:07:59] ah , k [11:08:33] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:08:52] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/469864 (https://phabricator.wikimedia.org/T208035) (owner: 10Rxy) [11:10:09] (03Merged) 10jenkins-bot: Remove global action related permissions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/469864 (https://phabricator.wikimedia.org/T208035) (owner: 10Rxy) [11:12:33] RECOVERY - puppet last run on lvs4005 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [11:14:46] rxy: the patch is at mwdebug1002, please test and let me know if it's ok to deploy it [11:15:01] I will testing that ... [11:15:30] (03CR) 10Muehlenhoff: [C: 032] Remove absented Diamond collector for Pybal [puppet] - 10https://gerrit.wikimedia.org/r/470358 (owner: 10Muehlenhoff) [11:17:09] (03PS12) 10Giuseppe Lavagetto: mediawiki::webserver: introduce profile, use it on mwdebug* [puppet] - 10https://gerrit.wikimedia.org/r/467644 [11:17:38] zeljkof: ok, It work correctly at mwdebug1002. Please deploy :) [11:18:05] rxy: ok [11:18:47] (03PS1) 10Muehlenhoff: Remove Diamond from LVSes [puppet] - 10https://gerrit.wikimedia.org/r/470359 (https://phabricator.wikimedia.org/T183454) [11:19:18] !log zfilipin@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:469864|Remove global action related permissions (T208035)]] (duration: 00m 48s) [11:19:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:19:22] T208035: Remove global action related permissions except meta wikimedia - https://phabricator.wikimedia.org/T208035 [11:21:09] rxy: it's deployed, please test [11:22:47] zeljkof: Request URL: https://en.wikipedia.org/wiki/Special:ListGroupRights ; server: mw1248.eqiad.wmnet -> ok, Request URL: https://meta.wikimedia.org/wiki/Special:ListGroupRights server: mw1322.eqiad.wmnet -> ok. It work correctly. Thanks :) [11:23:11] (03CR) 10Muehlenhoff: [C: 032] Remove Diamond from LVSes [puppet] - 10https://gerrit.wikimedia.org/r/470359 (https://phabricator.wikimedia.org/T183454) (owner: 10Muehlenhoff) [11:23:23] !log EU SWAT finished [11:23:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:26:13] (03PS1) 10Addshore: Wikibase BETA, use the same siteLinkGroups as prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470360 [11:26:15] (03PS1) 10Addshore: Wikibase, move some property lists to IS php files [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470361 [11:30:19] (03PS1) 10Addshore: Wikibase, Move wgArticlePlaceholderImageProperty to IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470362 [11:31:41] (03CR) 10jenkins-bot: Remove global action related permissions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/469864 (https://phabricator.wikimedia.org/T208035) (owner: 10Rxy) [11:34:27] (03CR) 10Tarrow: [C: 04-1] "Typo I think?" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470348 (owner: 10Addshore) [11:35:12] (03CR) 10Addshore: Wikibase, move dispatching settings to IS.php (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470348 (owner: 10Addshore) [11:35:14]