[00:11:53] (03PS3) 10Urbanecm: Change sitename of shnwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/473670 (https://phabricator.wikimedia.org/T206777) [00:12:35] (03PS2) 10Urbanecm: Close internalwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/468823 (https://phabricator.wikimedia.org/T205584) [00:43:11] PROBLEM - puppet last run on labtestservices2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:03:37] PROBLEM - Request latencies on neon is CRITICAL: instance=10.64.0.40:6443 verb={LIST,PATCH,PUT} https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:04:21] PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb={LIST,PATCH,POST,PUT} https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:04:25] PROBLEM - etcd request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 operation={compareAndSwap,get,list} https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:04:25] PROBLEM - etcd request latencies on neon is CRITICAL: instance=10.64.0.40:6443 operation={compareAndSwap,get,list} https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:05:21] PROBLEM - Request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 verb={GET,PUT} https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:05:35] PROBLEM - etcd request latencies on argon is CRITICAL: instance=10.64.32.133:6443 operation={compareAndSwap,get,list} https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:14:23] RECOVERY - puppet last run on labtestservices2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:06:47] RECOVERY - Request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:06:51] RECOVERY - etcd request latencies on chlorine is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:06:51] RECOVERY - etcd request latencies on neon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:47] RECOVERY - Request latencies on chlorine is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:08:01] RECOVERY - etcd request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:08:29] RECOVERY - Request latencies on neon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [03:36:07] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 917.32 seconds [03:49:13] PROBLEM - puppet last run on dumpsdata1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [04:15:07] RECOVERY - puppet last run on dumpsdata1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:21:27] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 638.96 seconds [04:28:37] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 163.96 seconds [06:15:41] RECOVERY - Disk space on notebook1004 is OK: DISK OK [07:28:46] 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Evaluate scalability and performance of PHP7 compared to HHVM - https://phabricator.wikimedia.org/T206341 (10Joe) Over the last week, I did a thorough benchmarking effort, with the setup being the following: - I tested real-life pages in our prod... [08:15:04] (03PS1) 10Smalyshev: Enable RDF dumps everywhere [puppet] - 10https://gerrit.wikimedia.org/r/477111 [10:03:43] PROBLEM - Check systemd state on an-coord1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [10:08:23] 10Operations, 10Maps, 10Discovery-Search (Current work): Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) - https://phabricator.wikimedia.org/T210940 (10Mathew.onipe) [10:33:23] 10Operations, 10Gadgets, 10MediaWiki-Cache, 10MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), and 4 others: Mcrouter periodically reports soft TKOs for mc[1,2]035 leading to MW Memcached exceptions - https://phabricator.wikimedia.org/T203786 (10aaron) >>! In T203786#4791225, @Nikerabbit wrote: > Is the value so... [10:49:43] (03PS1) 10Mathew.onipe: osm::planet_sync: change osm log dir permission [puppet] - 10https://gerrit.wikimedia.org/r/477118 (https://phabricator.wikimedia.org/T210940) [11:19:42] 10Operations, 10Gadgets, 10MediaWiki-Cache, 10MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), and 4 others: Mcrouter periodically reports soft TKOs for mc[1,2]035 leading to MW Memcached exceptions - https://phabricator.wikimedia.org/T203786 (10elukey) >>! In T203786#4791225, @Nikerabbit wrote: > Is the value so... [12:49:44] Hi, I'm getting "[XAPUzwpAICgAACQYk-kAAADB] 2018-12-02 12:49:20: Fatal exception of type MWException" trying to move a page in eswiki... [13:00:13] Anyone around? [13:08:03] It is a known issue. [13:08:25] https://phabricator.wikimedia.org/T210739 [13:09:37] Ok, thanks, Bsadowski1 [13:09:50] Let's wait then [14:51:35] (03CR) 10Giuseppe Lavagetto: profile::mediawiki::php: add opcache tuning for php-fpm (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/476499 (https://phabricator.wikimedia.org/T206341) (owner: 10Giuseppe Lavagetto) [15:19:01] (03CR) 10Paladox: profile::mediawiki::php: tune php-fpm parameters (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/476500 (https://phabricator.wikimedia.org/T206341) (owner: 10Giuseppe Lavagetto) [15:36:57] (03CR) 10Paladox: [C: 04-1] phabricator: add data types to all parameters (0314 comments) [puppet] - 10https://gerrit.wikimedia.org/r/471325 (owner: 10Dzahn) [16:38:25] jem: You ok if I quote you on: T210739 so it also gets reported? [16:38:25] T210739: Target deletion during page move fails - https://phabricator.wikimedia.org/T210739 [16:39:34] 10Operations, 10Math, 10Patch-For-Review: Clean up artifacts from LaTeX based math rendering - https://phabricator.wikimedia.org/T195847 (10Physikerwelt) [16:45:25] 10Operations, 10Math, 10Patch-For-Review: Clean up artifacts from LaTeX based math rendering - https://phabricator.wikimedia.org/T195847 (10Physikerwelt) [16:46:33] 10Operations, 10Math: Remove unused i18n messages for math extension - https://phabricator.wikimedia.org/T210948 (10Physikerwelt) p:05Triage>03Normal [16:46:40] 10Operations, 10Math: Remove unused i18n messages for math extension - https://phabricator.wikimedia.org/T210948 (10Physikerwelt) p:05Normal>03Low [17:01:14] 10Operations, 10Math, 10Patch-For-Review: Clean up artifacts from LaTeX based math rendering - https://phabricator.wikimedia.org/T195847 (10Physikerwelt) [17:53:17] PROBLEM - puppet last run on dns2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:15:11] PROBLEM - Disk space on druid1001 is CRITICAL: DISK CRITICAL - free space: / 1062 MB (3% inode=96%) [18:17:33] PROBLEM - Disk space on druid1001 is CRITICAL: DISK CRITICAL - free space: / 1062 MB (3% inode=96%) [18:22:23] PROBLEM - Disk space on druid1001 is CRITICAL: DISK CRITICAL - free space: / 1054 MB (3% inode=96%) [18:24:23] RECOVERY - puppet last run on dns2001 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [19:59:49] (03PS1) 10Zoranzoki21: Upload HD logos for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477136 (https://phabricator.wikimedia.org/T150618) [20:13:47] (03PS1) 10GTirloni: labsdb: Increase net_{read,write}_timeout [puppet] - 10https://gerrit.wikimedia.org/r/477137 (https://phabricator.wikimedia.org/T184126) [20:16:19] (!!) https://phabricator.wikimedia.org/T210962 [20:16:37] Any idea of some query or something? [20:17:06] (Please don't share the content, just answer there if you want to help) :) [20:34:07] (03PS1) 10Zoranzoki21: wgLogoHD: Fix sorting of chwiki and chywiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477138 (https://phabricator.wikimedia.org/T150618) [20:41:09] (03PS2) 10Zoranzoki21: Cleaning of wgLogoHD [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477138 (https://phabricator.wikimedia.org/T150618) [20:49:35] PROBLEM - puppet last run on labtestservices2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:49:53] (03PS1) 10Zoranzoki21: Use HD logos in InitialiseSettings.php for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477139 (https://phabricator.wikimedia.org/T150618) [20:50:17] (03PS2) 10Zoranzoki21: Use HD logos in InitialiseSettings.php for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477139 (https://phabricator.wikimedia.org/T150618) [20:51:07] (03CR) 10jerkins-bot: [V: 04-1] Use HD logos in InitialiseSettings.php for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477139 (https://phabricator.wikimedia.org/T150618) (owner: 10Zoranzoki21) [20:52:45] (03PS3) 10Zoranzoki21: Use HD logos in InitialiseSettings.php for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477139 (https://phabricator.wikimedia.org/T150618) [21:04:19] 10Operations, 10Traffic, 10Wikimedia-General-or-Unknown, 10media-storage: Loading full versions of larger images from Commons stucks / repeatedly gets interrupted after a few MBs - https://phabricator.wikimedia.org/T210890 (10Nemo_bis) We got a report from Canada on #wikimedia-tech and I confirm I see the... [21:06:55] (03CR) 10Urbanecm: [C: 04-1] "hrwikisource looks blurried. It looks like enlarged smaller PNG - this is not good. You must find a SVG and generate PNG of correct size f" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477136 (https://phabricator.wikimedia.org/T150618) (owner: 10Zoranzoki21) [21:15:29] RECOVERY - puppet last run on labtestservices2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:32:17] (03PS2) 10Zoranzoki21: Upload HD logos for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477136 (https://phabricator.wikimedia.org/T150618) [21:58:44] 10Operations, 10Gadgets, 10MediaWiki-Cache, 10MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), and 4 others: Mcrouter periodically reports soft TKOs for mc[1,2]035 leading to MW Memcached exceptions - https://phabricator.wikimedia.org/T203786 (10aaron) To clarify, the $useMutex logic in WAN cache never triggers d... [22:17:49] (03CR) 10Urbanecm: [C: 04-1] "hrwiktionary isn't transparent. Please improve." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477136 (https://phabricator.wikimedia.org/T150618) (owner: 10Zoranzoki21) [22:27:55] (03PS1) 10Urbanecm: Create namespace "Work" on bnwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477174 (https://phabricator.wikimedia.org/T210472) [22:30:28] (03PS1) 10Urbanecm: Create List namespace on euwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/477175 (https://phabricator.wikimedia.org/T209834) [22:39:52] !log addshore@mwmaint1002:~$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=testwikidatawiki addless # This is my account, and apparently I no longer have the 2fa for it [22:39:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:45:13] PROBLEM - Device not healthy -SMART- on db1063 is CRITICAL: cluster=mysql device=megaraid,7 instance=db1063:9100 job=node site=eqiad https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=db1063var-datasource=eqiad%2520prometheus%252Fops [23:47:02] * addshore is going to backport & deploy https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/477180/ for a security related ticket for wikidata in the next few mins [23:47:31] to avoid the chaos someone figuring that out would cause.... [23:52:42] its quite nice having the whole of CI to myself, hehe [23:57:49] * addshore twiddles thumbs