[00:06:27] 10Operations, 10MediaWiki-Vagrant: Import kibana package from jessie into stretch - https://phabricator.wikimedia.org/T183071#3842659 (10Paladox) [03:24:37] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 809.08 seconds [03:34:17] PROBLEM - puppet last run on analytics1053 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIPCity.dat.gz] [03:53:37] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 124.25 seconds [04:04:17] RECOVERY - puppet last run on analytics1053 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [07:03:38] PROBLEM - haproxy failover on dbproxy1010 is CRITICAL: CRITICAL check_failover servers up 2 down 1 [07:15:07] PROBLEM - Check HHVM threads for leakage on mw1307 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [07:30:37] PROBLEM - puppet last run on mw1275 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:30:37] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:30:37] PROBLEM - puppet last run on rhodium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:31:08] PROBLEM - puppet last run on elastic1043 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:31:08] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:31:48] PROBLEM - puppet last run on cp4025 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:31:48] PROBLEM - puppet last run on conf1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:31:48] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:32:07] PROBLEM - puppet last run on boron is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:32:08] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:32:27] PROBLEM - puppet last run on restbase1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:32:28] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:32:47] PROBLEM - puppet last run on mw1312 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:33:07] PROBLEM - puppet last run on labvirt1012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:33:08] PROBLEM - puppet last run on conf1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:33:17] PROBLEM - puppet last run on analytics1065 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:33:27] PROBLEM - puppet last run on mw1262 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:34:07] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:34:17] PROBLEM - puppet last run on analytics1051 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:34:27] PROBLEM - puppet last run on db1094 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:34:37] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:34:58] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:34:58] PROBLEM - puppet last run on mw1282 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:35:07] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:35:07] PROBLEM - puppet last run on rdb1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:39:47] RECOVERY - haproxy failover on dbproxy1010 is OK: OK check_failover servers up 2 down 0 [07:45:07] RECOVERY - Check HHVM threads for leakage on mw1307 is OK: OK [07:58:08] RECOVERY - puppet last run on labvirt1012 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [07:58:28] RECOVERY - puppet last run on mw1262 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:59:07] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:59:17] RECOVERY - puppet last run on analytics1051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:59:27] RECOVERY - puppet last run on db1094 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:59:37] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:59:57] RECOVERY - puppet last run on mw1282 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:59:58] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:00:07] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:00:07] RECOVERY - puppet last run on rdb1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:00:37] RECOVERY - puppet last run on mw1275 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:00:37] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:00:37] RECOVERY - puppet last run on rhodium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:01:07] RECOVERY - puppet last run on elastic1043 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:01:08] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:01:48] RECOVERY - puppet last run on cp4025 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [08:01:48] RECOVERY - puppet last run on conf1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [08:01:48] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [08:02:07] RECOVERY - puppet last run on boron is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [08:02:08] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [08:02:27] RECOVERY - puppet last run on restbase1013 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [08:02:28] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [08:02:47] RECOVERY - puppet last run on mw1312 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [08:03:08] RECOVERY - puppet last run on conf1003 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [08:03:17] RECOVERY - puppet last run on analytics1065 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [13:46:52] 10Operations, 10Wikimedia-Mailing-lists: Emails send by subscribers don't arrive on the mailing list Moderators-nl - https://phabricator.wikimedia.org/T181906#3843095 (10Multichill) This seems to be going on too long without progress. @RobH can you help us out here? Should be easy to find on the commandline wh... [14:44:57] PROBLEM - MariaDB Slave SQL: s5 on dbstore1001 is CRITICAL: CRITICAL slave_sql_state Slave_SQL_Running: No, Errno: 1062, Errmsg: Could not execute Update_rows_v1 event on table wikidatawiki.tag_summary: Duplicate entry 182949546 for key tag_summary_rev_id, Error_code: 1062: handler error HA_ERR_FOUND_DUPP_KEY: the events master log db1070-bin.001655, end_log_pos 808458079 [14:52:58] RECOVERY - MariaDB Slave SQL: s5 on dbstore1001 is OK: OK slave_sql_state Slave_SQL_Running: Yes [15:24:45] I have a problem.. I can not to upload patch on gerrit [15:25:13] I type my username and password and than I got error Authorization failed. [15:25:41] See screenshot: [15:25:43] One: https://snag.gy/Qe16En.jpg [15:25:59] Two: http://prntscr.com/hooa4t [15:26:24] Zoranzoki21 hi, you need to use the password from https://gerrit.wikimedia.org/r/#/settings/http-password [15:27:52] or use SSH instead of HTTPS [15:28:11] paladox: Thank you [15:28:14] Now is all ok [15:28:17] Thanks for help [15:28:18] your welcome. [15:28:53] not many people use windows around here [15:28:56] (03PS1) 10Zoranzoki21: Add xpda.com to $wgCopyUploadDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/398702 (https://phabricator.wikimedia.org/T183073) [15:28:58] and your one doesn't appear to be activated [15:29:22] Ok ok.. Now work. Patch is successfully uploaded. See up [15:29:51] (03CR) 10jerkins-bot: [V: 04-1] Add xpda.com to $wgCopyUploadDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/398702 (https://phabricator.wikimedia.org/T183073) (owner: 10Zoranzoki21) [15:30:46] (03PS2) 10Zoranzoki21: Add xpda.com to $wgCopyUploadDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/398702 (https://phabricator.wikimedia.org/T183073) [15:32:23] (03CR) 10Zoranzoki21: [C: 031] Enable Sandbox Extension at Atikamekw Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/398417 (https://phabricator.wikimedia.org/T182798) (owner: 10Jayprakash12345) [15:53:01] (03PS1) 10Ladsgroup: Move testwiki2 from group0 to group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/398704 (https://phabricator.wikimedia.org/T182326) [16:45:18] PROBLEM - Check Varnish expiry mailbox lag on cp4026 is CRITICAL: CRITICAL: expiry mailbox lag is 2040505 [16:55:20] RECOVERY - Check Varnish expiry mailbox lag on cp4026 is OK: OK: expiry mailbox lag is 0 [18:41:57] 10Operations, 10MediaWiki-Vagrant: Import kibana package from jessie into stretch - https://phabricator.wikimedia.org/T183071#3843316 (10bd808) [19:11:28] PROBLEM - Router interfaces on cr2-ulsfo is CRITICAL: CRITICAL: host 198.35.26.193, interfaces up: 76, down: 1, dormant: 0, excluded: 0, unused: 0 [19:15:37] RECOVERY - Router interfaces on cr2-ulsfo is OK: OK: host 198.35.26.193, interfaces up: 78, down: 0, dormant: 0, excluded: 0, unused: 0 [19:22:07] (03CR) 10Zoranzoki21: [C: 031] horizon: Update logos and file naming [puppet] - 10https://gerrit.wikimedia.org/r/398605 (https://phabricator.wikimedia.org/T168480) (owner: 10BryanDavis) [19:22:13] (03PS2) 10Zoranzoki21: horizon: Update logos and file naming [puppet] - 10https://gerrit.wikimedia.org/r/398605 (https://phabricator.wikimedia.org/T168480) (owner: 10BryanDavis) [19:25:47] PROBLEM - Router interfaces on cr2-ulsfo is CRITICAL: CRITICAL: host 198.35.26.193, interfaces up: 76, down: 1, dormant: 0, excluded: 0, unused: 0 [19:35:37] PROBLEM - Router interfaces on cr1-codfw is CRITICAL: CRITICAL: host 208.80.153.192, interfaces up: 120, down: 1, dormant: 0, excluded: 0, unused: 0 [20:10:57] RECOVERY - Router interfaces on cr1-codfw is OK: OK: host 208.80.153.192, interfaces up: 122, down: 0, dormant: 0, excluded: 0, unused: 0 [20:11:17] RECOVERY - Router interfaces on cr2-ulsfo is OK: OK: host 198.35.26.193, interfaces up: 78, down: 0, dormant: 0, excluded: 0, unused: 0 [22:26:52] 13,12▄10,09▄13,03▄13,12▄10,03▄08,12▄08,05▄03,04▄03,04▄04,07▄02,10▄13,05▄04,11▄05,03▄03 DID YOU GUYS KNOW TODAY WAS NIGGERS DAY?? SAY HI TO YOUR FAVORITY NIGGER IN #FREENODE!! quicktalkeh676te.onionuopkrvrubq: sir_lestaty mdholloway quiddity TheresNoTime hoo volans|off Krenair Nemo_bis icinga-wm godog matt_flaschen JollyOldStNick matthiasmullie omilu Zppix|Away masti yuri [22:26:52] 09,02▄05,06▄05,09▄08,05▄13,07▄07,11▄13,12▄12,10▄10,05▄11,12▄05,12▄03,08▄04,08▄05,04▄05,02▄06,09▄07 DID YOU GUYS KNOW TODAY WAS NIGGERS DAY?? SAY HI TO YOUR FAVORITY NIGGER IN #FREENODE!! quicktalkeh676te.oniontgfzk: ebernhardson ragesoss_ qchris_ MGC apergos eddiegp Jamesofur akl TheresNoTime Tobi_WMDE_SW_NA robh dmaza phedenskog musik raj matt_flaschen akosiaris [22:26:52] 06,04▄02,03▄08,11▄10,07▄11,05▄02,10▄05,08▄12,07▄04,10▄10,13▄02,07▄12,05▄06,04▄05,04▄09,06▄06,03▄10,04▄02,08▄04,06▄09,09▄12 DID YOU GUYS KNOW TODAY WAS NIGGERS DAY?? SAY HI TO YOUR FAVORITY NIGGER IN #FREENODE!! quicktalkeh676te.oniontvhim: sir_lestaty FlorianSW Betacommand guillom refeed[z] HaeB revi Trizek wm-bot ToAruShiroiNeko Raymond_ TheDrago [22:26:52] 09,13▄06,04▄05,09▄12,02▄05,13▄05,12▄09,02▄07,09▄02,06▄11,02▄02,07▄07,10▄09,11▄05,06▄11,09▄05,09▄05,06▄06 DID YOU GUYS KNOW TODAY WAS NIGGERS DAY?? SAY HI TO YOUR FAVORITY NIGGER IN #FREENODE!! quicktalkeh676te.onionxqdcofqv: DatGuy Lcawte refeed[z] wm-bot niko phedenskog godog greg-g urandom ebernhardson logmsgbot cormacparle SantaC Barras mutante raynor omil [23:19:04] (03CR) 10EddieGP: "> From the other side: It makes no sense to" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/397768 (https://phabricator.wikimedia.org/T182541) (owner: 10EddieGP) [23:23:09] (03CR) 10EddieGP: [C: 04-1] "I'm trying to figure out some way to test this, because I'm as unconfident as ops probably are with deploying this without thorough testin" [puppet] - 10https://gerrit.wikimedia.org/r/397770 (owner: 10EddieGP)