[00:00:05] <jouncebot>	 addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: It is that lovely time of the day again! You are hereby commanded to deploy Evening SWAT (Max 6 patches). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190215T0000).
[00:00:05] <jouncebot>	 No GERRIT patches in the queue for this window AFAICS.
[00:00:12] <mutante>	 marxarelli: uhmm. something about the footer lines.. probably a new line is expected
[00:00:19] <mutante>	 Line 12: Expected 'Bug:' to be in footer
[00:00:54] <marxarelli>	 oh maybe gerrit messed it up. i edited in the ui
[00:02:29] <wikibugs>	 (03PS5) 10Dduvall: ci: Permit git traffic between zuul mergers over ipv6 [puppet] - 10https://gerrit.wikimedia.org/r/490790 (https://phabricator.wikimedia.org/T216204)
[00:03:06] <wikibugs>	 (03CR) 10Dduvall: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/490790 (https://phabricator.wikimedia.org/T216204) (owner: 10Dduvall)
[00:03:23] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] ci: Permit git traffic between zuul mergers over ipv6 [puppet] - 10https://gerrit.wikimedia.org/r/490790 (https://phabricator.wikimedia.org/T216204) (owner: 10Dduvall)
[00:03:25] <wikibugs>	 (03CR) 10Dduvall: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/490790 (https://phabricator.wikimedia.org/T216204) (owner: 10Dduvall)
[00:03:39] <marxarelli>	 gah.
[00:04:10] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] ci: Permit git traffic between zuul mergers over ipv6 [puppet] - 10https://gerrit.wikimedia.org/r/490790 (https://phabricator.wikimedia.org/T216204) (owner: 10Dduvall)
[00:04:41] <paladox>	 marxarelli put a new line between Hosts and Bug
[00:05:43] <wikibugs>	 (03PS6) 10Dduvall: ci: Permit git traffic between zuul mergers over ipv6 [puppet] - 10https://gerrit.wikimedia.org/r/490790 (https://phabricator.wikimedia.org/T216204)
[00:06:54] <marxarelli>	 paladox: nice :)
[00:07:08] <paladox>	 :)
[00:08:56] <wikibugs>	 (03PS4) 10CDanis: WIP: puppet for rasdaemon testing [puppet] - 10https://gerrit.wikimedia.org/r/490787
[00:09:18] <marxarelli>	 mutante: the compiler diff looks right to me
[00:10:05] <marxarelli>	 it's only iptables. what could go wrong? :)
[00:10:19] <mutante>	 marxarelli: yea, heh, agree. let's merge :)
[00:10:49] <mutante>	 we can also revert by removing the generated ferm file, heh
[00:11:25] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] ci: Permit git traffic between zuul mergers over ipv6 [puppet] - 10https://gerrit.wikimedia.org/r/490790 (https://phabricator.wikimedia.org/T216204) (owner: 10Dduvall)
[00:12:15] <mutante>	 deploying on 2001
[00:13:54] <mutante>	 looks fine. ferm refreshed. though.. dont see in ip6tables -L yet.. sec
[00:14:12] <marxarelli>	 mutante: `telnet -6 contint2001.wikimedia.org 9418` from contint1001 works now!
[00:15:09] <marxarelli>	 and a `git fetch --ipv6` was speedy
[00:15:18] <jynus>	 !log setting labsdb1005 back into read-write
[00:15:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:15:20] <marxarelli>	 thanks for the quick review
[00:15:25] <mutante>	 marxarelli: oh, of course, i looked in the wrong place :)
[00:15:29] <mutante>	 glad it works, ok
[00:15:47] <mutante>	 deploying on 1001
[00:16:01] <marxarelli>	 that will speed up our service-jenkinsjob from 9 minutes to maybe 30 seconds
[00:16:36] <mutante>	 nice, can use that in an achievement list 
[00:16:47] <mutante>	 so measurable 
[00:16:56] <marxarelli>	 haha :)
[00:17:01] <marxarelli>	 greg-g: ^ {{done}}
[00:17:22] <wikibugs>	 10Operations, 10ops-eqiad, 10Patch-For-Review, 10cloud-services-team (Kanban): Degraded RAID on cloudvirt1020 - https://phabricator.wikimedia.org/T194855 (10bd808)
[00:18:00] <mutante>	 i'm always for adding more IPv6 records :)
[00:18:12] <mutante>	 we just have to remember the ferm part
[00:20:05] <greg-g>	 mutante: yay!
[00:20:30] <mutante>	 :)
[00:21:37] <greg-g>	 er, marxarelli yay!
[00:21:40] <greg-g>	 but both! :)
[00:23:10] <wikibugs>	 (03CR) 10Paladox: [C: 03+1] "This looks really nice (based on his test site). Im going to work on making it nice for all integration ci checks. Basing it on this (as i" [puppet] - 10https://gerrit.wikimedia.org/r/490640 (https://phabricator.wikimedia.org/T177868) (owner: 10Thcipriani)
[00:25:20] <marxarelli>	 greg-g: "SUCCESS in 33s" 3 seconds off our new KPI
[00:25:30] * marxarelli is fired
[00:26:45] <mutante>	 oh?
[00:27:35] <wikibugs>	 (03PS6) 10Dzahn: remove bast3003 prod IP, bast3002 has been repaired [dns] - 10https://gerrit.wikimedia.org/r/489104 (https://phabricator.wikimedia.org/T184936)
[00:28:25] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] "bast3003 never became actively used, then bast3002 was repaired, so 3002 is still the current one" [dns] - 10https://gerrit.wikimedia.org/r/489104 (https://phabricator.wikimedia.org/T184936) (owner: 10Dzahn)
[00:32:55] <wikibugs>	 10Operations, 10ops-esams, 10DC-Ops, 10decommission, 10Patch-For-Review: decom bast3003 (65R8Q4J, formerly amslvs4) - https://phabricator.wikimedia.org/T216199 (10Dzahn) I removed production DNS entries but kept mgmt because the host does not have mgmt entries by asset tag. The DRAC IP is: 10.21.0.109 an...
[00:35:42] <wikibugs>	 10Operations, 10ops-esams, 10DC-Ops, 10decommission: decom bast3003 (65R8Q4J, formerly amslvs4) - https://phabricator.wikimedia.org/T216199 (10Dzahn) a:03RobH
[00:36:51] <wikibugs>	 10Operations, 10ops-esams, 10DC-Ops, 10decommission: decom bast3003 (65R8Q4J, formerly amslvs4) - https://phabricator.wikimedia.org/T216199 (10Dzahn) I did not start the "non-interrupt steps" now or anything but i marked the first 2 check boxes there because it's already removed from puppet (was never in /...
[00:39:25] <mutante>	 !log puppetmaster1001: sudo puppet node clean bast3003.wikimedia.org ; sudo puppet node deactivate bast3003.wikimedia.org (T216199)
[00:39:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:39:28] <stashbot>	 T216199: decom bast3003 (65R8Q4J, formerly amslvs4) - https://phabricator.wikimedia.org/T216199
[00:40:25] <wikibugs>	 10Operations, 10ops-esams, 10DC-Ops, 10decommission: decom bast3003 (65R8Q4J, formerly amslvs4) - https://phabricator.wikimedia.org/T216199 (10Dzahn)
[00:49:09] <wikibugs>	 10Operations, 10ops-esams, 10DC-Ops, 10decommission: decom bast3003 (65R8Q4J, formerly amslvs4) - https://phabricator.wikimedia.org/T216199 (10Dzahn)
[00:49:26] <icinga-wm>	 PROBLEM - Logstash rate of ingestion percent change compared to yesterday on icinga2001 is CRITICAL: 141.5 ge 130 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1&panelId=2&fullscreen
[00:49:43] <wikibugs>	 10Operations, 10ops-esams, 10Patch-For-Review: install/designate other machine as esams bastion - https://phabricator.wikimedia.org/T184936 (10Dzahn) reverted in  T216199
[00:54:48] <wikibugs>	 (03CR) 10Smalyshev: [C: 04-1] "until testing is complete" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490648 (https://phabricator.wikimedia.org/T215684) (owner: 10Jforrester)
[00:55:18] <wikibugs>	 (03CR) 10Smalyshev: [C: 03+1] Deploy WikibaseCirrusSearch: Part II, InitialiseSettings.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490644 (owner: 10Jforrester)
[00:56:08] <wikibugs>	 (03CR) 10Smalyshev: [C: 03+1] Deploy WikibaseCirrusSearch: Part III, Wikibase.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490645 (owner: 10Jforrester)
[00:56:13] <wikibugs>	 (03CR) 10Smalyshev: [C: 03+1] [BETA] Enable WikibaseCirrusSearch on wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490646 (owner: 10Jforrester)
[01:00:26] <wikibugs>	 (03CR) 10Dzahn: "addressed both comments,should be resolved" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/451821 (owner: 10Dzahn)
[01:04:59] <wikibugs>	 (03PS8) 10Dzahn: puppetmaster/configmaster: convert from apache to httpd module [puppet] - 10https://gerrit.wikimedia.org/r/451821
[01:05:32] <wikibugs>	 (03PS1) 10Paladox: Fix replacing ${project-base-name} in gr-project [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/490796
[01:09:02] <mutante>	 bug: the bug reporting bug quits when there are too many bugs
[01:11:19] <wikibugs>	 (03PS9) 10Dzahn: puppetmaster/configmaster: convert from apache to httpd module [puppet] - 10https://gerrit.wikimedia.org/r/451821
[01:12:12] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] puppetmaster/configmaster: convert from apache to httpd module [puppet] - 10https://gerrit.wikimedia.org/r/451821 (owner: 10Dzahn)
[01:12:27] <wikibugs>	 (03CR) 10Dzahn: "manually rebased on top of prod due to changes in puppetmaster::passenger" [puppet] - 10https://gerrit.wikimedia.org/r/451821 (owner: 10Dzahn)
[01:13:01] <wikibugs>	 (03CR) 10Paladox: [V: 03+2 C: 03+2] "Verified that this builds." [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/490110 (owner: 10Paladox)
[01:13:35] <wikibugs>	 (03PS10) 10Dzahn: puppetmaster/configmaster: convert from apache to httpd module [puppet] - 10https://gerrit.wikimedia.org/r/451821
[01:14:39] <wikibugs>	 (03CR) 10Dzahn: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/451821 (owner: 10Dzahn)
[01:20:16] <wikibugs>	 (03CR) 10Dzahn: "see https://puppet-compiler.wmflabs.org/compiler1001/14689/" [puppet] - 10https://gerrit.wikimedia.org/r/451821 (owner: 10Dzahn)
[01:22:31] <wikibugs>	 10Operations, 10DC-Ops, 10Wikimedia-Incident, 10cloud-services-team (Kanban): Increase visibility of auto-generated tasks for RAID errors - https://phabricator.wikimedia.org/T216133 (10Bstorm)
[01:30:27] <wikibugs>	 (03PS11) 10Dzahn: puppetmaster/configmaster: convert from apache to httpd module [puppet] - 10https://gerrit.wikimedia.org/r/451821
[01:39:34] <wikibugs>	 (03CR) 10Dzahn: [C: 03+1] "should be good now. any concerns? could you take another look?  https://puppet-compiler.wmflabs.org/compiler1001/14689/" [puppet] - 10https://gerrit.wikimedia.org/r/451821 (owner: 10Dzahn)
[01:41:35] <wikibugs>	 (03CR) 10Dzahn: [C: 03+1] "you said you wanted to think about reorganizing this, right. maybe it would be ok to merge anyways and live with the duplication until we " [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[01:43:02] <wikibugs>	 (03PS4) 10Dzahn: noc: apache -> httpd module [puppet] - 10https://gerrit.wikimedia.org/r/416751
[01:46:57] <wikibugs>	 (03PS5) 10CDanis: rasdaemon: add hiera overrides for testing on stretch [puppet] - 10https://gerrit.wikimedia.org/r/490787 (https://phabricator.wikimedia.org/T205396)
[01:47:50] <wikibugs>	 (03CR) 10CDanis: "PCC for a stretch host with hiera, and a stretch and a buster host with no hiera:" [puppet] - 10https://gerrit.wikimedia.org/r/490787 (https://phabricator.wikimedia.org/T205396) (owner: 10CDanis)
[01:48:57] <wikibugs>	 10Operations, 10MediaWiki-Database, 10Performance-Team, 10Wikimedia-Logstash, and 6 others: MediaWiki errors overloading logstash - https://phabricator.wikimedia.org/T215611 (10aaron) I think Timo backported the change, so it should be live:  ` 18:01 krinkle@deploy1001: Synchronized php-1.33.0-wmf.17/inclu...
[01:49:31] <wikibugs>	 (03CR) 10Dzahn: "mediawiki has been converted, so this should be unblocked now, rebased" [puppet] - 10https://gerrit.wikimedia.org/r/416751 (owner: 10Dzahn)
[01:58:47] <wikibugs>	 (03CR) 10Dzahn: [C: 04-1] "not there yet, still need to convert  noc::php_engine as well" [puppet] - 10https://gerrit.wikimedia.org/r/416751 (owner: 10Dzahn)
[02:09:31] <wikibugs>	 (03CR) 10CDanis: "Oh, also -- I was quite careful to have Puppet revert to defaults if hiera is unset, but also am not sure if I implemented it well.  Let m" [puppet] - 10https://gerrit.wikimedia.org/r/490787 (https://phabricator.wikimedia.org/T205396) (owner: 10CDanis)
[02:14:44] <wikibugs>	 (03PS10) 10Paladox: Introduce gr-wikimedia-prettify-ci-comments [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/489483 (https://phabricator.wikimedia.org/T215658)
[02:35:01] <paladox>	 looks like the cpu has gone up on gerrit again. (dosen't seem to be impacting it)
[02:37:37] <wikibugs>	 (03PS1) 10Paladox: Gerrit: Remove socket config from log4j [puppet] - 10https://gerrit.wikimedia.org/r/490797
[02:40:10] <wikibugs>	 (03PS2) 10Paladox: Gerrit: Remove socket config from log4j [puppet] - 10https://gerrit.wikimedia.org/r/490797
[02:40:12] <wikibugs>	 (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/490797 (owner: 10Paladox)
[02:41:06] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Gerrit: Remove socket config from log4j [puppet] - 10https://gerrit.wikimedia.org/r/490797 (owner: 10Paladox)
[02:41:07] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Gerrit: Remove socket config from log4j [puppet] - 10https://gerrit.wikimedia.org/r/490797 (owner: 10Paladox)
[02:41:55] <wikibugs>	 (03PS3) 10Paladox: Gerrit: Remove socket config from log4j [puppet] - 10https://gerrit.wikimedia.org/r/490797
[02:43:57] <wikibugs>	 (03PS4) 10Paladox: Gerrit: Remove socket config from log4j [puppet] - 10https://gerrit.wikimedia.org/r/490797
[02:44:04] <wikibugs>	 (03CR) 10Paladox: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/490797 (owner: 10Paladox)
[02:49:38] <icinga-wm>	 RECOVERY - Logstash rate of ingestion percent change compared to yesterday on icinga2001 is OK: (C)130 ge (W)110 ge 108.3 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1&panelId=2&fullscreen
[04:39:51] <wikibugs>	 10Operations, 10ops-eqiad: Disk failure on labsdb1005 - https://phabricator.wikimedia.org/T216202 (10Bstorm) Manually dropped the disk from the RAID because the server is being very bad.  The database keeps locking up.
[05:04:46] <icinga-wm>	 PROBLEM - MegaRAID on labsdb1005 is CRITICAL: CRITICAL: 1 failed LD(s) (Partially Degraded)
[05:04:49] <icinga-wm>	 ACKNOWLEDGEMENT - MegaRAID on labsdb1005 is CRITICAL: CRITICAL: 1 failed LD(s) (Partially Degraded) nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T216223
[05:04:54] <wikibugs>	 10Operations, 10ops-eqiad: Degraded RAID on labsdb1005 - https://phabricator.wikimedia.org/T216223 (10ops-monitoring-bot)
[06:11:14] <wikibugs>	 (03PS2) 10Marostegui: Revert "dbproxy1011: Depool labsdb1009" [puppet] - 10https://gerrit.wikimedia.org/r/490636
[06:11:56] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Depool db1109 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490801 (https://phabricator.wikimedia.org/T210713)
[06:12:31] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] Revert "dbproxy1011: Depool labsdb1009" [puppet] - 10https://gerrit.wikimedia.org/r/490636 (owner: 10Marostegui)
[06:13:57] <marostegui>	 !log Reload haproxy on dbproxy11 to repool labsdb1009
[06:13:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:14:26] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1109 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490801 (https://phabricator.wikimedia.org/T210713) (owner: 10Marostegui)
[06:15:24] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1109 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490801 (https://phabricator.wikimedia.org/T210713) (owner: 10Marostegui)
[06:16:35] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 49s)
[06:16:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:17:48] <marostegui>	 !log Deploy schema change on db1109 - T210713
[06:17:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:17:51] <stashbot>	 T210713: Drop change_tag.ct_tag column in production - https://phabricator.wikimedia.org/T210713
[06:21:00] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Depool db1109 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490801 (https://phabricator.wikimedia.org/T210713) (owner: 10Marostegui)
[06:25:27] <wikibugs>	 (03PS2) 10Ammarpad: Increase default thumb size to 260px on Dutch Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490395 (https://phabricator.wikimedia.org/T215106)
[06:26:07] <wikibugs>	 (03PS3) 10Ammarpad: Set wgArticleCountMethod='any' for zhwikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/487115
[06:27:33] <wikibugs>	 (03PS10) 10Ammarpad: Add 'Author' namespace in Sanskrit Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486221 (https://phabricator.wikimedia.org/T214553)
[06:30:18] <marostegui>	 labsdb1005 is again with too many connections
[06:30:21] <marostegui>	 arturo: ^
[06:39:35] <marostegui>	 !log Restart labsdb1005 with max_user_connections = 20 T216208
[06:39:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:39:38] <stashbot>	 T216208: ToolsDB overload and cleanup - https://phabricator.wikimedia.org/T216208
[06:40:58] <marostegui>	 !log Stop puppet on labsdb1005 to leave "max_user_connections" on my.cnf - T216170 T216208
[06:41:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:41:02] <stashbot>	 T216170: toolsdb - Per-user connection limits - https://phabricator.wikimedia.org/T216170
[07:02:08] <wikibugs>	 (03PS1) 10Giuseppe Lavagetto: profile::services_proxy: make declaring services optional [puppet] - 10https://gerrit.wikimedia.org/r/490805 (https://phabricator.wikimedia.org/T216164)
[07:05:45] <wikibugs>	 10Operations, 10Analytics, 10Research-management, 10Patch-For-Review, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10elukey) https://rocm.github.io/hardware.html is very handy. As said previously, I'd stick with a GFX9 card, I'd say a RX Vega 64 or Radeon VII...
[07:06:49] <wikibugs>	 (03PS1) 10Marostegui: tools.my.cnf: Temporary max_user_connections [puppet] - 10https://gerrit.wikimedia.org/r/490806 (https://phabricator.wikimedia.org/T216170)
[07:08:11] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] tools.my.cnf: Temporary max_user_connections [puppet] - 10https://gerrit.wikimedia.org/r/490806 (https://phabricator.wikimedia.org/T216170) (owner: 10Marostegui)
[07:19:35] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [C: 03+2] profile::services_proxy: make declaring services optional [puppet] - 10https://gerrit.wikimedia.org/r/490805 (https://phabricator.wikimedia.org/T216164) (owner: 10Giuseppe Lavagetto)
[07:19:48] <wikibugs>	 (03PS2) 10Giuseppe Lavagetto: profile::services_proxy: make declaring services optional [puppet] - 10https://gerrit.wikimedia.org/r/490805 (https://phabricator.wikimedia.org/T216164)
[07:21:34] <wikibugs>	 10Operations, 10ops-eqiad, 10Data-Services, 10cloud-services-team: Degraded RAID on labsdb1005 - https://phabricator.wikimedia.org/T216223 (10Marostegui)
[07:23:10] <wikibugs>	 10Operations, 10ops-eqiad, 10Data-Services, 10cloud-services-team: Degraded RAID on labsdb1005 - https://phabricator.wikimedia.org/T216223 (10Marostegui) #cloud-services-team  I would suggest you coordinate with @Cmjohnson to get this disk replaced
[07:24:23] <wikibugs>	 10Operations, 10ops-eqiad, 10Toolforge, 10cloud-services-team: Degraded RAID on labsdb1005 - https://phabricator.wikimedia.org/T216223 (10Marostegui)
[07:36:12] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] "Thanks!" [dns] - 10https://gerrit.wikimedia.org/r/489251 (owner: 10Alexandros Kosiaris)
[07:36:19] <wikibugs>	 (03PS9) 10Alexandros Kosiaris: Add kubernetes pods PTR records for IPv4 [dns] - 10https://gerrit.wikimedia.org/r/489251
[07:36:24] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [V: 03+2 C: 03+2] Add kubernetes pods PTR records for IPv4 [dns] - 10https://gerrit.wikimedia.org/r/489251 (owner: 10Alexandros Kosiaris)
[07:55:40] <godog>	 !log bounce prometheus@ops on prometheus2004 to take a snapshot
[07:55:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:55:57] <godog>	 that'll cause a bunch of UNKNOWN for some minutes FYI
[08:06:03] <wikibugs>	 10Operations, 10Wikimedia-Logstash, 10Patch-For-Review, 10User-herron: Replace and expand Elasticsearch storage in eqiad and upgrade the cluster from Debian jessie to stretch - https://phabricator.wikimedia.org/T213898 (10fgiunchedi) On stretch by default we're installing `elasticsearch-curator` from `stre...
[08:17:45] <icinga-wm>	 PROBLEM - Host backup2001 is DOWN: PING CRITICAL - Packet loss = 100%
[08:17:59] <icinga-wm>	 RECOVERY - Host backup2001 is UP: PING OK - Packet loss = 0%, RTA = 0.48 ms
[08:20:03] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+1] Gerrit: comment styles for the pipeline [puppet] - 10https://gerrit.wikimedia.org/r/490640 (https://phabricator.wikimedia.org/T177868) (owner: 10Thcipriani)
[08:20:13] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] Gerrit: comment styles for the pipeline [puppet] - 10https://gerrit.wikimedia.org/r/490640 (https://phabricator.wikimedia.org/T177868) (owner: 10Thcipriani)
[08:20:21] <wikibugs>	 (03PS2) 10Alexandros Kosiaris: Gerrit: comment styles for the pipeline [puppet] - 10https://gerrit.wikimedia.org/r/490640 (https://phabricator.wikimedia.org/T177868) (owner: 10Thcipriani)
[08:20:23] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [V: 03+2 C: 03+2] Gerrit: comment styles for the pipeline [puppet] - 10https://gerrit.wikimedia.org/r/490640 (https://phabricator.wikimedia.org/T177868) (owner: 10Thcipriani)
[08:28:54] <moritzm>	 !log rolling restart of apertium to pick up Python 3.4 security update
[08:28:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:30:48] <wikibugs>	 (03PS1) 10Filippo Giunchedi: logstash: force use elasticsearch-curator 5 [puppet] - 10https://gerrit.wikimedia.org/r/490809 (https://phabricator.wikimedia.org/T213898)
[08:37:19] <wikibugs>	 (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1109" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490810
[08:37:25] <wikibugs>	 10Operations, 10hardware-requests: GPU for stat1005 - https://phabricator.wikimedia.org/T216226 (10elukey)
[08:38:22] <wikibugs>	 10Operations, 10Analytics, 10Research-management, 10Patch-For-Review, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10elukey) Opened T216226 to discuss hw requirements for the new GPU (everybody interested please subscribe/chime-in!), let's use this task to deb...
[08:38:37] <wikibugs>	 10Operations, 10Analytics, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10elukey)
[08:38:39] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] Revert "db-eqiad.php: Depool db1109" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490810 (owner: 10Marostegui)
[08:39:42] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1109" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490810 (owner: 10Marostegui)
[08:40:40] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 46s)
[08:40:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:42:29] <akosiaris>	 !log restart gerrit to pick up https://gerrit.wikimedia.org/r/490640 T177868
[08:42:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:42:34] <stashbot>	 T177868: Define pipeline failure developer feedback - https://phabricator.wikimedia.org/T177868
[08:45:29] <wikibugs>	 (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1109" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490810 (owner: 10Marostegui)
[08:45:33] <icinga-wm>	 PROBLEM - puppet last run on webperf2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_performance/docroot]
[08:45:43] <icinga-wm>	 PROBLEM - puppet last run on cumin2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_operations/cookbooks]
[08:48:17] <icinga-wm>	 PROBLEM - puppet last run on webperf1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_performance/docroot]
[08:49:32] <wikibugs>	 (03CR) 10Filippo Giunchedi: "LGTM, typo in commit message" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/490695 (https://phabricator.wikimedia.org/T214608) (owner: 10Herron)
[08:54:07] <wikibugs>	 (03PS2) 10Elukey: Define the zookeeper rmstore path on each Hadoop cluster [puppet] - 10https://gerrit.wikimedia.org/r/490575
[09:02:46] <wikibugs>	 (03CR) 10Muehlenhoff: "The conditional case should not be necessary; mcelog operated on /dev/mcelog and rasdaemon on debugfs, so they can coexist along each othe" [puppet] - 10https://gerrit.wikimedia.org/r/490787 (https://phabricator.wikimedia.org/T205396) (owner: 10CDanis)
[09:08:19] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Define the zookeeper rmstore path on each Hadoop cluster [puppet] - 10https://gerrit.wikimedia.org/r/490575 (owner: 10Elukey)
[09:12:41] <wikibugs>	 (03CR) 10Volans: [C: 04-1] "There's a typo, looks good otherwise. Please test it on the test instance first." (031 comment) [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/490700 (owner: 10CRusnov)
[09:14:56] <wikibugs>	 (03PS1) 10Elukey: Set the Yarn rmstore zookeeper path for the Hadoop testing cluster [puppet] - 10https://gerrit.wikimedia.org/r/490826
[09:16:05] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Set the Yarn rmstore zookeeper path for the Hadoop testing cluster [puppet] - 10https://gerrit.wikimedia.org/r/490826 (owner: 10Elukey)
[09:16:38] <icinga-wm>	 RECOVERY - puppet last run on webperf2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures
[09:16:48] <icinga-wm>	 RECOVERY - puppet last run on cumin2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures
[09:19:48] <icinga-wm>	 RECOVERY - puppet last run on webperf1001 is OK: OK: Puppet is currently enabled, last run 7 minutes ago with 0 failures
[09:26:10] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] "I had to follow up with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/490826/, since the rmstore test cluster path was 'rmstore-c" [puppet] - 10https://gerrit.wikimedia.org/r/490575 (owner: 10Elukey)
[09:33:20] <moritzm>	 !log imported php-defaults debs to thirdparty/php72
[09:33:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:44:55] <gehel>	 !log repool maps100[12]
[09:44:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:48:12] <wikibugs>	 10Operations, 10Puppet, 10Discovery-Search, 10Maps, 10Patch-For-Review: Fix maps puppet to make sure apt-get update runs after configuration change - https://phabricator.wikimedia.org/T214073 (10Gehel) 05Open→03Resolved a:03Gehel
[09:55:12] <icinga-wm>	 PROBLEM - DPKG on contint1001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[09:58:26] <icinga-wm>	 RECOVERY - DPKG on contint1001 is OK: All packages OK
[10:03:24] <wikibugs>	 10Operations, 10MediaWiki-Database, 10Performance-Team, 10Wikimedia-Logstash, and 6 others: MediaWiki errors overloading logstash - https://phabricator.wikimedia.org/T215611 (10fgiunchedi) Apologies for the late reply -- and thanks to all who have helped in fixing the issue!  >>! In T215611#4950057, @bd808...
[10:07:43] <wikibugs>	 (03PS1) 10Elukey: profile::analytics::refinery:job::data_purge: absent mediawiki-geoeditors-drop-month [puppet] - 10https://gerrit.wikimedia.org/r/490829
[10:07:54] <wikibugs>	 (03PS2) 10ArielGlenn: misc dumps: report names of most recent failed wikis if we bail out [dumps] - 10https://gerrit.wikimedia.org/r/488261
[10:08:36] <wikibugs>	 (03CR) 10Joal: [C: 03+1] "Thanks Luca :)" [puppet] - 10https://gerrit.wikimedia.org/r/490829 (owner: 10Elukey)
[10:08:43] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] profile::analytics::refinery:job::data_purge: absent mediawiki-geoeditors-drop-month [puppet] - 10https://gerrit.wikimedia.org/r/490829 (owner: 10Elukey)
[10:09:41] <elukey>	 sorry jenkins
[10:09:42] <elukey>	 fixing
[10:11:13] <wikibugs>	 (03PS2) 10Elukey: profile::analytics::refinery:job::data_purge: absent timer [puppet] - 10https://gerrit.wikimedia.org/r/490829
[10:12:11] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] profile::analytics::refinery:job::data_purge: absent timer [puppet] - 10https://gerrit.wikimedia.org/r/490829 (owner: 10Elukey)
[10:21:59] <wikibugs>	 (03CR) 10Gehel: [C: 04-1] "see comments inline." (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/490809 (https://phabricator.wikimedia.org/T213898) (owner: 10Filippo Giunchedi)
[10:33:33] <wikibugs>	 (03PS1) 10GTirloni: cloudvirt1019/1020 - Reimage with Stretch [puppet] - 10https://gerrit.wikimedia.org/r/490831 (https://phabricator.wikimedia.org/T193264)
[10:33:50] <wikibugs>	 10Operations, 10Discovery-Search, 10Elasticsearch: cleanup reprepro configuration for elasticsearch-curator - https://phabricator.wikimedia.org/T216235 (10Gehel)
[10:35:23] <gtirloni>	 !log reboot cloudvirt1019
[10:35:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:35:57] <wikibugs>	 (03PS2) 10ArielGlenn: showcrcs: util to write out crc information from a bzip2 file [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/490299 (https://phabricator.wikimedia.org/T216009)
[10:42:57] <akosiaris>	 !log upgrade docker on contint1001 to 18.06.2 T216236
[10:42:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:48:36] <akosiaris>	 !log upgrade docker on contint2001 to 18.06.2 T216236
[10:48:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:51:32] <wikibugs>	 (03PS2) 10Filippo Giunchedi: logstash: force use elasticsearch-curator 5 [puppet] - 10https://gerrit.wikimedia.org/r/490809 (https://phabricator.wikimedia.org/T213898)
[10:52:06] <wikibugs>	 (03CR) 10Filippo Giunchedi: logstash: force use elasticsearch-curator 5 (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/490809 (https://phabricator.wikimedia.org/T213898) (owner: 10Filippo Giunchedi)
[10:52:26] <wikibugs>	 10Operations, 10Discovery-Search, 10Elasticsearch, 10User-fgiunchedi: cleanup reprepro configuration for elasticsearch-curator - https://phabricator.wikimedia.org/T216235 (10fgiunchedi)
[10:54:14] <wikibugs>	 (03PS3) 10Muehlenhoff: Drop requires_os checks for trusty [puppet] - 10https://gerrit.wikimedia.org/r/489625
[10:54:57] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: Upgrade docker in CI [puppet] - 10https://gerrit.wikimedia.org/r/490832 (https://phabricator.wikimedia.org/T216236)
[10:56:00] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] Upgrade docker in CI [puppet] - 10https://gerrit.wikimedia.org/r/490832 (https://phabricator.wikimedia.org/T216236) (owner: 10Alexandros Kosiaris)
[10:56:02] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+2] Drop requires_os checks for trusty [puppet] - 10https://gerrit.wikimedia.org/r/489625 (owner: 10Muehlenhoff)
[10:56:30] <wikibugs>	 10Operations, 10Traffic, 10VisualEditor, 10Wikimedia-Apache-configuration: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10Schnark)
[10:58:51] <wikibugs>	 10Operations, 10hardware-requests: eqiad: (1) Ganeti VM for testing Kerberos in Production - https://phabricator.wikimedia.org/T216238 (10elukey)
[10:59:56] <wikibugs>	 10Operations, 10hardware-requests: eqiad: (1) Ganeti VM for testing Kerberos in Production - https://phabricator.wikimedia.org/T216238 (10elukey)
[11:16:53] <wikibugs>	 (03PS1) 10Filippo Giunchedi: prometheus: don't require Prometheus::Server when writing k8s token [puppet] - 10https://gerrit.wikimedia.org/r/490834 (https://phabricator.wikimedia.org/T187987)
[11:20:41] <wikibugs>	 (03CR) 10Volans: [C: 03+2] "Thanks for the info and the patch Bryan, appreciated!" [software/spicerack] - 10https://gerrit.wikimedia.org/r/490774 (owner: 10BryanDavis)
[11:21:23] <wikibugs>	 (03PS2) 10Arturo Borrero Gonzalez: cloudvirt1019/1020 - Reimage with Stretch [puppet] - 10https://gerrit.wikimedia.org/r/490831 (https://phabricator.wikimedia.org/T193264) (owner: 10GTirloni)
[11:22:05] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] cloudvirt1019/1020 - Reimage with Stretch [puppet] - 10https://gerrit.wikimedia.org/r/490831 (https://phabricator.wikimedia.org/T193264) (owner: 10GTirloni)
[11:23:15] <wikibugs>	 10Operations, 10ops-eqiad: ms-be1033 down and not powering up - https://phabricator.wikimedia.org/T215998 (10fgiunchedi) Thanks @Cmjohnson ! Did HP provide an ETA for shipment/resolution ?
[11:24:52] <wikibugs>	 10Operations, 10Thumbor, 10serviceops, 10Patch-For-Review, and 2 others: Upgrade Thumbor servers to Stretch - https://phabricator.wikimedia.org/T170817 (10fgiunchedi)
[11:26:23] <wikibugs>	 (03Merged) 10jenkins-bot: Change !log formatting to match Stashbot expectations [software/spicerack] - 10https://gerrit.wikimedia.org/r/490774 (owner: 10BryanDavis)
[11:27:22] <wikibugs>	 (03CR) 10jenkins-bot: Change !log formatting to match Stashbot expectations [software/spicerack] - 10https://gerrit.wikimedia.org/r/490774 (owner: 10BryanDavis)
[11:29:40] <wikibugs>	 (03PS4) 10Muehlenhoff: Drop requires_os checks for trusty [puppet] - 10https://gerrit.wikimedia.org/r/489625
[11:30:23] <wikibugs>	 10Operations, 10serviceops, 10Kubernetes, 10Patch-For-Review, 10User-fsero: Upgrade calico in production to version 2.4+ - https://phabricator.wikimedia.org/T207804 (10akosiaris)
[11:43:35] <wikibugs>	 (03PS38) 10Jbond: Create module for managing ulogd [puppet] - 10https://gerrit.wikimedia.org/r/486513 (https://phabricator.wikimedia.org/T116011)
[11:44:18] <wikibugs>	 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Access request: Ladsgroup to analytics-wmde-users - https://phabricator.wikimedia.org/T215938 (10Ladsgroup) Thanks!
[11:52:02] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: hiera: codfw: openstack: delete leftover file [puppet] - 10https://gerrit.wikimedia.org/r/490835
[11:53:09] <wikibugs>	 (03PS1) 10Jbond: Enable ulog module. [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011)
[11:53:39] <wikibugs>	 (03PS2) 10Jbond: Enable ulog module. [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011)
[11:54:11] <wikibugs>	 (03PS39) 10Jbond: Create module for managing ulogd [puppet] - 10https://gerrit.wikimedia.org/r/486513 (https://phabricator.wikimedia.org/T116011)
[12:00:12] <wikibugs>	 (03PS1) 10Jbond: Load apt when testing base::puppet [puppet] - 10https://gerrit.wikimedia.org/r/490837
[12:00:24] <wikibugs>	 10Operations, 10ops-eqiad, 10DC-Ops, 10decommission, 10cloud-services-team (Kanban): decommission labvirt101[01].eqiad.wmnet (Dec 2018 lease return) - https://phabricator.wikimedia.org/T210735 (10aborrero) Could you please refresh netbox status for these 2 servers: * https://netbox.wikimedia.org/dcim/dev...
[12:03:52] <wikibugs>	 (03PS40) 10Jbond: Create module for managing ulogd [puppet] - 10https://gerrit.wikimedia.org/r/486513 (https://phabricator.wikimedia.org/T116011)
[12:05:48] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] hiera: codfw: openstack: delete leftover file [puppet] - 10https://gerrit.wikimedia.org/r/490835 (owner: 10Arturo Borrero Gonzalez)
[12:08:13] <wikibugs>	 (03CR) 10Jbond: "Ready for review:" (037 comments) [puppet] - 10https://gerrit.wikimedia.org/r/486513 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[12:18:32] <wikibugs>	 10Operations, 10Analytics, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10Shilad) I've been lurking on this issue, and just wanted chime in with one bit of information I learned through experience. I'm not sure what model of Dell server you are using, but there a...
[12:23:16] <wikibugs>	 (03PS1) 10Elukey: profile::hue: set default database password (to ease testing) d [puppet] - 10https://gerrit.wikimedia.org/r/490838
[12:24:14] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] profile::hue: set default database password (to ease testing) d [puppet] - 10https://gerrit.wikimedia.org/r/490838 (owner: 10Elukey)
[12:28:29] <icinga-wm>	 PROBLEM - ensure kvm processes are running on labvirt1009 is CRITICAL: PROCS CRITICAL: 0 processes with regex args /usr/bin/kvm
[12:29:44] <icinga-wm>	 ACKNOWLEDGEMENT - ensure kvm processes are running on labvirt1009 is CRITICAL: PROCS CRITICAL: 0 processes with regex args /usr/bin/kvm Arturo Borrero Gonzalez draining
[12:30:31] <arturo>	 !log T216239 schedule 1week of icinga downtime for labvirt1019
[12:30:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:30:34] <stashbot>	 T216239: CloudVPS: drain and rebuild labvirt1009 as cloudvirt1009 - https://phabricator.wikimedia.org/T216239
[12:30:50] <jynus>	 !log stop db2089 mysql instances for reboot testing T216240
[12:30:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:30:53] <stashbot>	 T216240: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240
[12:30:59] <icinga-wm>	 RECOVERY - ensure kvm processes are running on labvirt1009 is OK: PROCS OK: 1 process with regex args /usr/bin/kvm
[12:42:23] <moritzm>	 !log installing squid3 security updates
[12:42:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:06:38] <moritzm>	 !log installing NSS security updates
[13:06:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:09:30] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: Enable ulog module. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[13:11:02] <arturo>	 !log T216239 labvirt1019 has been drained of any workload
[13:11:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:11:05] <stashbot>	 T216239: CloudVPS: drain and rebuild labvirt1009 as cloudvirt1009 - https://phabricator.wikimedia.org/T216239
[13:12:40] <gtirloni>	 !log reboot cloudvirt1020
[13:12:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:13:36] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: cloudvps: main: depool labvirt1009 [puppet] - 10https://gerrit.wikimedia.org/r/490845 (https://phabricator.wikimedia.org/T216239)
[13:15:00] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] cloudvps: main: depool labvirt1009 [puppet] - 10https://gerrit.wikimedia.org/r/490845 (https://phabricator.wikimedia.org/T216239) (owner: 10Arturo Borrero Gonzalez)
[13:15:17] <wikibugs>	 (03CR) 10Faidon Liambotis: "> > Patch Set 2:" [dns] - 10https://gerrit.wikimedia.org/r/489103 (owner: 10Dzahn)
[13:16:33] <wikibugs>	 (03PS5) 10Paladox: Gerrit: Remove socket config from log4j [puppet] - 10https://gerrit.wikimedia.org/r/490797
[13:17:05] <wikibugs>	 10Operations, 10Security: update tar - https://phabricator.wikimedia.org/T216242 (10jbond)
[13:19:44] <wikibugs>	 (03CR) 10Gehel: "This change is ready for review." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[13:19:49] <moritzm>	 !log rolling reboot of mwdebug servers in eqiad to pick up SSBD-enabled qemu
[13:19:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:22:42] <wikibugs>	 (03CR) 10Gehel: [C: 04-1] "minor comment inline, otherwise LGTM" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/490809 (https://phabricator.wikimedia.org/T213898) (owner: 10Filippo Giunchedi)
[13:27:10] <wikibugs>	 (03CR) 10Muehlenhoff: Enable ulog module. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[13:29:42] <wikibugs>	 10Operations, 10Security: update tar - https://phabricator.wikimedia.org/T216242 (10jbond) jessie ========== DebDiff Report ========== iff -Nru tar-1.27.1/debian/changelog tar-1.27.1/debian/changelog -- tar-1.27.1/debian/changelog  2016-10-30 06:48:55.000000000 +0000 ++ tar-1.27.1/debian/changelog  2018-12-31...
[13:33:52] <wikibugs>	 10Operations, 10Security: update tar - https://phabricator.wikimedia.org/T216242 (10jbond) bug 919493 dose not affect jessie
[13:36:44] <wikibugs>	 10Operations, 10Security: update tar - https://phabricator.wikimedia.org/T216242 (10jbond)
[13:40:01] <wikibugs>	 (03PS6) 10CDanis: rasdaemon: add hiera overrides for testing on stretch [puppet] - 10https://gerrit.wikimedia.org/r/490787 (https://phabricator.wikimedia.org/T205396)
[13:41:48] <wikibugs>	 10Operations, 10Security: update tar - https://phabricator.wikimedia.org/T216242 (10jbond) patch is missing additional tests but this is not an issue
[13:43:38] <wikibugs>	 (03PS7) 10CDanis: rasdaemon: add hiera overrides for testing on stretch [puppet] - 10https://gerrit.wikimedia.org/r/490787 (https://phabricator.wikimedia.org/T205396)
[13:43:59] <wikibugs>	 10Operations, 10Security: update tar - https://phabricator.wikimedia.org/T216242 (10jbond)
[13:46:03] <jbond42>	 !log install tar security updates
[13:46:03] <wikibugs>	 10Operations, 10ops-eqiad, 10Patch-For-Review, 10cloud-services-team (Kanban): Degraded RAID on cloudvirt1020 - https://phabricator.wikimedia.org/T194855 (10GTirloni) @Cmjohnson cloudvirt1020 is reporting a disk missing:  ` => ctrl slot=1 pd all show         Smart Array P840 in Slot 1     Unassigned...
[13:46:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:46:38] <wikibugs>	 10Operations, 10Patch-For-Review: Tracking and Reducing cron-spam to root@ - https://phabricator.wikimedia.org/T132324 (10jcrespo)
[13:46:40] <wikibugs>	 10Operations: cronspam for slow queries in PageAssessments - https://phabricator.wikimedia.org/T197564 (10jcrespo)
[13:47:50] <wikibugs>	 (03PS8) 10Mathew.onipe: Add wdqs data transfer cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/488256 (https://phabricator.wikimedia.org/T213401)
[13:49:42] <wikibugs>	 (03CR) 10Mathew.onipe: Add wdqs data transfer cookbook (037 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/488256 (https://phabricator.wikimedia.org/T213401) (owner: 10Mathew.onipe)
[13:50:09] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add wdqs data transfer cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/488256 (https://phabricator.wikimedia.org/T213401) (owner: 10Mathew.onipe)
[13:50:41] <wikibugs>	 10Operations, 10ops-eqiad, 10Patch-For-Review, 10cloud-services-team (Kanban): Degraded RAID on cloudvirt1020 - https://phabricator.wikimedia.org/T194855 (10GTirloni) cloudvirt1020 is also 5x slower to enter the BIOS menu (ESC+9) than cloudvirt1019. Not sure what that means.  The server is not doing anythi...
[13:51:21] <wikibugs>	 10Operations, 10MediaWiki-Maintenance-scripts: cron spam for slow queries on mwmaint /usr/local/bin/foreachwiki initSiteStats.php --update > /dev/null - https://phabricator.wikimedia.org/T216243 (10jcrespo)
[13:54:03] <wikibugs>	 (03PS1) 10Gehel: elasticsearch: check size of all replicas, not just primary shards [puppet] - 10https://gerrit.wikimedia.org/r/490852
[13:54:43] <moritzm>	 !log rebooting mwlog2001 for kernel security update
[13:54:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:55:05] <wikibugs>	 (03CR) 10DCausse: [C: 03+1] elasticsearch: check size of all replicas, not just primary shards [puppet] - 10https://gerrit.wikimedia.org/r/490852 (owner: 10Gehel)
[13:56:12] <wikibugs>	 (03PS9) 10Mathew.onipe: Add wdqs data transfer cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/488256 (https://phabricator.wikimedia.org/T213401)
[13:56:26] <wikibugs>	 (03CR) 10Mathew.onipe: [C: 03+1] elasticsearch: check size of all replicas, not just primary shards [puppet] - 10https://gerrit.wikimedia.org/r/490852 (owner: 10Gehel)
[13:58:25] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add wdqs data transfer cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/488256 (https://phabricator.wikimedia.org/T213401) (owner: 10Mathew.onipe)
[13:59:08] <wikibugs>	 10Operations, 10Analytics, 10Research, 10serviceops, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10akosiaris) >>! In T213566#4934895, @Ottomata wrote: >> they will also not allow them to send the SYN/ACK packet required for the secon...
[14:01:15] <cdanis>	 moritzm: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/490787 is ready for another look, if you have a minute
[14:02:12] <moritzm>	 cdanis: will have a look in ~15 mins
[14:02:36] <wikibugs>	 (03PS10) 10Mathew.onipe: Add wdqs data transfer cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/488256 (https://phabricator.wikimedia.org/T213401)
[14:03:37] <wikibugs>	 10Operations, 10Security: update tar - https://phabricator.wikimedia.org/T216242 (10jbond)
[14:04:13] <wikibugs>	 10Operations, 10Patch-For-Review: Tracking and Reducing cron-spam to root@ - https://phabricator.wikimedia.org/T132324 (10jcrespo)
[14:04:15] <wikibugs>	 10Operations, 10MediaWiki-Maintenance-scripts: cron spam for slow queries on mwmaint /usr/local/bin/foreachwiki initSiteStats.php --update > /dev/null - https://phabricator.wikimedia.org/T216243 (10jcrespo)
[14:04:49] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add wdqs data transfer cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/488256 (https://phabricator.wikimedia.org/T213401) (owner: 10Mathew.onipe)
[14:06:11] <moritzm>	 !log rebooting mwlog1001 for kernel security update
[14:06:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:07:13] <wikibugs>	 (03PS3) 10Jbond: Enable ulog module. [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011)
[14:10:59] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: Enable ulog module. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[14:11:28] <icinga-wm>	 PROBLEM - Check systemd state on webperf2002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[14:12:24] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: Create module for managing ulogd (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/486513 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[14:15:34] <wikibugs>	 (03CR) 10Gehel: [C: 04-1] Add wdqs data transfer cookbook (033 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/488256 (https://phabricator.wikimedia.org/T213401) (owner: 10Mathew.onipe)
[14:17:46] <icinga-wm>	 RECOVERY - Check systemd state on webperf2002 is OK: OK - running: The system is fully operational
[14:18:31] <wikibugs>	 (03CR) 10Muehlenhoff: "One thing to consider; ulogd isn't available in trusty, so we can only apply the profile to jessie and later." [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[14:20:49] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] "Looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/490787 (https://phabricator.wikimedia.org/T205396) (owner: 10CDanis)
[14:21:54] <wikibugs>	 (03CR) 10Muehlenhoff: "Nevermind, it is available also on trusty, I got confused by the source package name change." [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[14:22:10] <wikibugs>	 (03CR) 10CDanis: [C: 03+2] rasdaemon: add hiera overrides for testing on stretch [puppet] - 10https://gerrit.wikimedia.org/r/490787 (https://phabricator.wikimedia.org/T205396) (owner: 10CDanis)
[14:26:56] <icinga-wm>	 PROBLEM - puppet last run on thumbor1004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Package[rasdaemon]
[14:29:48] <wikibugs>	 (03PS1) 10CDanis: thumbor1004: can't install rasdaemon if there's no stretch [puppet] - 10https://gerrit.wikimedia.org/r/490855
[14:30:34] <wikibugs>	 (03PS41) 10Jbond: Create module for managing ulogd [puppet] - 10https://gerrit.wikimedia.org/r/486513 (https://phabricator.wikimedia.org/T116011)
[14:30:52] <wikibugs>	 (03PS2) 10CDanis: thumbor1004: can't install rasdaemon if there's no stretch [puppet] - 10https://gerrit.wikimedia.org/r/490855 (https://phabricator.wikimedia.org/T205396)
[14:31:01] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 04-1] "OK, -1 for a while as a stall." [deployment-charts] - 10https://gerrit.wikimedia.org/r/488800 (owner: 10Alexandros Kosiaris)
[14:31:24] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] thumbor1004: can't install rasdaemon if there's no stretch [puppet] - 10https://gerrit.wikimedia.org/r/490855 (https://phabricator.wikimedia.org/T205396) (owner: 10CDanis)
[14:31:54] <wikibugs>	 (03PS3) 10CDanis: thumbor1004: can't install rasdaemon if there's no stretch [puppet] - 10https://gerrit.wikimedia.org/r/490855 (https://phabricator.wikimedia.org/T205396)
[14:32:31] <wikibugs>	 (03CR) 10CDanis: [C: 03+2] thumbor1004: can't install rasdaemon if there's no stretch [puppet] - 10https://gerrit.wikimedia.org/r/490855 (https://phabricator.wikimedia.org/T205396) (owner: 10CDanis)
[14:34:53] <moritzm>	 cdanis: it should be straightforward to build rasdaemon for jessie-wikimedia, our kernels on jessie and stretch are both 4.9, so the debugfs is identical
[14:35:14] <moritzm>	 just mentioning because thumbor1004 is go good at being broken :-)
[14:35:33] <cdanis>	 moritzm: it is very good at that :) there are also other hosts almost as good
[14:35:45] <moritzm>	 ok :-)
[14:36:23] <cdanis>	 I'll enable the hiera on two of them and then probably also create a jessie backport
[14:37:22] <icinga-wm>	 RECOVERY - puppet last run on thumbor1004 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[14:40:40] <wikibugs>	 10Operations, 10ops-codfw, 10DBA: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10jcrespo) a:05jcrespo→03Papaul @Papaul from next week, please also upgrade firmware/BIOS of db2089 (**only that one for now**). I will put it back to product...
[14:45:31] <moritzm>	 !log removed labvirt1012 from debmonitor (got renamed to cloudvirt1012) (T216190)
[14:45:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:45:34] <stashbot>	 T216190: Rebuild labvirt1012 as cloudvirt1012 - https://phabricator.wikimedia.org/T216190
[14:49:27] <wikibugs>	 (03PS1) 10CDanis: mw2206: install rasdaemon [puppet] - 10https://gerrit.wikimedia.org/r/490856 (https://phabricator.wikimedia.org/T214529)
[14:50:10] <moritzm>	 !log installing unbound update from stretch point release
[14:50:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:50:15] <wikibugs>	 (03PS2) 10CDanis: mw2206: install rasdaemon [puppet] - 10https://gerrit.wikimedia.org/r/490856 (https://phabricator.wikimedia.org/T205396)
[14:51:57] <wikibugs>	 (03CR) 10CDanis: [C: 03+2] mw2206: install rasdaemon [puppet] - 10https://gerrit.wikimedia.org/r/490856 (https://phabricator.wikimedia.org/T205396) (owner: 10CDanis)
[14:58:52] <wikibugs>	 (03PS1) 10GTirloni: wmcs: Rename labvirt1009 to cloudvirt1009 [puppet] - 10https://gerrit.wikimedia.org/r/490858 (https://phabricator.wikimedia.org/T216239)
[14:59:12] <wikibugs>	 (03PS1) 10Muehlenhoff: Add library hint for unbound [puppet] - 10https://gerrit.wikimedia.org/r/490859
[15:02:19] <wikibugs>	 (03PS1) 10GTirloni: wmcs: Rename labvirt1009 to cloudvirt1009 [dns] - 10https://gerrit.wikimedia.org/r/490860 (https://phabricator.wikimedia.org/T216239)
[15:02:37] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+2] Add library hint for unbound [puppet] - 10https://gerrit.wikimedia.org/r/490859 (owner: 10Muehlenhoff)
[15:10:08] <wikibugs>	 (03CR) 10GTirloni: [C: 03+2] wmcs: Rename labvirt1009 to cloudvirt1009 [dns] - 10https://gerrit.wikimedia.org/r/490860 (https://phabricator.wikimedia.org/T216239) (owner: 10GTirloni)
[15:11:34] <wikibugs>	 (03PS2) 10GTirloni: wmcs: Rename labvirt1009 to cloudvirt1009 [puppet] - 10https://gerrit.wikimedia.org/r/490858 (https://phabricator.wikimedia.org/T216239)
[15:12:41] <wikibugs>	 (03CR) 10GTirloni: [C: 03+2] wmcs: Rename labvirt1009 to cloudvirt1009 [puppet] - 10https://gerrit.wikimedia.org/r/490858 (https://phabricator.wikimedia.org/T216239) (owner: 10GTirloni)
[15:18:35] <anomie>	 marostegui: FYI: Re T216208, I'm seeing processes on toolsdb seeming to be stuck in the "Opening tables" state again, despite the max_user_connections limit. I see talk in #wikimedia-cloud that seems like others are noticing too.
[15:18:35] <stashbot>	 T216208: ToolsDB overload and cleanup - https://phabricator.wikimedia.org/T216208
[15:19:39] <cdanis>	 anomie: I think marostegui is offline for the weekend already -- you might want to ask around in #wikimedia-databases ?
[15:27:53] <wikibugs>	 (03PS1) 10Gehel: elasticsearch: retry on TransportError while waiting for node to be up [software/spicerack] - 10https://gerrit.wikimedia.org/r/490866 (https://phabricator.wikimedia.org/T207920)
[15:31:18] <wikibugs>	 (03PS2) 10Gehel: elasticsearch: retry on TransportError while waiting for node to be up [software/spicerack] - 10https://gerrit.wikimedia.org/r/490866 (https://phabricator.wikimedia.org/T207920)
[15:38:17] <wikibugs>	 (03Abandoned) 10WMDE-leszek: DNM Use "entity source based federation" on Beta Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490658 (https://phabricator.wikimedia.org/T214557) (owner: 10WMDE-leszek)
[15:38:32] <wikibugs>	 (03Abandoned) 10WMDE-leszek: Added wmgUseEntitySourceBasedFederation setting to switch the federation mechanism [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490634 (https://phabricator.wikimedia.org/T214557) (owner: 10WMDE-leszek)
[15:45:28] <moritzm>	 !log rebooting auth1001 for kernel security update
[15:45:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:55:01] <wikibugs>	 (03PS1) 10GTirloni: cloudvirt1009 - Define partman recipe [puppet] - 10https://gerrit.wikimedia.org/r/490871 (https://phabricator.wikimedia.org/T216239)
[15:57:45] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 03+1] cloudvirt1009 - Define partman recipe [puppet] - 10https://gerrit.wikimedia.org/r/490871 (https://phabricator.wikimedia.org/T216239) (owner: 10GTirloni)
[15:58:37] <wikibugs>	 (03CR) 10GTirloni: [C: 03+2] cloudvirt1009 - Define partman recipe [puppet] - 10https://gerrit.wikimedia.org/r/490871 (https://phabricator.wikimedia.org/T216239) (owner: 10GTirloni)
[16:01:51] <wikibugs>	 (03CR) 10Mathew.onipe: [C: 03+1] elasticsearch: retry on TransportError while waiting for node to be up [software/spicerack] - 10https://gerrit.wikimedia.org/r/490866 (https://phabricator.wikimedia.org/T207920) (owner: 10Gehel)
[16:08:29] <wikibugs>	 (03PS4) 10Jbond: Enable ulog module. [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011)
[16:09:03] <wikibugs>	 10Operations, 10Wikimedia-Logstash: Retire udp2log: onboard its producers and consumers to the logging pipeline - https://phabricator.wikimedia.org/T205856 (10fgiunchedi) This is the outline of the plan to move mediawiki logging off udp2log and logging pipeline's kafka (cc @bd808 @aaron @Ottomata)  = Transport...
[16:09:13] <wikibugs>	 (03CR) 10Dzahn: [C: 03+1] "looked at it and thought about it..i'm afraid not, that turns it into a major refactor and using the same role but doing different things " [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[16:10:57] <wikibugs>	 10Operations, 10hardware-requests: requesting WMF7426 as phabricator system in eqiad - https://phabricator.wikimedia.org/T215335 (10faidon) a:05faidon→03RobH @Dzahn that's all fine, but we should have that documented in a separate Phabricator task tracking this work, if one doesn't exist already :) Separat...
[16:11:06] <wikibugs>	 (03CR) 10Dzahn: [C: 04-1] dumps:nfs: add data types, move variables to parameters and Hiera [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[16:11:39] <wikibugs>	 10Operations, 10hardware-requests: eqiad: requesting dual cpu misc host for icinga1001 replacement - https://phabricator.wikimedia.org/T215837 (10faidon) a:05faidon→03RobH If that's still needed, that's approved, and it takes priority over phab1002. And let's replenish our spare pool indeed!
[16:13:20] <wikibugs>	 (03CR) 10Jbond: Create module for managing ulogd (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/486513 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[16:13:26] <wikibugs>	 (03CR) 10Dzahn: [C: 04-1] "fwiw, the amount of duplication of hiera values would be the same before and after this change, just in different files" [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[16:13:44] <wikibugs>	 (03CR) 10Herron: [C: 03+1] "Looks good to me!" [puppet] - 10https://gerrit.wikimedia.org/r/490809 (https://phabricator.wikimedia.org/T213898) (owner: 10Filippo Giunchedi)
[16:14:17] <wikibugs>	 (03CR) 10Jbond: Enable ulog module. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[16:17:24] <wikibugs>	 10Operations, 10MediaWiki-Debug-Logger, 10Wikimedia-Logstash, 10monitoring: MediaWiki logging & encryption - https://phabricator.wikimedia.org/T126989 (10fgiunchedi) >>! In T126989#4934746, @fgiunchedi wrote: > Status update: mw logs that were going to logstash in plaintext now are being sent via localhost...
[16:18:45] <wikibugs>	 (03PS1) 10Mforns: Temporarily deactivate refine_eventlogging for maintenance [puppet] - 10https://gerrit.wikimedia.org/r/490875 (https://phabricator.wikimedia.org/T214384)
[16:19:49] <wikibugs>	 (03PS8) 10Dzahn: dumps:nfs: add data types, move variables to parameters and Hiera [puppet] - 10https://gerrit.wikimedia.org/r/479335
[16:20:55] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] dumps:nfs: add data types, move variables to parameters and Hiera [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[16:23:40] <wikibugs>	 (03CR) 10Dzahn: "ok, i tried to make this much simpler by ONLY adding the data types and not moving stuff to hiera.. but this gets downvotes because "Param" [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[16:24:42] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C: 03+1] "LGTM! I'd say the class documentation is minor at this point" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/486513 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[16:26:37] <wikibugs>	 (03PS9) 10Dzahn: dumps:nfs: add data types, move variables to parameters and Hiera [puppet] - 10https://gerrit.wikimedia.org/r/479335
[16:29:11] <bblack>	 !log reprepro: uploaded gdnsd-3.0.0-1~wmf1 to stretch-wikimedia
[16:29:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:32:12] <wikibugs>	 (03CR) 10Dzahn: "@ArielGlenn Reduced it to the most basic and original point of this, adding data types. No more moving parameters to Hiera or other change" [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[16:33:21] <wikibugs>	 (03PS1) 10Elukey: hue.wikimedia.org: serve static contents directly via httpd [puppet] - 10https://gerrit.wikimedia.org/r/490877
[16:33:23] <wikibugs>	 (03CR) 10Dzahn: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[16:34:29] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] hue.wikimedia.org: serve static contents directly via httpd [puppet] - 10https://gerrit.wikimedia.org/r/490877 (owner: 10Elukey)
[16:39:03] <wikibugs>	 (03CR) 10Dzahn: [C: 04-1] "this is also not working because port numbers are strings but should be numbers" [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[16:40:43] <wikibugs>	 (03PS10) 10Dzahn: dumps:nfs: add data types, move variables to parameters and Hiera [puppet] - 10https://gerrit.wikimedia.org/r/479335
[16:43:04] <wikibugs>	 (03CR) 10Dzahn: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[16:45:08] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Temporarily deactivate refine_eventlogging for maintenance [puppet] - 10https://gerrit.wikimedia.org/r/490875 (https://phabricator.wikimedia.org/T214384) (owner: 10Mforns)
[16:45:15] <wikibugs>	 (03PS2) 10Elukey: Temporarily deactivate refine_eventlogging for maintenance [puppet] - 10https://gerrit.wikimedia.org/r/490875 (https://phabricator.wikimedia.org/T214384) (owner: 10Mforns)
[16:45:58] <wikibugs>	 (03PS1) 10Muehlenhoff: profile::prometheus::nutcracker_exporter: Remove trusty support [puppet] - 10https://gerrit.wikimedia.org/r/490881
[16:46:04] <wikibugs>	 (03CR) 10Dzahn: [C: 03+1] "better now: https://puppet-compiler.wmflabs.org/compiler1001/118/" [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[16:47:22] <wikibugs>	 (03Abandoned) 10Dzahn: dumps:nfs: add data types, move variables to parameters and Hiera [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[16:48:04] <cdanis>	 !log adding backported rasdaemon 0.6.0-1.2+deb8u1 to jessie-wikimedia
[16:48:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:48:15] <wikibugs>	 (03PS1) 10Muehlenhoff: confd: Remove support for upstart [puppet] - 10https://gerrit.wikimedia.org/r/490882
[16:52:25] <cdanis>	 !log correction, needed to increment version; adding backported rasdaemon 0.6.0-1.2+deb8u2 to jessie-wikimedia
[16:52:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:53:42] <wikibugs>	 10Operations, 10ops-eqiad, 10Toolforge, 10cloud-services-team (Kanban): Degraded RAID on labsdb1005 - https://phabricator.wikimedia.org/T216223 (10colewhite)
[17:02:59] <wikibugs>	 (03PS1) 10Mforns: Reactivate refine_eventlogging_analytics after maintenance [puppet] - 10https://gerrit.wikimedia.org/r/490887 (https://phabricator.wikimedia.org/T214384)
[17:04:12] <icinga-wm>	 PROBLEM - Check systemd state on an-coord1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[17:06:58] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Reactivate refine_eventlogging_analytics after maintenance [puppet] - 10https://gerrit.wikimedia.org/r/490887 (https://phabricator.wikimedia.org/T214384) (owner: 10Mforns)
[17:07:34] <elukey>	 an-coord1001 is me
[17:09:44] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at esams on icinga2001 is CRITICAL: cluster=cache_text site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[17:10:04] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at codfw on icinga2001 is CRITICAL: cluster=cache_text site=codfw https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[17:10:34] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at ulsfo on icinga2001 is CRITICAL: cluster=cache_text site=ulsfo https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[17:10:58] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at esams on icinga2001 is CRITICAL: job=varnish-text site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[17:11:28] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at codfw on icinga2001 is CRITICAL: job=varnish-text site=codfw https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[17:12:20] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at eqiad on icinga2001 is CRITICAL: job=varnish-text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[17:12:56] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at eqsin on icinga2001 is CRITICAL: job=varnish-text site=eqsin https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[17:13:16] <icinga-wm>	 PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1004 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=eqiad&var-cache_type=All&var-status_type=5
[17:13:51] <godog>	 looks like a 503s spike, over already tho
[17:13:55] <godog>	 bblack: ^
[17:14:54] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at codfw on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[17:15:13] <godog>	 cp1077 perhaps
[17:15:34] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at esams on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[17:15:46] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at codfw on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[17:15:50] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at eqiad on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[17:16:20] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at ulsfo on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[17:17:22] <wikibugs>	 (03PS1) 10GTirloni: cloudvirt1009 - Define correct role [puppet] - 10https://gerrit.wikimedia.org/r/490889 (https://phabricator.wikimedia.org/T216239)
[17:18:56] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at esams on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[17:20:12] <wikibugs>	 10Operations, 10monitoring, 10Patch-For-Review: Evaluate/integrate rasdaemon as a replacement for mcelog - https://phabricator.wikimedia.org/T205396 (10CDanis) we got one: {P8089}
[17:20:40] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at eqsin on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[17:20:53] <wikibugs>	 (03CR) 10GTirloni: [C: 03+2] cloudvirt1009 - Define correct role [puppet] - 10https://gerrit.wikimedia.org/r/490889 (https://phabricator.wikimedia.org/T216239) (owner: 10GTirloni)
[17:21:24] <bblack>	 yeah looking around in the aggregate-client-status-codes, seems to have been a brief 503 spike local to cache_text, across all DCs
[17:21:29] <bblack>	 https://grafana.wikimedia.org/d/myRmf1Pik/varnish-aggregate-client-status-codes?orgId=1&from=now-1h&to=now&var-site=All&var-cache_type=text&var-status_type=5
[17:21:40] <icinga-wm>	 RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1004 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=eqiad&var-cache_type=All&var-status_type=5
[17:23:34] <bblack>	 ("across all DCs" meaning the problem was either in real services, or in eqiad backend caches, most likely)
[17:23:43] <herron>	 was it anything to do with this? https://grafana.wikimedia.org/d/000000304/prometheus-varnish-dc-stats?panelId=20&fullscreen&orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=cache_upload&var-layer=backend&var-layer=frontend
[17:24:10] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to stat1007 for kharlan - https://phabricator.wikimedia.org/T216258 (10kostajh)
[17:25:04] <wikibugs>	 (03Restored) 10Dzahn: dumps:nfs: add data types, move variables to parameters and Hiera [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[17:25:12] <wikibugs>	 (03PS5) 10Dzahn: noc: apache -> httpd module [puppet] - 10https://gerrit.wikimedia.org/r/416751
[17:25:44] <wikibugs>	 (03PS1) 10CDanis: thumbor1004: install rasdaemon post-jessie-backport [puppet] - 10https://gerrit.wikimedia.org/r/490890
[17:25:59] <godog>	 quite possible, from this I see it is most likely cp1077 https://logstash.wikimedia.org/goto/bd1992b243532ae86979a7bf0a3aedf9
[17:26:07] <wikibugs>	 (03PS2) 10CDanis: thumbor1004: install rasdaemon post-jessie-backport [puppet] - 10https://gerrit.wikimedia.org/r/490890 (https://phabricator.wikimedia.org/T205396)
[17:26:08] <bblack>	 herron: quite possibly, a large enough purge spike on quite hot objects could cause 503s as well
[17:27:04] <wikibugs>	 (03CR) 10CDanis: [C: 03+2] thumbor1004: install rasdaemon post-jessie-backport [puppet] - 10https://gerrit.wikimedia.org/r/490890 (https://phabricator.wikimedia.org/T205396) (owner: 10CDanis)
[17:35:37] <wikibugs>	 (03PS2) 10Bstorm: wiki replicas: Remove reference to old comment fields [puppet] - 10https://gerrit.wikimedia.org/r/489242 (https://phabricator.wikimedia.org/T212972) (owner: 10Anomie)
[17:36:18] <bblack>	 at the varnish layer, cp1077 was definitely hitting hard caps (reaching configured limits for backwards-facing connections/threads)
[17:36:41] <bblack>	 it's often quite difficult to pin down causes based on stats alone though :/
[17:37:54] <bblack>	 https://grafana.wikimedia.org/d/000000439/varnish-backend-connections?orgId=1&from=1550250287161&to=1550250927474
[17:38:03] <bblack>	 ^ this is a different view, but also I'm not sure how helpful
[17:38:21] <wikibugs>	 (03CR) 10Bstorm: "It seems have rebased away my change.  Trying again." [puppet] - 10https://gerrit.wikimedia.org/r/489242 (https://phabricator.wikimedia.org/T212972) (owner: 10Anomie)
[17:39:19] <wikibugs>	 (03PS3) 10Bstorm: wiki replicas: Remove reference to old comment fields [puppet] - 10https://gerrit.wikimedia.org/r/489242 (https://phabricator.wikimedia.org/T212972) (owner: 10Anomie)
[17:39:45] <bblack>	 the failures do seem to be specific to cp1077, but that could be because it's the chash destination for specific URLs that were temporarily problematic, or it could've been a real cp1077 problem (e.g. network blip or daemon malfunction)
[17:40:22] <wikibugs>	 (03CR) 10Dzahn: "works now https://puppet-compiler.wmflabs.org/compiler1001/14702/mwmaint1002.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/416751 (owner: 10Dzahn)
[17:40:22] <wikibugs>	 10Operations, 10monitoring, 10Patch-For-Review: Evaluate/integrate rasdaemon as a replacement for mcelog - https://phabricator.wikimedia.org/T205396 (10CDanis) in under 10 minutes after installing rasdaemon on thumbor1004 we also saw one there.  that machine is such a consistent performer: {P8090}
[17:40:33] <wikibugs>	 (03PS4) 10Bstorm: wiki replicas: Remove reference to old comment fields [puppet] - 10https://gerrit.wikimedia.org/r/489242 (https://phabricator.wikimedia.org/T212972) (owner: 10Anomie)
[17:41:09] <wikibugs>	 10Operations, 10DC-Ops, 10Wikimedia-Incident, 10cloud-services-team (Kanban): Increase visibility of auto-generated tasks for RAID errors - https://phabricator.wikimedia.org/T216133 (10Marostegui) For what is worth, I do have a Herald rule that automatically subscribes me to any degraded RAID ticket for th...
[17:42:18] <wikibugs>	 (03CR) 10Bstorm: "There was one where the script won't tidily allow the table aliases.  There I left a comment in case we add that to the script." [puppet] - 10https://gerrit.wikimedia.org/r/489242 (https://phabricator.wikimedia.org/T212972) (owner: 10Anomie)
[17:42:31] <wikibugs>	 (03PS5) 10Bstorm: wiki replicas: Remove reference to old comment fields [puppet] - 10https://gerrit.wikimedia.org/r/489242 (https://phabricator.wikimedia.org/T212972) (owner: 10Anomie)
[17:44:51] <wikibugs>	 10Operations, 10DC-Ops, 10Wikimedia-Incident, 10cloud-services-team (Kanban): Increase visibility of auto-generated tasks for RAID errors - https://phabricator.wikimedia.org/T216133 (10Bstorm) Because I had no clue what those are: https://www.mediawiki.org/wiki/Phabricator/Help/Herald_Rules :)
[17:45:12] <jynus>	 bstorm_: so what is the state right now and what do you think we should do?
[17:45:55] <bstorm_>	 Last I checked, every query is stuck in "opening tables".  we are trying to get the cloudvirts to virtualize the database on up and ready so that we can try using the new hardware.
[17:46:20] <bstorm_>	 At this point, we figure the hardware is probably somehow shot or the database just wants to retire in it's old age.
[17:47:27] <jynus>	 so I think it is "just" an overload- more resources are required than available, and that snowballs
[17:47:47] <bstorm_>	 Well, now it is limited to 20 conn per user, but that doesn't matter
[17:47:49] <jynus>	 specially if the overload leads to crashes
[17:47:56] <jynus>	 and the crashes to corruption
[17:47:59] <wikibugs>	 (03CR) 10Anomie: [C: 03+1] wiki replicas: Remove reference to old comment fields [puppet] - 10https://gerrit.wikimedia.org/r/489242 (https://phabricator.wikimedia.org/T212972) (owner: 10Anomie)
[17:48:03] <jynus>	 and the corruption to more slowdown
[17:48:13] <bstorm_>	 Oh I'm sure corruption isn't helping
[17:48:32] <jynus>	 20 per user still enough to get 500+ open connections
[17:48:34] <bstorm_>	 Overall, though, the overloads have been due to slowness.  The slowness doesn't seem to have a strong relationship to growth in toolforge
[17:48:54] <bstorm_>	 It slows down, connections never finish, new connections happen
[17:49:11] <jynus>	 yeah, innodb complains on the error log, but it comnplains of slowdown
[17:49:15] <bstorm_>	 Restarting causes it to work for a an hour or more.
[17:49:41] <bstorm_>	 There will be very few connections, but every query, no matter how gentle or great the plan, hangs
[17:49:52] <bstorm_>	 Killed queries just sit there
[17:50:09] <jynus>	 Thread 140617210279680 has waited at dict0dict.cc line 1134 for 25.000 seconds the semaphore
[17:50:11] <bstorm_>	 It's maddening.  I watched it slowly march to its doom last night.
[17:50:37] <bstorm_>	 I watched the process list for hours. 
[17:50:39] <jynus>	 that means innodb is blocked on the data dictionary
[17:50:49] <bstorm_>	 Yeah, lots of that
[17:51:02] <jynus>	 Mutex at 0x7fe86a02ca68 '&dict_sys->mutex', lock var 1
[17:51:32] <bstorm_>	 But nothing seems to fix that...and the quality and number of queries don't seem to matter.
[17:52:07] <bstorm_>	 restart fixes it briefly...it could be that something kills it when it is under almost no processor or connection load.  But what could do that??
[17:52:41] <jynus>	 in general, what we would do when weirdness is observed on production
[17:52:43] <bstorm_>	 I worry that moving it will not fix thing because the database has been damaged.  However, it is a reasonable attempt, I think.
[17:52:51] <bstorm_>	 That's that?
[17:52:56] <jynus>	 is to save the data by exporting it
[17:52:58] <jynus>	 and rebuild it
[17:53:24] <bstorm_>	 Yeah.   I think that's a good idea.  We are hoping to rebuild it on a VM so we can throw out the server, though that does introduce a new variable :)
[17:53:54] <jynus>	 so I belive it more to be software than hardware, in any case your solution would fix both
[17:56:13] <bstorm_>	 gtirloni appears to have stepped away for a bit.  When the server is ready, we'll probably be glad of some help to correctly export the data and such unless you have a playbook for us to follow :)
[17:56:54] <bstorm_>	 jynus: ^
[17:57:35] <wikibugs>	 10Operations, 10Analytics, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10RobH) Thanks for the input @shilad, its much appreciated!  That info is EXACTLY the kind of info we need (and why this task exists!)   To echo what some of us discussed about this in irc ye...
[17:58:10] <jynus>	 given the issue with the current server I would suggest to shutdown the replica (and be careful with it, as it may be the only good copy existing), and just copy it to the new place
[17:58:47] <jynus>	 copying the original server could make things worse, even if it was run with a logical dump
[17:59:48] <jynus>	 in theory transfer.py can do that, but it is not tested on jessie
[18:00:01] <jynus>	 do you want me to try it, do we have disk somewhere to try?
[18:01:03] <icinga-wm>	 RECOVERY - Check systemd state on an-coord1001 is OK: OK - running: The system is fully operational
[18:03:52] <wikibugs>	 (03PS6) 10Dzahn: noc: apache -> httpd module [puppet] - 10https://gerrit.wikimedia.org/r/416751
[18:04:27] <icinga-wm>	 PROBLEM - Check systemd state on an-coord1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[18:04:41] <bstorm_>	 jynus: I'm not sure anything is up and ready to receive it yet.  :-/   I'll poke around
[18:13:28] <wikibugs>	 (03CR) 10Dzahn: [V: 03+1 C: 03+1] "i used cumin to check for existence of /etc/ImageMagick (without the 6) and it was found on none of mw*" [puppet] - 10https://gerrit.wikimedia.org/r/487888 (owner: 10Muehlenhoff)
[18:16:05] <wikibugs>	 (03CR) 10Dzahn: [C: 03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/489631 (owner: 10Muehlenhoff)
[18:21:55] <wikibugs>	 10Operations, 10Release-Engineering-Team, 10Wikimedia-production-error: HHVM CPU usage when deploying MediaWiki - https://phabricator.wikimedia.org/T208549 (10Krinkle)
[18:22:34] <chaomodus>	 that systemd thing is `eventlogging_to_druid_navigationtiming_hourly.service` presumably this autorecovers since it seemed to?
[18:26:07] <wikibugs>	 (03PS1) 10GTirloni: cloudvirt1019 - Fix network config [puppet] - 10https://gerrit.wikimedia.org/r/490896 (https://phabricator.wikimedia.org/T193264)
[18:26:34] <mutante>	 elukey: ^ fyi
[18:26:53] <mutante>	 chaomodus: my first guess would be something fails and then puppet starts it again and it's called "hourly"
[18:26:57] <mutante>	 flapping?
[18:27:21] <chaomodus>	 i was just looking for a phab task about it but i dun see any
[18:27:47] <mutante>	 same. but i know it's analytics 
[18:27:53] <mutante>	 and the server was setup in https://phabricator.wikimedia.org/T204970
[18:28:01] <mutante>	 that's why i pinged luca
[18:28:26] <wikibugs>	 (03CR) 10GTirloni: [C: 03+2] cloudvirt1019 - Fix network config [puppet] - 10https://gerrit.wikimedia.org/r/490896 (https://phabricator.wikimedia.org/T193264) (owner: 10GTirloni)
[18:28:53] <mutante>	 if you just search "druid navigationtiming" you get some things
[18:29:04] <davidwbarratt>	 question about the replicas... let's say there was a namespace that was created that was non-public and should not be included in the replicas.. would it be possible to remove pages/revisions of that namespace before it goes into the replicas? I assume so, but wanted to double check
[18:29:17] <chaomodus>	 yah
[18:29:18] <elukey>	 an-coord1001 is due to a systemd timer failing sorry
[18:29:24] <chaomodus>	 oke doke
[18:29:25] <elukey>	 will check later on
[18:29:30] <mutante>	 'k, thx :)
[18:29:48] <chaomodus>	 o/\o icinga-wm campers
[18:29:52] <elukey>	 basically we have an alarm if the service unit, triggered by the hourly timer, fails
[18:30:03] <elukey>	 before that it was all crons :D
[18:30:34] <chaomodus>	 timers sure on everyone's mind lately
[18:31:28] <wikibugs>	 (03CR) 10Herron: [C: 03+1] "Looks good to me.  If you need any help with roll out I'm game." [puppet] - 10https://gerrit.wikimedia.org/r/451821 (owner: 10Dzahn)
[18:34:14] <wikibugs>	 (03CR) 10Dzahn: [C: 03+1] "thanks! i would be but the "Friday right before vacation and affects puppetmasters in labs who have other trouble right now" keeps me back" [puppet] - 10https://gerrit.wikimedia.org/r/451821 (owner: 10Dzahn)
[18:35:19] <wikibugs>	 (03CR) 10Dzahn: [C: 03+1] "that being said, properly paranoid because either it's a noop or it isn't and we have already switched basically everything else" [puppet] - 10https://gerrit.wikimedia.org/r/451821 (owner: 10Dzahn)
[18:38:04] <wikibugs>	 (03CR) 10Herron: [C: 03+1] "> that being said, properly paranoid because either it's a noop or it" [puppet] - 10https://gerrit.wikimedia.org/r/451821 (owner: 10Dzahn)
[18:38:59] <bd808>	 davidwbarratt: yes, that is possible. New tables are not exposed on the wiki replicas until they have a security/privacy impact review. We have means for both scrubbing at the point of replication as well as the view layer for redaction of things that are not readily scrubbed. I have a TODO to write up a description of the process to help folks like you understand who to talk to and when about getting things exposed safely.
[18:39:28] <wikibugs>	 10Operations, 10hardware-requests: Two test hosts for SREs - https://phabricator.wikimedia.org/T214024 (10RobH) a:03faidon So we are down to just one single cpu spare misc host.  I'm creating a task to order more spare servers, but for now I can only allocate 1 system for this.   wmf7622 in eqiad: Single Int...
[18:39:52] <wikibugs>	 10Operations, 10hardware-requests: Two test hosts for SREs - https://phabricator.wikimedia.org/T214024 (10RobH) Once this is approved, assign back to me and I'll get it allocated and spun up, then stall this task until a second single cpu misc system arrives for approval for the second of the two systems.
[18:39:53] <davidwbarratt>	 bd808 thanks! yeah we are exploring the idea of having an access-controlled namespace and I just thought I should check before assuming that that was even possible
[18:39:54] <bd808>	 The starting point today is a phab task asking to expose a table and then we work from there to determine impact and mitigation
[18:40:06] <davidwbarratt>	 bd808 ah! perfect
[18:42:42] <bawolff>	 davidwbarratt: namespace as in mediawiki namespace?
[18:42:55] <wikibugs>	 10Operations, 10hardware-requests: Two test hosts for SREs - https://phabricator.wikimedia.org/T214024 (10RobH) Please note T216269 tracks the order of new single cpu spare pool systems.  Once we have those ordered, a second system can be allocated via this task.
[18:43:04] <wikibugs>	 10Operations, 10hardware-requests: Two test hosts for SREs - https://phabricator.wikimedia.org/T214024 (10RobH)
[18:44:16] <bawolff>	 davidwbarratt: There's some hard coded ugliness/edge cases in mediawiki, that make having a secret namespace be sketch and probably not safe
[18:44:17] <bstorm_>	 !log stopped replication and then mariadb on labsdb1004
[18:44:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:46:01] <bawolff>	 davidwbarratt: So if you mean a MW namespace, that's probably a no-go, unless you spend a lot of time guaranteeing its safe in all edge cases
[18:49:32] <wikibugs>	 (03PS1) 10Thcipriani: scap prep: ensure old directories exist, add logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490900
[18:51:20] <wikibugs>	 (03CR) 10Dzahn: [C: 03+1] "ltgm now  https://puppet-compiler.wmflabs.org/compiler1001/14705/" [puppet] - 10https://gerrit.wikimedia.org/r/416751 (owner: 10Dzahn)
[18:51:34] <arturo>	 !log T193264 icinga downtime cloudvirt1019 for 1 week
[18:51:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:51:37] <stashbot>	 T193264: Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020 - https://phabricator.wikimedia.org/T193264
[18:52:33] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: hiera: cloudvirt1009: fix interface name [puppet] - 10https://gerrit.wikimedia.org/r/490901 (https://phabricator.wikimedia.org/T193264)
[18:53:35] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists, 10Patch-For-Review, 10User-herron: Ban recurrent spam to Wikimedia mailing lists (January 2019) - https://phabricator.wikimedia.org/T215251 (10herron) Hi @MarcoAurelio, has this situation improved for you with the above patches merged?
[18:53:49] <wikibugs>	 10Operations, 10ops-eqiad, 10Patch-For-Review, 10cloud-services-team (Kanban): Degraded RAID on cloudvirt1019 - https://phabricator.wikimedia.org/T196507 (10aborrero) 05Open→03Resolved To make it clear that we are now putting this HW into service, I'm closing this task. Please re-open if you have any o...
[18:55:04] <wikibugs>	 10Operations, 10vm-requests: eqiad: (1) Ganeti VM for testing Kerberos in Production - https://phabricator.wikimedia.org/T216238 (10elukey)
[18:55:24] <wikibugs>	 (03PS1) 10GTirloni: cloudvirt1009 - Network config fixes [puppet] - 10https://gerrit.wikimedia.org/r/490902 (https://phabricator.wikimedia.org/T216239)
[18:56:05] <wikibugs>	 (03PS2) 10Dzahn: mediawiki: add data types to parameters [puppet] - 10https://gerrit.wikimedia.org/r/485104
[18:56:28] <wikibugs>	 (03CR) 10Dzahn: [C: 03+1] "restored :p so it's back in the queue in a simplified form" [puppet] - 10https://gerrit.wikimedia.org/r/479335 (owner: 10Dzahn)
[18:56:45] <wikibugs>	 (03PS2) 10GTirloni: cloudvirt1009 - Network config fixes [puppet] - 10https://gerrit.wikimedia.org/r/490902 (https://phabricator.wikimedia.org/T216239)
[18:57:03] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mediawiki: add data types to parameters [puppet] - 10https://gerrit.wikimedia.org/r/485104 (owner: 10Dzahn)
[18:57:19] <wikibugs>	 (03CR) 10GTirloni: [C: 03+2] cloudvirt1009 - Network config fixes [puppet] - 10https://gerrit.wikimedia.org/r/490902 (https://phabricator.wikimedia.org/T216239) (owner: 10GTirloni)
[18:58:12] <wikibugs>	 10Operations, 10Elasticsearch, 10Discovery-Search (Current work), 10Patch-For-Review: Create Icinga check for failed shard allocation - https://phabricator.wikimedia.org/T212850 (10debt) 05Open→03Resolved
[18:58:28] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists, 10Patch-For-Review, 10User-herron: Ban recurrent spam to Wikimedia mailing lists (January 2019) - https://phabricator.wikimedia.org/T215251 (10MarcoAurelio) Hello @herron - Since the merge of the patch that got to be reverted (Feb 5/6) I've not received any spam ema...
[18:58:52] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to stat1007 for kharlan - https://phabricator.wikimedia.org/T216258 (10marcella) Approved.
[19:00:03] <wikibugs>	 (03PS3) 10Dzahn: mediawiki: add data types to parameters [puppet] - 10https://gerrit.wikimedia.org/r/485104
[19:00:06] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] hiera: cloudvirt1009: fix interface name [puppet] - 10https://gerrit.wikimedia.org/r/490901 (https://phabricator.wikimedia.org/T193264) (owner: 10Arturo Borrero Gonzalez)
[19:00:17] <wikibugs>	 (03PS2) 10Arturo Borrero Gonzalez: hiera: cloudvirt1009: fix interface name [puppet] - 10https://gerrit.wikimedia.org/r/490901 (https://phabricator.wikimedia.org/T193264)
[19:00:21] <icinga-wm>	 RECOVERY - Check systemd state on an-coord1001 is OK: OK - running: The system is fully operational
[19:00:43] <wikibugs>	 10Operations, 10Cloud-VPS, 10cloud-services-team, 10Discovery-Search (Current work), 10Patch-For-Review: Setup elasticsearch on cloudelastic100[1-4] - https://phabricator.wikimedia.org/T214921 (10debt)
[19:00:46] <wikibugs>	 10Operations, 10Cloud-VPS, 10SRE-Access-Requests, 10cloud-services-team, and 2 others: Create cloudelastic-root group - https://phabricator.wikimedia.org/T214922 (10debt) 05Open→03Resolved a:03debt
[19:04:51] <mutante>	 nuria: re: https://phabricator.wikimedia.org/T215555  if we have confirmed the 3 listed users are already in the right group ..does that resolve it or should we check for more search team members 
[19:04:53] <icinga-wm>	 PROBLEM - Check systemd state on an-coord1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[19:06:56] <icinga-wm>	 ACKNOWLEDGEMENT - Check systemd state on an-coord1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. daniel_zahn per IRC with Luca, persistent comment
[19:07:21] <elukey>	 thanks mutante !
[19:07:27] <elukey>	 I was about to do it
[19:07:39] <mutante>	 :) yw
[19:09:12] <mutante>	 elukey: i see you were also involved in "access to turnilo for members of search team" and asked the same question as i did above
[19:09:58] <mutante>	 i think i'll resolve it. the 3 listed users are in the group. confirmed
[19:15:46] <elukey>	 yeah I think we can resolve and re-open if necessart
[19:19:10] <mutante>	 ok, ack
[19:20:25] <wikibugs>	 10Operations, 10ORES, 10Release Pipeline, 10Scoring-platform-team, 10Release-Engineering-Team (Backlog): The continuous release pipeline should support more than one service per repo - https://phabricator.wikimedia.org/T210267 (10thcipriani)
[19:22:59] <wikibugs>	 10Operations: New cronspam from db clusters - https://phabricator.wikimedia.org/T216273 (10colewhite)
[19:31:49] <wikibugs>	 10Operations, 10ops-eqiad, 10Toolforge, 10cloud-services-team (Kanban): Degraded RAID on labsdb1005 - https://phabricator.wikimedia.org/T216223 (10colewhite) p:05Triage→03Normal
[19:32:40] <wikibugs>	 10Operations: New cronspam from db clusters - https://phabricator.wikimedia.org/T216273 (10colewhite)
[19:32:42] <wikibugs>	 10Operations, 10ops-eqiad, 10Toolforge, 10cloud-services-team (Kanban): Degraded RAID on labsdb1005 - https://phabricator.wikimedia.org/T216223 (10Bstorm) Directly related to T216202
[19:32:45] <wikibugs>	 10Operations, 10Patch-For-Review: Tracking and Reducing cron-spam to root@ - https://phabricator.wikimedia.org/T132324 (10colewhite)
[19:33:05] <wikibugs>	 10Operations: New cronspam from db clusters - https://phabricator.wikimedia.org/T216273 (10colewhite) p:05Triage→03Normal
[19:33:27] <wikibugs>	 10Operations, 10MediaWiki-Maintenance-scripts: cron spam for slow queries on mwmaint /usr/local/bin/foreachwiki initSiteStats.php --update > /dev/null - https://phabricator.wikimedia.org/T216243 (10colewhite) p:05Triage→03Normal
[19:33:56] <wikibugs>	 10Operations, 10ops-eqiad, 10Toolforge, 10cloud-services-team (Kanban): Degraded RAID on labsdb1005 - https://phabricator.wikimedia.org/T216223 (10Bstorm) This was triggered when I manually offlined the failing disk in order to recover the database filesystem.  So far, it hasn't helped as much as I was rea...
[19:34:32] <wikibugs>	 10Operations, 10ops-eqiad: Disk failure on labsdb1005 - https://phabricator.wikimedia.org/T216202 (10Bstorm)
[19:34:34] <wikibugs>	 10Operations, 10ops-eqiad, 10Toolforge, 10cloud-services-team (Kanban): Degraded RAID on labsdb1005 - https://phabricator.wikimedia.org/T216223 (10Bstorm)
[19:34:54] <wikibugs>	 10Operations, 10ops-eqiad: Disk failure on labsdb1005 - https://phabricator.wikimedia.org/T216202 (10colewhite) p:05Triage→03Normal
[19:35:15] <wikibugs>	 (03CR) 10CRusnov: [C: 03+2] Add .gitreview file [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/488909 (owner: 10Hashar)
[19:35:48] <wikibugs>	 10Operations, 10Analytics, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10colewhite) p:05Triage→03Normal
[19:36:55] <wikibugs>	 10Operations, 10Discovery-Search, 10Elasticsearch, 10User-fgiunchedi: cleanup reprepro configuration for elasticsearch-curator - https://phabricator.wikimedia.org/T216235 (10colewhite) p:05Triage→03Normal
[19:38:55] <wikibugs>	 (03PS3) 10CRusnov: Make oldhardware report exclued cablemgmt and storagebin [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/490700
[19:41:46] <icinga-wm>	 PROBLEM - ElasticSearch shard size check - eqiad-search- on search.svc.eqiad.wmnet is CRITICAL: CRITICAL - commonswiki_content_1538078672(61gb)
[19:41:55] <arturo>	 !log T193264 reimaging cloudvirt1019 to get mitaka/stretch
[19:41:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:41:59] <stashbot>	 T193264: Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020 - https://phabricator.wikimedia.org/T193264
[19:54:08] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] "SORRY wrong commit title. This is about cloudvirt1019." [puppet] - 10https://gerrit.wikimedia.org/r/490901 (https://phabricator.wikimedia.org/T193264) (owner: 10Arturo Borrero Gonzalez)
[20:00:22] <icinga-wm>	 RECOVERY - Check systemd state on an-coord1001 is OK: OK - running: The system is fully operational
[20:08:15] <wikibugs>	 10Operations, 10Analytics, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10EBernhardson) >>! In T216226#4957729, @RobH wrote: > Thanks for the input @shilad, its much appreciated!  That info is EXACTLY the kind of info we need (and why this task exists!) >  >  > T...
[20:09:38] <wikibugs>	 (03PS4) 10CRusnov: Make oldhardware report exclued cablemgmt and storagebin [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/490700
[20:10:17] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Make oldhardware report exclued cablemgmt and storagebin [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/490700 (owner: 10CRusnov)
[20:10:19] <wikibugs>	 (03CR) 10CRusnov: "> Patch Set 2: Code-Review-1" (031 comment) [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/490700 (owner: 10CRusnov)
[20:10:21] <wikibugs>	 10Operations, 10Parsoid, 10RESTBase, 10Traffic, and 5 others: Consider stashing data-parsoid for VE - https://phabricator.wikimedia.org/T215956 (10mobrovac) A possible alternative is to differentiate VE's requests based on the UA header: VE [sets both `Api-User-Agent` and `User-Agent`](https://github.com/w...
[20:11:53] <wikibugs>	 (03CR) 10Dzahn: [C: 04-1] "Class[Mediawiki::Syslog]: parameter 'forward_syslog' expects a Boolean value, got String" [puppet] - 10https://gerrit.wikimedia.org/r/485104 (owner: 10Dzahn)
[20:13:47] <wikibugs>	 (03PS5) 10CRusnov: Make oldhardware report exclued cablemgmt and storagebin [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/490700
[20:15:18] <wikibugs>	 (03PS4) 10Dzahn: mediawiki: add data types to parameters [puppet] - 10https://gerrit.wikimedia.org/r/485104
[20:21:57] <wikibugs>	 (03CR) 10Dzahn: "works now, at least on mwdebug: https://puppet-compiler.wmflabs.org/compiler1001/14707/mwdebug1001.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/485104 (owner: 10Dzahn)
[20:34:07] <wikibugs>	 10Operations, 10Parsoid, 10RESTBase, 10Traffic, and 5 others: Consider stashing data-parsoid for VE - https://phabricator.wikimedia.org/T215956 (10Pchelolo) I don't think complicating Varnish logic worths here, I don't see how UA is better than a query parameter regarding DDOs. We can mitigate the DDOS pos...
[20:40:12] <andrewbogott>	 !log enabled virtualization (all three settings) on cloudvirt1019
[20:40:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:41:18] <wikibugs>	 10Operations, 10ops-eqiad, 10Patch-For-Review, 10cloud-services-team (Kanban): Degraded RAID on cloudvirt1020 - https://phabricator.wikimedia.org/T194855 (10Andrew) note that I just now had to enable virtualization and associated settings on cloudvirt1019.  so 1020 might need this as well -- probably best...
[20:42:49] <wikibugs>	 10Operations, 10Parsoid, 10RESTBase, 10Traffic, and 5 others: Consider stashing data-parsoid for VE - https://phabricator.wikimedia.org/T215956 (10mobrovac) >>! In T215956#4958249, @Pchelolo wrote: > I don't think complicating Varnish logic worths here, I don't see how UA is better than a query parameter r...
[20:51:51] <wikibugs>	 10Operations, 10ORES, 10Release Pipeline, 10Scoring-platform-team, 10Release-Engineering-Team (Backlog): The continuous release pipeline should support more than one service per repo - https://phabricator.wikimedia.org/T210267 (10dduvall) I went a little crazy with a new config proposal in anticipation o...
[21:05:21] <wikibugs>	 10Operations, 10Maps, 10Traffic, 10Reading-Infrastructure-Team-Backlog (Kanban): Decide on Cache-Control headers for map tiles - https://phabricator.wikimedia.org/T186732 (10Mholloway) @Jhernandez Looks like this one has a change waiting to roll out when the Stretch migration is complete, so per our discus...
[21:06:12] <wikibugs>	 (03PS1) 10Andrew Bogott: purge references to labvirt1009 [dns] - 10https://gerrit.wikimedia.org/r/490930 (https://phabricator.wikimedia.org/T216239)
[21:07:06] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 03+2] purge references to labvirt1009 [dns] - 10https://gerrit.wikimedia.org/r/490930 (https://phabricator.wikimedia.org/T216239) (owner: 10Andrew Bogott)
[21:08:52] <wikibugs>	 10Operations, 10ops-eqiad, 10cloud-services-team (Kanban): Update label and switch to rename labvirt1009 to cloudvirt1009 - https://phabricator.wikimedia.org/T216281 (10Andrew) p:05Triage→03High
[21:19:03] <wikibugs>	 (03PS7) 10Paladox: Add support for "recheck" as button in PolyGerrit's ui [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/489482 (https://phabricator.wikimedia.org/T214631)
[21:27:55] <wikibugs>	 (03PS8) 10Jforrester: Add support for "recheck" as button in PolyGerrit's ui [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/489482 (https://phabricator.wikimedia.org/T214631) (owner: 10Paladox)
[21:29:55] <wikibugs>	 (03CR) 10Legoktm: [C: 03+1] "Can't comment on the technical aspects of the change, but fully supportive of the idea :)" [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/489482 (https://phabricator.wikimedia.org/T214631) (owner: 10Paladox)
[21:30:38] <wikibugs>	 (03CR) 10Paladox: [V: 03+2 C: 03+2] "Tested locally and works: https://phabricator.wikimedia.org/F28225474" [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/489482 (https://phabricator.wikimedia.org/T214631) (owner: 10Paladox)
[21:34:12] <wikibugs>	 10Operations, 10Parsoid, 10RESTBase, 10Traffic, and 5 others: Consider stashing data-parsoid for VE - https://phabricator.wikimedia.org/T215956 (10BBlack) Correct me if I'm wrong, but I would think all VE traffic would already be uncacheable at the Varnish level anyways, since it happens in the context of...
[21:34:45] <wikibugs>	 (03PS11) 10Paladox: Introduce gr-wikimedia-prettify-ci-comments [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/489483 (https://phabricator.wikimedia.org/T215658)
[21:48:52] <wikibugs>	 (03PS1) 10Cwhite: admin: add Angela Muigai to ldap_only_users [puppet] - 10https://gerrit.wikimedia.org/r/490932 (https://phabricator.wikimedia.org/T216101)
[21:50:53] <wikibugs>	 (03PS12) 10Paladox: Introduce gr-wikimedia-prettify-ci-comments [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/489483 (https://phabricator.wikimedia.org/T215658)
[21:50:55] <wikibugs>	 (03PS9) 10Paladox: Add support for cherry picking with merge conflicts from the UI (PolyGerrit) [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/490225
[21:59:49] <icinga-wm>	 PROBLEM - IPv6 ping to eqiad on ripe-atlas-eqiad IPv6 is CRITICAL: CRITICAL - failed 42 probes of 390 (alerts on 35) - https://atlas.ripe.net/measurements/1790947/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23RIPE_alerts
[22:10:15] <icinga-wm>	 RECOVERY - IPv6 ping to eqiad on ripe-atlas-eqiad IPv6 is OK: OK - failed 19 probes of 390 (alerts on 35) - https://atlas.ripe.net/measurements/1790947/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23RIPE_alerts
[22:44:48] <wikibugs>	 (03CR) 10Dzahn: [V: 03+1 C: 03+1] admin: add Angela Muigai to ldap_only_users [puppet] - 10https://gerrit.wikimedia.org/r/490932 (https://phabricator.wikimedia.org/T216101) (owner: 10Cwhite)
[22:47:24] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "LGTM" [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/490700 (owner: 10CRusnov)
[22:48:20] <wikibugs>	 (03PS1) 10Bstorm: toolsdb: refactoring some of the mariadb things for toolsdb [puppet] - 10https://gerrit.wikimedia.org/r/490946 (https://phabricator.wikimedia.org/T193264)
[22:49:17] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] toolsdb: refactoring some of the mariadb things for toolsdb [puppet] - 10https://gerrit.wikimedia.org/r/490946 (https://phabricator.wikimedia.org/T193264) (owner: 10Bstorm)
[22:56:09] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to stat1007 for kharlan - https://phabricator.wikimedia.org/T216258 (10Dzahn) Hi, generally access is granted based on admin groups instead of individual host names. So it can move to new servers without having to make changes.  https://wikitech.wikimedia....
[22:56:39] <wikibugs>	 (03PS3) 10ArielGlenn: showcrcs: util to write out crc information from a bzip2 file [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/490299 (https://phabricator.wikimedia.org/T216009)
[22:57:32] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to stat1007 for kharlan - https://phabricator.wikimedia.org/T216258 (10Dzahn) p:05Triage→03Normal
[23:03:02] <wikibugs>	 (03PS1) 10BryanDavis: wmcs: Allow wmcs-roots full sudo on wmcs non-replica dbs [puppet] - 10https://gerrit.wikimedia.org/r/490948
[23:05:21] <wikibugs>	 (03CR) 10BryanDavis: "Added _joe_ as a reviews because he is currently listed for clinic duty. This should get a review at the next SRE meeting. Hopefully it wi" [puppet] - 10https://gerrit.wikimedia.org/r/490948 (owner: 10BryanDavis)
[23:13:26] <wikibugs>	 (03PS2) 10BryanDavis: toolsdb: refactoring some of the mariadb things for toolsdb [puppet] - 10https://gerrit.wikimedia.org/r/490946 (https://phabricator.wikimedia.org/T193264) (owner: 10Bstorm)
[23:24:11] <wikibugs>	 (03CR) 10CRusnov: [C: 03+2] Make oldhardware report exclued cablemgmt and storagebin [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/490700 (owner: 10CRusnov)
[23:24:28] <wikibugs>	 (03PS3) 10BryanDavis: toolsdb: refactoring some of the mariadb things for toolsdb [puppet] - 10https://gerrit.wikimedia.org/r/490946 (https://phabricator.wikimedia.org/T193264) (owner: 10Bstorm)
[23:25:25] <wikibugs>	 (03PS6) 10CRusnov: Make oldhardware report exclued cablemgmt and storagebin [software/netbox-reports] - 10https://gerrit.wikimedia.org/r/490700
[23:31:19] <wikibugs>	 (03CR) 10Bstorm: [C: 03+2] toolsdb: refactoring some of the mariadb things for toolsdb [puppet] - 10https://gerrit.wikimedia.org/r/490946 (https://phabricator.wikimedia.org/T193264) (owner: 10Bstorm)
[23:46:50] <_joe_>	 bd808: I'm not on clinic duty. Whoever is (I don't remember) forgot to change the topic for one full week apparently
[23:56:30] <bd808>	 _joe_: heh. sorry :)
[23:56:46] <_joe_>	 bd808: not your fault :P