[00:23:48] 06Operations, 10Beta-Cluster-Infrastructure, 07HHVM: Beta-cluster web server fills up /var/log with Apache logs - https://phabricator.wikimedia.org/T75262#755023 (10AlexMonk-WMF) I haven't seen this issue in a long while... [00:50:23] 06Operations, 06Labs, 10Labs-Infrastructure: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#2552368 (10AlexMonk-WMF) Was this completed? ```krenair@bastion-01:~$ ldapsearch -x "(&(objectClass=novauser)(!(loginShell=/bin/bash)))" # extended LDIF # # LDAPv3 # base... [00:54:23] 06Operations, 06Labs, 10Labs-Infrastructure: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#2552371 (10AlexMonk-WMF) Ugh, wrong objectClass: ```krenair@bastion-01:~$ ldapsearch -x "(&(objectClass=person)(!(loginShell=/bin/bash)))" | grep dn: | grep ou=people | gr... [00:59:03] 06Operations, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Set up LVS for labs dns recursors - https://phabricator.wikimedia.org/T119660#2552373 (10AlexMonk-WMF) See also T133389 [01:19:17] (03PS12) 10BryanDavis: [WIP] Provision Striker via scap3 [puppet] - 10https://gerrit.wikimedia.org/r/301505 (https://phabricator.wikimedia.org/T141014) [01:27:19] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [50.0] [01:29:19] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [01:41:26] 06Operations, 06Labs, 10Labs-Infrastructure, 07IPv6: Enable ipv6 on labs - https://phabricator.wikimedia.org/T37947#399081 (10AlexMonk-WMF) >>! In T37947#399351, @scfc wrote: > http://permalink.gmane.org/gmane.org.wikimedia.labs/2651: > > | > Of particular interest would be to hear if there are plans to I... [02:24:40] !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 21s) [02:24:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:26:33] 06Operations, 06Labs, 10Labs-Infrastructure, 10Monitoring: Have a paging check for Nova API accessible - https://phabricator.wikimedia.org/T133656#2238563 (10AlexMonk-WMF) It's not just as simple as checking whether http://labnet1002.eqiad.wmnet:8774 is up, is it? [02:30:30] !log l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 15 02:30:30 UTC 2016 (duration 5m 51s) [02:30:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [03:27:38] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 978.33 seconds [03:35:38] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 235.71 seconds [05:38:16] LALALAALLALA [05:38:17] JEM [05:38:23] SEXO CON ASMOCBOT [06:34:45] (03PS1) 10Yuvipanda: k8s: Restart k8s master components when they crash [puppet] - 10https://gerrit.wikimedia.org/r/304778 [06:46:11] (03CR) 10Muehlenhoff: Disable unprivileged user namespaces on trusty systems (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/304474 (https://phabricator.wikimedia.org/T142567) (owner: 10Muehlenhoff) [06:53:27] PROBLEM - puppet last run on elastic2013 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:19] PROBLEM - puppet last run on stat1004 is CRITICAL: CRITICAL: Puppet has 1 failures [07:03:20] 06Operations, 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Rename specific account in LDAP, Wikitech, Gerrit and Phabricator - https://phabricator.wikimedia.org/T85913#2552603 (10adrianheine) @hashar: Since I left WMDE, both addresses will be bouncing. Just remove me from the list. [07:18:49] RECOVERY - puppet last run on elastic2013 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:23:42] (03CR) 10Muehlenhoff: [C: 032] gerrit (2.12.3-wmf.1) jessie-wikimedia; urgency=low [debs/gerrit] - 10https://gerrit.wikimedia.org/r/304486 (owner: 10Chad) [07:24:38] RECOVERY - puppet last run on stat1004 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:29:51] !log upgrading openjdk on maps clusters (along with cassandra restart) [07:29:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [07:35:25] (03PS1) 10Merlijn van Deen: toollabs: install pdf2djvu [puppet] - 10https://gerrit.wikimedia.org/r/304788 (https://phabricator.wikimedia.org/T130138) [07:39:05] (03PS2) 10Merlijn van Deen: toollabs: install pdf2djvu [puppet] - 10https://gerrit.wikimedia.org/r/304788 (https://phabricator.wikimedia.org/T130138) [08:16:27] PROBLEM - cassandra CQL 10.192.32.146:9042 on maps2003 is CRITICAL: Connection refused [08:18:19] RECOVERY - cassandra CQL 10.192.32.146:9042 on maps2003 is OK: TCP OK - 0.037 second response time on port 9042 [08:19:19] PROBLEM - puppet last run on mw2075 is CRITICAL: CRITICAL: Puppet has 1 failures [08:44:57] RECOVERY - puppet last run on mw2075 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [09:00:32] 06Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests: Renaming the Aramaic (arc) Wikipedia to the Syriac (syc) Wikipedia - https://phabricator.wikimedia.org/T28725#2552717 (10Liuxinyu970226) Not sure if this is still needed or not, one user created Wp/syc on Incubator, which is probably planni... [09:43:21] 06Operations, 10Traffic, 13Patch-For-Review: Investigate TCP Fast Open for tlsproxy - https://phabricator.wikimedia.org/T108827#2552754 (10ema) 05Open>03Resolved The high rate of failed incoming TFO connections in esams seems to have stopped since [[ https://gerrit.wikimedia.org/r/297418| we switched th... [09:45:33] 06Operations, 10Traffic: Sideways Only-If-Cached on misses at a primary DC - https://phabricator.wikimedia.org/T142841#2552771 (10ema) p:05Triage>03Normal [09:46:38] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [50.0] [09:48:37] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [09:53:54] (03PS1) 10Muehlenhoff: Fix malformed changelog entry (timezone is mandatory) [debs/gerrit] - 10https://gerrit.wikimedia.org/r/304796 [09:54:43] (03CR) 10Muehlenhoff: [C: 032] Fix malformed changelog entry (timezone is mandatory) [debs/gerrit] - 10https://gerrit.wikimedia.org/r/304796 (owner: 10Muehlenhoff) [10:06:04] !log rolling restart of cassandra on restbase2* to pick up openjdk security update [10:06:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [10:13:54] (03CR) 10Paladox: "Thanks." [debs/gerrit] - 10https://gerrit.wikimedia.org/r/304486 (owner: 10Chad) [10:26:18] (03PS1) 10Ema: cache_upload backend: convert range requests into pass [puppet] - 10https://gerrit.wikimedia.org/r/304802 (https://phabricator.wikimedia.org/T142076) [10:33:28] 06Operations, 06Discovery, 06Maps, 10Maps-data, 10hardware-requests: 2 servers for maps-beta cluster - https://phabricator.wikimedia.org/T138600#2552869 (10Gehel) 05Open>03Resolved a:03Gehel Maps servers are in place. Initial configuration is tracked by T138092. Closing this. [10:38:13] 06Operations, 06Discovery, 06Maps, 10Maps-data, 10hardware-requests: 2 servers for maps-beta cluster - https://phabricator.wikimedia.org/T138600#2552877 (10Gehel) 05Resolved>03Open Oops, I mixed up requests. The maps-beta cluster is not yet ready and discussion still needs to happen. We should at le... [10:45:37] 06Operations, 06Discovery, 06Maps: Maps - remove multiple JVM versions from maps servers - https://phabricator.wikimedia.org/T142977#2552901 (10Gehel) [10:46:54] 06Operations, 06Discovery, 06Maps: Maps - remove multiple JVM versions from maps servers - https://phabricator.wikimedia.org/T142977#2552921 (10Gehel) [10:49:42] !log restbase restring RB in codfw following the jdk upgrade and cassandra restarts [10:49:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [10:54:10] !log uploaded gerrit 2.12.3 to apt.wikimedia.org [10:54:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [10:54:22] ^ paladox, ostriches [11:04:28] PROBLEM - Juniper alarms on asw-d-eqiad.mgmt.eqiad.wmnet is CRITICAL: JNX_ALARMS CRITICAL - Received genError(5) error-status at error-index 1 [11:06:27] RECOVERY - Juniper alarms on asw-d-eqiad.mgmt.eqiad.wmnet is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms [11:07:07] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet has 1 failures [11:08:17] !log rolling restart of cassandra on restbase1* to pick up openjdk security update [11:08:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [11:13:07] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:25:51] 06Operations, 06Discovery, 06Maps: Maps - remove multiple JVM versions from maps servers - https://phabricator.wikimedia.org/T142977#2552978 (10Gehel) After a refresher on how depbian package dependencies work and some digging into it: * osmosis depends on `default-jre-headless | java6-runtime-headless` * o... [11:42:13] (03PS1) 10Gehel: Maps - ensure Osmosis is only installed after JRE is available [puppet] - 10https://gerrit.wikimedia.org/r/304806 (https://phabricator.wikimedia.org/T142977) [11:42:32] moritzm thanks :) [11:45:07] PROBLEM - check_mysql on fdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2658 [11:50:08] PROBLEM - check_mysql on fdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2958 [11:51:16] (03CR) 10Merlijn van Deen: "'Archived projects with orphaned tasks'?" [puppet] - 10https://gerrit.wikimedia.org/r/303500 (https://phabricator.wikimedia.org/T142347) (owner: 10Aklapper) [11:53:56] (03CR) 10BBlack: [C: 031] Switch India & BIOT to esams (4) [dns] - 10https://gerrit.wikimedia.org/r/257843 (owner: 10Faidon Liambotis) [11:55:09] RECOVERY - check_mysql on fdb2001 is OK: Uptime: 1624328 Threads: 1 Questions: 29320206 Slow queries: 9963 Opens: 2683 Flush tables: 2 Open tables: 593 Queries per second avg: 18.050 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [11:56:31] 06Operations, 06Services, 06Services-next, 05Security, 15User-mobrovac: Productize the Electron PDF render service & create a REST API end point - https://phabricator.wikimedia.org/T142226#2552992 (10Lea_WMDE) > If you take this on, would somebody from WMDE be able to take the lead during your absence? @... [12:07:41] (03CR) 10Danny B.: "> 'Archived projects with orphaned tasks'?" [puppet] - 10https://gerrit.wikimedia.org/r/303500 (https://phabricator.wikimedia.org/T142347) (owner: 10Aklapper) [12:07:58] (03CR) 10Muehlenhoff: [C: 04-1] "default-jre-headless in jessie has a Depends: on openjdk-7-jre-headless, independant of any ordering this will always lead to the installa" [puppet] - 10https://gerrit.wikimedia.org/r/304806 (https://phabricator.wikimedia.org/T142977) (owner: 10Gehel) [12:27:57] (03CR) 10Muehlenhoff: "I missed the fact that openjdk-8 still provides java6-runtime-headless." [puppet] - 10https://gerrit.wikimedia.org/r/304806 (https://phabricator.wikimedia.org/T142977) (owner: 10Gehel) [12:36:26] (03PS2) 10Ema: cache_upload backend: convert range requests into pass [puppet] - 10https://gerrit.wikimedia.org/r/304802 (https://phabricator.wikimedia.org/T142076) [12:37:32] (03PS3) 10Ema: cache_upload backend: convert range requests into pass [puppet] - 10https://gerrit.wikimedia.org/r/304802 (https://phabricator.wikimedia.org/T142076) [12:46:24] 06Operations: Review lists of config/sysctl recommendations by "kernel self-protection project" - https://phabricator.wikimedia.org/T142984#2553120 (10MoritzMuehlenhoff) [13:05:11] (03CR) 10Faidon Liambotis: [C: 032] Switch India & BIOT to esams (4) [dns] - 10https://gerrit.wikimedia.org/r/257843 (owner: 10Faidon Liambotis) [13:59:25] !log upgrading mw1293 to firejail 0.9.40 [13:59:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [14:01:58] 06Operations: Encrypt all the things - https://phabricator.wikimedia.org/T111653#2553328 (10Jgreen) [14:06:51] (03CR) 10Gehel: "LGTM. I don't know enough about our networking constraints to actually validate this. elasticsearch production servers should only be acce" [puppet] - 10https://gerrit.wikimedia.org/r/304483 (owner: 10Muehlenhoff) [14:08:38] (03PS4) 10Ema: cache_upload backend: convert range requests into pass [puppet] - 10https://gerrit.wikimedia.org/r/304802 (https://phabricator.wikimedia.org/T142076) [14:12:15] (03CR) 10BBlack: [C: 031] cache_upload backend: convert range requests into pass [puppet] - 10https://gerrit.wikimedia.org/r/304802 (https://phabricator.wikimedia.org/T142076) (owner: 10Ema) [14:14:00] 06Operations: Encrypt all the things - https://phabricator.wikimedia.org/T111653#2553354 (10Jgreen) [14:14:03] (03CR) 10Ema: [C: 032] cache_upload backend: convert range requests into pass [puppet] - 10https://gerrit.wikimedia.org/r/304802 (https://phabricator.wikimedia.org/T142076) (owner: 10Ema) [14:17:55] (03PS1) 10Ottomata: Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9 [puppet] - 10https://gerrit.wikimedia.org/r/304821 (https://phabricator.wikimedia.org/T138265) [14:17:57] (03PS1) 10Ottomata: Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9 [puppet] - 10https://gerrit.wikimedia.org/r/304822 (https://phabricator.wikimedia.org/T138265) [14:19:48] PROBLEM - puppet last run on labnodepool1001 is CRITICAL: CRITICAL: Puppet last ran 4 days ago [14:21:47] RECOVERY - puppet last run on labnodepool1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:23:23] (03PS2) 10Ottomata: Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9 [puppet] - 10https://gerrit.wikimedia.org/r/304821 (https://phabricator.wikimedia.org/T138265) [14:23:38] (03PS2) 10Ottomata: Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9 [puppet] - 10https://gerrit.wikimedia.org/r/304822 (https://phabricator.wikimedia.org/T138265) [14:24:55] !log upgrading firejail on mw* to 0.9.40 [14:24:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [14:37:38] (03PS1) 10Ottomata: Remove now unused kafka module [puppet] - 10https://gerrit.wikimedia.org/r/304827 (https://phabricator.wikimedia.org/T138265) [14:38:12] (03CR) 10Ottomata: "To be merged after main-eqiad upgrade and https://gerrit.wikimedia.org/r/#/c/304822/" [puppet] - 10https://gerrit.wikimedia.org/r/304827 (https://phabricator.wikimedia.org/T138265) (owner: 10Ottomata) [14:42:57] (03PS1) 10Muehlenhoff: xhgui: Restrict to domain networks [puppet] - 10https://gerrit.wikimedia.org/r/304831 [14:47:46] (03CR) 10EBernhardson: [C: 031] "everything should be good for giving nobelium back." [puppet] - 10https://gerrit.wikimedia.org/r/304112 (https://phabricator.wikimedia.org/T142581) (owner: 10Dzahn) [14:48:33] 06Operations, 10Phabricator: Renew phab.wmfusercontent.org https certificate - https://phabricator.wikimedia.org/T142951#2553421 (10RobH) 05Open>03Resolved a:03RobH This is already being handled on task T140649. That task involves purchase authorization, so it is a private task. Thus this task is a dup... [14:49:59] (03PS1) 10Milimetric: Remove newline from rendered password [puppet] - 10https://gerrit.wikimedia.org/r/304833 [14:50:19] ACKNOWLEDGEMENT - HTTPS-wmfusercontent on phab.wmfusercontent.org is CRITICAL: SSL CRITICAL - Certificate *.wmfusercontent.org valid until 2016-09-12 13:41:12 +0000 (expires in 27 days) rhalsell new cert being ordered on T140649 [14:51:27] (03CR) 10EBernhardson: [C: 031] "lgtm" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304204 (https://phabricator.wikimedia.org/T142705) (owner: 10Gehel) [14:54:11] (03CR) 10jenkins-bot: [V: 04-1] Remove newline from rendered password [puppet] - 10https://gerrit.wikimedia.org/r/304833 (owner: 10Milimetric) [14:57:38] (03PS2) 10Milimetric: Remove newline from rendered password [puppet] - 10https://gerrit.wikimedia.org/r/304833 [14:59:12] (03CR) 10Ottomata: [C: 032] Remove newline from rendered password [puppet] - 10https://gerrit.wikimedia.org/r/304833 (owner: 10Milimetric) [15:00:05] anomie, ostriches, thcipriani, hashar, and twentyafterfour: Dear anthropoid, the time has come. Please deploy Morning SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160815T1500). [15:00:22] nothing on the deployments page today. [15:00:38] crazy! [15:00:53] Convenient, since I need to restart gerrit for the upgrade :) [15:01:09] RELEASE THE CHAOS MONKEY [15:01:30] * bd808 waits for js fixes with great anticipation [15:01:43] bd808: I pulled that fix from the new version just to bug you ;-) [15:01:45] jk <3 [15:02:08] ostriches: that would be a great troll [15:03:09] !log gerrit: upgrading 2.12.2 -> 2.12.3. Quick restart will happen. [15:03:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:04:17] And we're back! [15:04:18] :) [15:04:18] gerrit upgrade, again :D Gerrit gets unusual love :D :D [15:07:58] PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:18] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Puppet has 2 failures [15:14:56] I can copy with keyboard shortcuts again. [15:15:08] * bd808 hugs ostriches [15:15:56] [ and ] still aren't working in the unified diff view [15:16:19] I think that's fixed in 2.12.4 which isn't out just yet. [15:16:29] *nod* I'll survive [15:17:10] FlorianSW: Minor point upgrade. Biggest problem was something about keyboard shortcuts and Firefox :p [15:18:15] ostriches: On a german keyboard, shortcuts weren't working for years, and I'm using Chrome, so :D But thanks for your maintenance work all the time! :) [15:19:47] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [15:20:50] FlorianSW: Oh no, this was a little different from that. [15:21:04] It was that gerrit's shortcuts were *taking over* your system ones like copy/paste. [15:22:07] ostriches: aha, that's bad :/ Great that this is fixed :) [15:23:19] 06Operations, 10hardware-requests, 10Continuous-Integration-Infrastructure (phase-out-gallium): Allocate contint1001 to releng and allocate to a vlan - https://phabricator.wikimedia.org/T140257#2553490 (10thcipriani) >>! In T140257#2491705, @faidon wrote: > I've deliberated this a little bit and honestly my... [15:23:55] 06Operations, 10ops-requests: Servers for CirrusSearch's Elasticsearch Instances - https://phabricator.wikimedia.org/T83046#2553493 (10RobH) [15:24:13] 06Operations, 10ops-eqiad: Rack Elastic servers in eqiad - https://phabricator.wikimedia.org/T83209#2553500 (10RobH) [15:24:27] 06Operations: Setup/Provision elastic search in eqiad - https://phabricator.wikimedia.org/T83212#2553505 (10RobH) [15:30:40] !log Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1010.eqiad.wmnet [15:30:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:34:48] RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:35:24] 06Operations, 10Traffic, 13Patch-For-Review: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2553546 (10BBlack) [15:37:49] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [50.0] [15:38:05] !log Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1011.eqiad.wmnet [15:38:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:41:47] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [15:44:58] 06Operations, 10Ops-Access-Requests, 10Analytics: Add analytics team members to group aqs-admins to be able to deploy pageview APi - https://phabricator.wikimedia.org/T142101#2553564 (10RobH) 05Open>03stalled The new deploy-aqs group allows for sudo/deployment access for these services. As such, this re... [15:45:30] !log Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1012.eqiad.wmnet [15:45:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:46:27] 06Operations, 10Traffic, 13Patch-For-Review: Convert upload cluster to Varnish 4 - https://phabricator.wikimedia.org/T131502#2553571 (10ema) [15:46:31] 06Operations, 10Traffic, 13Patch-For-Review: Analyze Range requests on cache_upload frontend - https://phabricator.wikimedia.org/T142076#2553569 (10ema) 05Open>03Resolved [15:47:09] yuvipanda: my merge and yours were both waitin gon palladium [15:47:12] i pushed [15:47:22] (so you dont wonder wtf happened to your patch ;) [15:47:23] thanks robh [15:47:34] np [15:52:46] !log Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1013.eqiad.wmnet [15:52:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:56:32] 06Operations, 10Ops-Access-Requests: Requesting access to stat1002/stat1004 for Jdlrobson - https://phabricator.wikimedia.org/T141811#2553625 (10RobH) 05Open>03Resolved a:03RobH The 3 day wait passed without objection. Alex already created the patchset, so a quick rebase and its now live on the cluster.... [15:57:08] 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: Requesting access to stat1003, stat1002 and fluorine for chelsyx - https://phabricator.wikimedia.org/T142648#2553631 (10RobH) 05Open>03Resolved a:03RobH No objections during the three day wait, so this is now merged live. It can take up to 30 mi... [15:59:09] 06Operations, 10Ops-Access-Requests: Requesting access to stat1003, stat1002 and bast1001 for ovasileva - https://phabricator.wikimedia.org/T142502#2553668 (10RobH) a:03ovasileva @ovasileva: I've assigned this request back to you for the input requested by Alex, namely we need the following: Then please cr... [16:00:13] !log Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1014.eqiad.wmnet [16:00:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:00:47] 06Operations, 10Ops-Access-Requests, 10Fundraising-Backlog: Access request: AWight access to iridium - https://phabricator.wikimedia.org/T142446#2553685 (10RobH) 05Open>03declined It appears that this is declined, with the reason being this data is better accessed via hadoop. I'm going to close this tas... [16:02:02] 06Operations, 10Ops-Access-Requests, 10Fundraising-Backlog: Access request: AWight access to iridium - https://phabricator.wikimedia.org/T142446#2535387 (10Ottomata) FYI, this is in Hive in the `wmf.webrequest` table in the `webrequest_source='misc'` partition. [16:06:37] 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: Add marktraceur to statistics-privatedata-users for access to stat1002 - https://phabricator.wikimedia.org/T140132#2454200 (10RobH) It appears this has been awaiting manager approval. I've chatted with @MarkTraceur via IRC and he has already pinged so... [16:07:39] !log Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1015.eqiad.wmnet [16:07:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:08:17] grrrit-wm: doesn't seem to be talking.. maybe related to the upgrade? [16:08:20] ostriches: ^ [16:08:33] 06Operations, 10Ops-Access-Requests, 10Analytics: Add analytics team members to group aqs-admins to be able to deploy pageview APi - https://phabricator.wikimedia.org/T142101#2553701 (10Ottomata) Add me and Luca as well please. We won't gain any extra sudo permissions, but this group will be used to grant a... [16:08:45] Glaisher: Someone who knows how needs to restart the bot [16:09:07] FRIENDLY REMINDER: I have a standing $50 bribe to get someone to make the bot not be stupid and reconnect itself :P [16:09:13] That would be legoktm or yuvipanda? ;) [16:09:48] I think there's others, I dunno [16:09:59] Everyone seems to think I know how to do it :p [16:10:26] https://wikitech.wikimedia.org/wiki/Grrrit-wm ;) [16:10:30] :D [16:10:38] (am doing) [16:14:25] (03PS1) 10Mobrovac: Citoid: increase the number of redirects to 10 [puppet] - 10https://gerrit.wikimedia.org/r/304845 (https://phabricator.wikimedia.org/T115108) [16:15:01] yuvipanda: You so silly. Me? Read docs?! [16:15:24] ;) [16:16:29] !log restbase restarting in eqiad after cassandra restarts for the openjdk upgrade [16:16:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:19:27] (03CR) 10Mobrovac: "PCC OK - https://puppet-compiler.wmflabs.org/3706/" [puppet] - 10https://gerrit.wikimedia.org/r/304845 (https://phabricator.wikimedia.org/T115108) (owner: 10Mobrovac) [16:23:01] (03PS1) 10BBlack: acme-setup: pep8 fixups [puppet] - 10https://gerrit.wikimedia.org/r/304847 [16:23:03] (03PS1) 10BBlack: acme-setup: sort the subjects early [puppet] - 10https://gerrit.wikimedia.org/r/304848 (https://phabricator.wikimedia.org/T134447) [16:25:43] (03CR) 10BBlack: [C: 032] acme-setup: pep8 fixups [puppet] - 10https://gerrit.wikimedia.org/r/304847 (owner: 10BBlack) [16:28:01] (03PS2) 10BBlack: acme-setup: sort the subjects early [puppet] - 10https://gerrit.wikimedia.org/r/304848 (https://phabricator.wikimedia.org/T134447) [16:29:26] (03CR) 10BBlack: [C: 032 V: 032] acme-setup: sort the subjects early [puppet] - 10https://gerrit.wikimedia.org/r/304848 (https://phabricator.wikimedia.org/T134447) (owner: 10BBlack) [16:36:12] (03PS1) 10Kaldari: Change sorting for mkwiki from uppercase to uca-mk-u-kn [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304851 (https://phabricator.wikimedia.org/T26953) [16:44:24] 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: Add marktraceur to statistics-privatedata-users for access to stat1002 - https://phabricator.wikimedia.org/T140132#2454200 (10TrevorParscal) I approve giving Mark access to stat1002 [16:46:19] (03PS2) 10RobH: admin: add marktraceur to statistics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/299115 (https://phabricator.wikimedia.org/T140132) (owner: 10Filippo Giunchedi) [16:48:01] (03CR) 10RobH: [C: 032] admin: add marktraceur to statistics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/299115 (https://phabricator.wikimedia.org/T140132) (owner: 10Filippo Giunchedi) [16:50:08] 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: Add marktraceur to statistics-privatedata-users for access to stat1002 - https://phabricator.wikimedia.org/T140132#2553864 (10RobH) 05stalled>03Resolved a:03RobH So with the 3 days LONG past and no objections, plus mgmt approval, the patchset for... [16:52:24] (03PS1) 10Ottomata: Add pv -l alias to otto's .bash_aliases [puppet] - 10https://gerrit.wikimedia.org/r/304854 [16:53:49] (03CR) 10Ottomata: [C: 032] Add pv -l alias to otto's .bash_aliases [puppet] - 10https://gerrit.wikimedia.org/r/304854 (owner: 10Ottomata) [16:59:28] * gehel preemptively waves to jouncebot [17:00:05] gehel: Respected human, time to deploy Weekly Wikidata query service deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160815T1700). Please do the needful. [17:00:05] SMalyshev: A patch you scheduled for Weekly Wikidata query service deployment window is about to be deployed. Please be available during the process. [17:03:05] (03PS2) 10Ema: common VCL: use FQDN for backend naming [puppet] - 10https://gerrit.wikimedia.org/r/276529 (https://phabricator.wikimedia.org/T138546) (owner: 10BBlack) [17:03:10] !log deploying latest wdqs on wdqs100[12].eqiad.wmnet [17:03:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:04:37] ^ SMalyshev: new version LGTM (test queries run, UI looks good). Feel free to do additional tests... [17:07:07] ostriches and bd808 hi and [ and ] are not fixed in unified diff view. [17:07:32] I don't use unified diff view so *shrug* [17:07:47] paladox: known [17:10:01] Yep [17:10:25] It has a note in the release notes that this is not fixed in unified view but is fixed in side by side [17:11:02] I doint know if there is a bug filed upstream for unified view [17:12:19] ostriches how did you manage to upgrade gerrit? [17:12:31] since when i did apt-get update it updated the package [17:12:45] but it didnt init it so it wasent using the latest gerrit.war in bin [17:12:59] MAGIC! [17:12:59] I'm a wizard! [17:13:00] :D [17:13:04] I had to mv bin and index to a different folder [17:13:08] and re ran puppet [17:13:11] LOL [17:14:05] Just run init with the correct jar instead of the one in bin/ [17:14:10] bin/ doesn't upgrade until init is run [17:14:38] Oh [17:15:10] is there a way we can get puppet to automatically detect we update the gerrit.war outside of review_site and so it can re run init [17:15:15] woops italics LOL [17:15:47] anways or can we do it through heira with a config please? [17:16:37] (03PS3) 10Ema: common VCL: use FQDN for backend naming [puppet] - 10https://gerrit.wikimedia.org/r/276529 (https://phabricator.wikimedia.org/T138546) (owner: 10BBlack) [17:17:13] 06Operations, 10Phabricator: Renew phab.wmfusercontent.org https certificate - https://phabricator.wikimedia.org/T142951#2553965 (10Paladox) Thanks. [17:21:32] ostriches ^^ would make updating gerrit a brez if we do that [17:21:33] :) [17:21:59] No no no. [17:22:04] I don't want puppet to auto-update things [17:22:10] That's dangerous if we're looking at a schema change. [17:22:31] Oh [17:22:37] Maybe a config then [17:22:47] that we can apply when we want it to update [17:22:48] ? [17:24:15] Why bother with a config thing? Just do it on the host... [17:30:31] ok [17:30:43] or just move bin and index [17:30:46] to different folder [17:30:49] and re run puppet [17:30:50] lol [17:48:20] 06Operations, 10Cassandra, 10hardware-requests, 07Wikimedia-Incident: Staging / Test environment(s) for RESTBase - https://phabricator.wikimedia.org/T136340#2554074 (10GWicke) > Owing to their location, data-center support would be minimal This looks like the biggest issue to me. Since the staging cluste... [18:10:20] 06Operations, 10Traffic, 07HTTPS, 13Patch-For-Review: letsencrypt puppetization: upgrade for scalability - https://phabricator.wikimedia.org/T134447#2554248 (10BBlack) [18:13:55] !log starting main-eqiad Kafka upgrade to confluent 0.9, will be stopping and starting brokers on kafka1001 and kafka1002 [18:14:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:14:59] (03CR) 10Ottomata: [C: 032] Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9 [puppet] - 10https://gerrit.wikimedia.org/r/304821 (https://phabricator.wikimedia.org/T138265) (owner: 10Ottomata) [18:15:03] (03PS3) 10Ottomata: Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9 [puppet] - 10https://gerrit.wikimedia.org/r/304821 (https://phabricator.wikimedia.org/T138265) [18:15:06] (03CR) 10Ottomata: [V: 032] Prepare for upgrading Kafka main-eqiad to Confluent Kafka 0.9 [puppet] - 10https://gerrit.wikimedia.org/r/304821 (https://phabricator.wikimedia.org/T138265) (owner: 10Ottomata) [18:16:09] (03PS3) 10Ottomata: Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9 [puppet] - 10https://gerrit.wikimedia.org/r/304822 (https://phabricator.wikimedia.org/T138265) [18:27:28] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [50.0] [18:28:27] (03CR) 10Ottomata: [C: 032] Finalize upgrade of Kafka main-eqiad to Confluent Kafka 0.9 [puppet] - 10https://gerrit.wikimedia.org/r/304822 (https://phabricator.wikimedia.org/T138265) (owner: 10Ottomata) [18:31:00] i don't think that MW exception alert is me, but it could be [18:31:05] trying to look into why [18:31:07] Pchelolo: ^^^ [18:32:00] i should look at mediawiki-errors in logstash, ja? [18:32:04] i don't see anything there related [18:32:29] ottomata: me neather [18:32:35] 06Operations, 10Phabricator: Search not finding task - https://phabricator.wikimedia.org/T143014#2554368 (10Danny_B) p:05Triage>03High Setting High priority because it is not the first time I am encountering this either myself or reported and it is very annoying. I guess some search index rebuild might wo... [18:32:44] ok [18:32:56] proceeding, need bounce brokers one more time to apply protocol version change [18:33:39] (03PS10) 10Mattflaschen: Change login cookies (for 'Keep me logged in') to a one year expiry. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/230954 (https://phabricator.wikimedia.org/T68699) [18:33:51] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [18:36:01] PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: puppet fail [18:43:06] 06Operations, 10Ops-Access-Requests, 06Labs, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2554426 (10RobH) a:05yuvipanda>03madhuvishy I signed via hangout, and @madhuvishy listed off her key fingerprint. I've signed and pushed to keyserv... [18:49:11] PROBLEM - puppet last run on mw2075 is CRITICAL: CRITICAL: Puppet has 1 failures [18:52:33] 06Operations, 10Ops-Access-Requests: Requesting access to stat1003, stat1002 and bast1001 for ovasileva - https://phabricator.wikimedia.org/T142502#2554487 (10ovasileva) Hi Rob, sorry for the delay! Here it is: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD3yvsX9fUZX7B14Rzko/VbDfdrDZvHUBP/SmKM9oxKI5bcCaIa6u7l+faDjE... [18:58:13] 06Operations, 06Labs, 10Labs-Infrastructure, 07Wikimedia-Incident: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2554505 (10greg) [18:59:50] RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:15:02] RECOVERY - puppet last run on mw2075 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:27:21] !log T140008: Staring user-defined compaction (10 tables, highest droppable tombstones), restbase2001-b.codfw.wmnet [19:27:22] T140008: High RESTBase storage utilization - https://phabricator.wikimedia.org/T140008 [19:27:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:39:57] !log T140008: Starting major compaction (WP parsoid html, split output) on restbase1007-a.eqiad.wmnet [19:39:58] T140008: High RESTBase storage utilization - https://phabricator.wikimedia.org/T140008 [19:40:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:47:43] 06Operations, 06Labs, 06Release-Engineering-Team, 10wikitech.wikimedia.org, 07LDAP: Rename specific account in LDAP, Wikitech and Gerrit - https://phabricator.wikimedia.org/T133968#2554836 (10demon) [19:52:41] 06Operations, 10Cassandra, 10RESTBase-Cassandra: Track/alert cassandra certs expiration - https://phabricator.wikimedia.org/T120662#2554867 (10Eevans) [20:00:04] gwicke, cscott, arlolra, subbu, bearND, mdholloway, halfak, and Amir1: Respected human, time to deploy Services – Parsoid / OCG / Citoid / Mobileapps / ORES / … (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160815T2000). Please do the needful. [20:00:50] (03PS1) 10BryanDavis: Extract /etc/profile.d/umask-wikidev.sh to a shared class [puppet] - 10https://gerrit.wikimedia.org/r/304885 [20:01:45] 06Operations, 10Phabricator: Search not finding task - https://phabricator.wikimedia.org/T143014#2554877 (10Danny_B) Observation for the record: Any combination of two out of those three words result in successfull match of T68699. [20:03:12] no deploy for ores today [20:03:28] !log starting parsoid deploy [20:03:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:04:19] 06Operations, 10Ops-Access-Requests, 06Labs: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2554886 (10RobH) [20:05:22] !log synced new parsoid code; restarted parsoid on wtp1001 as a canary [20:05:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:09:04] 06Operations, 10Cassandra, 06Services: Renew RESTBase self-signed root certificate authority - https://phabricator.wikimedia.org/T143044#2554914 (10Eevans) [20:09:55] 06Operations, 10Cassandra, 06Services: Renew RESTBase self-signed root certificate authority - https://phabricator.wikimedia.org/T143044#2554903 (10Eevans) [20:10:45] !log finished deploying parsoid sha f039dcf6 [20:10:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:10:50] time to verify [20:11:43] (03PS2) 10Ottomata: Remove now unused kafka module [puppet] - 10https://gerrit.wikimedia.org/r/304827 (https://phabricator.wikimedia.org/T138265) [20:16:59] (03CR) 10Ottomata: [C: 032] Remove now unused kafka module [puppet] - 10https://gerrit.wikimedia.org/r/304827 (https://phabricator.wikimedia.org/T138265) (owner: 10Ottomata) [20:17:11] !log labtestcontrol2001: Raise max_connections mysql global to 500 to match real-labs' on m5-master (db1009). old value was 151, see my comment at T132422 [20:17:12] T132422: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422 [20:17:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:20:48] (03PS6) 10Ottomata: Confluent MirrorMaker puppetization [puppet] - 10https://gerrit.wikimedia.org/r/300879 (https://phabricator.wikimedia.org/T134184) [20:22:52] (03CR) 10Ottomata: [C: 032] Confluent MirrorMaker puppetization [puppet] - 10https://gerrit.wikimedia.org/r/300879 (https://phabricator.wikimedia.org/T134184) (owner: 10Ottomata) [20:31:16] (03CR) 10Ladsgroup: Rename ores deploy repo (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/296687 (https://phabricator.wikimedia.org/T139008) (owner: 10Ladsgroup) [20:37:20] 06Operations, 06Services, 06Services-next, 05Security, 15User-mobrovac: Productize the Electron PDF render service & create a REST API end point - https://phabricator.wikimedia.org/T142226#2555023 (10cscott) I think we should avoid getting too far ahead of ourselves and making guesses about what the comm... [20:38:04] (03PS1) 10Gehel: Maps - use sources.prod2.yaml also on eqiad cluster [puppet] - 10https://gerrit.wikimedia.org/r/304888 [20:40:24] (03PS1) 10Yuvipanda: Add tcl base / web images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/304889 [20:41:10] (03CR) 10Gehel: [C: 032] Maps - use sources.prod2.yaml also on eqiad cluster [puppet] - 10https://gerrit.wikimedia.org/r/304888 (owner: 10Gehel) [20:42:50] (03PS1) 10Yuvipanda: Add tcl webservice type [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/304890 [20:45:03] (03CR) 10BryanDavis: [C: 032] Add tcl webservice type [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/304890 (owner: 10Yuvipanda) [20:47:09] (03Merged) 10jenkins-bot: Add tcl webservice type [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/304890 (owner: 10Yuvipanda) [20:48:01] (03CR) 10BryanDavis: [C: 031] Add tcl base / web images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/304889 (owner: 10Yuvipanda) [20:48:21] bd808 you have merge rights there too no? [20:48:37] yeah I probably do [20:48:47] (03CR) 10BryanDavis: [C: 032] Add tcl base / web images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/304889 (owner: 10Yuvipanda) [20:48:54] yuvipanda: {{done}} [20:49:13] thanks bd808 [20:55:59] (03CR) 10Dpatrick: [C: 031] "How can we now (with AuthManager) log bad authentication attempts? I think I read a note about this someplace, but I can't remember where." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303939 (owner: 10Gergő Tisza) [20:56:15] 06Operations, 10Ops-Access-Requests: Requesting access to stat1003, stat1002 and bast1001 for ovasileva - https://phabricator.wikimedia.org/T142502#2555134 (10RobH) @ovasileva: Excellent! Can you also tell me what your wikitech username is (its what we base your UID off of)? For access on analytics, it can... [20:58:49] (03PS1) 10Yuvipanda: Use lighttpd-plain for tcl [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/304908 [20:59:12] bd808 ^ [21:00:05] dapatrick and bawolff: Respected human, time to deploy Weekly Security deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160815T2100). Please do the needful. [21:00:38] yuvipanda: does the UNRELEASED in the changelog make a difference? [21:00:51] bd808 not sure, I just used 'dch' this time [21:00:53] I don't think it does [21:00:55] some weird debianism [21:01:05] at least the timestamp is correct [21:01:07] *nod* [21:01:20] also why the fuck did I think SF was -1300?! [21:01:30] (03CR) 10BryanDavis: [C: 032] Use lighttpd-plain for tcl [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/304908 (owner: 10Yuvipanda) [21:01:43] my life has been a lie [21:04:16] o.O [21:05:35] (03CR) 10Gergő Tisza: "The new logging code is in the next block, we cannot log passwords (or attributes of passwords) though. The note was probably T137194." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303939 (owner: 10Gergő Tisza) [21:11:30] (03Merged) 10jenkins-bot: Use lighttpd-plain for tcl [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/304908 (owner: 10Yuvipanda) [21:12:56] (03PS1) 10Ottomata: Mirror main-eqiad into main-codfw [puppet] - 10https://gerrit.wikimedia.org/r/304928 (https://phabricator.wikimedia.org/T134184) [21:18:38] (03CR) 10jenkins-bot: [V: 04-1] Mirror main-eqiad into main-codfw [puppet] - 10https://gerrit.wikimedia.org/r/304928 (https://phabricator.wikimedia.org/T134184) (owner: 10Ottomata) [21:28:12] (03PS2) 10Ottomata: Mirror main-eqiad into main-codfw [puppet] - 10https://gerrit.wikimedia.org/r/304928 (https://phabricator.wikimedia.org/T134184) [21:46:09] (03PS4) 10Rush: WIP labstore nfs: nfs client mount manager [puppet] - 10https://gerrit.wikimedia.org/r/304070 (https://phabricator.wikimedia.org/T140483) [21:53:18] (03PS5) 10Rush: WIP labstore nfs: nfs client mount manager [puppet] - 10https://gerrit.wikimedia.org/r/304070 (https://phabricator.wikimedia.org/T140483) [21:54:03] 06Operations, 06Labs, 10Labs-Infrastructure, 07Wikimedia-Incident: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2555374 (10AlexMonk-WMF) I've written a script to hopefully purge the vast majority of problematic entries in T120797 [22:10:57] (03PS2) 10Thcipriani: Add the fatalmonitor query to logstash_checker [puppet] - 10https://gerrit.wikimedia.org/r/304327 (https://phabricator.wikimedia.org/T142784) [22:11:55] 06Operations, 10Citoid, 10ContentTranslation-CXserver, 10RESTBase, and 3 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#2555433 (10Jdforrester-WMF) [22:12:44] 06Operations, 10Cassandra: Address abnormally wide partitions - https://phabricator.wikimedia.org/T143056#2555435 (10Peachey88) [22:25:10] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [22:28:43] Platonides: again ^ [22:29:21] PROBLEM - IPv4 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 22 probes of 393 (alerts on 19) - https://atlas.ripe.net/measurements/1791210/#!map [22:30:11] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [22:31:25] grr [22:35:10] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [22:35:20] RECOVERY - IPv4 ping to codfw on ripe-atlas-codfw is OK: OK - failed 2 probes of 393 (alerts on 19) - https://atlas.ripe.net/measurements/1791210/#!map [22:37:36] (03PS1) 10Alex Monk: Follow-up Ia3344e72: Fix VE namespaces in he, fa and ko Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304934 (https://phabricator.wikimedia.org/T142824) [22:38:09] poor andrew is gonna think the channel has it out for him [22:39:25] Platonides, I think that's your third false positive now? [22:40:11] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [22:40:15] do FPs against the same user count as different FPs? [22:40:37] (03PS2) 10Alex Monk: Follow-up Ia3344e72: Fix VE namespaces in he, fa and ko Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304934 (https://phabricator.wikimedia.org/T142824) [22:41:26] maybe I am simply biased against him ;) [22:45:02] (03PS3) 10Alex Monk: Follow-up Ia3344e72: Fix VE namespaces in he, fa and ko Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304934 (https://phabricator.wikimedia.org/T142824) [22:45:04] (03PS1) 10Alex Monk: Remove redundant svwiktionary wmgVisualEditorAvailableNamespaces config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304936 [22:45:11] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [22:45:52] mintaka is a host of ours? [22:46:27] ah, no wonder I didn't recognise it [22:46:28] codfw frack [22:50:10] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [22:51:09] (03PS1) 10Dereckson: Enable WikidataPageBanner on ro.wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304938 (https://phabricator.wikimedia.org/T142963) [22:55:10] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [22:58:55] (03PS1) 10Dereckson: Add autopatrolled and rollbacker user groups to it.wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304940 (https://phabricator.wikimedia.org/T142571) [23:00:04] RoanKattouw, ostriches, MaxSem, and Dereckson: Dear anthropoid, the time has come. Please deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160815T2300). [23:00:04] kaldari, aude, and Krenair: A patch you scheduled for Evening SWAT (Max 8 patches) is about to be deployed. Please be available during the process. [23:00:08] Hi. [23:00:09] 07Puppet, 10Beta-Cluster-Infrastructure: deployment-sca0[12] puppet failure due to issues involving /srv/deployment directory - https://phabricator.wikimedia.org/T143065#2555699 (10AlexMonk-WMF) [23:00:10] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [23:00:10] here [23:00:14] * Dereckson added two patches to the SWAT. [23:00:27] 07Puppet, 10Beta-Cluster-Infrastructure: deployment-sca0[12] puppet failure due to issues involving /srv/deployment directory - https://phabricator.wikimedia.org/T143065#2555699 (10AlexMonk-WMF) [23:00:32] hi [23:00:53] I can SWAT this evening. [23:01:17] 07Puppet, 10Beta-Cluster-Infrastructure: deployment-sca0[12] puppet failure due to issues involving /srv/deployment directory - https://phabricator.wikimedia.org/T143065#2555699 (10AlexMonk-WMF) [23:01:25] 07Puppet, 10Beta-Cluster-Infrastructure: deployment-sca0[12] puppet failure due to issues involving /srv/deployment directory - https://phabricator.wikimedia.org/T143065#2555699 (10AlexMonk-WMF) [23:01:49] kaldari: the locale exists in 1.28.0-wmf.14? [23:02:17] yep. this time it does :) [23:02:21] Good. [23:02:28] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304851 (https://phabricator.wikimedia.org/T26953) (owner: 10Kaldari) [23:02:55] (03Merged) 10jenkins-bot: Change sorting for mkwiki from uppercase to uca-mk-u-kn [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304851 (https://phabricator.wikimedia.org/T26953) (owner: 10Kaldari) [23:03:06] kaldari: live on mw1099 [23:04:17] https://mk.wikipedia.org/wiki/%D0%9A%D0%B0%D1%82%D0%B5%D0%B3%D0%BE%D1%80%D0%B8%D1%98%D0%B0:%D0%9A%D0%BB%D0%B0%D1%81%D0%B8%D1%84%D0%B8%D0%BA%D0%B0%D1%86%D0%B8%D1%98%D0%B0_%D0%BD%D0%B0_%D0%B3%D0%BB%D0%B0%D0%B2%D0%BD%D0%B8%D1%82%D0%B5_%D1%82%D0%B5%D0%BC%D0%B8 was the old URL submitted in the bug (I imagine they've added a lot of sort keys since that) [23:04:19] Dereckson: eh, it's another updateCollation.php thing, so I can't test it from mw1099. Will need to sync and run the maintanence script. [23:04:42] At least we can see category view doesn't throw a fatal error. [23:04:49] (it does when collation doesn't exist) [23:05:01] true, lemme check... [23:05:10] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [23:05:27] seems OK [23:05:36] use X-Wikimedia-Debug [23:06:26] use=using [23:08:21] !log dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set collation to uca-mk-u-kn on mk.wikipedia (T26953) (duration: 01m 00s) [23:08:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:08:27] T26953: Set $wgCategoryCollation to 'uca-mk-u-kn' on Macedonian wikis and rebuild category sort keys - https://phabricator.wikimedia.org/T26953 [23:08:51] aude: you've manually edited the map or used WikimediaMaintenance script? [23:09:20] (03PS2) 10Yuvipanda: labs: Don't collect network stats by default [puppet] - 10https://gerrit.wikimedia.org/r/304434 [23:09:46] Dereckson: So far so good. I see the new number headers: https://mk.wikipedia.org/wiki/%D0%9A%D0%B0%D1%82%D0%B5%D0%B3%D0%BE%D1%80%D0%B8%D1%98%D0%B0:16_%D0%B2%D0%B5%D0%BA. I'll start the maintenance script. Might take a little while. [23:10:02] kaldari: ack [23:10:09] eh? [23:10:10] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [23:10:12] Dereckson: dumpInterwiki [23:10:29] (03PS3) 10Dereckson: Update interwiki map [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304622 (owner: 10Aude) [23:10:34] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304622 (owner: 10Aude) [23:11:15] (03Merged) 10jenkins-bot: Update interwiki map [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304622 (owner: 10Aude) [23:11:22] kaldari: I acknowledge you're running the script [23:11:24] Dereckson: Yeah, it's a bit of a mess until the script has been run. Anything else you were acking about? [23:11:51] ah [23:12:00] ack, like acknowledged :) [23:12:08] not like Aaackk!! [23:12:27] Does ops have a technical term to differentiate "project" wikis (i.e., publicly editable, with or with the aim of having a community that interacts with them) from other wikis we run for various purposes (office, collab, wikitech, fr, etc.) ? [23:12:28] !log mwscript maintenance/updateCollation.php --wiki=mkwiki --force [23:12:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:12:34] Different ack indeed. [23:12:59] I don't think there's a standard AndyRussG [23:13:02] Specifically, all the wikis that may show community or FR announcements via CentralNotice... All just "WMF project wikis"? [23:13:05] fr = fundraising? [23:13:05] aude: live on mw1099 [23:13:08] ok [23:13:14] Just trying to think what to call them in some doc [23:13:20] Honestly I doubt most people here know that you run MW inside frack [23:13:22] Krenair: yeah :) [23:13:34] Heheh it's a state secret [23:13:34] jsk [23:13:36] jk [23:13:41] Krenair: each time I need an half second to disambiguate fundraising and French [23:13:47] Dereckson, :) [23:13:57] lf [23:14:12] (= levée de fonds) [23:14:29] sorry for confusingness [23:14:34] looks good https://test2.wikipedia.org/wiki/User:Aude [23:14:45] AndyRussG, so wikitech is kind of an odd one out here [23:14:51] it's not a fishbowl or private [23:14:58] ok [23:15:10] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [23:15:15] Yeah... I guess I'll go with "WMF community project wikis"? [23:15:17] there's the content projects like wikipedias, wiktionaries separated by languages [23:15:18] (03PS4) 10Dereckson: Follow-up Ia3344e72: Fix VE namespaces in he, fa and ko Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304934 (https://phabricator.wikimedia.org/T142824) (owner: 10Alex Monk) [23:15:27] wikitech does a lot more than being a wiki [23:15:29] there's even some misc wikis like arbcom-* under wikipedia.org that are private [23:15:39] (03PS3) 10Yuvipanda: labs: Enable only disk stats collection for labs instances [puppet] - 10https://gerrit.wikimedia.org/r/304434 [23:15:43] !log dereckson@tin Synchronized wmf-config/interwiki.php: Update interwiki map ([[Gerrit:304622]]) (duration: 00m 49s) [23:15:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:15:54] Krenair: hmmm ... I don't think we show banners on those [23:15:56] then we have a few community wikis under *.wikimedia.org that aren't fishbowl/private [23:16:02] neta [23:16:04] meat [23:16:06] rrg [23:16:08] meta [23:16:09] meta? [23:16:10] and commons [23:16:13] incubator [23:16:14] Dereckson: thanks [23:16:20] species too [23:16:22] (03CR) 10BryanDavis: [C: 031] labs: Enable only disk stats collection for labs instances [puppet] - 10https://gerrit.wikimedia.org/r/304434 (owner: 10Yuvipanda) [23:16:35] wikimania* [23:16:43] login and vote are special [23:16:58] outreach [23:16:58] (03CR) 10Yuvipanda: [C: 032 V: 032] labs: Enable only disk stats collection for labs instances [puppet] - 10https://gerrit.wikimedia.org/r/304434 (owner: 10Yuvipanda) [23:17:05] chapter wikis [23:17:36] Krenair: what about add in commit message translation sources? [23:18:31] Dereckson, what? [23:18:37] AndyRussG, I can't think of any more [23:18:40] Ah yeah there are zillions [23:18:43] they're just 'special' wikis [23:18:46] Krenair: that's cool, that's a big help!! [23:18:58] Krenair: in https://gerrit.wikimedia.org/r/#/c/304934/, we could add to commit message where the translations come from. [23:19:01] https://meta.wikimedia.org/wiki/Special:SiteMatrix may be helpful AndyRussG [23:19:41] Dereckson, we could, but why would we? [23:19:47] it's not like I'm setting up new namespaces [23:19:49] document, credit [23:19:50] Krenair: right on! thx :) [23:20:10] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [23:20:30] Dereckson, we're not creating new namespaces in this commit [23:20:46] these are just the existing names being reused in the same file [23:21:23] k [23:21:46] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304934 (https://phabricator.wikimedia.org/T142824) (owner: 10Alex Monk) [23:22:14] (03Merged) 10jenkins-bot: Follow-up Ia3344e72: Fix VE namespaces in he, fa and ko Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304934 (https://phabricator.wikimedia.org/T142824) (owner: 10Alex Monk) [23:22:23] AndyRussG, betawikiversity and sourceswiki are some special cases to look out for. species should probably be treated a bit like wikidata/mediawiki/etc? [23:22:49] Krenair: live on mw1099 [23:23:01] (03PS2) 10Dereckson: Remove redundant svwiktionary wmgVisualEditorAvailableNamespaces config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304936 (owner: 10Alex Monk) [23:24:01] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304936 (owner: 10Alex Monk) [23:24:03] (03PS1) 10Yuvipanda: labs: Enable libvirtkvm collector [puppet] - 10https://gerrit.wikimedia.org/r/304942 (https://phabricator.wikimedia.org/T141673) [23:24:23] ko good [23:24:26] he good [23:24:28] (03Merged) 10jenkins-bot: Remove redundant svwiktionary wmgVisualEditorAvailableNamespaces config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304936 (owner: 10Alex Monk) [23:24:36] fa.. hmm [23:25:11] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [23:25:38] seems to have not taken effect on fawiki? [23:27:04] RTL is always fun [23:27:14] Dereckson: script is done. Looks like the bug is resolved on the Macedonian wiki now :) [23:27:20] kaldari: :) [23:27:24] Indeed [23:27:36] Dereckson: thanks for the help. We'll be doing Swedish next week :) [23:27:38] oops [23:27:46] [Mon Aug 15 23:27:31 2016] [hphp] [9042:7fdfe17ff700:0:000001] [] Core dumped: Segmentation fault [23:27:50] that wasn't supposed to happen :) [23:28:01] * Josve05a heard Swedish... [23:28:33] Wow [23:28:36] (03CR) 10Yuvipanda: [C: 032 V: 032] labs: Enable libvirtkvm collector [puppet] - 10https://gerrit.wikimedia.org/r/304942 (https://phabricator.wikimedia.org/T141673) (owner: 10Yuvipanda) [23:28:57] Josve05a: https://phabricator.wikimedia.org/T142113 [23:28:58] I appear to have made HHVM reliably segfault by typing and pasting a certain combination of things into mwrepl? [23:29:21] there's going to be a couple of graphite metric creation warnings [23:29:32] should be ok tho [23:29:32] kaldari: oh that...real discussion about tha on sv.wp :p [23:29:37] Krenair: I confirm your string is the same than the one in wgNamespaces [23:29:38] that* [23:29:42] (extraNamespaces) [23:30:11] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [23:30:22] It's stopped segfaulting now [23:30:42] Krenair: I suggest to submit a fix to remove fix for fa, so we deploy it only for he/ko? [23:30:55] And then you'll be able to analyze the fa issue [23:31:05] Dereckson, why would we do that? [23:31:10] it's not like the change on fawiki is harmful [23:31:16] it just hasn't fixed anything yet [23:31:18] 23:28:58 < Krenair> I appear to have made HHVM reliably segfault by typing and pasting a certain combination of things into mwrepl? [23:31:24] That was seemingly separate [23:33:23] Krenair: http://tinyurl.com/wm-logstash-mw1099 [23:34:52] ostriches: About? [23:35:11] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [23:35:14] Can you look why https://gerrit.wikimedia.org/r/#/c/304168/ won't submit (user seeing no button) [23:35:33] I guess I am now hah [23:36:06] Krenair: so the two were when you used mwrepl? [23:36:58] Reedy: Eh, crappy inheritance rules. [23:36:59] Fixing. [23:37:03] Thanks [23:37:38] Try now [23:38:00] Krenair: you confirm the two crashes was only using mwrepl so? [23:38:57] Dereckson, mwrepl is entirely irrelevant please forget about it [23:39:07] ok [23:39:09] I'm trying to understand what's wrong with my commit at the moment [23:39:53] =MWNamespace::getCanonicalName( 118 ); [23:39:53] "\331\276\333\214\330\264\342\200\214\331\206\331\210\333\214\330\263" [23:39:57] =array_keys( $wgVisualEditorAvailableNamespaces )[0]; [23:39:57] "\331\276\333\214\330\264\331\206\331\210\333\214\330\263" [23:40:00] I must've copied it wrong [23:40:10] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [23:40:27] ah [23:40:53] Hmmmm strange, I cut the namespace line, pasted it and according git diff I had the same result than yours. [23:44:33] (03PS1) 10Alex Monk: Fix fawiki namespace name [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304945 (https://phabricator.wikimedia.org/T142824) [23:44:41] Dereckson, ^ [23:44:55] (03PS2) 10Dereckson: Fix fawiki namespace name [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304945 (https://phabricator.wikimedia.org/T142824) (owner: 10Alex Monk) [23:45:00] - 'پیشنویس' /* Draft */ => true // T118060 [23:45:01] + 'پیشنویس' /* Draft */ => true // T118060 [23:45:01] T118060: Enable Visual Editor in draft namespace in Persian Wikipedia - https://phabricator.wikimedia.org/T118060 [23:45:01] T118060: Enable Visual Editor in draft namespace in Persian Wikipedia - https://phabricator.wikimedia.org/T118060 [23:45:02] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304945 (https://phabricator.wikimedia.org/T142824) (owner: 10Alex Monk) [23:45:08] (03PS13) 10BryanDavis: [WIP] Provision Striker via scap3 [puppet] - 10https://gerrit.wikimedia.org/r/301505 (https://phabricator.wikimedia.org/T141014) [23:45:11] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [23:45:29] (03Merged) 10jenkins-bot: Fix fawiki namespace name [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304945 (https://phabricator.wikimedia.org/T142824) (owner: 10Alex Monk) [23:45:47] Krenair: live on mw1099 [23:47:50] That fixed it Dereckson [23:48:01] 304936 live too on mw1099 [23:50:10] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [23:50:30] according mwrepl: print_r($wmgVisualEditorAvailableNamespaces); : …[User] => 1… [23:50:33] so works fine [23:51:33] !log dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Fix VE namespaces in he, fa and ko Wikipedias (T118060). Remove redundant svwiktionary wmgVisualEditorAvailableNamespaces entry ([[Gerrit:304936]]). (duration: 00m 52s) [23:51:34] T118060: Enable Visual Editor in draft namespace in Persian Wikipedia - https://phabricator.wikimedia.org/T118060 [23:51:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:52:02] Krenair: here you're in prd ^ [23:53:01] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Provision Striker via scap3 [puppet] - 10https://gerrit.wikimedia.org/r/301505 (https://phabricator.wikimedia.org/T141014) (owner: 10BryanDavis) [23:53:15] Thanks for your patience Dereckson, this works on all wikis [23:53:20] ostriches: They say it's still broken... [23:53:49] ugh fml. [23:54:03] What is maven-release-user? [23:54:13] I told them to log out and in again incase it's being shitty [23:54:34] yawn try again [23:54:41] You want us to give them a submit button or something? [23:54:42] Krenair: A user for maven releasing [23:54:44] Clearly ;-) [23:54:52] ah, ostriches just did it. ok [23:55:05] ostriches, yeah, but... [23:55:08] whatever [23:55:10] PROBLEM - check_ipn_redir on mintaka is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 301 Moved Permanently [23:55:21] Unrelated with current deployment, but has spawned in fatalmonitor: https://phabricator.wikimedia.org/T143070 [23:55:30] You're welcome Krenair, thanks for the quick fix [23:55:37] Krenair: I only have the emotional capacity for "painfully obvious" and "snarky as fuck" right now. [23:55:39] Sorry ;-) [23:55:40] (03PS14) 10BryanDavis: [WIP] Provision Striker via scap3 [puppet] - 10https://gerrit.wikimedia.org/r/301505 (https://phabricator.wikimedia.org/T141014) [23:55:54] :D [23:56:04] ostriches: Fixed now, yus [23:56:10] taaaa [23:56:16] Dereckson, pfff fast? It took me like 15-20 minutes [23:56:17] Krenair: 10 parent, LightProcess exiting [23:56:29] Oh LightProcess. [23:56:31] Go to hell you [23:56:46] (03PS1) 10Yuvipanda: spaces for the pep8 gods [puppet] - 10https://gerrit.wikimedia.org/r/304947 [23:56:49] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Provision Striker via scap3 [puppet] - 10https://gerrit.wikimedia.org/r/301505 (https://phabricator.wikimedia.org/T141014) (owner: 10BryanDavis) [23:56:50] bd808 ^ [23:57:43] Dereckson, fun story from last year [23:58:01] I did a quick, theoretically perfectly fine, sync-file/sync-dir [23:58:22] Suddenly the logs were filled with that from what must've been half the cluster [23:58:51] This one? https://phabricator.wikimedia.org/T124956 [23:59:00] T124k? way too high