[00:58:27] !log tools.list switch to PHP 7.2 webservice [00:58:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.list/SAL [01:02:40] !log tools.blankpages switch to PHP 7.2 webservice [01:02:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.blankpages/SAL [01:42:03] !log git migrating project to eqiad1-r [01:42:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [01:56:15] Αⅼlɑh is ⅾoⅰᥒg [01:56:15] sun ⅰѕ nⲟt ԁoіᥒg Allah is dоinɡ [02:06:36] Aⅼⅼɑh iѕ ⅾoinɡ [02:33:30] Аⅼlah iѕ dοіng [02:55:50] Allɑһ іs doing [05:23:59] (03CR) 10Zhuyifei1999: "recheck" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460443 (https://phabricator.wikimedia.org/T192696) (owner: 10Zhuyifei1999) [05:24:13] (03CR) 10jerkins-bot: [V: 04-1] results: Fixes for memory consumption and bytes support [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460443 (https://phabricator.wikimedia.org/T192696) (owner: 10Zhuyifei1999) [05:31:20] (03PS3) 10Zhuyifei1999: Add ESLint [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460761 [05:32:19] (03CR) 10Zhuyifei1999: [C: 032] "I'm merging this cuz jenkins would complain on other patches." [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460761 (owner: 10Zhuyifei1999) [05:32:49] (03Merged) 10jenkins-bot: Add ESLint [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460761 (owner: 10Zhuyifei1999) [05:32:59] (03PS7) 10Zhuyifei1999: results: Fixes for memory consumption and bytes support [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460443 (https://phabricator.wikimedia.org/T192696) [05:33:40] (03PS2) 10Zhuyifei1999: output.py: Fix tsv delimiter [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460758 (https://phabricator.wikimedia.org/T204432) [05:39:45] (03PS4) 10Zhuyifei1999: Get rid of ?download=true [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460760 (https://phabricator.wikimedia.org/T204430) [05:40:30] (03PS6) 10Zhuyifei1999: Update username in database when a renamed user logins [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/398711 (https://phabricator.wikimedia.org/T73064) [05:41:37] Αllаh iѕ dоing [05:45:53] (03PS5) 10Zhuyifei1999: Get rid of ?download=true [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460760 (https://phabricator.wikimedia.org/T204430) [05:48:30] (03CR) 10Zhuyifei1999: [C: 04-1] Use flask.jsonify instead of json.dumps [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/455237 (owner: 10Framawiki) [05:59:03] Аllah is ⅾoing [06:28:38] Hi, is there a cloud outage? [06:28:42] I’m getting [06:28:46] 06:42 icinga2-wm: PROBLEM - apt_get on puppet-paladox.git.eqiad.wmflabs is CRITICAL: CHECK_NRPE: Error - Could not connect to 172.16.1.179: Connection reset by peer [07:23:47] (03PS1) 10QChris: Add .gitreview [labs/tools/vuln-checker] - 10https://gerrit.wikimedia.org/r/460798 [07:23:49] (03CR) 10QChris: [V: 032 C: 032] Add .gitreview [labs/tools/vuln-checker] - 10https://gerrit.wikimedia.org/r/460798 (owner: 10QChris) [07:24:40] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/vuln-checker] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/460799 [07:24:43] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/vuln-checker] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/460799 (owner: 10QChris) [07:33:44] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/map-of-monuments] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/460802 [07:33:47] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/map-of-monuments] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/460802 (owner: 10QChris) [07:34:05] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/map-of-monuments] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/460803 [07:34:08] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/map-of-monuments] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/460803 (owner: 10QChris) [07:53:24] Allaһ іs ԁοⅰᥒg [08:12:18] Hmm it has a neutron ip [08:12:42] I guess that’s why because I need to get the new ip and add it to the nagios allowed host field [08:13:32] Αⅼlɑh іs ԁoіng [08:13:32] sun іs not doing Allɑһ iѕ dⲟing [08:13:32] mοഠn iѕ nⲟt ԁoіnɡ Аⅼlaһ iѕ doіᥒg [08:29:32] Αlⅼаһ is doinɡ [09:13:06] !log tools T204481 aborrero@tools-mail:~$ sudo exiqgrep -i | xargs sudo exim -Mrm [09:13:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:13:12] T204481: tools-mail exim queue has lot of frozen messages - https://phabricator.wikimedia.org/T204481 [09:31:44] (03PS3) 10Lokal Profil: Make template addition dependent on starting parameter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460716 (https://phabricator.wikimedia.org/T204351) [09:34:09] (03CR) 10Lokal Profil: "While this part is resolved I would not say that we should re-activate the reports before we've addressed the issue of incorrect template " [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460716 (https://phabricator.wikimedia.org/T204351) (owner: 10Lokal Profil) [09:41:35] Аlⅼɑh ⅰs doing [09:41:35] ѕᥙn іѕ ᥒot dоinɡ Аlⅼah ⅰs doing [09:51:51] Allаh is dⲟing [09:51:51] ѕun is not ⅾoinɡ Αlⅼah iѕ doing [09:54:45] Alⅼɑһ іs ⅾoing [09:54:45] sun iѕ ᥒot doing Αllaһ іs dοⅰng [09:54:45] mഠoᥒ is not ԁoing Ꭺllah is ԁoinɡ [09:54:45] ѕtɑrѕ are ᥒot ԁⲟiᥒg Aⅼlah iѕ doіᥒg [11:15:48] (03PS1) 10Urbanecm: Basic readme file [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/460882 [11:16:05] (03CR) 10Urbanecm: [V: 032 C: 032] Basic readme file [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/460882 (owner: 10Urbanecm) [11:57:57] Allah іѕ doinɡ [12:23:51] (03PS1) 10Urbanecm: Make generate_monuments.py work on localhost [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/460892 [12:24:35] (03CR) 10Urbanecm: [V: 032 C: 032] Make generate_monuments.py work on localhost [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/460892 (owner: 10Urbanecm) [12:29:21] Alⅼɑh is ԁoing [12:29:21] suᥒ iѕ ᥒot ԁoiᥒɡ Aⅼⅼɑһ іs dοіᥒɡ [12:29:21] ⅿοοn is not doіᥒg Aⅼⅼaһ is ԁഠⅰᥒg [12:29:42] Good morning everyone :) o/ [12:30:36] hej hej! [12:30:51] andre__: Trying out german :P [12:32:22] Hmm? "hej" is not German, as far as I know :) [13:13:16] (03PS1) 10Urbanecm: Enable lint [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/460896 [13:16:08] Alⅼaһ is dⲟiᥒɡ [13:19:30] paladox: when you're around… can you have a look at the 'git' project and see how it looks post-migrate? It seems mostly good to me, although something is broken with puppetdb (which doesn't worry me that much since as far as I know you're the only one who uses it) [13:42:11] (03CR) 10Hashar: "recheck" [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/460896 (owner: 10Urbanecm) [13:42:59] (03CR) 10Urbanecm: [C: 032] Enable lint [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/460896 (owner: 10Urbanecm) [13:43:26] (03CR) 10Hashar: [C: 032] Enable lint [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/460896 (owner: 10Urbanecm) [13:43:28] (03Merged) 10jenkins-bot: Enable lint [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/460896 (owner: 10Urbanecm) [13:45:38] andrewbogott: yup, I’m pretty sure puppetdb failed due to a old package being installed. (I’m currently at college and will be home soon) [13:46:01] paladox: ok, just let me know if you think there's any new fallout from migration. Thanks! [13:46:10] Ok :) [13:48:16] andrewbogott: I see gerrit-jenkins.wmflabs.org working so at least that bug is fixed where it deleted the proxy after migrating :) [13:48:25] cool! [14:02:26] Aⅼlɑһ is ԁoinɡ [15:37:31] Alⅼaһ is ԁоiᥒg [15:55:07] Αⅼⅼɑһ іs ⅾοіnɡ [15:55:07] ѕuᥒ іs ᥒⲟt doing Аllah is doіnɡ [15:55:07] moοn iѕ not ԁⲟiᥒg Aⅼlaһ is ԁoinɡ [16:07:55] andrewbogott im now home and looking at the git project. [16:35:03] andrewbogott i think you should also add a check so that it will replace the old ip in /etc/hosts. [16:36:23] paladox: that's reasonable, I'll add that to the script [16:36:30] ok thanks :) [16:36:51] hm, or maybe that should be puppetized… I'll need to do some research [16:37:29] ok :) [16:37:38] im not sure if all instances have there ip in there [16:37:45] but at least for me it had the old ip [16:37:52] (which i doin't remember adding) [16:38:10] that is also part of the reason puppetdb broke [16:48:15] andrewbogott also after the migration, puppet always seems to say this: [16:48:29] https://phabricator.wikimedia.org/P7555 [16:48:36] presumingly because the ssh key changed? [16:53:53] (03PS2) 10Framawiki: default_config.yaml: fix resultset file extension [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460746 [16:56:56] (03CR) 10Framawiki: "Updated commit msg." [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460746 (owner: 10Framawiki) [16:57:24] (03CR) 10Framawiki: [C: 032] default_config.yaml: fix resultset file extension [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460746 (owner: 10Framawiki) [16:58:02] (03Merged) 10jenkins-bot: default_config.yaml: fix resultset file extension [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460746 (owner: 10Framawiki) [16:59:59] andrewbogott hmm https://gerrit.git.wmflabs.org dosen't seem to work [17:03:42] ah [17:03:58] it was because the ip in dns->zone was wrong [17:12:46] Yeah, I don't make any attempt to reassign dns IPs. That's a pretty rare use case. [17:14:52] so when you have an instance with a floating IP migrated you have to update all the A records to point to the new floating IP andrewbogott? [17:15:57] Yes, probably. I can look at automating that but I assume that automatic migration of proxies catches most cases outside of deployment-prep [17:19:26] Seems MW-Vagrant got only support for Apache, I'm running Nginx [17:19:34] So Nginx is not supported? [17:22:58] d3r1ck: we try to kind of keep the mediawiki-vagrant stuff aligned with WMF production stacks, and we mostly use Apache there. Nginx is used in the https role [17:23:32] bd808: Thanks, that answers a lot of other questions already :) [17:24:02] (03CR) 10Framawiki: [C: 032] Update username in database when a renamed user logins [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/398711 (https://phabricator.wikimedia.org/T73064) (owner: 10Zhuyifei1999) [17:24:47] (03Merged) 10jenkins-bot: Update username in database when a renamed user logins [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/398711 (https://phabricator.wikimedia.org/T73064) (owner: 10Zhuyifei1999) [17:25:29] Aⅼⅼah іѕ ԁoiᥒg [17:28:58] (03PS1) 10Paladox: Update ip for gerrit-mysql [labs/icinga2] - 10https://gerrit.wikimedia.org/r/460934 [17:29:28] (03PS2) 10Paladox: Update ip for gerrit-mysql [labs/icinga2] - 10https://gerrit.wikimedia.org/r/460934 [17:29:31] (03CR) 10Paladox: [V: 032 C: 032] Update ip for gerrit-mysql [labs/icinga2] - 10https://gerrit.wikimedia.org/r/460934 (owner: 10Paladox) [17:31:33] Alⅼɑһ іѕ ⅾoiᥒɡ [17:41:00] !log quarry deployed e8e6e02 to quarry-web-01 T73064 [17:41:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [17:41:04] T73064: Quarry should update username when a user is renamed - https://phabricator.wikimedia.org/T73064 [18:13:32] is it just me or is everything broken on Toolforge? [18:13:38] not just you [18:13:41] ops are working on it [18:13:46] okay, thanks :) [18:15:17] Hi all! There seems to be a problem on the toolservers, filesystem is not mounted via NFS or stale file handle [18:15:32] Wurgl, please read topics when joining channels [18:15:52] okay [18:21:24] https://quarry.wmflabs.org [18:21:32] Seems to have slow response [18:21:46] hmm I wonder if that one talks to NFS [18:21:56] its not tools [18:22:00] Doing an upgrade? [18:23:08] ShakespeareFan00, looks like the quarry project has an NFS mount [18:23:10] So I don't know much, but I've been told there is maintenance ongoing, but the impact happening was not scheduled [18:23:15] NFS has known issues right now [18:23:25] people are looking at them [18:24:02] The NFS change I was doing went very badly. Services should be restoring now. [18:24:08] After a rollback [18:25:43] ohhhh so you broke it :P [18:25:56] Yeah... 😅 [18:26:04] I broke all the things this time. [18:26:13] That happens [18:26:21] Things should be back up now. If this is not true, please let me know [18:26:26] piaget2 was losing it [18:26:49] Seems that it is already running fine [18:26:58] piaget2? [18:27:09] musikanimal, Wurgl, ShakespeareFan00: check now [18:27:15] https://usercontent.irccloud-cdn.com/file/4lO3ussB/image.png [18:27:26] every time we talked piaget went a little bananas [18:28:01] hm, it's cloaked [18:28:06] might be on tools [18:28:20] I have some webcam which talks NFS version 2 only. And every time I upgrade my Linux it is broken … [18:28:21] looks good, thanks [18:28:34] when bstorm_ said they were fixing something or had piaget2 nipped out and back and seems to not be crazy now [18:28:57] Krenair: I was also seeing a lot of replication lag [18:29:16] ShakespeareFan00, prod replication lag shouldn't be caused by this [18:30:26] coibot is back [18:30:40] :) [18:31:03] NFS is not doing [18:34:04] Krenair [18:34:15] : https://quarry.wmflabs.org/query/29748 [18:34:34] Seeing entries in the results that shoud NOT be being returned [18:34:55] Perhaps you can take a look and sanity check the logic? [18:47:05] ShakespeareFan00: https://tools.wmflabs.org/replag/ is reporting reasonably short lag at the moment. I saw a few small spikes of 1-2 minutes, but nothing strange. [18:50:07] bd808: have you tried putting coffee in the server? [18:50:16] maybe 5 hour energy? [19:02:05] (03CR) 10Framawiki: [C: 032] results: Fixes for memory consumption and bytes support [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460443 (https://phabricator.wikimedia.org/T192696) (owner: 10Zhuyifei1999) [19:02:33] (03Merged) 10jenkins-bot: results: Fixes for memory consumption and bytes support [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460443 (https://phabricator.wikimedia.org/T192696) (owner: 10Zhuyifei1999) [19:20:43] (03CR) 10Framawiki: [C: 032] output.py: Fix tsv delimiter [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460758 (https://phabricator.wikimedia.org/T204432) (owner: 10Zhuyifei1999) [19:21:15] (03Merged) 10jenkins-bot: output.py: Fix tsv delimiter [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460758 (https://phabricator.wikimedia.org/T204432) (owner: 10Zhuyifei1999) [19:27:49] (03PS2) 10Zhuyifei1999: Get rid of unicodecsv dependency [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460559 [19:28:36] (03CR) 10Framawiki: [C: 032] Get rid of unicodecsv dependency [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460559 (owner: 10Zhuyifei1999) [19:29:02] (03Merged) 10jenkins-bot: Get rid of unicodecsv dependency [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460559 (owner: 10Zhuyifei1999) [19:31:52] Aⅼⅼah is doіng [19:31:52] suᥒ is not ԁoіᥒɡ Allɑһ іs dⲟiᥒɡ [19:31:53] mooᥒ ⅰs ᥒot dഠіng Aⅼlah is ԁοіᥒɡ [19:39:13] (03PS6) 10Zhuyifei1999: Get rid of ?download=true [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460760 (https://phabricator.wikimedia.org/T204430) [19:56:55] (03CR) 10Framawiki: [C: 032] Get rid of ?download=true [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460760 (https://phabricator.wikimedia.org/T204430) (owner: 10Zhuyifei1999) [19:57:29] (03Merged) 10jenkins-bot: Get rid of ?download=true [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/460760 (https://phabricator.wikimedia.org/T204430) (owner: 10Zhuyifei1999) [20:06:39] !log quarry deployed till e59152e T192696 T204432 T204226 T204430 to quarry-web-01 [20:06:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [20:06:56] T192696: Quarry results serialization (web) should have its memory usage upper-bounded and streamed - https://phabricator.wikimedia.org/T192696 [20:06:57] T204430: HTTP header encoding error - https://phabricator.wikimedia.org/T204430 [20:06:58] T204432: TSV results are CSV, rows aren't separated with tabs - https://phabricator.wikimedia.org/T204432 [20:06:58] T204226: Get rid of this ?download=true parameter - https://phabricator.wikimedia.org/T204226 [20:42:16] (03PS1) 10Lokal Profil: Do not pass unnecessary parameters [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460974 [20:45:14] (03CR) 10Lokal Profil: "a similar simplification could be done to the statistics dicts that get produced, but that would be more appropriate in a separate patch." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460974 (owner: 10Lokal Profil) [21:05:59] (03CR) 10Jean-Frédéric: [C: 032] "Oh, geez, that makes *so much sense*. How come we never noticed before? :D" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460974 (owner: 10Lokal Profil) [21:07:33] (03Merged) 10jenkins-bot: Do not pass unnecessary parameters [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460974 (owner: 10Lokal Profil) [21:08:38] (03CR) 10Jean-Frédéric: [C: 032] "100% code coverage ? Niiiiice :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/459405 (owner: 10Lokal Profil) [21:10:01] (03CR) 10jenkins-bot: Do not pass unnecessary parameters [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/460974 (owner: 10Lokal Profil) [21:11:01] (03Merged) 10jenkins-bot: Refactor table generation into its own class [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/459405 (owner: 10Lokal Profil) [21:13:15] (03CR) 10jenkins-bot: Refactor table generation into its own class [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/459405 (owner: 10Lokal Profil) [22:08:03] !log admin Granted gtirloni project roles of admin, projectadmin, and user [22:08:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:08:17] gtirloni: ^ in theory... that should do it [22:08:58] bd808: awesome, thanks a lot. I see the admin project now \o/ [22:32:22] Αⅼlah iѕ doіᥒg [22:32:22] suᥒ iѕ ᥒⲟt ԁοⅰnɡ Αlⅼaһ iѕ ԁഠⅰnɡ [22:32:22] moon іs nⲟt ԁoiᥒɡ Αlⅼaһ is ԁoiᥒg [23:15:15] !log puppet shut down trusty instance "puppet-mailman" (T204558) [23:15:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet/SAL [23:15:19] T204558: cloudvps: puppet project trusty deprecation - https://phabricator.wikimedia.org/T204558 [23:15:46] !log planet shut down instance design-apache, as used for netbox, not anymore [23:15:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Planet/SAL [23:21:41] !log puppet shut down instance trusty-update (T204558) [23:21:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet/SAL [23:21:46] T204558: cloudvps: puppet project trusty deprecation - https://phabricator.wikimedia.org/T204558