[00:02:14] Krenair: is banbot yours? can we get it in #wikipedia-de too? [00:04:18] annika: it's Platonides's [00:05:18] ty [00:16:11] ��}�K*7Q�8�� [00:18:57] �L��+��I��!(�R��>+ [00:18:57] &�/�|��O��=.��> ��(/Y�Fw7��mNҪag⠒Q䐋�.9� [00:21:15] Krenair: shinken has left again [00:37:07] e��+^�V�� [01:38:09] ԝ�_y��d05��sk/��ZO��tZƥPz�ݏd��_ [02:45:35] ��46`*+��|��1��8��:� [05:14:11] Anyone awake? [05:18:33] !ask [05:18:33] Hi, how can we help you? Just ask your question. [05:21:23] thanks legoktm [05:21:56] !ask WinSCP permission denied [05:21:56] Hi, how can we help you? Just ask your question. [05:22:02] or [05:22:04] yeah [05:25:33] Operator873: I mean, just ask your question [05:25:43] and if people are around who know the answer, they will [05:26:38] I was working with arturo and zhuyifei1999_ eariler trying to figure out why I couldn't sweettalk WinSCP into connecting to login.tools.wmflabs.org [05:27:21] I've been testing ssh -v login-tools.wmflabs.org and have noticed I'm connecting, everything looks good until the key exchange [05:27:47] After all keys are attempted, "Permission denied (publickey, hostbased). [05:28:10] I've refreshed the tooladmin key to make sure I didn't bork that up some how, still no success. [05:32:37] ��: [05:48:15] Disregard previous questions and nick pings arturo and zhuyifei1999_ . Problem solved; access gained. [07:08:40] (03PS2) 10Lokal Profil: Move config filtering logic to monuments_config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462821 [07:10:33] (03PS2) 10Lokal Profil: Make use og monuments_config.filtered_countries [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462823 [07:13:47] (03PS1) 10Lokal Profil: Exclude wlpa from database statistics [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462861 (https://phabricator.wikimedia.org/T203349) [07:17:05] (03PS3) 10Lokal Profil: Make use of monuments_config.filtered_countries [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462823 [07:29:20] ��O@�&ѯX�E�P��\n�n��|̉��}`['K/�ET7��R��sY [07:44:21] �|m[�u� [08:10:45] (03CR) 10Jean-Frédéric: [C: 032] Exclude wlpa from database statistics [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462861 (https://phabricator.wikimedia.org/T203349) (owner: 10Lokal Profil) [09:17:03] PSA! If you're using HexChat and you're hearing "ding" sounds on your computer randomly, it's the spammers sending the bell code in order to ding you on IRC. Enter "/set input_filter_beep on" in order to turn that off. [09:17:21] ��L+�CZ��F�� [09:18:04] !log wikimania-support T204745 migrating project to eqiad1 [09:18:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikimania-support/SAL [09:18:08] T204745: cloudvps: migrate projects from main to eqiad1 - https://phabricator.wikimedia.org/T204745 [10:10:38] !log admin restart neutron-server in cloudcontrol1003, investigating T205524 [10:10:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:10:42] T205524: cloudvps: neutron: agents failed to communicate with server - https://phabricator.wikimedia.org/T205524 [10:11:17] o.0 [10:13:25] !log admin T205524 restart all agents in cloudnet1004.eqiad.wmnet [10:13:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:19:03] 4Q61|��毷��ߏ��Ł5@?�^�r��Y�pń��7��2�z��TE�foK��W��2��S�e�d�ca��L��[��n�*�h=��ۋ�ͭ��/s3�ퟍs��j��Q��L�0�0+0�X� [10:20:16] !log admin T205524 stop/start all neutron agents in cloudnet1003.eqiad.wmnet [10:20:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:20:20] T205524: cloudvps: neutron: agents failed to communicate with server - https://phabricator.wikimedia.org/T205524 [10:37:05] !log ores 1b5efba is going to staging [10:37:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [10:40:32] !log admin T205524 all sorts of restarts in all neutron daemons [10:40:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:40:39] T205524: cloudvps: neutron: agents failed to communicate with server - https://phabricator.wikimedia.org/T205524 [10:46:18] (03CR) 10Jean-Frédéric: [C: 032] Move config filtering logic to monuments_config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462821 (owner: 10Lokal Profil) [10:47:34] (03CR) 10Jean-Frédéric: [C: 032] Make use of monuments_config.filtered_countries [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462823 (owner: 10Lokal Profil) [10:47:49] !log ores 1b5efba is being deployed [10:47:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [10:49:05] (03Merged) 10jenkins-bot: Move config filtering logic to monuments_config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462821 (owner: 10Lokal Profil) [10:50:19] (03Merged) 10jenkins-bot: Make use of monuments_config.filtered_countries [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462823 (owner: 10Lokal Profil) [10:51:29] (03CR) 10jenkins-bot: Move config filtering logic to monuments_config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462821 (owner: 10Lokal Profil) [10:53:27] (03CR) 10jenkins-bot: Make use of monuments_config.filtered_countries [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462823 (owner: 10Lokal Profil) [12:45:19] (03PS1) 10Elukey: add ::passwords::recommendationapi::mysql fake passwords [labs/private] - 10https://gerrit.wikimedia.org/r/463057 [12:45:52] (03CR) 10Elukey: [V: 032 C: 032] add ::passwords::recommendationapi::mysql fake passwords [labs/private] - 10https://gerrit.wikimedia.org/r/463057 (owner: 10Elukey) [13:02:34] (03PS2) 10Lokal Profil: Exclude wlpa from database statistics [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462861 (https://phabricator.wikimedia.org/T203349) [13:04:25] �� VDVWg�=@�� ?��DsR��?O��k7�+�v�pB�6c��噩�~��!�� &�ZL;}�6b5��˪4��sn,{@.M�롰5�ud��W�� [13:10:55] Operator873: how did you fix it? [13:24:29] !log quarry restarted mariadb on -db-01 after max_allowed_packet fix [13:24:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [13:32:50] �o�n��R5��T ��uԔ6b��Ky!�c��F��ǭrO�z�)~K��d��V�n_ ��8��?��(>��L�q [13:32:50] ��P�WX��8�� (03CR) 10Lokal Profil: [C: 032] "trigger merge" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462861 (https://phabricator.wikimedia.org/T203349) (owner: 10Lokal Profil) [13:55:41] (03Merged) 10jenkins-bot: Exclude wlpa from database statistics [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462861 (https://phabricator.wikimedia.org/T203349) (owner: 10Lokal Profil) [13:57:48] (03CR) 10jenkins-bot: Exclude wlpa from database statistics [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/462861 (https://phabricator.wikimedia.org/T203349) (owner: 10Lokal Profil) [14:00:41] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @amir1 & @Pablo_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:03:58] (03PS1) 10Lokal Profil: [WIP] use mconfig.countries rather than get_countries [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/463074 [14:05:40] (03CR) 10Lokal Profil: "@JeanFred: Can't figure out how to deal with the mock for this in a way that also allows me to switch in an empty dictionary." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/463074 (owner: 10Lokal Profil) [14:07:34] (03CR) 10jerkins-bot: [V: 04-1] [WIP] use mconfig.countries rather than get_countries [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/463074 (owner: 10Lokal Profil) [14:26:31] 1�F��0i�귃w/T�O�빙5u_� [14:50:42] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @amir1 & @Pablo_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:48:44] !log wikimania-support T204745 actually migrating project to eqiad1 [15:48:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikimania-support/SAL [15:48:48] T204745: cloudvps: migrate projects from main to eqiad1 - https://phabricator.wikimedia.org/T204745 [15:50:18] !log aicaptcha deleting VM and project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_aicaptcha [15:50:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Aicaptcha/SAL [15:52:21] !log bots deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_bots [15:52:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/SAL [15:57:16] !log contributors deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_contributors [15:57:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Contributors/SAL [15:59:27] !log wikidata-page-banner deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_wikidata-page-banner [15:59:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-page-banner/SAL [15:59:46] !log wikisource-tools deleting project as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_wikisource-tools [15:59:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisource-tools/SAL [16:13:36] zhuyifei1999_: I couldn't figure out what the issue was so I just made another key and went from the top of the how to page. [16:13:41] This time it worked. [16:13:51] huh o.O [16:24:03] !log wikimania-support T204745 project migrated to eqiad1 but the webservice isn't working (lxc issue) [16:24:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikimania-support/SAL [16:24:08] T204745: cloudvps: migrate projects from main to eqiad1 - https://phabricator.wikimedia.org/T204745 [17:06:12] ��y5�!�9��l��%�u�g�Gn�ٟ��v [17:06:12] does it support gettext? [19:29:52] Their prices are very reasonable too [19:29:54] i should learn how all that works [19:29:58] [@FormationXtof] Bienvenue sur http://indieweb.me @Warzeur@indieweb.me (http://twtr.io/1i_d0rhvhw0) [19:30:00] baka mplayer got a load of options, perfect [19:30:47] Krenair / TheresNoTime: ^^^ [19:33:10] lots of bots recently eh bd808? [19:33:42] yeah, its freenode wide abuse [19:33:50] trolls being trolls [19:34:15] yeah, i know. Wonder why the uptick suddenly between these and the allah is doing thing [19:36:08] the usual cycle is: somebody decides to "play", they get k-lined, they get mad, they recruit others to try and overwhelm admins with boring bots [19:37:21] hence the name "lamers" [20:21:31] Blondie101010 do you have that same opinion with the new design? [20:21:34] kind of a shame with all that potential shielding work it's only cat5 [20:21:36] Never even heard of BRE. [20:21:38] latest version is 0.23.79 :) [20:21:42] "No sexualized language" is a requirement in literally anything you would write at any science/academia job and most private industry jobs. I mean really? [20:21:50] bugger [20:36:43] pawiecki disable automatic submission [20:36:45] where column='text' [20:36:47] we have. [20:36:50] no... no stabbing [20:36:53] smellsLikeGoatSp: he doesn't know you are able to switch back and forth succesfully [21:16:57] PROBLEM - HTTPS-toolserver on www.toolserver.org is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused [21:16:59] > ERR_CONNECTION_REFUSED [21:19:37] it is a uart serial not usb serial GrandPa-G ? [21:19:44] what do you think tomreyn ? [21:19:44] would do the trick* [21:19:44] you'd need an adapter [21:19:49] on kile, is live preview a exclusive feature for pdfLaTeX? [21:23:08] Krinkle: same for me. Let me see if I can figure out what's going on with that proxy [21:28:43] ugh. the apache2 config is messed up [21:31:13] (and "special case") [21:31:15] So who is? [21:31:18] \x I will sell you a bitcoin for 5 xmr [21:31:21] CerebralIRC[m]: also, the general idea is to not use much any kind of additional software in dom0, qubes comes with a mostly working build env for building dom0 pkgs in an appvm. [21:35:54] mutante: do you know what I need to do to force the toolserver_legacy server to get a new letsencrypt cert? Apache2 is failing to start with a complaint about cert + key mismatch [21:39:00] I'd generate a self-signed pair in its place to get apache starting, then run LE bd808 [21:39:30] puppet should have auto renewed it [21:39:34] unless it was failing [21:39:42] yeah but it won't be able to do that if apache is failing to start [21:39:54] yeah. its a chicken + egg problem [21:41:18] you can either disable SSL support on the webserver, or feed it self-signed certs, to get it to start [21:41:49] wait. we already fixed this [21:41:57] why would it be broken again now [21:41:57] then you can get proper certs, replace the self-signed ones, and possibly re-enable SSL stuff if needed [21:42:06] i copied the certs manually to fix this [21:42:20] mutante: did you fix it after failing back to the eqiad vm? [21:42:36] apache2 was dead there with 2 different errors [21:42:43] first is addressed by https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/463146/ [21:42:50] and now this cert thing [21:43:05] i dont understand. toolserver.org was working [21:43:26] the docroot had to be changed? eh... [21:43:31] You need something like https://github.com/wikimedia/puppet/blob/production/modules/gerrit/templates/apache.erb#L38 here https://github.com/wikimedia/puppet/blob/production/modules/toolserver_legacy/templates/www.toolserver.org.erb#L22 [21:47:56] i am really confused by this. i am positive the site as up after it was moved.. and now there are things missing that make it look like it could have never worked [21:48:06] stretch or no stretch [21:48:15] checking the cert thing first [21:52:46] mutante: thanks. I'll stop piddling and let you poke around. Note I disabled puppet and hand hacked 50-www-toolserver-org.conf to match my gerrit patch [21:53:31] bd808: ok, i will enable puppet again and attempt to fix [21:53:39] <3 [22:00:37] !log toolserver-legacy - copied mismatching toolserver.crt over from backup from old trusty instance, started apache, site is up again [22:00:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolserver-legacy/SAL [22:01:32] fwiw, it worked without the document root change [22:01:42] i just fixed the cert thing and it came up fine [22:01:59] then i ran puppet and it applied the docroot change.. wasnt there before but started [22:02:15] well, and then puppet broke it again.. hmm [22:02:53] i seee. that change wasn't actually merged yet.. on it [22:06:14] iCookie: https://www.youtube.com/watch?v=APo_Pmr-JIk [22:06:17] ketas: yes the capacitance and inductance get in the way [22:06:20] i'm using a lot of pointers in my program and it's getting real tiring to keep typing -> over and over. i guess that's just me? [22:06:23] zenix_2k2: you seem to be starting from the incorrect assumption that drivers can only be for hardware and working backwards [22:06:26] if its another client, you may wish to consider a bot like sigyn [22:08:48] paladox: Krenair: fwiw, i can always fix it by manually copying the cert .. and on the next puppet run it breaks and crt and key are mismatching again.. not working as expected [22:08:58] oh [22:09:53] tries again, copies key and cert.. apache is fine.. runs puppet [22:10:10] where is puppet getting the cert from? [22:11:21] need more debug output of Exec[acme-setup-acme-toolserver] to answer it [22:11:33] I guess puppet agent -tv --debug? [22:12:39] Executing '/usr/local/sbin/acme-setup -i toolserver -s toolserver.org,www.toolserver.org,wiki.toolserver.org,stable.toolserver.org --key-user root --key-group root -m acme -w apache2' [22:12:53] Cert renewal on or after 2018-09-26T20:33:32 [22:13:00] Getting ACME cert /etc/acme/cert/toolserver.crt [22:13:11] so it tries to auto-renew.. this is the new part [22:13:26] and then the challenge fails [22:13:51] sounds like something is wrong with acme-setup [22:13:56] Error signing certificate: [22:14:05] Error creating new cert :: too many certificates already issued for exact set of domains: [22:14:05] yeh [22:14:08] arrrg [22:14:10] damn [22:14:25] exact set of domains: stable.toolserver.org,toolserver.org,wiki.toolserver.org,www.toolserver.org: s [22:14:33] see https://letsencrypt.org/docs/rate-limits/", [22:14:33] it's broken and has been a while [22:14:36] and keeps trying? [22:15:38] is there a way to revoke the old one? [22:15:42] to get under the limit again [22:15:46] nope [22:15:54] I doubt revocations count in favour of limits [22:15:57] not that i am aware of anyways, i tryed to do that. [22:16:03] I believe you can revoke but.... [22:16:08] well, it says the problem is too many "are issue" for it [22:16:12] issued [22:16:47] "Revoking certificates does not reset rate limits, because the resources involved in issuing the certificates have already been used." [22:16:54] garrr [22:17:10] https://letsdebug.net/toolserver.org/5821 [22:17:11] "If you are a large hosting provider or organization working on a Let’s Encrypt integration, we have a rate limiting form that can be used to request a higher rate limit. It takes a few weeks to process requests, so this form is not suitable if you just need to reset a rate limit faster than it resets on its own." [22:17:12] * Reedy waits [22:17:48] you could have FXO/FXS with all sorts of weird connector types [22:17:51] yeah that worked wal -f "$(rofi -dmenu < themefile)" [22:17:53] so what do you suggest? [22:17:58] pepee: when you have more time later, please see if this is potentially the same issue (run 'lsblk -a' and 'nvme list' from the live / installer iso): https://bugs.launchpad.net/ubuntu/bionic/+source/linux/+bug/1778765 [22:18:00] bonus points if you don't use pillar ;) [22:18:20] meh the bots have stored our coversations and now saying it at random. [22:18:23] things changed.. now the puppet run fails [22:18:27] and does NOT break apache anymore [22:18:29] in return [22:19:59] paladox: sounds like that will end in recursion where the bots are citing other bots [22:20:14] heh [22:21:35] "To overcome the issue wait for this week period to pass and reissue certificate." [22:21:35] lol [22:22:10] mutante: Apparently just adding another subdomain will let it go through [22:25:47] !log toolserver-relic adding foo.toolserver.org fake subdomain just to trick LE into issue a new cert, running acme-setup manually [22:25:48] mutante: Unknown project "toolserver-relic" [22:25:56] !log toolserver-legacy adding foo.toolserver.org fake subdomain just to trick LE into issue a new cert, running acme-setup manually [22:25:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolserver-legacy/SAL [22:26:07] Did it work? [22:26:09] Reedy: good idea.. kind of works.. just also need a puppet change [22:26:19] hahaah [22:26:23] or next run it tries to run it without the foo. domain again [22:26:35] how often is it running? [22:28:22] every 30 min [22:29:06] ... [22:29:07] Why? [22:29:20] Puppet or the lets encrypt part? [22:29:49] puppet [22:30:13] well, puppet runs le [22:31:05] I presume it doesn't actually cause a cert renewal attempt every 30 mins? [22:31:38] I believe it does? [22:31:53] :( [22:31:55] that's stupid [22:32:08] it needs running... Every month or so [22:32:18] Not 48 times a day [22:32:39] Or does hte LE tool bail out and go "not yet"... Which I think some do [22:32:42] copying the old one worked at first.. then 3 days went by and it expired the first time [22:32:50] this would explain something [22:33:19] it should not renew unless it's 15 days near expiry Reedy [22:33:28] Right [22:33:34] But the fact it's caused itself to be rate limited [22:33:38] Suggests something is broken [22:36:18] so i see in google that there was once status.toolserver.org [22:36:23] but we never added that to relic [22:36:29] let's just add it now :p [22:36:36] trick LE :) [22:37:06] https://www.mediawiki.org/wiki/Toolserver:Status.toolserver.org [22:40:00] it exists in DNS but wasn't a ServerAlias.. perfect [22:44:28] Hm.. seems the 404 handler isn't how it used be be. known issue? [22:44:43] I thought the home page "We've moved" was also served to other paths that are unknkown [22:44:47] https://www.toolserver.org/tsstatus/ [22:44:55] has default error page instead [22:46:19] only the ones that are individually listed [22:46:21] compare https://www.toolserver.org/~alexxw/Denkmalliste [22:47:34] https://github.com/wikimedia/puppet/blob/production/modules/toolserver_legacy/templates/www.toolserver.org.erb [22:47:49] from line 70 on [22:53:12] Reedy: well.. that worked :) [22:53:35] we succesfully tricked it into a new cert by just adding a new ServerAlias (which we kind of needed anyways and existed before). so.. cool , heh [22:53:55] puppet and apache both happy [22:54:53] bd808: fixed. sorry if i broke it. i dont fully understand how it happened.. but now it's fine [22:55:18] mutante: no worries. thanks for jumping back in to work on it more! [22:55:28] and bonus is more URLs work than before .. status.toolserver.org [22:55:30] * bd808 blames Puppet for everything [22:59:56] you could do them in bulk that way [22:59:59] it sans the -ldap subpackage doesnt depend on the main library which is probably in the -main subpackage [23:00:02] it tends to only ever happen when changing versions [23:00:05] ikr... at least a chance to keep it up when something breaks i guess [23:03:24] ntnn: what do you mean with 'shell lines'? [23:03:27] the wisdom of this fucking monologue [23:03:29] __init__() got an unexpected keyword argument 'capture_output': TypeError [23:03:32] like 2x 1070 [23:34:15] how can i exclude spesific file extension in git diff [23:34:18] Soni: Federated. So like mastodon and matrix. Can you elaborate on your differences? [23:34:20] or slock, or a script [23:34:24] also, line 5 acts on information you got by subtracting mbr_var_v with newV, so you could test diff for negativity and know if add should be true or not [23:34:27] aidrocsid: which ubuntu release is it?