[00:03:11] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:03:19] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:11:12] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures  
[00:13:18] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[01:35:11] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpFLj
[01:35:12] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 03075a0c0 - BOT: Updating services config for wikis
[06:54:02] <ZppixBot>	 Hello Guest23! If you have any questions feel free to ask and someone should answer soon.
[08:25:55] <psiconauta>	 good morning
[08:28:00] <psiconauta>	 regarding custom domains, is there any trouble if the parent zone provider NS records are not ns.miraheze.org?
[08:29:56] <psiconauta>	 I already tried to update these records to ns.miraheze.org with no luck
[08:30:33] <psiconauta>	 after some minutes, these records are replaced by my DNS hosting provider
[08:31:39] <psiconauta>	 maybe that's because they're acting as an authoritative server?
[08:34:46] <psiconauta>	 should I request them to change it to act as a slave?
[08:35:28] <psiconauta>	 or hidden primary
[08:45:14] <psiconauta>	 sadly, Miraheze dns server (gdnsd) does not support dnssec (https://github.com/gdnsd/gdnsd/issues/65)
[08:45:15] <ZppixBot>	 Title: [ Issues · gdnsd/gdnsd · GitHub ] - github.com
[08:50:52] <psiconauta>	 would be nice to replace gdnsd with nsd or bind (both support dnssec)... someday in the distant future xD
[10:23:53] <ZppixBot>	 Hello BCMatsuyama! If you have any questions feel free to ask and someone should answer soon.
[10:25:03] <BCMatsuyama>	 Hi, who do I talk to to report a new problem with my wiki's images?
[10:25:29] <BCMatsuyama>	 .svg images that were okay when we uploaded them weeks ago are now turning into just solid black fill objects
[10:25:43] <BCMatsuyama>	 Example: https://anterra.miraheze.org/wiki/File:Ares_Antina_da-01.svg
[10:25:44] <ZppixBot>	 Title: [ File:Ares Antina da-01.svg - Anterra ] - anterra.miraheze.org
[10:36:30] <psiconauta>	 open a task on phabricator
[10:39:51] <psiconauta>	 btw, does it happen to new uploaded .svgs?
[10:40:06] <psiconauta>	 or just on some images?
[10:53:50] <BCMatsuyama>	 the .svg someone uploaded 2 Dec seems to be okay. The .svgs that came to our attention were uploaded at different points in October
[10:57:19] <BCMatsuyama>	 actually upon further looking  into it it seems only one user is affected
[10:57:27] <BCMatsuyama>	 but when those images were initially uploaded they displayed fine
[11:03:48] <psiconauta>	 have you tried to replace them?
[11:03:56] <psiconauta>	 upload them again
[11:05:51] <psiconauta>	 https://anterra.miraheze.org/wiki/index.php?title=Special:Upload&wpDestFile=Ares_Antina_da-01.svg&wpForReUpload=1
[11:05:52] <ZppixBot>	 Title: [ Index.php - Anterra ] - anterra.miraheze.org
[11:05:59] <psiconauta>	 try to reupload this one
[11:06:14] <psiconauta>	 or delete it and upload it again
[11:08:40] <psiconauta>	 hummm, I tried to upload that Ares Antina image
[11:08:53] <psiconauta>	 and get the same result than you
[11:09:35] <psiconauta>	 maybe there's some issue with rendering svgs
[11:09:56] <psiconauta>	 please open a task on phabricator
[11:10:19] <psiconauta>	 so the op team can analize this
[11:29:19] <BCMatsuyama>	 alright thank you
[14:35:12] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpF5z
[14:35:13] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 035019ee6 - BOT: Updating services config for wikis
[15:22:55] <icinga-miraheze>	 PROBLEM - misc2 Prometheus on misc2 is CRITICAL: connect to address 81.4.127.174 and port 9090: Connection refused  
[15:25:09] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[15:26:32] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpFxQ
[15:26:33] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03f58fe21 - Install TwoFactorAuth on matomo
[15:35:21] <icinga-miraheze>	 RECOVERY - misc2 Prometheus on misc2 is OK: TCP OK - 0.001 second response time on 81.4.127.174 port 9090  
[15:37:08] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[15:37:37] <paladox>	 !log rebooted misc2 (seems it had a OOM)
[15:38:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:41:08] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 4785  
[15:53:15] <icinga-miraheze>	 PROBLEM - misc2 Prometheus on misc2 is CRITICAL: connect to address 81.4.127.174 and port 9090: Connection refused  
[15:55:15] <icinga-miraheze>	 RECOVERY - misc2 Prometheus on misc2 is OK: TCP OK - 0.001 second response time on 81.4.127.174 port 9090  
[16:07:48] <paladox>	 !log upgrading prometheus on misc2 from 1.x to 2.x (from stretch backports)
[16:07:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:09:13] <icinga-miraheze>	 PROBLEM - misc2 Prometheus on misc2 is CRITICAL: connect to address 81.4.127.174 and port 9090: Connection refused  
[16:10:55] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpbfI
[16:10:56] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03df552e6 - Update prometheus config to prometheus 2.x
[16:11:20] <psiconauta>	 prometheus?
[16:11:23] <psiconauta>	 nice
[16:11:35] <paladox>	 it's what we use to gather metrics
[16:11:43] <paladox>	 which is displayed on grafana.miraheze.org
[16:12:21] <psiconauta>	 yeah, seems like a good piece of software
[16:12:48] <psiconauta>	 actually I'm planning on using it on my VPS
[16:13:13] <psiconauta>	 *to using
[16:13:14] <icinga-miraheze>	 RECOVERY - misc2 Prometheus on misc2 is OK: TCP OK - 0.001 second response time on 81.4.127.174 port 9090  
[16:13:24] <paladox>	 :)
[16:15:15] <psiconauta>	 I said earlier that you guys someday should migrate a DNS server with DNSSEC support (like nsd or bind)
[16:15:24] <psiconauta>	 *your
[16:15:49] <psiconauta>	 in case of interest 
[16:16:32] <psiconauta>	 *to a DNS server -.-
[16:20:00] <paladox>	 heh
[16:20:08] <paladox>	 dosen't gdnsd support that?
[16:20:40] <paladox>	 oh nope it does not
[16:20:47] <psiconauta>	 nope
[16:57:04] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[16:59:16] <JohnLewis>	 gdnsd does not support it and I'm not dropping gdnsd because of a lack of DNSSEC support
[17:14:59] <paladox>	 !log upgrade node-exporter on all servers
[17:15:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:43:49] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpbqO
[17:43:50] <Not-8d6a>	 [02miraheze/puppet] 07paladox 036a925f8 - Pin prometheus-node-exporter to debian stretch backports
[17:44:57] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpbqc
[17:44:58] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0304bcf5e - Pin prometheus to debian stretch back ports
[17:45:51] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 59507 MB (16% inode=96%);  
[18:26:54] <icinga-miraheze>	 PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[18:28:50] <icinga-miraheze>	 RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 44833 bytes in 0.083 second response time   
[19:16:36] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 033 commits to 03REL1_32 [+0/-0/±8] 13https://git.io/fpbnI
[19:16:38] <Not-8d6a>	 [02miraheze/mediawiki] 0720after4 039633c6d - bump wgVersion to 1.32.0-rc.1 Change-Id: Ib779ef76e851d7e430cd87f57d3fab2f7122290e
[19:16:39] <Not-8d6a>	 [02miraheze/mediawiki] 07plstand 035f1bf2c - PostgreSQL: Drop three unused columns The columns dropped in this commit are: * category.cat_hidden (for MySQL, dropped in 1.20 / 0cbc85494d47cdeb) * site_stats.ss_admins (for MySQL, dropped in 1.21 / bd602d20833f435f) * recentchanges.rc_cur_time (for MySQL, dropped in 1.24 / 2b441eba406f2c49) Bug: T64103 Change-Id: I7a7e9a2ac5066a9c9ab2583419d507e14fdd8c2c (cherry
[19:16:39] <Not-8d6a>	 picked from commit ba1207d5d582f4ce9cb8a9c0cfe8f083fd9e366f)
[19:16:41] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 03b96541e - Merge branch 'REL1_32' of https://github.com/wikimedia/mediawiki into REL1_32
[19:16:42] <ZppixBot>	 Title: [ GitHub - wikimedia/mediawiki: 🌻 The collaborative editing software that runs Wikipedia. This is a mirror from gerrit.wikimedia.org. See https://www.mediawiki.org/wiki/Developer_access for contributing ] - github.com
[19:17:36] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±3] 13https://git.io/fpbnm
[19:17:37] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 03cb26730 - Update Cargo and SocialProfile
[19:20:20] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±3] 13https://git.io/fpbnB
[19:20:22] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 03d426229 - Update Cargo and SocialProfile
[19:38:19] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fpbCZ
[19:38:21] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 03f4778b0 - Update Flow
[19:42:21] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fpbCr
[19:42:22] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 038da688e - Update Maps
[19:45:20] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fpbCd
[19:45:22] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 03db1e992 - Update Maps
[19:55:13] <Reception123>	 paladox: heh getting close to 1.32 :)
[19:55:19] <paladox>	 yeh
[19:55:27] <paladox>	 1.32rc1 is likley the last rc release
[19:55:30] <paladox>	 according to mukunda
[19:55:46] <Reception123>	 ok
[19:55:54] <Reception123>	 so we can expect 1.32 in the next week or so
[19:57:20] <Reception123>	 paladox: https://meta.miraheze.org/wiki/Community_noticeboard#Users_can.27t_register_because_their_accounts_were_already_created_during_wiki_import
[19:57:22] <ZppixBot>	 Title: [ Community noticeboard - Miraheze Meta ] - meta.miraheze.org
[19:59:32] <paladox>	 Reception123 yup.
[19:59:46] <paladox>	 Correct, the users were registered so the import would work properly.
[20:01:06] <Reception123>	 please answer since you took care of the case before :)
[20:01:47] <paladox>	 ok
[20:01:48] <paladox>	 done
[20:30:28] <Not-8d6a>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpb4M
[20:30:30] <Not-8d6a>	 [02miraheze/puppet] 07Reception123 0361ecd09 - add fifamwiki backup monthly T3889
[22:24:01] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpbaK
[22:24:03] <Not-8d6a>	 [02miraheze/puppet] 07paladox 039ce4b16 - Backup meeusenwiki monthly fixes T3878
[23:00:14] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpbrL
[23:00:16] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 033a5274c - BOT: Updating services config for wikis
[23:03:03] <paladox>	 hmm
[23:03:10] <paladox>	 someone disabled vs from test1
[23:04:02] <Not-8d6a>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpbr0
[23:04:04] <Not-8d6a>	 [02miraheze/mw-config] 07MacFan4000 03f45447b - Update LocalSettings.php
[23:20:13] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpbo2
[23:20:14] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 0302e2c84 - BOT: Updating services config for wikis
[23:27:26] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpbop
[23:27:28] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03c01e2b2 - php::php_fpm: Make class a little bit more presentable Also add's a few more params.
[23:35:12] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpbKR
[23:35:13] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 035a2260d - BOT: Updating services config for wikis
[23:56:07] <icinga-miraheze>	 HTTP CRITICAL - Unable to open TCP socket