[00:00:21] RECOVERY - misc1 Current Load on misc1 is OK: OK - load average: 0.14, 0.07, 1.43 [00:03:23] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3846 MB (19% inode=93%); [01:24:24] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [01:26:31] PROBLEM - bacula2 Bacula Static on bacula2 is WARNING: WARNING: Full, 6831642 files, 655.2GB, 2020-06-10 12:11:00 (2.2 weeks ago) [01:39:27] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [01:40:00] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [01:41:50] PROBLEM - bacula2 Bacula Static on bacula2 is WARNING: WARNING: Full, 6831642 files, 655.2GB, 2020-06-10 12:11:00 (2.2 weeks ago) [01:41:58] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Full, 82779 files, 3.261GB, 2020-06-07 15:19:00 (2.6 weeks ago) [03:11:02] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:01] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Thu 20 Aug 2020 14:59:34 GMT +0000. [03:35:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfjWp [03:35:14] [02miraheze/services] 07MirahezeSSLBot 035dd164a - BOT: Updating services config for wikis [03:45:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfjlC [03:45:15] [02miraheze/services] 07MirahezeSSLBot 0339f1513 - BOT: Updating services config for wikis [04:42:33] Hey [04:43:47] Is user [[User:Reception3611]] imposter account of Reception123 and can be (b)locked or not? [05:18:22] @MrJaroslavik thanks for letting me know, will lock for impersonation [05:19:55] Not a problem :) [05:47:06] [02mw-config] 07Reception123 commented on pull request 03#3116: Add '.*scratch' to rw blacklist - 13https://git.io/Jfj4j [06:11:13] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.61, 1.75, 1.09 [06:12:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.90, 16.32, 11.76 [06:13:13] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.10, 1.47, 1.07 [06:14:21] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.56, 16.11, 12.23 [06:55:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfj0J [06:55:15] [02miraheze/services] 07MirahezeSSLBot 03321ea2a - BOT: Updating services config for wikis [07:20:31] [ANNOUNCEMENT] Hey everyone, We'll be upgrading python between 9am and 10am. This will cause about half an hour of downtime while we upgrade. --RhinosF1 [08:09:09] PROBLEM - mw6 HTTPS on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:29] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:10:03] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:10:03] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw6 [08:10:23] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:10:40] Oh f [08:10:43] Reception123: ^ [08:10:51] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw6 [08:10:51] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [08:10:52] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw6 [08:11:10] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw6 [08:11:16] Reception123: can you depool mw6? [08:12:25] paladox, PuppyKun, SPF|Cloud: mw6 is failing [08:15:43] ssh is fine [08:19:43] https://www.irccloud.com/pastebin/2uajchWy/ [08:19:44] [ Snippet | IRCCloud ] - www.irccloud.com [08:19:52] curl shows a 503 [08:20:16] icinga shows a HTTPS alert [08:27:15] JohnLewis: mw6 has depooled, icinga shows HTTPS error but no clue why [08:28:59] No access atm, in around 30 mins I can look. [08:28:59] Checked nginx’s logs? [08:29:29] JohnLewis: cat: error.log: Permission denied [08:29:41] As www-data? [08:30:13] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 5.155 second response time [08:30:30] JohnLewis: I see a lot of no such file or directory errors [08:30:57] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15533 bytes in 1.026 second response time [08:31:02] Mkay, looks like it recovered on it owns anyway [08:31:06] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [08:31:27] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy [08:31:33] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.008 second response time [08:31:43] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [08:31:48] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.340 second response time [08:32:03] RECOVERY - mw6 HTTPS on mw6 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 0.007 second response time [08:32:07] JohnLewis: that's still weird [08:32:18] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [08:33:09] JohnLewis: see pm [08:35:28] based on icinga first fail was 09:06:21, went green at 09:32:03 [08:46:17] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [08:46:25] ^ known [08:59:02] ^ me manually running [09:01:31] * RhinosF1 pasted error in -staff [09:04:48] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [09:05:10] JohnLewis: ^ mw4 just gone as well [09:05:33] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [09:05:39] and mw7 [09:05:51] so just mw5 left to fail [09:17:40] !log umount /mnt/mediawiki-static on mw4/6/7 (mounted automatically after) [09:17:41] ^ RhinosF1 [09:17:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [09:19:04] Reception123: ty [09:19:49] Np [09:20:01] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [09:20:01] Reception123: mw6 passed [09:22:29] Reception123: mw4 passed, mw7 running [09:22:49] Reception123: let's hope 5 doesn't break [09:22:53] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:23:25] yey all passed [09:23:38] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [12:07:38] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03fix-rcmaxage [+0/-0/±1] 13https://git.io/JfjPT [12:07:39] [02miraheze/mw-config] 07RhinosF1 03bbdefb9 - Enforce interger on RCMaxAge caused 2020-06-26 07:25:42 mw4 atbwwiki: [2834ecc379da4985c90a545e] /w/index.php?title=2023&action=info&useformat=mobile Wikimedia\Timestamp\TimestampException from line 240 of /srv/mediawiki/w/vendor/wikimedia/timestamp/src/ConvertibleTimestamp.php: Wikimedia\Timestamp\ConvertibleTimestamp::setTimestamp: Invalid timestamp - [12:07:39] -99999998406843657 [12:07:41] [02mw-config] 07RhinosF1 created branch 03fix-rcmaxage - 13https://git.io/vbvb3 [12:07:44] [02mw-config] 07RhinosF1 opened pull request 03#3126: Enforce interger on RCMaxAge - 13https://git.io/JfjPI [12:08:03] JohnLewis: will ManageWiki validate everything fine [12:08:42] PROBLEM - misc1 SSH on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:56] PROBLEM - misc1 APT on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:08:57] :( [12:09:03] Reception123: misc1 is going [12:09:10] PROBLEM - ping6 on misc1 is CRITICAL: PING CRITICAL - Packet loss = 100% [12:09:14] miraheze/mw-config/fix-rcmaxage/bbdefb9 - RhinosF1 The build passed. https://travis-ci.org/miraheze/mw-config/builds/702351386 [12:09:16] PROBLEM - misc1 Current Load on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:09:36] :( [12:09:44] PROBLEM - Host misc1 is DOWN: PING CRITICAL - Packet loss = 100% [12:11:44] Reception123: I think you might want to recover ^ first [12:12:02] RECOVERY - Host misc1 is UP: PING OK - Packet loss = 0%, RTA = 12.91 ms [12:12:03] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:12:03] RECOVERY - misc1 Current Load on misc1 is OK: OK - load average: 0.96, 0.21, 0.07 [12:12:05] PROBLEM - misc1 Disk Space on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:12:05] PROBLEM - misc1 SMTP on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:05] PROBLEM - ping4 on misc1 is CRITICAL: PING CRITICAL - Packet loss = 100% [12:12:05] PROBLEM - misc1 IMAP on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:08] weird system [12:12:13] PROBLEM - ping4 on misc1 is CRITICAL: PING CRITICAL - Packet loss = 100% [12:12:15] RECOVERY - ping4 on misc1 is OK: PING OK - Packet loss = 0%, RTA = 12.92 ms [12:12:15] RECOVERY - misc1 Disk Space on misc1 is OK: DISK OK - free space: / 34658 MB (85% inode=99%); [12:12:20] RECOVERY - misc1 IMAP on misc1 is OK: IMAP OK - 0.112 second response time on 185.52.1.76 port 143 [* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.] [12:12:33] RECOVERY - misc1 SMTP on misc1 is OK: SMTP OK - 0.084 sec. response time [12:12:35] RhinosF1: always fixes itself somehow [12:13:08] Reception123: that's the second server today to decide to go walkies [12:13:10] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [12:13:18] RECOVERY - misc1 SSH on misc1 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u7 (protocol 2.0) [12:13:26] RECOVERY - misc1 APT on misc1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:13:45] RECOVERY - ping6 on misc1 is OK: PING OK - Packet loss = 0%, RTA = 19.78 ms [12:19:23] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfjPu [12:19:25] [02miraheze/mw-config] 07RhinosF1 032cd9497 - Update ManageWikiSettings.php [12:19:33] paladox: ^ [12:23:25] Thanks [12:29:53] [02miraheze/ManageWiki] 07RhinosF1 pushed 031 commit to 03fix-rc-max-age [+1/-0/±0] 13https://git.io/JfjPd [12:29:54] [02miraheze/ManageWiki] 07RhinosF1 03d015241 - Create fixRCMaxAge.php [12:29:56] [02ManageWiki] 07RhinosF1 created branch 03fix-rc-max-age - 13https://git.io/vpSns [12:29:58] [02ManageWiki] 07RhinosF1 opened pull request 03#171: Create fixRCMaxAge.php - 13https://git.io/JfjPb [12:30:08] paladox: ^ [12:32:07] [02ManageWiki] 07RhinosF1 assigned pull request 03#171: Create fixRCMaxAge.php - 13https://git.io/JfjPb [12:32:50] miraheze/ManageWiki/fix-rc-max-age/d015241 - RhinosF1 The build failed. https://travis-ci.org/miraheze/ManageWiki/builds/702356563 [12:34:34] !log sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --wiki allthetropeswiki --full > /home/reception/allthetropeswiki26062020.xml (part of wikibackups) [12:34:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:24:17] PROBLEM - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ricochetuniverse.com' expires in 15 day(s) (Sun 12 Jul 2020 13:21:16 GMT +0000). [13:28:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfjMn [13:28:35] [02miraheze/ssl] 07MirahezeSSLBot 035f2b8b0 - Bot: Update SSL cert for wiki.ricochetuniverse.com [13:32:27] RECOVERY - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ricochetuniverse.com' will expire on Thu 24 Sep 2020 12:28:26 GMT +0000. [13:38:48] RhinosF1; integers don’t need to be validated, only numbers can be entered [13:41:03] [02mw-config] 07JohnFLewis closed pull request 03#3114: fix simcitywiki configuration - 13https://git.io/JfHo7 [13:41:05] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfjDe [13:41:06] [02miraheze/mw-config] 07Amanda-Catherine 0333fe6cd - fix simcitywiki configuration (#3114) * fix simcitywiki configuration * give abusefilter checkuser rights back to stewards [13:41:56] JohnLewis: so that'll work fine as long as we do a script to fix the wrong values [13:44:30] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [13:46:01] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out [13:47:00] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Full, 82779 files, 3.261GB, 2020-06-07 15:19:00 (2.7 weeks ago) [13:47:59] PROBLEM - bacula2 Bacula Static on bacula2 is WARNING: WARNING: Full, 6831642 files, 655.2GB, 2020-06-10 12:11:00 (2.3 weeks ago) [13:50:25] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfjDV [13:50:27] [02miraheze/services] 07MirahezeSSLBot 03df55248 - BOT: Updating services config for wikis [14:04:51] PROBLEM - jobrunner1 Disk Space on jobrunner1 is WARNING: DISK WARNING - free space: / 2352 MB (8% inode=75%); [14:08:52] PROBLEM - jobrunner1 Disk Space on jobrunner1 is CRITICAL: DISK CRITICAL - free space: / 1677 MB (5% inode=75%); [14:14:51] Hey, is better Abstain or Neutral for voting template? :) [14:15:45] Either [14:18:21] @MrJarsolavlik: why not just do both? [14:19:30] There was redirect now, i think. You mean two separated templates? :) [14:20:31] @MrJarosolavik: yes [14:20:43] Okay [14:20:51] Thanks 👍 [14:22:43] @MrJaroslavik: You need to get your sysop extended indefinately as well [14:23:17] Ooh you have requested [14:23:19] * RhinosF1 votes [14:24:13] !log increase jobrunner1 disk by 10g [14:24:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:25:06] :-) [14:27:19] Thanks for your vote RhinosF1 [14:29:15] RECOVERY - jobrunner1 Disk Space on jobrunner1 is OK: DISK OK - free space: / 7085 MB (19% inode=80%); [14:29:41] !log increase jobrunner1 disk by 10g [14:29:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:38:42] [02ManageWiki] 07paladox synchronize pull request 03#171: Create fixRCMaxAge.php - 13https://git.io/JfjPb [14:38:43] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03fix-rc-max-age [+0/-0/±1] 13https://git.io/Jfj9H [14:38:45] [02miraheze/ManageWiki] 07paladox 0354164ae - Update fixRCMaxAge.php [14:39:32] [02ManageWiki] 07paladox synchronize pull request 03#171: Create fixRCMaxAge.php - 13https://git.io/JfjPb [14:39:33] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03fix-rc-max-age [+0/-0/±1] 13https://git.io/Jfj9F [14:39:35] [02miraheze/ManageWiki] 07paladox 03cf2bbb8 - Update fixRCMaxAge.php [14:39:53] [02mw-config] 07paladox closed pull request 03#3126: Enforce interger on RCMaxAge - 13https://git.io/JfjPI [14:39:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfj9b [14:39:56] [02miraheze/mw-config] 07RhinosF1 038321914 - Enforce interger on RCMaxAge (#3126) caused 2020-06-26 07:25:42 mw4 atbwwiki: [2834ecc379da4985c90a545e] /w/index.php?title=2023&action=info&useformat=mobile Wikimedia\Timestamp\TimestampException from line 240 of /srv/mediawiki/w/vendor/wikimedia/timestamp/src/ConvertibleTimestamp.php: Wikimedia\Timestamp\ConvertibleTimestamp::setTimestamp: Invalid timestamp - [14:39:56] -99999998406843657 [14:39:57] [02mw-config] 07paladox deleted branch 03fix-rcmaxage - 13https://git.io/vbvb3 [14:39:59] [02miraheze/mw-config] 07paladox deleted branch 03fix-rcmaxage [14:40:57] paladox: the script needs to validate the number is between minint and maxint as set in MWS.php [14:41:37] RhinosF1 you'll need to create a script then that output each value for each wiki [14:41:45] and then see if it matches the value you had set [14:41:57] (as in if it's between min and max) [14:42:14] miraheze/ManageWiki/fix-rc-max-age/cf2bbb8 - paladox The build was fixed. https://travis-ci.org/miraheze/ManageWiki/builds/702402980 [14:42:49] paladox: I don't know how to do it. You have the value in that script. So check if it's lower than 1 and set to 1 and above 15552000 and set to 155520000 [14:43:10] RhinosF1 pretty easy [14:43:26] RhinosF1 if (config < 1 ) set ... [14:43:38] if ( config > 15552000) set... [14:44:21] paladox: let me try but mobile so might need some cleaning [14:44:29] ok [14:50:31] [02miraheze/ManageWiki] 07RhinosF1 pushed 031 commit to 03fix-rc-max-age [+0/-0/±1] 13https://git.io/JfjHR [14:50:33] [02miraheze/ManageWiki] 07RhinosF1 0395a9acc - Update fixRCMaxAge.php [14:50:34] [02ManageWiki] 07RhinosF1 synchronize pull request 03#171: Create fixRCMaxAge.php - 13https://git.io/JfjPb [14:50:58] paladox: ^ will need tabs v spaces as I'm mobile but code should be right [14:51:06] ok [14:52:34] paladox: can you? [14:52:58] miraheze/ManageWiki/fix-rc-max-age/95a9acc - RhinosF1 The build was broken. https://travis-ci.org/miraheze/ManageWiki/builds/702406104 [14:53:02] [02miraheze/ManageWiki] 07RhinosF1 pushed 031 commit to 03fix-rc-max-age [+0/-0/±1] 13https://git.io/JfjHw [14:53:04] [02miraheze/ManageWiki] 07RhinosF1 03af57ffb - Update fixRCMaxAge.php [14:53:05] [02ManageWiki] 07RhinosF1 synchronize pull request 03#171: Create fixRCMaxAge.php - 13https://git.io/JfjPb [14:53:12] paladox: ^ [14:54:13] [02ManageWiki] 07paladox synchronize pull request 03#171: Create fixRCMaxAge.php - 13https://git.io/JfjPb [14:54:15] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03fix-rc-max-age [+0/-0/±1] 13https://git.io/JfjHP [14:54:16] [02miraheze/ManageWiki] 07paladox 030fcd05a - Update fixRCMaxAge.php [14:55:32] miraheze/ManageWiki/fix-rc-max-age/af57ffb - RhinosF1 The build was fixed. https://travis-ci.org/miraheze/ManageWiki/builds/702406856 [14:55:36] [02ManageWiki] 07paladox synchronize pull request 03#171: Create fixRCMaxAge.php - 13https://git.io/JfjPb [14:55:38] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03fix-rc-max-age [+0/-0/±1] 13https://git.io/JfjH9 [14:55:39] [02miraheze/ManageWiki] 07paladox 03ca19246 - Update fixRCMaxAge.php [14:56:20] RhinosF1 looks good to run now [14:56:30] we won't merge the change, but you would run the script [14:57:19] paladox: so cd into ManageWiki and cherry pick it? [14:57:32] And then I can run when I have access [14:57:55] Yes but do not cherry pick it, just copy the file [14:58:04] e.g by using vi and copying and pasting [14:58:22] miraheze/ManageWiki/fix-rc-max-age/ca19246 - paladox The build was broken. https://travis-ci.org/miraheze/ManageWiki/builds/702407594 [14:58:23] paladox: ok cool [14:58:50] paladox: parse error [14:59:08] [02ManageWiki] 07paladox synchronize pull request 03#171: Create fixRCMaxAge.php - 13https://git.io/JfjPb [14:59:10] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03fix-rc-max-age [+0/-0/±1] 13https://git.io/JfjHN [14:59:11] [02miraheze/ManageWiki] 07paladox 0347b4ac1 - Update fixRCMaxAge.php [15:01:45] miraheze/ManageWiki/fix-rc-max-age/47b4ac1 - paladox The build was fixed. https://travis-ci.org/miraheze/ManageWiki/builds/702408615 [15:01:59] Yey! [15:02:07] * RhinosF1 will run tonight [15:03:27] paladox: the RCMaxAge box doesn't fit every charecter in [15:03:39] what do you mean? [15:03:57] paladox: it hides the last charecter [15:04:02] huh? [15:04:19] https://usercontent.irccloud-cdn.com/file/28UFU5fO/IMG_6051.PNG [15:04:23] paladox: ^ [15:04:33] oh ok [15:04:39] That's 15552000 [15:04:44] It's missing 2 off [15:04:49] you should file a bug i guess. [15:04:51] Can we make the box bigger [15:07:30] I have no idea :) [15:07:32] maybe [15:07:35] this is OOUI [15:11:00] * RhinosF1 tries to find what defines the box for an integer [15:11:19] paladox: any clue what part of the code creates that box? [15:11:25] no [15:13:39] JohnLewis: so I can provide information in this bug? [15:32:22] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jfj7l [15:32:24] [02miraheze/ssl] 07paladox 03edecc30 - Add wiki.erisly.com ssl certificate [15:32:25] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:32:27] [02ssl] 07paladox opened pull request 03#327: Add wiki.erisly.com ssl certificate - 13https://git.io/Jfj78 [15:32:59] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/Jfj7R [15:33:00] [02miraheze/ssl] 07paladox 03f729eb9 - Create wiki.erisly.com.crt [15:33:02] [02ssl] 07paladox synchronize pull request 03#327: Add wiki.erisly.com ssl certificate - 13https://git.io/Jfj78 [15:33:44] [02ssl] 07paladox closed pull request 03#327: Add wiki.erisly.com ssl certificate - 13https://git.io/Jfj78 [15:33:44] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jfj72 [15:33:46] [02miraheze/ssl] 07paladox 03b94966c - Add wiki.erisly.com ssl certificate (#327) * Add wiki.erisly.com ssl certificate * Create wiki.erisly.com.crt [15:33:47] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:33:49] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [15:35:55] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jfj7P [15:35:57] [02miraheze/ssl] 07paladox 03b27181c - Change domain to ao90.pinho.org [15:35:58] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:36:00] [02ssl] 07paladox opened pull request 03#328: Change domain to ao90.pinho.org - 13https://git.io/Jfj7X [15:36:50] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://git.io/Jfj7D [15:36:52] [02miraheze/ssl] 07paladox 03162df8e - Update and rename contraao.com.crt to ao90.pinho.org.crt [15:36:53] [02ssl] 07paladox synchronize pull request 03#328: Change domain to ao90.pinho.org - 13https://git.io/Jfj7X [15:37:08] [02ssl] 07paladox closed pull request 03#328: Change domain to ao90.pinho.org - 13https://git.io/Jfj7X [15:37:10] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/Jfj7Q [15:37:11] [02miraheze/ssl] 07paladox 03e0a1f25 - Change domain to ao90.pinho.org (#328) * Change domain to ao90.pinho.org * Update and rename contraao.com.crt to ao90.pinho.org.crt [15:37:24] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [15:37:26] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:40:23] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfj7h [15:40:25] [02miraheze/services] 07MirahezeSSLBot 03467bd79 - BOT: Updating services config for wikis [15:41:42] !log upgrade ca-certificates - gluster1 [15:41:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:41:52] [02ManageWiki] 07JohnFLewis commented on pull request 03#171: Create fixRCMaxAge.php - 13https://git.io/Jfj5f [15:42:10] !log upgrade ca-certificates - gluster2 [15:42:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:43:33] [02ManageWiki] 07paladox commented on pull request 03#171: Create fixRCMaxAge.php - 13https://git.io/Jfj5T [15:44:28] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[contraao.com],File[contraao.com_private] [15:44:55] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[contraao.com],File[contraao.com_private] [15:44:57] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[contraao.com],File[contraao.com_private] [15:45:03] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[contraao.com],File[contraao.com_private] [15:45:08] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[contraao.com],File[contraao.com_private] [15:45:39] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[contraao.com],File[contraao.com_private] [15:46:10] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[contraao.com],File[contraao.com_private] [15:46:13] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): File[contraao.com],File[contraao.com_private] [15:48:02] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[contraao.com],File[contraao.com_private] [15:51:06] paladox: ^ [15:51:14] hmm [15:51:53] paladox: think you need to get rid of the private key [15:51:55] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jfj50 [15:51:57] [02miraheze/ssl] 07paladox 031f5f69a - Remove wiki.contraao.com [15:51:58] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:52:00] [02ssl] 07paladox opened pull request 03#329: Remove wiki.contraao.com - 13https://git.io/Jfj5E [15:52:01] I don't know why sometimes it gets mad about that, sometimes it doesn't mind [15:52:07] [02ssl] 07paladox closed pull request 03#329: Remove wiki.contraao.com - 13https://git.io/Jfj5E [15:52:09] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfj5u [15:52:10] [02miraheze/ssl] 07paladox 03bbf9ed6 - Remove wiki.contraao.com (#329) [15:52:12] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [15:52:13] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [15:52:35] the private key is gone [15:52:38] but that's not the problem [15:54:32] [02ManageWiki] 07JohnFLewis commented on pull request 03#171: Create fixRCMaxAge.php - 13https://git.io/Jfj5V [16:02:51] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:03:02] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:03:04] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [16:03:05] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [16:03:18] JohnLewis: how do we do it then? [16:03:50] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [16:03:57] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:04:13] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:04:29] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:05:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfj5x [16:05:16] [02miraheze/services] 07MirahezeSSLBot 0386c4e22 - BOT: Updating services config for wikis [16:07:13] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:10:58] RhinosF1; see ManageWikiSettings.php [16:12:42] paladox: ^ [16:12:48] ? [16:13:04] paladox: that's how we actually do it [16:13:11] ok [16:15:01] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [16:16:46] paladox: ^ needs mounting? [16:17:20] yes [16:17:21] and done [16:17:36] !log remounted /mnt/mediawiki-static on mw6 [16:17:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:17:44] paladox: it did that like 6 hours ago [16:17:47] On all but mw5 [16:18:06] ok [16:18:13] i'm now mobile [16:22:55] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:31:21] JohnLewis: should we allow mw-admins to unmount static? [16:34:02] hi, i'm from https://tuscriaturas.miraheze.org/ [16:34:03] [ Bestiario del Hypogripho ] - tuscriaturas.miraheze.org [16:34:33] PROBLEM - spiral.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:54] how do i make an interwiki so i can connect an article written in spanish with the same article written on english on a different page? RhinosF1 [16:34:58] ^ not broke [16:35:18] avengium: I wonder if WikiBase would be better here [16:36:09] i don't know what is wikibase [16:36:28] RECOVERY - spiral.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiral.wiki' will expire on Thu 20 Aug 2020 15:07:11 GMT +0000. [16:37:05] avengium: you have a central wiki, and then each page has a page on that wiki which specifies the link for each language. Each language then appears in the sidebar. [16:38:14] that could work. The same wiki but different languages [16:39:01] avengium: each language has its own wiki [16:39:17] ok [16:39:25] But they are linked by a central wiki [16:40:10] avengium: if you request a wiki for each language and the central wiki, then create a phab task. I'll set it up. [16:40:40] If you give me an example of 1 page that has an entry on all languages, I can add that so you can see how it works. [16:41:01] but i need to specify the amount of languages in the phab ticket i want or is something dynamic? [16:41:22] avengium: we can add new languages at any time [16:41:47] You can add them yourself once you see how but don't worry, WikiBase can look scary at first. [16:43:09] ok. so i need to send a phabricator ticket. What info i need to provide on the phab ticket [16:45:05] avengium: just give me a list of the wikis and a page that is about the same thing on each wiki but in each language on phab [16:47:00] At the moment, we have only created content in Spanish, I have no copies in English on miraheze. Do I need to give you an article in English published on our wiki? [16:47:56] avengium: It would be good to be able to show you how it works if you had the 2 languages already created. [16:48:24] ok, i'm going to talk to another admin and tell him that we need an article in english [16:48:56] avengium: you set it up so you have a wiki for each language [16:50:43] sorry if i'm being slow, or something, but what do you mean? what is next step i need to do before sending the ticket on phabricator? [16:52:00] which is the next step* [16:57:24] Just request the wikis [16:58:47] ok. the current one tuscriaturas. Do I need to say the name of the subdomain that I want to request? [17:00:04] On the request yes [17:01:26] and the central one is an usable wiki or is just an "admin" wiki or something not visible for common users [17:03:26] the central wiki is something common users see or only visible for admins? [17:03:37] * by admins [17:04:54] It's public [17:05:05] Anyone can edit it but you're able to change that [17:07:11] ok [17:59:22] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.40, 1.85, 1.34 [18:01:25] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.93, 1.65, 1.34 [18:46:45] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:47:13] * RhinosF1 sighs at icinga-miraheze [18:48:49] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Thu 20 Aug 2020 14:59:34 GMT +0000. [18:58:51] Question: must be pages like https://meta.miraheze.org/wiki/Training_modules/Dealing_with_online_harassment/slides/what-kind-of-non-miraheze-support-can-you-direct-someone-to in the translation system? Its currently/still used? [18:58:52] [ Training modules/Dealing with online harassment/slides/what-kind-of-non-miraheze-support-can-you-direct-someone-to - Miraheze Meta ] - meta.miraheze.org [19:00:39] @MrJaroslavik: Yes, I maintain them. [19:01:16] Okay, thanks for the info. [19:02:33] They make up https://meta.miraheze.org/wiki/Training_modules/Online_harassment [19:02:34] [ Training modules/Online harassment - Miraheze Meta ] - meta.miraheze.org [19:03:14] Its too long :O [19:03:51] It's made to be for advanced users & sysadmins [19:04:10] or those that comes across harrasment [19:21:56] Any steward around? [19:22:27] !log upgrade ca-certificates - bacula2 [19:22:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:23:44] !log upgrade grafana - mon1 [19:23:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:24:55] @MrJaroslavik is it urgent? [19:25:30] Not urgent, its about missed requests at SN (already archived) [19:26:06] Oh, probably best to at least post a note about them, and I'll take a look later today [19:26:49] And you can nudge me tomorrow if I forget again [19:27:19] !log remove icinga* from mon1 - accidentally removed just realised (thought it was misc1) [19:27:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:29:02] First is request for delete translator group (https://meta.miraheze.org/wiki/Stewards%27_noticeboard/Archive_11#Delete_.22translator.22_user_group), second is request for inactivity exempt for me (https://meta.miraheze.org/wiki/Stewards%27_noticeboard/Archive_11#Request_for_inactivity_exempt) [19:29:03]