[00:03:25] Hello apap! If you have any questions, feel free to ask and someone should answer soon. [00:54:27] Hello apap_04! If you have any questions, feel free to ask and someone should answer soon. [02:41:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:41:41] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [02:41:46] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [02:42:00] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:44:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:44:16] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [02:44:17] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [02:44:42] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [05:49:23] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [05:51:56] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [06:26:31] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2993 MB (12% inode=94%); [07:36:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [07:39:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:48:26] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [07:48:37] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [07:48:38] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3 [07:48:39] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [07:50:59] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [07:51:11] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [07:51:12] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [07:51:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [08:42:41] !log reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php rebuildall.php --wiki kkutuwiki [08:42:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:15:02] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [10:15:03] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [10:15:05] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [10:15:24] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:16:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [10:16:37] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:17:13] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:17:36] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:17:48] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [10:18:26] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [10:18:27] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [10:18:28] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [10:18:37] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.003 second response time [10:19:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:19:37] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24592 bytes in 0.004 second response time [10:19:53] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24570 bytes in 0.714 second response time [10:20:08] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24570 bytes in 0.394 second response time [10:20:21] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [10:32:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [10:35:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:04:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZA2 [13:04:15] [02miraheze/puppet] 07paladox 03ab5cf34 - Add timeout to /php-status [13:09:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZAw [13:09:58] [02miraheze/puppet] 07paladox 03841587d - varnish: Remove fastcgi timeout options from varnish mediawiki template [13:13:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZAi [13:13:36] [02miraheze/puppet] 07paladox 03f399096 - lizardfs: Increase NR_OF_NETWORK_WORKERS and NR_OF_HDD_WORKERS_PER_NETWORK_WORKER to 8 [13:15:56] !log restart lizardfs-chunkserver on lizardfs[45] [13:16:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:20:17] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [13:25:26] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:43:00] PROBLEM - panadev.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'panadev.ir' expires in 15 day(s) (Tue 15 Oct 2019 01:38:37 PM GMT +0000). [13:43:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZxT [13:43:17] [02miraheze/ssl] 07MirahezeSSLBot 03e9d3d32 - Bot: Update SSL cert for panadev.ir [13:52:53] RECOVERY - panadev.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'panadev.ir' will expire on Sat 28 Dec 2019 12:43:08 PM GMT +0000. [14:15:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZxy [14:15:10] [02miraheze/services] 07MirahezeSSLBot 0321e4df3 - BOT: Updating services config for wikis [14:19:54] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [14:34:28] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/JeZxj [14:34:30] [02miraheze/mediawiki] 07paladox 03c5e3ff6 - Update WikiForum [15:27:10] !log rm /mnt/mediawiki-static/metawiki/lockdir/obm4i1gmrsn12ek8soh4eorwgw4t4kh.lock [15:27:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:27:56] !log rm /mnt/mediawiki-static/allthetropeswiki/lockdir/*.lock [15:28:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:33:11] hello. One user uploads multiple images under false claims of CC-BY-SA. [17:34:08] Packages, book covers, or pictures grabbed from copyrighted sites. [17:35:41] https://commons.miraheze.org/wiki/Special:ListFiles/Penarc1?ilshowall=1 [17:35:45] [ File list - Miraheze Commons ] - commons.miraheze.org [17:36:12] qq[IrcCity]: looking [17:37:09] I already had a negative impression of this person since interaction in Wikimedia. [17:38:04] qq[IrcCity]: are they definitely all Copyright Violations? [17:38:15] possibly not all, but many. [17:38:31] should I do case-by-case analysis? [17:38:51] qq[IrcCity]: that would be easier - send the information to staff[at]Miraheze.org [17:39:51] Does a Special:EmailUser interface exists, by the way? [17:40:20] qq[IrcCity]: Not for staff but I suppose we could set it up [17:40:28] There's one for operations only [18:02:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeZjA [18:02:35] [02miraheze/puppet] 07paladox 03f002bdf - Revert "Add timeout to /php-status" This reverts commit ab5cf34f5272f5445d695f585cba88148573cc91. [19:52:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JenvC [19:53:00] [02miraheze/puppet] 07paladox 03473c781 - lizardfs: Set NO_ATIME to 1 [19:55:46] !log depool and repool mw1 [19:55:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:56:13] !log restart lizardfs-master - misc3 [20:16:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jenvu [20:16:18] [02miraheze/puppet] 07paladox 037a65bc0 - Update mediawiki.pp [20:16:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jenvz [20:16:44] [02miraheze/puppet] 07paladox 03053db72 - Revert "lizardfs: Set NO_ATIME to 1" This reverts commit 473c781a25552cc900a9e308b463dd5daae4749c. [20:22:18] https://commons.miraheze.org/wiki/Miraheze_Commons:Files_uploaded_by_Penarc1 [20:22:19] [ Miraheze Commons:Files uploaded by Penarc1 - Miraheze Commons ] - commons.miraheze.org [20:23:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JenvV [20:23:21] [02miraheze/puppet] 07paladox 03b028eaa - Revert "Update mediawiki.pp" This reverts commit 7a65bc0d5268656532e477dfff5c7db8def318cf. [20:30:13] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [20:35:29] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:29:46] hi [21:30:44] RhinosF1: You here? [22:46:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.10, 6.36, 4.88 [22:48:33] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.81, 5.31, 4.69 [23:01:13] Examknow: I am around whats ip [23:01:16] s/ip/up [23:01:16] !log sudo usermod -a -G ssl-cert www-data - mw3 [23:01:16] Zppix meant to say: Examknow: I am around whats up [23:01:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:01:53] !log sudo usermod -a -G ssl-cert www-data - mw2 [23:01:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:02:14] paladox: I see paladox then usermod, is it bad i immediately get concerned? [23:02:45] [02mw-config] 07Southparkfan closed pull request 03#2660: Enable CirrusSearch globally + config clean up - 13https://git.io/fjtx7 [23:02:48] heh, just fixing something that prevented www-data from reading a file. [23:07:26] !log depool and repool mw[23] [23:07:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master