[00:06:29] <icinga-wm>	 RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[00:08:49] <icinga-wm>	 RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[00:12:49] <icinga-wm>	 PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:37:50] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 032] Fix typo in realm.pp [puppet] - 10https://gerrit.wikimedia.org/r/335548 (owner: 10Tim Landscheidt)
[00:41:49] <icinga-wm>	 RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[00:42:35] <wikibugs>	 06Operations, 10Domains, 10Traffic, 10Wikimedia-Site-requests: Consider mw.org being added as a redirect to mediawiki.wikimedia.org - https://phabricator.wikimedia.org/T158490#3038669 (10SamanthaNguyen) @Zppix: Instead of mediawiki.wikimedia.org, did you mean https://www.mediawiki.org?
[00:46:42] <wikibugs>	 06Operations, 10Domains, 10Traffic, 10Wikimedia-Site-requests: Consider mw.org being added as a redirect to mediawiki.wikimedia.org - https://phabricator.wikimedia.org/T158490#3038649 (10Reedy) mw.org seems to be a reserved address. And AFAIK, the WMF don't own it  ``` mw.org is already registered*  Name i...
[01:09:09] <icinga-wm>	 PROBLEM - puppet last run on cp3041 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[01:37:09] <icinga-wm>	 RECOVERY - puppet last run on cp3041 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[01:45:43] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 032] "retest" [puppet] - 10https://gerrit.wikimedia.org/r/335548 (owner: 10Tim Landscheidt)
[01:52:31] <Zppix>	 andrewbogott: fyi its recheck
[01:53:10] <wikibugs>	 (03CR) 10Reedy: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/335548 (owner: 10Tim Landscheidt)
[01:55:06] <andrewbogott>	 Yep, seems happy now
[01:55:54] <Zppix>	 andrewbogott: it happens it probably lost connection to gerrits changes for a sec and didnt see it
[02:31:49] <logmsgbot>	 !log l10nupdate@tin scap sync-l10n completed (1.29.0-wmf.12) (duration: 13m 08s)
[02:31:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[02:33:42] <wikibugs>	 (03CR) 10Tim Landscheidt: "@DatGuy: Sorry for the late response; I meant to say that you should use the Redirect directive.  I have submitted Ic79292fdc3c1d54ba58b32" [puppet] - 10https://gerrit.wikimedia.org/r/330178 (https://phabricator.wikimedia.org/T113696) (owner: 10DatGuy)
[02:34:08] <wikibugs>	 (03PS1) 10Tim Landscheidt: toolserver_legacy: Use Redirect instead of RedirectMatch [puppet] - 10https://gerrit.wikimedia.org/r/338610
[02:37:07] <logmsgbot>	 !log l10nupdate@tin ResourceLoader cache refresh completed at Sun Feb 19 02:37:07 UTC 2017 (duration 5m 18s)
[02:37:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[03:20:39] <icinga-wm>	 PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 752.17 seconds
[03:28:41] <icinga-wm>	 RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 276.10 seconds
[03:33:19] <icinga-wm>	 PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIP2-City.mmdb.gz]
[03:43:39] <icinga-wm>	 PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[03:55:39] <icinga-wm>	 PROBLEM - puppet last run on restbase1012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[03:55:49] <icinga-wm>	 PROBLEM - Make sure enwiki dumps are not empty on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/dumps - 288 bytes in 0.013 second response time
[04:01:09] <icinga-wm>	 RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[04:01:49] <icinga-wm>	 PROBLEM - Disk space on elastic1024 is CRITICAL: DISK CRITICAL - free space: /srv 60751 MB (12% inode=99%)
[04:12:39] <icinga-wm>	 RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[04:23:39] <icinga-wm>	 RECOVERY - puppet last run on restbase1012 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[05:02:19] <icinga-wm>	 PROBLEM - puppet last run on labservices1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[05:15:49] <icinga-wm>	 RECOVERY - Disk space on elastic1024 is OK: DISK OK
[05:24:09] <icinga-wm>	 PROBLEM - Disk space on elastic1026 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 63960 MB (12% inode=99%)
[05:32:19] <icinga-wm>	 RECOVERY - puppet last run on labservices1002 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[05:56:10] <icinga-wm>	 PROBLEM - Disk space on elastic1026 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 64588 MB (12% inode=99%)
[06:41:09] <icinga-wm>	 PROBLEM - Disk space on elastic1026 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 65632 MB (12% inode=99%)
[06:50:59] <icinga-wm>	 PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[07:08:09] <icinga-wm>	 PROBLEM - Disk space on elastic1026 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 64042 MB (12% inode=99%)
[07:18:59] <icinga-wm>	 RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[07:21:59] <icinga-wm>	 PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[07:39:49] <icinga-wm>	 PROBLEM - Disk space on elastic1024 is CRITICAL: DISK CRITICAL - free space: /srv 61192 MB (12% inode=99%)
[07:48:09] <icinga-wm>	 PROBLEM - Disk space on elastic1026 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 64980 MB (12% inode=99%)
[07:49:59] <icinga-wm>	 RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[08:08:09] <icinga-wm>	 PROBLEM - Disk space on elastic1026 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 64518 MB (12% inode=99%)
[08:16:49] <icinga-wm>	 PROBLEM - Disk space on elastic1024 is CRITICAL: DISK CRITICAL - free space: /srv 60829 MB (12% inode=99%)
[08:29:09] <icinga-wm>	 RECOVERY - Disk space on elastic1026 is OK: DISK OK
[08:30:49] <icinga-wm>	 PROBLEM - Disk space on elastic1024 is CRITICAL: DISK CRITICAL - free space: /srv 62144 MB (12% inode=99%)
[08:56:49] <icinga-wm>	 PROBLEM - Disk space on elastic1024 is CRITICAL: DISK CRITICAL - free space: /srv 61755 MB (12% inode=99%)
[09:07:49] <icinga-wm>	 RECOVERY - Disk space on elastic1024 is OK: DISK OK
[09:35:55] <wikibugs>	 (03CR) 10Nemo bis: [C: 031] "Ready to go now: over one month has passed from the latest comment on the RfC. Let's pick a date on https://wikitech.wikimedia.org/wiki/De" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/333860 (https://phabricator.wikimedia.org/T63729) (owner: 10MarcoAurelio)
[10:11:59] <icinga-wm>	 RECOVERY - Make sure enwiki dumps are not empty on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.007 second response time
[10:32:09] <icinga-wm>	 PROBLEM - puppet last run on ms-be1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[10:57:59] <icinga-wm>	 PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[11:00:09] <icinga-wm>	 RECOVERY - puppet last run on ms-be1020 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[11:09:10] <icinga-wm>	 PROBLEM - Disk space on elastic1026 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 64695 MB (12% inode=99%)
[11:26:59] <icinga-wm>	 RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[11:50:09] <icinga-wm>	 RECOVERY - Disk space on elastic1026 is OK: DISK OK
[12:08:59] <icinga-wm>	 PROBLEM - Host mw2256 is DOWN: PING CRITICAL - Packet loss = 100%
[12:09:19] <icinga-wm>	 RECOVERY - Host mw2256 is UP: PING OK - Packet loss = 0%, RTA = 36.21 ms
[12:15:09] <icinga-wm>	 PROBLEM - puppet last run on mw1299 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:25:29] <icinga-wm>	 PROBLEM - cassandra-a SSL 10.192.16.165:7001 on restbase2002 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused
[12:26:09] <icinga-wm>	 PROBLEM - cassandra-a CQL 10.192.16.165:9042 on restbase2002 is CRITICAL: connect to address 10.192.16.165 and port 9042: Connection refused
[12:26:19] <icinga-wm>	 PROBLEM - cassandra-b SSL 10.192.48.69:7001 on restbase2012 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused
[12:27:09] <icinga-wm>	 PROBLEM - cassandra-b CQL 10.192.48.69:9042 on restbase2012 is CRITICAL: connect to address 10.192.48.69 and port 9042: Connection refused
[12:27:09] <icinga-wm>	 PROBLEM - Check systemd state on restbase2012 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[12:27:09] <icinga-wm>	 PROBLEM - cassandra-b service on restbase2012 is CRITICAL: CRITICAL - Expecting active but unit cassandra-b is failed
[12:27:19] <icinga-wm>	 PROBLEM - cassandra-a service on restbase2002 is CRITICAL: CRITICAL - Expecting active but unit cassandra-a is failed
[12:28:09] <icinga-wm>	 PROBLEM - Check systemd state on restbase2002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[12:31:09] <icinga-wm>	 PROBLEM - Disk space on elastic1026 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 65371 MB (12% inode=99%)
[12:33:00] <icinga-wm>	 PROBLEM - puppet last run on rdb1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[12:34:06] <paladox>	 there seems to be alot of warnings about elastic* out of storage ^^
[12:42:09] <icinga-wm>	 RECOVERY - puppet last run on mw1299 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[12:43:09] <icinga-wm>	 RECOVERY - Check systemd state on restbase2012 is OK: OK - running: The system is fully operational
[12:43:19] <icinga-wm>	 RECOVERY - cassandra-b service on restbase2012 is OK: OK - cassandra-b is active
[12:44:19] <icinga-wm>	 RECOVERY - cassandra-b SSL 10.192.48.69:7001 on restbase2012 is OK: SSL OK - Certificate restbase2012-b valid until 2017-11-17 00:54:33 +0000 (expires in 270 days)
[12:45:09] <icinga-wm>	 RECOVERY - cassandra-b CQL 10.192.48.69:9042 on restbase2012 is OK: TCP OK - 0.036 second response time on 10.192.48.69 port 9042
[12:45:09] <icinga-wm>	 RECOVERY - Check systemd state on restbase2002 is OK: OK - running: The system is fully operational
[12:45:19] <icinga-wm>	 RECOVERY - cassandra-a service on restbase2002 is OK: OK - cassandra-a is active
[12:47:09] <icinga-wm>	 RECOVERY - cassandra-a CQL 10.192.16.165:9042 on restbase2002 is OK: TCP OK - 0.036 second response time on 10.192.16.165 port 9042
[12:47:29] <icinga-wm>	 RECOVERY - cassandra-a SSL 10.192.16.165:7001 on restbase2002 is OK: SSL OK - Certificate restbase2002-a valid until 2017-09-12 15:35:00 +0000 (expires in 205 days)
[12:57:59] <icinga-wm>	 PROBLEM - puppet last run on restbase1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[13:00:59] <icinga-wm>	 RECOVERY - puppet last run on rdb1007 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[13:13:09] <icinga-wm>	 PROBLEM - Disk space on elastic1026 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 65228 MB (12% inode=99%)
[13:17:09] <icinga-wm>	 PROBLEM - Disk space on elastic1026 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 64054 MB (12% inode=99%)
[13:26:59] <icinga-wm>	 RECOVERY - puppet last run on restbase1007 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[13:30:09] <icinga-wm>	 PROBLEM - Disk space on elastic1026 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 64367 MB (12% inode=99%)
[13:58:09] <icinga-wm>	 RECOVERY - Disk space on elastic1026 is OK: DISK OK
[14:03:09] <icinga-wm>	 PROBLEM - All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 185 bytes in 0.416 second response time
[14:03:59] <icinga-wm>	 PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[14:30:59] <icinga-wm>	 RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[14:31:10] <icinga-wm>	 RECOVERY - All k8s worker nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.325 second response time
[14:49:09] <icinga-wm>	 PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[14:58:52] <SmellyPirate>	 !ops
[15:06:05] <SmellyPirate>	 !os
[15:06:07] <SmellyPirate>	 !ops
[15:06:28] <SmellyPirate>	 g o a t s e x * g o a t s e x * g o a t s e x * g o a t s e x * g o a t s e x * g o a t s e x * g o a t s e x * g o a t s e x * g o a t s e x * 
[15:06:31] <SmellyPirate>	 Heil Hitler und Das Dritte Reich!  Allle heil der allmächtige Deutsch rasse!  |  Hail Hitler and The Third Reich!  All hail the almighty German race!
[15:07:00] <SmellyPirate>	 hi grumble 
[15:07:09] <SmellyPirate>	 should i stop now?
[15:07:17] <SmellyPirate>	 i think i should stop now
[15:07:17] <vere>	 lool
[15:07:20] * SmellyPirate stops now
[15:07:22] <Joseph>	 SmellyPirate: Yes. Stop.
[15:07:28] <Qaei>	 Smelly Pirate,yup, please
[15:07:30] <vere>	 u tell him joe
[15:14:21] <UaMaol>	 topic Meow! Meow! Meow! Meow! Meow! Meow! Meow! Meow! Meow! Meow! Meow! Meow! 
[15:15:00] <UaMaol>	 wut :o
[15:15:08] <SmellyPirate>	 omg u vandalized it
[15:15:09] <icinga-wm>	 PROBLEM - Disk space on elastic1023 is CRITICAL: DISK CRITICAL - free space: /srv 60287 MB (12% inode=99%)
[15:15:20] <UaMaol>	 permaban much
[15:15:21] <vere>	 ^^^ send all ops!
[15:15:40] <SmellyPirate>	 lolkek, just set mode +t
[15:16:22] <UaMaol>	 if you let non op'd users change the topic, it's your own death!
[15:16:32] <UaMaol>	 ay cap'n?
[15:16:46] <vere>	 SmellyPirate: this ur bitch? ^
[15:16:52] <Joseph>	 I think the reason -t is set is so that bots can update the topic without being opped
[15:16:56] <Joseph>	 but I'm not 100% sure
[15:17:26] <SmellyPirate>	 Joseph huh
[15:17:35] <SmellyPirate>	 then you give bots +t chanserv flag
[15:17:38] <UaMaol>	 Can't you just half op or even voice bots?
[15:17:45] <SmellyPirate>	 and program them to update the topic thru chanserv
[15:17:47] <Joseph>	 they need a freenode account for that. 
[15:17:49] <UaMaol>	 or do that
[15:17:52] <Joseph>	 UaMaol: half-op doesn't exist here
[15:17:53] <SmellyPirate>	 UaMaol: freenode has no halfop
[15:17:58] <UaMaol>	 wat :o
[15:17:58] <NotASpy>	 there's a lack of IRC technical know-how from some of the WMF staffers who need to update the topic, nothing to do with ops etc. 
[15:18:01] <Joseph>	 (the HOP template does, but it isn't half-op)
[15:18:01] <SmellyPirate>	 and voice cannot change topic when +t set
[15:18:09] <icinga-wm>	 RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[15:18:22] <SmellyPirate>	 NotASpy: just tell them add /msg chanserv
[15:18:23] <UaMaol>	 freenode is gei
[15:18:28] <SmellyPirate>	 to the beginning of the command
[15:18:37] <SmellyPirate>	 and then just make sure they are +t in chanserv
[15:18:47] <Joseph>	 Again, that needs a freenode account
[15:18:51] <Joseph>	 Look at icinga-wm for example
[15:18:54] <Joseph>	 They don't have an account
[15:19:00] <NotASpy>	 SmellyPirate: doing all this with the web client and without a Freenode account ?
[15:19:04] <Sagan>	 I think we can drop that topic now
[15:19:40] <SmellyPirate>	 NotASpy: no freenode account = make a damned account or dont expect to perform staff roles on any channel ever
[15:19:40] <UaMaol>	 hey carl
[15:19:46] <vere>	 NotASpy: you could /msg chanserv flags #wikimedia-operations *!*@*wikimedia.org +t
[15:19:49] <vere>	 or something near that
[15:20:00] <SmellyPirate>	 vere: that too
[15:20:22] <SmellyPirate>	 but they need to then be connecting from wikimedia network with rDNS set up
[15:20:51] <UaMaol>	 or have a dummy op that relays messaages from bots
[15:21:29] <Joseph>	 Anyway I think this conversation has run its course, this channel is for serious platform operations, you can take up the recommended procedures for WMF staff to implemente here elsewhere, I guess
[15:21:35] <UaMaol>	 maybe using curl and a protected wikipage
[15:22:09] <NotASpy>	 again, it's making something more complex. If people weren't being disruptive and respected the topic being -t for simplicity, none of this would be needed. 
[15:22:10] <icinga-wm>	 PROBLEM - puppet last run on db1076 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[15:22:27] <grumble>	 vere: that exact mask sounds like a bad idea because people could register somethingsomethingwikimedia.org and gain flag +t
[15:22:41] <vere>	 NotASpy: it is not in the slightest complex
[15:23:00] <vere>	 grumble: obviously you'd want the .wikimedia.org
[15:23:22] <NotASpy>	 it's more complex than leaving the channel -t and expecting infantile users to leave the channel alone
[15:23:27] <Joseph>	 ^
[15:23:29] <grumble>	 are there clear IP ranges?
[15:23:43] <vere>	 NotASpy: good luck with that
[15:24:08] <vere>	 geeknerd will be back many a times to ensure you've fixed all the loopholes, it's the way of nature
[15:24:23] <Joseph>	 Or they'll be back to see if problematic users have left.
[15:24:25] <SmellyPirate>	 who the fuck is geeknerd?
[15:25:01] <NotASpy>	 I've been idling in this channel for years, and I've seen the channel changed disruptively precisely once before today. 
[15:26:06] <vere>	 yeah but the other million wiki?edia channels aren't as lucky
[15:26:17] <grumble>	 you could set the IP ranges +t.   Due to technical limitations, IP range masks won't apply to web gateways other than our webchat, though. That one can be fixed by converting IP ranges to hexadecimal ranges (would look like *!abcde???@gateway/web/*) 
[15:28:45] <NotASpy>	 vere: most of the other channels are +t or invite only, this is about the only one I know that isn't +t or invite only. 
[15:37:09] <icinga-wm>	 PROBLEM - Disk space on elastic1023 is CRITICAL: DISK CRITICAL - free space: /srv 61905 MB (12% inode=99%)
[15:39:09] <icinga-wm>	 PROBLEM - Disk space on elastic1023 is CRITICAL: DISK CRITICAL - free space: /srv 60990 MB (12% inode=99%)
[15:39:29] <icinga-wm>	 PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[15:51:09] <icinga-wm>	 RECOVERY - puppet last run on db1076 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[15:54:45] <hashar>	 !log Restarted Zuul to clear out a stall function in Gearman server.
[15:54:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:05:59] <wikibugs>	 06Operations, 06Labs, 10Tool-Labs-tools-Other: Move geohack to production - https://phabricator.wikimedia.org/T102960#3039207 (10scfc) 05Open>03declined Usages of GeoHack should be replaced by #MediaWiki-Extensions-Kartographer or, ad interim, #Mediawiki-Extensions-MapSources.  The latter //is// effectiv...
[16:08:09] <icinga-wm>	 RECOVERY - Disk space on elastic1023 is OK: DISK OK
[16:08:29] <icinga-wm>	 RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[16:48:48] <wikibugs>	 06Operations, 10Domains, 10Traffic, 10Wikimedia-Site-requests: Consider mw.org being added as a redirect to mediawiki.wikimedia.org - https://phabricator.wikimedia.org/T158490#3039228 (10Zppix) @Reedy maybe WMF could talk to ICANN? and @SamanthaNguyen Yes i did.
[16:51:10] <Zppix>	 valhallasw`cloud: fyi the wikibugs bot isnt in ##Zppix-Wikipedia even though its in its config?
[16:52:55] <valhallasw`cloud>	 Zppix: iirc it only joins when it actually has to say something
[17:09:29] <icinga-wm>	 PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[17:27:29] <icinga-wm>	 PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[17:31:00] <wikibugs>	 06Operations, 10Domains, 10Traffic, 10Wikimedia-Site-requests: Consider mw.org being added as a redirect to mediawiki.org - https://phabricator.wikimedia.org/T158490#3039263 (10SamanthaNguyen)
[17:37:29] <icinga-wm>	 RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[17:56:29] <icinga-wm>	 RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[18:15:49] <wikibugs>	 06Operations, 10Domains, 10Traffic, 10Wikimedia-Site-requests: Consider mw.org being added as a redirect to mediawiki.org - https://phabricator.wikimedia.org/T158490#3039269 (10Aklapper) > Since mediawiki is commonly abbrev'd as MW I don't believe that...
[18:46:29] <icinga-wm>	 PROBLEM - puppet last run on cp3040 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[19:08:29] <wikibugs>	 (03Abandoned) 10MarcoAurelio: 'shellmanagers' to 'shellmanager' on wikitech.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/338340 (https://phabricator.wikimedia.org/T158039) (owner: 10MarcoAurelio)
[19:14:29] <icinga-wm>	 RECOVERY - puppet last run on cp3040 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[19:17:45] <wikibugs>	 (03PS1) 10MarcoAurelio: Removing the 'shellmanagers' group from Wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/338632 (https://phabricator.wikimedia.org/T158482)
[19:17:51] <wikibugs>	 (03CR) 10ArielGlenn: [C: 032] fix private table dumping in dumps [dumps] - 10https://gerrit.wikimedia.org/r/336643 (owner: 10ArielGlenn)
[19:27:57] <wikibugs>	 (03PS1) 10Hashar: build: allow usage of a different puppet version [puppet] - 10https://gerrit.wikimedia.org/r/338633
[19:28:20] <wikibugs>	 06Operations, 10Domains, 10Traffic, 10Wikimedia-Site-requests: Consider mw.org being added as a redirect to mediawiki.org - https://phabricator.wikimedia.org/T158490#3039326 (10Zppix) @Aklapper I meant like if abbrev'd its usually as MW and seeing as enwiki has enwp.org, shouldnt mediawiki.org have mw.org?
[19:31:07] <wikibugs>	 (03PS2) 10Hashar: build: allow usage of a different puppet version [puppet] - 10https://gerrit.wikimedia.org/r/338633
[19:39:15] <wikibugs>	 (03PS2) 10ArielGlenn: report failed dump runs correctly [dumps] - 10https://gerrit.wikimedia.org/r/336811 (https://phabricator.wikimedia.org/T157669)
[19:40:20] <wikibugs>	 (03CR) 10ArielGlenn: [C: 032] report failed dump runs correctly [dumps] - 10https://gerrit.wikimedia.org/r/336811 (https://phabricator.wikimedia.org/T157669) (owner: 10ArielGlenn)
[19:40:29] <icinga-wm>	 PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[20:08:15] <logmsgbot>	 !log ariel@tin Started deploy [dumps/dumps@364470e]: fix private table dumping, report failed runs correctly
[20:08:18] <logmsgbot>	 !log ariel@tin Finished deploy [dumps/dumps@364470e]: fix private table dumping, report failed runs correctly (duration: 00m 03s)
[20:08:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:08:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:09:29] <icinga-wm>	 RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[20:13:19] <icinga-wm>	 PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[20:41:19] <icinga-wm>	 RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[21:03:09] <icinga-wm>	 PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[21:10:17] <wikibugs>	 06Operations, 10Domains, 10Traffic, 10Wikimedia-Site-requests: Consider mw.org being added as a redirect to mediawiki.org - https://phabricator.wikimedia.org/T158490#3039415 (10Reedy) We could speak to ICANN etc.  Though, we actively removed redirects after we forced HTTPS, as we'd need separate SSL certs...
[21:13:29] <icinga-wm>	 PROBLEM - puppet last run on mw1271 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[21:16:42] <wikibugs>	 06Operations, 10Domains, 10Traffic, 10Wikimedia-Site-requests: Consider mw.org being added as a redirect to mediawiki.org - https://phabricator.wikimedia.org/T158490#3039418 (10Zppix) @Reedy can't we use selfsigned certs or use letsencrypt like you mentioned
[21:18:31] <wikibugs>	 06Operations, 10Domains, 10Traffic, 10Wikimedia-Site-requests: Consider mw.org being added as a redirect to mediawiki.org - https://phabricator.wikimedia.org/T158490#3039419 (10Reedy) >>! In T158490#3039418, @Zppix wrote: > @Reedy can't we use selfsigned certs or use letsencrypt like you mentioned  Self si...
[21:20:18] <wikibugs>	 06Operations, 10Domains, 10Traffic, 10Wikimedia-Site-requests: Consider mw.org being added as a redirect to mediawiki.org - https://phabricator.wikimedia.org/T158490#3039421 (10Zppix) Okay, let's wait to see what the input from #operations is then well go from there.
[21:21:40] <Zppix>	 jouncebot: next
[21:21:40] <jouncebot>	 No deployments scheduled for the forseeable future!
[21:23:19] <icinga-wm>	 PROBLEM - puppet last run on mx1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[21:31:09] <icinga-wm>	 RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[21:38:59] <icinga-wm>	 PROBLEM - Start a job and verify on Precise on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/grid/start/precise - 185 bytes in 0.858 second response time
[21:42:29] <icinga-wm>	 RECOVERY - puppet last run on mw1271 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[21:43:59] <icinga-wm>	 RECOVERY - Start a job and verify on Precise on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.779 second response time
[21:44:27] <paladox>	 jouncebot: refresh
[21:44:29] <jouncebot>	 I refreshed my knowledge about deployments.
[21:44:31] <paladox>	 jouncebot: next
[21:44:31] <jouncebot>	 No deployments scheduled for the forseeable future!
[21:45:23] <wikibugs>	 06Operations, 10Domains, 10Traffic, 10Wikimedia-Site-requests: Consider mw.org being added as a redirect to mediawiki.org - https://phabricator.wikimedia.org/T158490#3038649 (10Platonides) MW is the country-code of Malawi (ISO 3166-1), so I find unlikely we would be allowed to use it.
[21:46:16] <Zppix>	 bd808: do you mind if i update the realname with the phab/gerrit repo link?
[21:46:23] <Zppix>	 For jouncebot
[21:52:19] <icinga-wm>	 RECOVERY - puppet last run on mx1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[22:00:29] <icinga-wm>	 PROBLEM - puppet last run on francium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[22:16:29] <icinga-wm>	 PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[22:25:21] <wikibugs>	 (03Draft2) 10Zppix: Update the realname from github repo url --> phabricator diffusion [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/338700
[22:29:29] <icinga-wm>	 RECOVERY - puppet last run on francium is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[22:44:29] <icinga-wm>	 RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[22:49:08] <wikibugs>	 (03CR) 10EddieGP: [C: 031] Update the realname from github repo url --> phabricator diffusion [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/338700 (owner: 10Zppix)
[23:34:39] <icinga-wm>	 PROBLEM - puppet last run on ganeti1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[23:39:19] <icinga-wm>	 PROBLEM - puppet last run on db1075 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[23:39:39] <icinga-wm>	 PROBLEM - tools homepage -admin tool- on tools.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 20 seconds
[23:44:19] <icinga-wm>	 RECOVERY - tools homepage -admin tool- on tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 3677 bytes in 0.036 second response time
[23:51:26] <wikibugs>	 06Operations, 10Domains, 10Traffic, 10Wikimedia-Site-requests: Consider mw.org being added as a redirect to mediawiki.org - https://phabricator.wikimedia.org/T158490#3038649 (10Matthewrbowker) >>! In T158490#3039326, @Zppix wrote: > @Aklapper I meant like if abbrev'd its usually as MW and seeing as enwiki...