[00:19:43] PROBLEM - Puppet freshness on tungsten is CRITICAL: Last successful Puppet run was Sun 08 Dec 2013 09:19:01 PM UTC [02:00:36] !log resync labsdb dewiki.pagelinks from upstream [02:00:54] Logged the message, Master [02:11:34] !log LocalisationUpdate completed (1.23wmf5) at Mon Dec 9 02:11:34 UTC 2013 [02:11:50] Logged the message, Master [02:13:36] (03PS1) 10Springle: switch research slave cnames to (mostly) 12th floor pmtpa masters [operations/dns] - 10https://gerrit.wikimedia.org/r/100337 [02:15:23] (03CR) 10Springle: [C: 032] switch research slave cnames to (mostly) 12th floor pmtpa masters [operations/dns] - 10https://gerrit.wikimedia.org/r/100337 (owner: 10Springle) [02:21:01] !log LocalisationUpdate completed (1.23wmf6) at Mon Dec 9 02:21:01 UTC 2013 [02:21:16] Logged the message, Master [02:36:36] PROBLEM - Puppet freshness on sq80 is CRITICAL: Last successful Puppet run was Sat 07 Dec 2013 11:31:16 PM UTC [02:39:46] (03PS1) 10Springle: depool es1007 for upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100339 [02:42:50] (03CR) 10Springle: [C: 032] depool es1007 for upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100339 (owner: 10Springle) [02:43:52] !log springle synchronized wmf-config/db-eqiad.php 'depool es1007 for upgrade' [02:44:07] Logged the message, Master [02:49:21] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Dec 9 02:49:20 UTC 2013 [02:49:37] Logged the message, Master [02:57:11] (03PS1) 10Springle: switch es1007 to mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/100340 [02:58:32] (03CR) 10Springle: [C: 032] switch es1007 to mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/100340 (owner: 10Springle) [03:10:28] (03PS1) 10Springle: repool es1007 after upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100341 [03:11:01] (03CR) 10Springle: [C: 032] repool es1007 after upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100341 (owner: 10Springle) [03:12:06] !log springle synchronized wmf-config/db-eqiad.php 'repool es1007 after upgrade, max_connections lowered during warm up' [03:12:19] Logged the message, Master [03:19:08] (03PS1) 10Springle: depool es1010 for upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100342 [03:20:30] PROBLEM - Puppet freshness on tungsten is CRITICAL: Last successful Puppet run was Sun 08 Dec 2013 09:19:01 PM UTC [03:20:46] (03CR) 10Springle: [C: 032] depool es1010 for upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100342 (owner: 10Springle) [03:21:44] !log springle synchronized wmf-config/db-eqiad.php 'depool es1010 for upgrade' [03:21:57] Logged the message, Master [03:29:31] (03PS1) 10Springle: switch es1010 to mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/100343 [03:34:57] (03CR) 10coren: [C: 04-1] "There are unguarded stack overflows in export.c that need fixing." (033 comments) [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/100329 (owner: 10Ori.livneh) [03:38:45] (03CR) 10Springle: [C: 032] switch es1010 to mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/100343 (owner: 10Springle) [04:23:02] (03PS5) 10MZMcBride: Create "Draft" namespace on the English Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 [04:23:46] (03CR) 10MZMcBride: "PS5: Added the isAnon() check, per Matt F." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [04:32:52] (03PS1) 10Andrew Bogott: Add proper nova auth to labsstatus.rb [operations/puppet] - 10https://gerrit.wikimedia.org/r/100345 [04:35:56] (03PS2) 10Andrew Bogott: Add proper nova auth to labsstatus.rb [operations/puppet] - 10https://gerrit.wikimedia.org/r/100345 [04:37:23] (03PS1) 10Springle: repool es1010 after upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100346 [04:37:24] (03CR) 10Andrew Bogott: [C: 032] Add proper nova auth to labsstatus.rb [operations/puppet] - 10https://gerrit.wikimedia.org/r/100345 (owner: 10Andrew Bogott) [04:37:57] (03CR) 10Springle: [C: 032] repool es1010 after upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100346 (owner: 10Springle) [04:38:49] !log springle synchronized wmf-config/db-eqiad.php 'repool es1010 after upgrade, max_connections lowered during warm up' [04:39:04] Logged the message, Master [04:44:45] (03CR) 10Ori.livneh: "Thanks for reviewing. These issues have waited eight years for someone to notice them :D (03PS1) 10Andrew Bogott: When reporting labs hostname, use $::instancename [operations/puppet] - 10https://gerrit.wikimedia.org/r/100348 [04:52:41] (03CR) 10Andrew Bogott: [C: 032] When reporting labs hostname, use $::instancename [operations/puppet] - 10https://gerrit.wikimedia.org/r/100348 (owner: 10Andrew Bogott) [05:13:10] (03PS2) 10Ori.livneh: Use glib's GHashTable instead of persistent BDB [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/100329 [05:13:40] (03CR) 10Ori.livneh: "Dox: " [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/100329 (owner: 10Ori.livneh) [05:13:51] Coren: ^ [05:17:45] it qualifies me for the "fixed a domas bug" badge [05:17:46] ori-l: That definitely alleviates my concern. :-) I don't really have the time to do a whole review, though; I'm about to go to bed. Want me to give it a good read tomorrow? [05:18:09] Coren: it's not pressing at all; get some sleep [05:18:36] ori-l: Well, it wasn't a "bug" so much as a risk; with the current source there could have been no overflow (the source was limited to 127 characters) but it was a timebomb. :-) [05:19:14] yeah, code should be obviously safe, not well-actually safe [05:19:41] PROBLEM - SSH on cp4011 is CRITICAL: Server answer: [05:20:41] RECOVERY - SSH on cp4011 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [05:20:42] ok, thai food time! thanks again for looking and good night [05:36:42] PROBLEM - Puppet freshness on sq80 is CRITICAL: Last successful Puppet run was Sat 07 Dec 2013 11:31:16 PM UTC [05:41:12] PROBLEM - Varnish HTCP daemon on cp4011 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:42:42] PROBLEM - SSH on cp4011 is CRITICAL: Server answer: [05:43:12] RECOVERY - Varnish HTCP daemon on cp4011 is OK: PROCS OK: 1 process with UID = 111 (vhtcpd), args vhtcpd [05:43:42] RECOVERY - SSH on cp4011 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [06:08:05] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [06:10:23] PROBLEM - LDAPS on sanger is CRITICAL: Connection refused [06:10:53] PROBLEM - LDAP on sanger is CRITICAL: Connection refused [06:12:03] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (204471) [06:13:03] PROBLEM - Varnish traffic logger on cp4019 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:14:03] RECOVERY - Varnish traffic logger on cp4019 is OK: PROCS OK: 2 processes with command name varnishncsa [06:14:03] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [06:19:03] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (205043) [06:20:33] PROBLEM - Puppet freshness on tungsten is CRITICAL: Last successful Puppet run was Sun 08 Dec 2013 09:19:01 PM UTC [06:21:03] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [06:22:43] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [06:27:53] PROBLEM - udp2log log age for lucene on oxygen is CRITICAL: CRITICAL: log files /a/log/lucene/lucene.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [06:29:54] RECOVERY - udp2log log age for lucene on oxygen is OK: OK: all log files active [06:33:13] PROBLEM - Varnish HTCP daemon on cp4019 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:34:13] RECOVERY - Varnish HTCP daemon on cp4019 is OK: PROCS OK: 1 process with UID = 111 (vhtcpd), args vhtcpd [06:42:42] PROBLEM - SSH on cp4011 is CRITICAL: Server answer: [06:43:12] PROBLEM - Varnish HTCP daemon on cp4011 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:43:42] RECOVERY - SSH on cp4011 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [06:44:12] RECOVERY - Varnish HTCP daemon on cp4011 is OK: PROCS OK: 1 process with UID = 111 (vhtcpd), args vhtcpd [06:50:53] PROBLEM - SSH on cp4019 is CRITICAL: Server answer: [06:51:53] RECOVERY - SSH on cp4019 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [06:53:12] PROBLEM - RAID on cp4019 is CRITICAL: NRPE: Unable to read output [06:54:12] RECOVERY - RAID on cp4019 is OK: OK: no RAID installed [07:04:22] PROBLEM - RAID on cp4019 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [07:04:53] PROBLEM - SSH on cp4019 is CRITICAL: Server answer: [07:05:12] RECOVERY - RAID on cp4019 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [07:05:53] RECOVERY - SSH on cp4019 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [07:07:22] PROBLEM - Puppet freshness on cp4011 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 04:06:44 AM UTC [07:18:22] PROBLEM - Puppet freshness on cp4019 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 04:17:36 AM UTC [07:35:12] PROBLEM - DPKG on cp4019 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [07:36:12] PROBLEM - Puppet freshness on cp4012 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 04:34:56 AM UTC [07:36:12] RECOVERY - DPKG on cp4019 is OK: All packages OK [07:37:02] RECOVERY - Puppet freshness on cp4011 is OK: puppet ran at Mon Dec 9 07:36:52 UTC 2013 [07:48:22] RECOVERY - Puppet freshness on tungsten is OK: puppet ran at Mon Dec 9 07:48:15 UTC 2013 [07:51:07] (03CR) 10Ori.livneh: "As I understand it, a severe Puppet failure could prevent the report from running, which could leave 'puppetstatus' stuck in an OK state. " [operations/puppet] - 10https://gerrit.wikimedia.org/r/100221 (owner: 10Andrew Bogott) [07:53:12] PROBLEM - RAID on cp4019 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [07:54:12] RECOVERY - RAID on cp4019 is OK: OK: no RAID installed [08:05:55] PROBLEM - RAID on db1047 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) [08:05:56] PROBLEM - SSH on cp4019 is CRITICAL: Server answer: [08:06:56] RECOVERY - SSH on cp4019 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [08:08:05] PROBLEM - SSH on cp4012 is CRITICAL: Server answer: [08:10:05] RECOVERY - SSH on cp4012 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [08:10:35] PROBLEM - Varnish HTCP daemon on cp4019 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [08:11:35] RECOVERY - Varnish HTCP daemon on cp4019 is OK: PROCS OK: 1 process with UID = 111 (vhtcpd), args vhtcpd [08:14:05] PROBLEM - SSH on cp4012 is CRITICAL: Server answer: [08:15:15] PROBLEM - DPKG on cp4012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:17:05] RECOVERY - DPKG on cp4012 is OK: All packages OK [08:17:05] RECOVERY - SSH on cp4012 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [08:31:45] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [08:34:56] RECOVERY - Puppet freshness on cp4012 is OK: puppet ran at Mon Dec 9 08:34:53 UTC 2013 [08:37:35] PROBLEM - Puppet freshness on sq80 is CRITICAL: Last successful Puppet run was Sat 07 Dec 2013 11:31:16 PM UTC [08:48:56] PROBLEM - Disk space on cp4019 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [08:48:56] PROBLEM - puppet disabled on cp4019 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [08:48:56] PROBLEM - SSH on cp4019 is CRITICAL: Server answer: [08:49:56] RECOVERY - Disk space on cp4019 is OK: DISK OK [08:49:56] RECOVERY - puppet disabled on cp4019 is OK: OK [08:49:56] RECOVERY - SSH on cp4019 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [08:55:45] RECOVERY - Host ms-be1006 is UP: PING OK - Packet loss = 0%, RTA = 0.66 ms [09:01:33] !log powercycled ms-be1006, unresponsive, lots of "BUG: soft lockup - CPU#0 stuck for 22s!" etc from time it stopped responding (16.00 utc Sunday) [09:01:48] Logged the message, Master [09:08:26] PROBLEM - Host sq80 is DOWN: PING CRITICAL - Packet loss = 100% [09:08:46] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [09:08:56] RECOVERY - Host sq80 is UP: PING OK - Packet loss = 0%, RTA = 35.55 ms [09:09:16] RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Mon Dec 9 09:09:10 UTC 2013 [09:10:36] PROBLEM - Varnish HTCP daemon on cp4019 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [09:12:18] !log powercycled sq80, i/o errors for every command, fs complaints on various filesystems in dmesg, smells like controller but it booted ok so... [09:12:34] Logged the message, Master [09:12:36] RECOVERY - Varnish HTCP daemon on cp4019 is OK: PROCS OK: 1 process with UID = 111 (vhtcpd), args vhtcpd [09:24:56] RECOVERY - LDAPS on sanger is OK: TCP OK - 0.039 second response time on port 636 [09:24:56] RECOVERY - LDAP on sanger is OK: TCP OK - 0.038 second response time on port 389 [09:25:44] !log restarted opendj on sanger [09:25:58] Logged the message, Master [09:35:59] !log restart backend varnish on cp4019, out of memory (fork failed and other nice things) [09:36:05] PROBLEM - Varnish HTTP mobile-frontend on cp4019 is CRITICAL: Connection refused [09:36:07] (03PS1) 10Hashar: beta: rename {/mnt,/data/project}/upload7 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100355 [09:36:15] Logged the message, Master [09:36:23] (03CR) 10Hashar: [C: 032] beta: rename {/mnt,/data/project}/upload7 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100355 (owner: 10Hashar) [09:37:28] (03Merged) 10jenkins-bot: beta: rename {/mnt,/data/project}/upload7 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100355 (owner: 10Hashar) [09:40:05] RECOVERY - Varnish HTTP mobile-frontend on cp4019 is OK: HTTP OK: HTTP/1.1 200 OK - 262 bytes in 0.149 second response time [09:41:35] PROBLEM - Varnish traffic logger on cp4019 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [09:43:35] RECOVERY - Varnish traffic logger on cp4019 is OK: PROCS OK: 2 processes with command name varnishncsa [09:47:01] (03CR) 10Akosiaris: [C: 032] Add a content parameter to ferm::conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/99704 (owner: 10Akosiaris) [09:48:05] RECOVERY - Puppet freshness on cp4019 is OK: puppet ran at Mon Dec 9 09:47:58 UTC 2013 [10:04:57] should have done that earlier [10:05:00] (03PS2) 10Hashar: icinga: raise timeout of check_job_queue nrpe command [operations/puppet] - 10https://gerrit.wikimedia.org/r/99411 [10:05:18] (03CR) 10Hashar: "Raising the check_job_queue check is done with https://gerrit.wikimedia.org/r/#/c/99411/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/99410 (owner: 10Hashar) [10:07:02] apergos: Hi! How about fixing https://bugzilla.wikimedia.org/show_bug.cgi?id=44464 then? :) [10:17:35] (03CR) 10Akosiaris: [C: 032] change bugzilla role classes [operations/puppet] - 10https://gerrit.wikimedia.org/r/99788 (owner: 10Dzahn) [10:22:17] Nemo_bis: regarding http://dumps.wikimedia.org/other/pagecounts-raw/ , do you happen to know where the HTML for it is ? [10:23:02] we should probably make it a git repo :D [10:23:34] (03PS1) 10Dan-nl: gwtoolset-whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100356 [10:26:13] (03PS2) 10Dan-nl: gwtoolset-whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100356 [10:30:23] (03CR) 10Hashar: gwtoolset-whitelist (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100356 (owner: 10Dan-nl) [10:30:37] hashar: need to double check with apergos but i think it may be already [10:31:34] (03PS3) 10Hashar: beta: gwtoolset-whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100356 (owner: 10Dan-nl) [10:32:02] (03CR) 10Hashar: [C: 032] beta: gwtoolset-whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100356 (owner: 10Dan-nl) [10:32:13] (03Merged) 10jenkins-bot: beta: gwtoolset-whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100356 (owner: 10Dan-nl) [10:33:02] hashar: AFAIK it's on that machine only; Ariel knows :) I can't judge how hard/worth puppetizing it is [10:33:22] (03CR) 10Odder: [C: 04-1] "The submitted favicon is different from the current one, which is avaialble at -- please" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100326 (owner: 10Gerrit Patch Uploader) [10:33:52] p858snake|l: Nemo_bis if the HTML files/templates are in some git repository, it would be very nice to have them slightly enhanced :] [10:34:04] the monobook look'n feel is getting dated [10:34:21] nah, most users still use monobook [10:34:55] what's monobook-like in that page though? O_o [10:37:34] hashar: those are generated, scripts and html live on fenari [10:37:58] they need to be moved off (and stuffed into puppet or some other repo) soon [10:38:08] uh maybe I should have pinged Nemo_bis for that [10:38:09] anyways [10:38:32] and yeah it's on my plate to make sure any download/dumps related stuff is in puppet and off fenari by end of dec [10:38:49] nice! [10:38:59] oh, ok, so I guess we can send patches after that :) [10:39:08] the 'main stuff' is already but there's a bunch of crap all smaller stuff that meh [10:39:08] Nemo_bis: sorry was referring to some parent page that has the mono book look [10:39:24] sure, or if it's urgent send it now [10:39:36] hashar: ah, yeah, a nice look [10:41:56] I (or was it Timo) integration twitter bootstrap for https://integration.wikimedia.org [10:42:00] yet another look'n feel :( [10:43:14] (03PS1) 10Matanya: mysql_multi_instance : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/100357 [10:43:45] what's twittery in that, the top bar? that's an epidemic pattern [10:43:52] yup :( [10:44:13] we could possibly get our designers to work on a twitter bootstrap theme that will give us a different feeling [10:45:01] like http://unicorn.wmflabs.org/navmod/ ? [11:20:43] ACKNOWLEDGEMENT - RAID on db1047 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) Sean Pringle RT 6463 [11:31:17] (03PS2) 10Hashar: Update favicon wikispecies.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99590 (owner: 10Tholam) [11:33:16] (03CR) 10Hashar: [C: 031] "will merge it" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99590 (owner: 10Tholam) [11:33:40] (03PS2) 10Hashar: Update favicon wikiveristy.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99595 (owner: 10Tholam) [11:34:23] (03CR) 10Hashar: [C: 031] "will merge it" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99595 (owner: 10Tholam) [11:35:42] (03PS2) 10Hashar: Update favicon wmf.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99601 (owner: 10Tholam) [11:35:55] (03CR) 10Hashar: [C: 031] "will merge it" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99601 (owner: 10Tholam) [11:36:24] (03PS2) 10Hashar: Update favicon wikimania.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99602 (owner: 10Tholam) [11:38:21] (03PS2) 10Hashar: Update favicon chapcom.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99605 (owner: 10Tholam) [11:38:50] (03CR) 10Hashar: [C: 031] "I find that whitebox to be horrible, but that is not worth than before :-] Will merge it." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99605 (owner: 10Tholam) [11:39:59] (03PS9) 10Hashar: Update favicon mediawiki.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99756 (owner: 10Vldandrew) [11:41:49] (03CR) 10Hashar: [C: 031] "icon looks fine to me, will merge it." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99756 (owner: 10Vldandrew) [11:43:50] (03PS2) 10Hashar: Update favicon wikinews.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100123 (owner: 10M4tx) [11:44:10] (03CR) 10Hashar: [C: 031] "Looks fine, will merge it." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100123 (owner: 10M4tx) [11:44:44] (03PS2) 10Hashar: Update favicon incubator.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100133 (owner: 10Vldandrew) [11:45:40] (03CR) 10Hashar: [C: 031] "looks fine, will merge it" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100133 (owner: 10Vldandrew) [11:45:58] (03PS2) 10Hashar: Update favicon wikisource.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100139 (owner: 10Vldandrew) [11:46:57] (03CR) 10Hashar: [C: 031] "looks fine, will merge it" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100139 (owner: 10Vldandrew) [11:47:22] (03PS3) 10Hashar: Update favicon wikivoyage.ico. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100142 (owner: 10Vldandrew) [11:48:13] (03CR) 10Hashar: [C: 031] "will merge it." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100142 (owner: 10Vldandrew) [11:48:25] (03PS2) 10Akosiaris: Template ferm's defs.production [operations/puppet] - 10https://gerrit.wikimedia.org/r/99705 [11:48:31] (03PS5) 10Hashar: Update favicon wikidata.ico. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100144 (owner: 10Vldandrew) [11:49:37] (03CR) 10Hashar: [C: 031] "looks fine, will merge it." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100144 (owner: 10Vldandrew) [11:51:07] (03PS3) 10Hashar: Update favicon piece.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100161 (owner: 10M4tx) [11:51:39] (03CR) 10Hashar: [C: 031] "looks fine to me :-) will merge it" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100161 (owner: 10M4tx) [11:52:07] (03PS2) 10Hashar: Bug: 58183 Updated with new favicon, wikiquote.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100212 (owner: 10Gerrit Patch Uploader) [11:52:59] (03CR) 10Hashar: [C: 031] "looks fine. Will merge it." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100212 (owner: 10Gerrit Patch Uploader) [11:56:59] (03PS1) 10Hashar: update several favicons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100360 [11:57:36] (03CR) 10Hashar: [C: 032] "deploying" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100360 (owner: 10Hashar) [12:00:56] apergos: you shouldn't had closed RT #6447 [12:01:11] this means that we'll have paging alerts for ulsfo on Dec 12th [12:01:17] which we can ignore [12:01:31] oh [12:01:33] you didn't [12:01:35] nevermind, sorry :) [12:01:48] no worries [12:02:04] still, let's have it mind [12:02:07] yep [12:05:57] hashar: thanks for your work on the favicons [12:06:10] twkozlowski: i haven't done anything :-] I am merely approving hehe [12:06:44] still, with the 3 week delay in approving anything in operations/mediawiki-config, that's helpful [12:13:04] (03PS2) 10Hashar: update several favicons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100360 [12:13:27] gerrit piss me off [12:23:43] (03CR) 10QChris: "ping" [operations/puppet] - 10https://gerrit.wikimedia.org/r/98499 (owner: 10QChris) [12:25:13] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [12:27:23] (03PS3) 10Akosiaris: Template ferm's defs.production [operations/puppet] - 10https://gerrit.wikimedia.org/r/99705 [12:30:22] (03CR) 10Akosiaris: "b) Has been addressed and hopefully with other types of hosts in mind. c) Has been addressed by specifically populating the production rul" [operations/puppet] - 10https://gerrit.wikimedia.org/r/99705 (owner: 10Akosiaris) [12:37:21] (03CR) 10Akosiaris: [C: 04-1] mysql_multi_instance : lint cleanup (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100357 (owner: 10Matanya) [12:41:57] (03PS2) 10Matanya: mysql_multi_instance : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/100357 [12:44:12] (03PS3) 10Hashar: Update favicon incubator.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100133 (owner: 10Vldandrew) [12:44:13] (03PS4) 10Hashar: Update favicon piece.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100161 (owner: 10M4tx) [12:44:14] (03PS10) 10Hashar: Update favicon mediawiki.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99756 (owner: 10Vldandrew) [12:44:15] (03PS3) 10Hashar: Update favicon wikispecies.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99590 (owner: 10Tholam) [12:44:16] (03PS3) 10Hashar: Update favicon wikiveristy.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99595 (owner: 10Tholam) [12:44:17] (03PS4) 10Hashar: Update favicon wikivoyage.ico. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100142 (owner: 10Vldandrew) [12:44:18] (03PS3) 10Hashar: Update favicon wikisource.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100139 (owner: 10Vldandrew) [12:44:19] (03PS3) 10Hashar: Update favicon wikimania.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99602 (owner: 10Tholam) [12:44:20] (03PS3) 10Hashar: Update favicon wmf.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99601 (owner: 10Tholam) [12:44:21] (03PS3) 10Hashar: Bug: 58183 Updated with new favicon, wikiquote.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100212 (owner: 10Gerrit Patch Uploader) [12:44:22] (03PS6) 10Hashar: Update favicon wikidata.ico. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100144 (owner: 10Vldandrew) [12:44:23] (03PS3) 10Hashar: Update favicon chapcom.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99605 (owner: 10Tholam) [12:44:24] (03PS3) 10Hashar: Update favicon wikinews.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100123 (owner: 10M4tx) [12:44:58] (03PS4) 10Hashar: Update favicon wikiquote.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100212 (owner: 10Gerrit Patch Uploader) [12:45:17] (03CR) 10Hashar: [C: 032] Update favicon wikispecies.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99590 (owner: 10Tholam) [12:45:25] (03CR) 10Hashar: [C: 032] Update favicon wikiveristy.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99595 (owner: 10Tholam) [12:45:30] (03CR) 10Hashar: [C: 032] Update favicon wmf.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99601 (owner: 10Tholam) [12:45:35] (03CR) 10Hashar: [C: 032] Update favicon wikimania.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99602 (owner: 10Tholam) [12:45:40] (03CR) 10Hashar: [C: 032] Update favicon chapcom.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99605 (owner: 10Tholam) [12:45:46] (03CR) 10Hashar: [C: 032] Update favicon mediawiki.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99756 (owner: 10Vldandrew) [12:45:52] (03CR) 10Hashar: [C: 032] Update favicon wikinews.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100123 (owner: 10M4tx) [12:45:58] (03CR) 10Hashar: [C: 032] Update favicon incubator.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100133 (owner: 10Vldandrew) [12:46:05] (03CR) 10Hashar: [C: 032] Update favicon wikisource.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100139 (owner: 10Vldandrew) [12:46:18] (03CR) 10Hashar: [C: 032] Update favicon wikivoyage.ico. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100142 (owner: 10Vldandrew) [12:46:25] (03CR) 10Hashar: [C: 032] Update favicon wikidata.ico. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100144 (owner: 10Vldandrew) [12:46:29] (03CR) 10Hashar: [C: 032] Update favicon piece.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100161 (owner: 10M4tx) [12:46:36] (03CR) 10Hashar: [C: 032] Update favicon wikiquote.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100212 (owner: 10Gerrit Patch Uploader) [12:46:37] Hashar :-] [12:46:44] yeah that is really stupid [12:46:50] I sent a nice octopus merge [12:46:58] why didn't you merge all those patches one by one? [12:47:00] but gerrit would not let me merge it in since the dependencies are not submitted :-( [12:47:23] (03Merged) 10jenkins-bot: Update favicon wikispecies.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99590 (owner: 10Tholam) [12:47:46] (03Merged) 10jenkins-bot: Update favicon wikiveristy.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99595 (owner: 10Tholam) [12:47:55] (03Merged) 10jenkins-bot: Update favicon wmf.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99601 (owner: 10Tholam) [12:48:19] (03Merged) 10jenkins-bot: Update favicon wikimania.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99602 (owner: 10Tholam) [12:48:29] (03Merged) 10jenkins-bot: Update favicon chapcom.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99605 (owner: 10Tholam) [12:48:45] (03Merged) 10jenkins-bot: Update favicon mediawiki.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99756 (owner: 10Vldandrew) [12:48:53] (03Merged) 10jenkins-bot: Update favicon wikinews.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100123 (owner: 10M4tx) [12:49:02] hey, the chapcom one wasn't ready yet. [12:49:14] (03Merged) 10jenkins-bot: Update favicon incubator.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100133 (owner: 10Vldandrew) [12:49:23] (03Merged) 10jenkins-bot: Update favicon wikisource.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100139 (owner: 10Vldandrew) [12:49:26] (03PS2) 10QChris: Backup geowiki's data-private bare repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/98499 [12:49:33] (03Abandoned) 10Hashar: update several favicons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100360 (owner: 10Hashar) [12:49:44] (03Merged) 10jenkins-bot: Update favicon wikivoyage.ico. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100142 (owner: 10Vldandrew) [12:49:53] (03Merged) 10jenkins-bot: Update favicon wikidata.ico. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100144 (owner: 10Vldandrew) [12:50:09] (03Merged) 10jenkins-bot: Update favicon piece.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100161 (owner: 10M4tx) [12:50:16] (03Merged) 10jenkins-bot: Update favicon wikiquote.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100212 (owner: 10Gerrit Patch Uploader) [12:50:59] need to commute to my coworking place [12:51:12] will deploy those favicon updates whenever I arrive (aka in half an hour [12:51:17] brb [12:53:32] what's with all the favicons? [12:54:26] paravoid: Google Code-In students went on a spree [12:54:32] re-created almost all of our favicons [12:54:38] \o/ [12:58:02] (03CR) 10Odder: "I didn't want to have this merged because I was still waiting for feedback from AffCom per bug 58076 comment 2, but it seems they don't ca" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99605 (owner: 10Tholam) [13:04:33] paravoid: do we use svn? or is it still a leftover? [13:15:29] (03CR) 10ArielGlenn: [C: 031] remove outdated tesla subnet from dhcpd [operations/puppet] - 10https://gerrit.wikimedia.org/r/96489 (owner: 10Dzahn) [13:18:51] !log updating favicons by updating operations/mediawiki-config.git ( b0d4645..0812087 ) [13:19:08] Logged the message, Master [13:19:42] !log hashar synchronized docroot/bits/favicon 'updating favicons ( b0d4645..0812087 )' [13:23:41] Logged the message, Master [13:34:30] (03Abandoned) 10Hashar: Mostly remove need for $wmgPrivateWikiUploads [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/63702 (owner: 10Reedy) [13:34:58] hashar: \o/ [13:35:00] (03Abandoned) 10Hashar: add wikidata redirector script entry point (DO NOT MERGE) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/67123 (owner: 10Aude) [13:35:13] twkozlowski: yeah I deployed some favicons a few minutes ago [13:35:21] moaar need to be done thgouh [13:35:36] moaar favicons? [13:35:48] yup [13:36:04] there are some changes/bug still pending [13:36:14] such as the wikipedia favicon iirc [13:36:22] but he we got like 13 of them updated today. [13:36:43] hashar: Yes, I know. Those are awaiting proper patches or more feedback. [13:37:21] (03Abandoned) 10Hashar: (bug 29902) Tidied up CommonSettings and InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/65860 (owner: 10Hazard-SJ) [13:37:38] hashar: and there a few more that haven't been converted into GCI tasks yet. [13:40:43] (03Abandoned) 10Hashar: Fix various path inflexibilities and inconsistencies (attempt 2) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/72562 (owner: 10Krinkle) [13:41:48] (03CR) 10Hashar: "YuviPanda, can you confirm this is still needed? And if so rebase it. Else I guess we can abandon it." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76946 (owner: 10Yuvipanda) [13:43:10] * Nemo_bis waits for own config patches [13:46:46] (03CR) 10Odder: "@Isarra: Is this good to go?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99768 (owner: 10M4tx) [13:52:04] (03CR) 10Hashar: [C: 04-1] "Why do you want search engines to browse bits ? Seems rather useless to me. Can you amend the commit message to explain why would that b" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95548 (owner: 10Dr0ptp4kt) [13:52:22] nothing worth than https://gerrit.wikimedia.org/r/#/c/94598/ [13:52:38] changing 300+ lines in various files grbmblbl [13:53:59] * twkozlowski doesn't envy hashar having to go through that particular patch [13:56:00] (03CR) 10Hashar: [C: 04-1] "no clue what it is for, something to note is that 'ota' does not exist in mw/core languages/Names.php" (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96771 (owner: 10Dereckson) [13:59:26] (03PS4) 10Hashar: skwikisource: Restrict uploads to sysops [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86643 (owner: 10Danny B.) [14:00:02] (03CR) 10Hashar: [C: 032] skwikisource: Restrict uploads to sysops [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86643 (owner: 10Danny B.) [14:00:13] (03Merged) 10jenkins-bot: skwikisource: Restrict uploads to sysops [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86643 (owner: 10Danny B.) [14:01:08] !log hashar synchronized wmf-config/InitialiseSettings.php '{{gerrit|86643}} skwikisource: Restrict uploads to sysops' [14:01:11] (03CR) 10Hashar: "deployed" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86643 (owner: 10Danny B.) [14:01:22] Logged the message, Master [14:02:48] (03PS2) 10Hashar: Change favicon for angwiktionary to ['w] icon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97458 (owner: 10TTO) [14:02:59] (03CR) 10Hashar: [C: 032] Change favicon for angwiktionary to ['w] icon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97458 (owner: 10TTO) [14:03:11] (03Merged) 10jenkins-bot: Change favicon for angwiktionary to ['w] icon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97458 (owner: 10TTO) [14:03:55] !log hashar synchronized wmf-config/InitialiseSettings.php '{{gerrit|97458}} Change favicon for angwiktionary to w icon' [14:04:05] I should do that more often [14:04:10] Logged the message, Master [14:04:57] (03PS2) 10Hashar: Bug: 58187 Fixed internals.ico favicon. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100326 (owner: 10Gerrit Patch Uploader) [14:05:08] (03PS3) 10Hashar: Update internal.ico favicon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100326 (owner: 10Gerrit Patch Uploader) [14:06:09] (03PS2) 10Hashar: nomcom is dead. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99030 (owner: 10MZMcBride) [14:06:29] (03CR) 10Hashar: [C: 032] "And it is in deleted.dblist:" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99030 (owner: 10MZMcBride) [14:06:38] (03Merged) 10jenkins-bot: nomcom is dead. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99030 (owner: 10MZMcBride) [14:07:10] errm hashar [14:07:27] !log hashar synchronized wmf-config/InitialiseSettings.php '{{gerrit|99030}} nomcom clean up' [14:07:33] twkozlowski: yup? [14:07:44] Logged the message, Master [14:07:51] you +2'd https://gerrit.wikimedia.org/r/#/c/100326/ ? [14:08:03] oh no. [14:08:08] I am so blind today :-P [14:08:25] ah no just changed the commit message and rebased it [14:08:26] sorry [14:08:30] that's the problem with the e-mails [14:08:37] they don't tell you exactly what was changed :-/ [14:16:19] (03CR) 10Hashar: "I don't think secure.wikimedia.org should emit any 404. That should be the responsibility of the domain redirected to." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99024 (owner: 10Tim Starling) [14:20:13] (03CR) 10Hashar: [C: 031] "I still have no clue how to enable new extensions on wiki so added Reedy as reviewer. Maybe I will pair with him to learn how to do it pr" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99739 (owner: 10Ebrahim) [14:21:51] (03PS5) 10Hashar: Disable display of wikibase parser function errors [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99681 (owner: 10Aude) [14:23:25] (03PS1) 10Matanya: add protection level to articles in hewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100372 [14:24:14] (03Abandoned) 10Aude: Disable display of wikibase parser function errors [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99681 (owner: 10Aude) [14:28:06] hashar: anyone else i need to add as reviewer on the patch above? [14:30:17] !log hashar synchronized php-1.23wmf5/.git/refs/heads/wmf/1.23wmf5 'bump Special:Version for aude' [14:30:34] Logged the message, Master [14:31:30] matanya: no idea :-) [14:31:45] matanya: just BTW, you left a trailing whitespace behind. [14:32:05] thanks twkozlowski i'll fix this [14:32:06] (03CR) 10Hashar: [C: 04-1] add protection level to articles in hewiki (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100372 (owner: 10Matanya) [14:32:39] lol [14:32:54] (03PS2) 10Matanya: add protection level to articles in hewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100372 [14:33:12] matanya: you can add 'git diff --check' in your git pre commit hooks :-] [14:33:14] hashar: you are cute :) [14:33:58] apparently https://www.mediawiki.org/wiki/User:Hashar#mediaviewer/File:Hashar-wikipediaglobe.JPG/0 [14:35:45] matanya: more seriously, you might want to talk about such change with the hewiki community [14:35:55] ah https://he.wikipedia.org/wiki/ויקיפדיה:מזנון#.D7.94.D7.A6.D7.91.D7.A2.D7.94 [14:35:57] .. [14:36:04] hashar: ... :) [14:36:07] look at all the green mark voting +1 [14:36:13] though voting for something I can't even read [14:36:29] maybe the question is "I will give you all free pizza at next community event" [14:36:31] +1 :D [14:36:40] (03PS3) 10Hashar: add protection level to articles in hewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100372 (owner: 10Matanya) [14:36:41] hashar: you got me there [14:36:46] deploying [14:36:56] (03CR) 10Hashar: [C: 032] add protection level to articles in hewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100372 (owner: 10Matanya) [14:37:05] (03Merged) 10jenkins-bot: add protection level to articles in hewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100372 (owner: 10Matanya) [14:37:06] thanks a lot hashar [14:38:10] !log hashar synchronized wmf-config/InitialiseSettings.php '{{gerrit|100372}} add protection level to articles in hewiki' [14:38:24] Logged the message, Master [14:38:28] (03CR) 10Siebrand: New extra language for wikidata: Ottoman Turkish (ota) (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96771 (owner: 10Dereckson) [14:39:29] siebrand: thank you for the confirmation :-] [14:39:49] hashar: hmm? [14:39:59] siebrand: 'ota' language for wikibase/wikidata [14:40:01] hey I want my patched deployed too oneoneone! [14:40:06] which doesn't exist in Names.php [14:40:37] hashar: Using extra names because it will never be a meidawiki locale code, but they want support in Wikidata for it. [14:43:48] (03PS4) 10Akosiaris: Template ferm's defs.production [operations/puppet] - 10https://gerrit.wikimedia.org/r/99705 [14:55:44] (03CR) 10Hashar: [C: 04-1] "Some code is duplicate dwith CommonSettings-labs.php from Gerrit change 99600." (033 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [14:58:59] (03PS2) 10Hashar: (bug 57395) Add AbuseFilter rights to sysops on fiwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96967 (owner: 10Odder) [14:59:07] (03CR) 10Hashar: [C: 032] (bug 57395) Add AbuseFilter rights to sysops on fiwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96967 (owner: 10Odder) [14:59:19] (03Merged) 10jenkins-bot: (bug 57395) Add AbuseFilter rights to sysops on fiwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96967 (owner: 10Odder) [14:59:51] hashar: That review is nearly incomprehensible. [15:00:08] !log hashar synchronized wmf-config/abusefilter.php '{{gerrit|96967}} (bug 57395) Add AbuseFilter rights to sysops on fiwiki' [15:00:12] Gloria: do ask there so [15:00:13] sorry [15:00:24] Logged the message, Master [15:00:31] !log hashar synchronized wmf-config/InitialiseSettings.php 'touch' [15:00:44] Logged the message, Master [15:01:25] Gloria: maybe the comments in the inline diff are a bit better ? [15:10:01] siebrand, happy with https://gerrit.wikimedia.org/r/#/c/100327/ ? [15:10:05] hashar: Your comments only displayed ignorance of the change. Which is fine, but it probably means you shouldn't be reviewing it... [15:11:21] Gloria: at least there is a code duplication that needs to be addressed :D [15:11:45] hashar: yes [15:11:55] and the title permission hack could use a bit better comment than "shitty hack" imo [15:12:09] hashar: What code duplication are you talking about? [15:12:21] the hook registration [15:12:24] siebrand: thanks! [15:12:30] got introduced in one of the -labs.php file eaerlier [15:12:43] hashar: This change is about the production files, not the lab files. [15:12:47] You realize that? [15:12:51] they are shared [15:12:58] They shouldn't be. [15:13:03] on beta cluster the production settings are applied first (including hook registration) [15:13:09] then get overridden in the labs file [15:13:13] so you don't need to register the hook twice [15:13:23] I don't know what you're talking about, sorry. [15:14:32] It sounds like you're suggesting putting production configuration in a -labs.php file. [15:15:14] will reply in Gerrit [15:15:21] Okay. [15:23:52] (03CR) 10Hashar: "clarified my review for MZ" (033 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [15:24:00] Gloria: clarified hopefully [15:29:23] (03CR) 10MZMcBride: Create "Draft" namespace on the English Wikipedia (033 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [15:40:00] (03PS1) 10Ryan Lane: Only install mosh on bastion if precise or above [operations/puppet] - 10https://gerrit.wikimedia.org/r/100380 [15:42:17] (03CR) 10Andrew Bogott: [C: 031] "Yes! This has been annnoying me as well." [operations/puppet] - 10https://gerrit.wikimedia.org/r/100380 (owner: 10Ryan Lane) [15:42:21] (03CR) 10Ryan Lane: [C: 032] Only install mosh on bastion if precise or above [operations/puppet] - 10https://gerrit.wikimedia.org/r/100380 (owner: 10Ryan Lane) [15:44:46] hi andrewbogott :) [15:45:05] 'morning [15:49:24] (03PS5) 10Akosiaris: Template ferm's defs.production [operations/puppet] - 10https://gerrit.wikimedia.org/r/99705 [15:50:54] PROBLEM - Varnish HTCP daemon on cp1065 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:51:04] PROBLEM - Varnish HTTP text-backend on cp1065 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:04] PROBLEM - Varnish traffic logger on cp1065 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:54:49] !log rebooting cp1065 (kmem_alloc) [15:55:02] (03CR) 10Akosiaris: [C: 032] mysql_multi_instance : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/100357 (owner: 10Matanya) [15:55:04] Logged the message, Master [15:55:24] PROBLEM - Host cp1065 is DOWN: PING CRITICAL - Packet loss = 100% [15:56:54] RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 1.40 ms [15:56:54] RECOVERY - Varnish traffic logger on cp1065 is OK: PROCS OK: 2 processes with command name varnishncsa [15:57:11] apergos: thanks [15:57:13] I just saw [15:57:53] sure [15:57:54] RECOVERY - Varnish HTTP text-backend on cp1065 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 0.004 second response time [15:59:02] (03PS1) 10Ottomata: debian/varnishkafka.logrotate - fixing postrotate reload command [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/100385 [16:00:27] (03CR) 10Edenhill: [C: 031] debian/varnishkafka.logrotate - fixing postrotate reload command [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/100385 (owner: 10Ottomata) [16:05:38] ori-l: are you around? [16:15:55] (03PS3) 10Akosiaris: Modularize misc::install-server classes [operations/puppet] - 10https://gerrit.wikimedia.org/r/89687 [16:20:13] akosiaris: is there any docs on svn? [16:29:31] matanya: meaning ? [16:30:27] akosiaris: i see we have a svn manifest, and some servers use it in site.pp, but i don't recall any project using it. i wonder if it is redundant or if any docs can tell me who uses it [16:32:36] (03CR) 10Expi1: "I was unable to find a higher resolution version of the internal.ico favicon, and it was difficult to increase the size of the icon to 32p" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100326 (owner: 10Gerrit Patch Uploader) [16:34:35] matanya: it is in a limbo state. The repos are read-only AFAIK, but still served [16:34:50] limbo state was not correct [16:35:18] it is deprecated but still used and will continue to be [16:35:46] i see akosiaris. is it worthwhile to convert all that into a module? [16:37:15] I think yes. It is not going away anytime soon so probably. Depends on the amount of work [16:37:50] if it is too much hassle not, but if it is relatively easy yes [16:38:52] i had a look at it, it look relatively simple. [16:40:18] akosiaris: what about the application servers? what should they include? it is a bit hard for me to guess what are the plans for the future and try to pupptize stuff to adhere to that [16:43:05] as it is for everybody.... but I don't think you should expect a change in the app servers anytime soon. I have not heard anything related [16:46:30] thanks akosiaris [17:05:15] (03PS1) 10BryanDavis: [WIP] Logstash puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 [17:08:37] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Logstash puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 (owner: 10BryanDavis) [17:10:46] ^d: morning. I'm late to build the change sets to push [17:10:56] I'm reclonging core and it is taking forever [17:11:57] <^d> Recloning core? :\ [17:12:21] yeah - I was confused about some of the diffs I was getting.... :( [17:12:32] and didn't realize that meant downloading 1 gig of data [17:12:50] what with submodules [17:13:55] (03CR) 10BryanDavis: [WIP] Logstash puppet class (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 (owner: 10BryanDavis) [17:14:43] (03CR) 10BryanDavis: [WIP] Logstash puppet class (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 (owner: 10BryanDavis) [17:16:30] bd808: do you know there is a deb package for logstash? [17:17:07] matanya: No. At least I didn't know that there was one that we could use yet. [17:18:14] bd808: http://build.logstash.net/job/logstash.jar-flat.daily/ [17:19:13] bd808: we need to add it to apt.wikimedia.org, and then we can use it [17:19:39] i think adding it in get on operations/debs/logstash will do it [17:19:44] *git [17:20:30] (03PS1) 10Manybubbles: make cirrus primary for mw.org [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100399 [17:24:20] matanya: That would be great. I think ori-l was going to look into make sure we had a package from somewhere. [17:24:50] But if there already was one by the time he found time to work on that all the better :) [17:27:36] (03CR) 10BryanDavis: [WIP] Logstash puppet class (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 (owner: 10BryanDavis) [17:28:22] bd808: i can push that to git, if it will help. [17:29:29] matanya: That sounds good to me, but I'm just a lowly software dev :) [17:30:04] i'll try to get ori-l to say a word here before i work on this [17:30:19] Sounds like a reasonable plan [17:30:36] a word [17:30:51] that's two words! [17:31:34] what do you think ori-l ? [17:32:01] should this come from the logstash site and packaged by us? or you prefer some other method? [17:32:30] ask paravoid, what do i know about packaging [17:33:00] not having to repeat work would be nice, but i don't know if the available packages are good enough [17:33:46] ^d: finally submitted everything but it sounds like marktraceur is trying to hunt down Howie to check on something to do with Cirrus and beta features. not sure what [17:34:16] <^d> Lemme find him and see what's up :) [17:34:30] ori-l: i guess it is not, since it is not a native debian package but built using fpm [17:34:32] ^d: also, it look like wmf5 never got the fix for cirrus fatals [17:34:58] i'm not familiar with fpm :/ [17:35:07] so paravoid 's opinion here is vital [17:35:13] matanya: i was going to check out https://github.com/TagManCode/logstash-deb/tree/master/logstash/debian [17:35:17] opinion on what? [17:35:31] ^d: that, or the diff for wmf5 is busted for me [17:35:31] the quality / suitability of readymade debs for logstash [17:35:38] link? [17:35:40] http://build.logstash.net/job/logstash.jar-flat.daily/ [17:35:58] <^d> manybubbles: I'll look. [17:36:15] fpm is an abomination, but it was written by sissel iirc [17:36:24] and if the debs are okay, we don't have to touch fpm [17:36:32] ^d, manybubbles, for logs: Howie doesn't know, but go anyway for now. [17:36:58] andrewbogott: what's the current preferred puppet way to set up an apache vhost site(s) with https, (in labs)? [17:37:02] apache module? [17:37:24] marktraceur: thansk. we're running down more confusion for now but should be good to go soon [17:37:39] James_F: You may have the ability to weigh in on the BetaFeaturesisation of CirrusSearch [17:37:40] paravoid: this is your call here. i used those deb's in our company and no issues so far. but don't take that for granted [17:37:49] <^d> manybubbles: Everything +2'd on gerrit. [17:38:08] (03CR) 10Chad: [C: 032] make cirrus primary for mw.org [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100399 (owner: 10Manybubbles) [17:38:17] ottomata: Yeah, I'd think that apache_site would do it. [17:38:22] ^d: bot is being slow:) [17:38:29] ori-l: so, no more BS with naming conventions: https://github.com/trebuchet-deploy/trigger [17:38:33] ottomata: Although in general you shouldn't need to set up https in labs, unless you're specifically testing the s aspect. [17:38:36] everything is now trebuchet [17:38:39] ^d: https://gerrit.wikimedia.org/r/#/c/99747/ [17:38:48] ottomata: otherwise just do http and use the proxy as your ssl terminator. [17:38:51] ^d: https://gerrit.wikimedia.org/r/#/c/98046/ [17:39:16] ori-l: I'm setting up that portion of the code to have drivers and plugins, so that it's possible to change the mechanism of deployment without modifying the core code base much [17:39:21] Ryan_Lane: \o. [17:39:25] \o/ even! [17:39:30] <^d> manybubbles: 747 hadn't made it into master, so it wasn't part of the submodule update. [17:39:32] and to allow extra things like scaptraps in the command line [17:39:32] <^d> That's why. [17:39:56] ottomata: In case that last bit was obscure… when you set up a proxy here ( https://wikitech.wikimedia.org/wiki/Special:NovaProxy ) https requests that hit the proxy are redirected to http on the back end. [17:40:01] (03Merged) 10jenkins-bot: make cirrus primary for mw.org [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100399 (owner: 10Manybubbles) [17:40:05] if we really want to drive the nail in the coffin of naming, we could use: git trigger start/sync/traps/abort/etc [17:40:31] ^d: we can get it later I believe then [17:40:38] shouldn't be a problem, right? [17:40:56] really it could named anything, though. we could do git scap start/sync/etc. :D [17:41:02] hmm, right ok [17:41:06] (03CR) 10Chad: [C: 032] Enable Cirrus as a beta feature [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98046 (owner: 10Legoktm) [17:41:09] thanks andrewbogott will try that first [17:41:26] ori-l: also, now the license is apache2 throughout the entire codebase [17:41:46] oh bd808 i also wanted to ask you why you retain udp2log, wouldn't it be better let logstash do it itself? [17:42:20] ok so andrewbogott, I should use apache_site over apache::vhost? [17:42:39] matanya: supporting the existing udp2log feed will be the quickest way for us to get data [17:42:41] wait that's just a symlink? [17:42:46] ottomata oh! Um, no idea... [17:42:53] * andrewbogott looks [17:44:02] !log manybubbles synchronized php-1.23wmf6/extensions/BetaFeatures/ 'Update BetaFeatures to master so Cirrus can be a beta feature' [17:44:16] ottomata: looks like apache::vhost is being actively used in other projects, so seems like a safe bet. [17:44:17] Logged the message, Master [17:44:29] the deb is so-and-so [17:44:45] ottomata: sorry to be obscure… so far attempts to standarize on One True Solution for puppet and apache have failed :( [17:44:48] matanya: I'm not sure yet if we want to or should run logstash as a log tailer on fluorine. Eventually we thihk we will provide a patch to core (or extension) that lets MW write json logs directly to redis to transport them to the logstash indexer [17:44:49] (03Merged) 10jenkins-bot: Enable Cirrus as a beta feature [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98046 (owner: 10Legoktm) [17:44:56] !log manybubbles synchronized php-1.23wmf6/extensions/CirrusSearch/ 'Update Cirrus to master' [17:45:07] ships empty useless files, ships the jar into /opt, has no docs/copyright statement/license [17:45:12] Logged the message, Master [17:45:25] technically we can't even redistribute as it comes with no license to do so [17:45:39] paravoid: I was thinking we could build our own using a fork of https://github.com/TagManCode/logstash-deb [17:45:44] (or even use) [17:45:50] ^d and marktraceur: test2wiki looks good with new code so I'll sync wmf5 now [17:45:55] 'kay [17:46:08] bd808: that sounds like best approched to me, though on seperate hardware, not fluorine [17:46:24] But I now notice that repo does not itself have a declared license :( [17:46:52] it has bd808 : https://github.com/TagManCode/logstash-deb/blob/master/logstash/debian/copyright [17:47:39] which is wrong :P [17:47:54] yeah, now i noticed :/ [17:48:10] That should be the copyright for the logstash package shouldn't it? [17:48:23] yes [17:49:16] !log manybubbles synchronized php-1.23wmf5/extensions/BetaFeatures 'Update BetaFeatures so Cirrus can be a beta feature' [17:49:26] I propose that we use the build.logstash.net for now (if they have a release train in addition to the nightly ones, that is) [17:49:32] Logged the message, Master [17:49:39] and make a note to create proper packages as we go forward [17:49:44] !log manybubbles synchronized php-1.23wmf5/extensions/CirrusSearch 'Update Cirrus to master' [17:49:47] marktraceur: Sure. Where? [17:50:00] Logged the message, Master [17:50:06] but let's get some results first, might help with allocating some time into the project [17:50:10] maybe some of my time even :-) [17:50:18] James_F: Like...here [17:50:26] James_F: manybubbles and ^d are pushing it now [17:50:36] I'm fighting my perfectionism with pragmatism [17:51:08] James_F and marktraceur: huh? [17:51:11] paravoid: so you wnat me to push the one from logstash to operations/debs/logstash? [17:51:16] no [17:51:24] I'd like to get java support into trebuchet for things like this and gerrit [17:51:24] no point in doing that [17:51:26] manybubbles: James_F is involved in BF too [17:51:33] so that we don't necessarily need to package every java thing we need [17:51:34] does upstream provide an apt repo? [17:51:48] no paravoid , there is an open bug on that [17:52:17] if there's a deb file available, making an apt repo is as simple as setting up a PPA on launchpad [17:52:20] do they provide a debian source package in addition to .debs, somewhere? [17:52:21] marktraceur and James_F: well good news! you are going to get another beta features this morning! unless you don't want it and tell me now:) [17:53:02] I continue cautious support [17:53:25] not that i know of paravoid. i asked jordan sissel, he said he sees no point in that since fpm doesn't create them [17:56:51] !log manybubbles synchronized wmf-config/InitialiseSettings.php 'Set mediawiki.org as primary and prepare for cirrus as a beta feature' [17:57:08] !log manybubbles synchronized wmf-config/CirrusSearch-common.php 'Finally make Cirrus a beta feature' [17:57:08] Logged the message, Master [17:57:22] Logged the message, Master [17:57:26] marktraceur: Ah, sounds good. [17:57:32] marktraceur: Pushing to beta? [17:57:33] weeee, cirrus as beta feature :) [17:57:53] Or pushing to production? [17:58:06] I thought we were going to put all Beta Features on beta labs first for a week? [17:58:20] hmm [17:58:21] https://en.wikisource.org/wiki/Special:Preferences#mw-prefsection-betafeatures [17:58:21] James_F: marktraceur or manybubbles? ;) [17:58:22] > Here are some new features we're considering for $2. Please try them out and give us your thoughts, so we can improve them based on your feedback. [17:58:43] James_F: first I've heard of that. cirrus has been in beta for a long time [17:58:44] legoktm: I see it [17:58:50] legoktm: l10n cache broken again? [17:58:52] also, cirrus isn't in the list [17:58:54] marktraceur: It was greg-g's commandment. [17:59:07] let's say, guideline [17:59:12] MatmaRex: seems like a missing parameter [17:59:12] Lo, and Cirrus was a BetaFeature. And it was good. [17:59:15] * greg-g channels johnny depp [17:59:26] ah yeah, i'm dumb [17:59:34] legoktm and ^d: look like sync didn't update some messages and has made a little mess [17:59:35] it should be sitename there. [17:59:50] <^d> manybubbles: Sync doesn't update messages. [18:00:00] <^d> With 2 extensions + config + messages, shoulda scapped. [18:00:14] greg-g: So we have a special request from the GWToolset folk that we be allowed to push it to (phase|stage)0 wikis on Thursday with a paltry 6 days of beta cluster residence. [18:00:21] I'm not sure why that message would have changed though... [18:00:30] ^d: so, should I do that now? [18:00:34] ah [18:00:34] https://github.com/wikimedia/mediawiki-extensions-BetaFeatures/commit/0e6d3a9cdbbff42d469398896a7149ba3d934528#diff-42a960c066608f67f28561247bf3bb2a [18:00:36] yeah [18:00:38] needs scap [18:01:04] <^d> Run a full scap, what legoktm said :) [18:01:21] k. doing it! [18:02:17] ^d: it is doing things now [18:02:30] ^d: the wiki says this will take a long time [18:02:42] <^d> Get a cup of coffee. [18:02:45] <^d> :) [18:04:04] ^d: ah, missed seomthing. can I ctrl-c it? [18:04:48] marktraceur: I wouldn't :) [18:05:19] greg-g and ^d: well it was the config files. I forgot to rebase to get them. I'll check if they were synced when it is done. I imagine they were [18:05:29] sorry, will be [18:05:57] k [18:06:06] !log manybubbles started scap: Updating messages after extension deploys missed it by syncing files and directories [18:06:23] Logged the message, Master [18:06:40] man, there is a big delay there [18:06:53] (before it logs to channel) [18:07:57] yeah! [18:12:41] (03PS2) 10Dr0ptp4kt: WIP: DO NOT MERGE YET. Allow Google's bots to scrape bits. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95548 [18:13:58] that is pretty funny: during scap the option to enable cirrus as a beta feature keeps appearing and disappearing [18:14:47] ja, ok ,thanks andrewbogott, that's why i was asking [18:14:47] danke [18:15:10] manybubbles: maybe you're hitting different servers :P [18:15:21] MatmaRex: I'm sure of it [18:15:31] I should look up what scap actually does now that I'm doing it [18:16:08] <^d> scap does lots of scary things. [18:16:56] ^d: looks like it is copying to a couple of machines at a time. I assume it is depooling them, copying, and repooling or something [18:20:35] <^d> No depooling. [18:20:46] <^d> But yeah, it spreads out over a couple machines at a time. [18:22:16] "One user has enabled this feature." \o/ [18:22:17] legoktm and ^d and marktraceur and James_F and greg-g: modulo the scap cirrus is working correctly as a beta feature. No extra fatals/exception/warnings I can see and I can set the setting and see it take effect (modulo scap, again) [18:22:24] that'd be me, probably [18:22:28] woo :D [18:22:46] and the message is fixed [18:23:44] marktraceur: Cool. What wikis are getting it? [18:24:31] (03Abandoned) 10Yuvipanda: Move UploadWizard tracking categories out of AutoAdd [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76946 (owner: 10Yuvipanda) [18:25:19] James_F: AFAIK stage0, === test, test2, mw.o, maybe testwikidatawiki [18:25:23] James_F and marktraceur: see this https://www.mediawiki.org/wiki/Search#Wikis Check wikis that don't have cirrus already as a primary but _do_ have it as a secondary [18:25:31] <^d> legoktm, manybubbles: Just enabled it on test.wp for myself :) [18:25:48] manybubbles: That would explain why I don't see it on MW.org. :-) [18:25:51] manybubbles: Thanks! [18:25:54] MW.org doesn't need it [18:25:57] yeah [18:26:04] cirrus is funky because it came before this project started [18:26:11] "this project" meaning betafeatures [18:26:43] I wish scap had a progress meter or something [18:27:29] greg-g: ^^ [18:27:35] not a bad idea [18:27:41] manybubbles: In the near-to-mid future it would be fantastic to have an icon for the feature [18:28:01] marktraceur: hmmmm that sounds fun! [18:28:07] Designers can halp [18:28:53] yurik, not yet. once i'm done with my email and small tasks, i will work on the review. thank you for emailing on that [18:29:21] manybubbles: I've sent mail to howie + a bunch of others [18:29:59] yay! [18:30:35] when the scap finishes I'll restart indexing wikidata and send an email to ops and aude [18:30:58] !log manybubbles finished scap: Updating messages after extension deploys missed it by syncing files and directories [18:31:07] speak of the devil! [18:31:12] scap completed in 28m 48s. [18:31:14] Logged the message, Master [18:31:15] not too bad [18:31:24] greg-g: I'm off of tin [18:31:30] no more syncing planned [18:33:08] manybubbles: sweet [18:37:14] PROBLEM - Puppet freshness on labsdb1002 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 03:36:38 PM UTC [18:41:30] (03PS5) 10Ottomata: [not ready for review] Productionizing Wikimetrics [operations/puppet] - 10https://gerrit.wikimedia.org/r/96042 (owner: 10Milimetric) [18:41:34] ^d and greg-g: it looks like wikiata doesn't have cirrus now.... [18:41:36] bug! [18:41:39] what?! [18:41:41] why! [18:41:48] you proooo missed [18:42:56] * paravoid looks at logstash's makefile and cries [18:43:17] god, someone needs to teach all these devops people about unix [18:44:11] (03PS1) 10Manybubbles: give wikidata cirrus as an alternate again [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100412 [18:44:20] https://raw.github.com/logstash/logstash/master/Makefile fwiw [18:44:24] PROBLEM - Host cp1065 is DOWN: PING CRITICAL - Packet loss = 100% [18:44:41] paravoid: there is so much unix to learn, though [18:45:07] this makefile is awful [18:45:22] besides being syntactically awful, it fetches a number of unverified files from all kinds of places [18:46:30] (03CR) 10Chad: [C: 032] give wikidata cirrus as an alternate again [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100412 (owner: 10Manybubbles) [18:46:46] (03Merged) 10jenkins-bot: give wikidata cirrus as an alternate again [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100412 (owner: 10Manybubbles) [18:46:50] greg-g and ^d: I'm going to sync that [18:47:55] !log powercycling cp1065, unresposinve, no output on console [18:48:02] interesting [18:48:09] Logged the message, Master [18:48:11] ^d: I'm not really sure how we lost it [18:48:12] !log manybubbles synchronized wmf-config/InitialiseSettings.php 'readd cirrus to wikidata' [18:48:13] but whatever [18:48:16] not it should be back [18:48:29] Logged the message, Master [18:49:51] ^d: grr - that didn't work. what have I done wrong? [18:50:39] <^demon> Logging in. [18:50:54] RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 0.62 ms [18:51:44] <^demon> manybubbles: Duh :) [18:51:53] <^demon> removing => false isn't the same as setting => true [18:51:59] <^demon> When the default is => false [18:52:13] * ^demon needs caffeine [18:52:17] it is in the list thought! [18:52:20] though [18:53:04] oh weird, it works when I do it in a page request.... [18:53:14] PROBLEM - Puppet freshness on db1053 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 03:52:08 PM UTC [18:54:42] ^d: found it [18:54:44] damn it [18:55:44] ^d: it looks like arsenic is busted [18:55:53] <^d> Yeah it's pretty busted. [18:56:01] <^d> That explains it :) [18:56:05] manybubbles@arsenic:~$ ls /a/common/ | grep php- [18:56:06] <^d> Because it looks fine everywhere else. [18:56:08] wait! [18:56:12] I was supposed to do this on terbium [18:56:14] I forgot! [18:58:00] everything is fine! [18:58:08] greg-g: I'm well and truely done [18:58:16] ^d: everything is running well now [18:59:39] manybubbles: :) [18:59:51] paravoid, we are ready to upgrade to node 0.10 [18:59:56] greg-g: squeaked in this time [19:00:55] paravoid: we have a deploy window between 1 and 2pm pacific, could roll out node 0.10 to the parsoid cluster then [19:01:36] gwicke: in a meeting, sorry [19:02:14] PROBLEM - Puppet freshness on db1054 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 04:01:16 PM UTC [19:02:14] PROBLEM - Puppet freshness on db1057 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 04:01:22 PM UTC [19:03:11] paravoid: np, ping me when you have time [19:05:14] PROBLEM - Puppet freshness on labsdb1003 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 04:04:47 PM UTC [19:20:14] PROBLEM - Puppet freshness on labsdb1001 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 04:19:28 PM UTC [19:21:14] PROBLEM - MySQL Replication Heartbeat on db73 is CRITICAL: CRIT replication delay 316 seconds [19:21:24] PROBLEM - MySQL Slave Delay on db73 is CRITICAL: CRIT replication delay 322 seconds [19:23:38] (03CR) 10Ebrahim: "This also needed I guess https://gerrit.wikimedia.org/r/#/c/100418/" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99739 (owner: 10Ebrahim) [19:36:14] PROBLEM - MySQL Replication Heartbeat on db73 is CRITICAL: CRIT replication delay 306 seconds [19:36:24] PROBLEM - MySQL Slave Delay on db73 is CRITICAL: CRIT replication delay 315 seconds [19:39:24] PROBLEM - MySQL Slave Delay on db73 is CRITICAL: CRIT replication delay 302 seconds [19:42:31] holy... [19:42:38] large traffic spike [19:43:01] esp. on mobile [19:43:21] (03CR) 10Hashar: "Sorry odder I have been to fast. My reasoning is that the new size is not worse than the smaller one :-D" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99605 (owner: 10Tholam) [19:44:05] mobile was 80% up, desktop 15% [19:45:08] (03CR) 10Odder: "Sure; AffCom had three days to comment on this, and they didn't, so it's their loss :-)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99605 (owner: 10Tholam) [19:46:39] (03PS6) 10Ottomata: [not ready for review] Productionizing Wikimetrics [operations/puppet] - 10https://gerrit.wikimedia.org/r/96042 (owner: 10Milimetric) [19:49:14] RECOVERY - MySQL Replication Heartbeat on db73 is OK: OK replication delay 137 seconds [19:49:24] RECOVERY - MySQL Slave Delay on db73 is OK: OK replication delay 145 seconds [19:50:07] grace hopper seems to be high enough [19:50:14] the grace hopper article [19:50:16] google doodle? [19:50:45] I don't see a google doodle here, maybe US only? [19:53:46] Coren: reminder re https://gerrit.wikimedia.org/r/#/c/100329/ :) [19:53:56] gwicke: don't forget to set the proxy option today [19:54:29] ori-l: Ah, right. I'm on it. [19:57:59] so [19:58:02] that traffic spike [19:58:12] filled amslvs1's gigabit [19:58:14] how nice :) [19:59:27] (03CR) 10coren: [C: 032] "There's a (very minor) quibble about a pointless variable in the SIGCHLD handler, but not a reason to not +2 IMO." (031 comment) [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/100329 (owner: 10Ori.livneh) [20:04:37] (03PS7) 10Ottomata: [not ready for review] Productionizing Wikimetrics [operations/puppet] - 10https://gerrit.wikimedia.org/r/96042 (owner: 10Milimetric) [20:14:45] (03CR) 10Tholam: "I can get rid of the white background if it looks bad. I just left it in because it was there in the original icon." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99605 (owner: 10Tholam) [20:23:23] (03CR) 10Hashar: "Odder: to be fair, 3 days over the weekend is not that long. I don't think it is going to be too disruptive for them :-] After all the fav" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99605 (owner: 10Tholam) [20:30:05] Coren: thanks! [20:32:29] (03CR) 10Odder: "I was thinking about changing 'chapcom' into 'affcom' if the Committee wanted that, but since they didn't say anything, I'm OK with how th" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99605 (owner: 10Tholam) [20:34:14] (03PS3) 10Ori.livneh: Use glib's GHashTable instead of persistent BDB [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/100329 [20:34:35] (03CR) 10Ori.livneh: [C: 032 V: 032] "Based on Coren's +2" [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/100329 (owner: 10Ori.livneh) [20:35:08] (03PS8) 10Ottomata: [not ready for review] Productionizing Wikimetrics [operations/puppet] - 10https://gerrit.wikimedia.org/r/96042 (owner: 10Milimetric) [20:35:09] paravoid, yup [20:35:38] paravoid: what is your thinking re node 0.10 roll-out? [20:35:47] I'd need root help for that [20:38:02] (03PS9) 10Ottomata: [not ready for review] Productionizing Wikimetrics [operations/puppet] - 10https://gerrit.wikimedia.org/r/96042 (owner: 10Milimetric) [20:40:36] Ryan_Lane: i think deployment minions incorrectly report successful deployment when there are local modifications to the deployment target [20:40:46] not suggestion you sanction local changes, but presumably there should be a loud failure [20:40:56] (03CR) 10Hashar: Create "Draft" namespace on the English Wikipedia (032 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [20:41:19] or even better, precede every deployment with a hard reset and a git clean -fd [20:52:12] (03PS1) 10Ottomata: rxbytes should be a counter in VarnishkafkaLogster.py [operations/puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/100430 [20:52:23] (03CR) 10Ottomata: [C: 032 V: 032] rxbytes should be a counter in VarnishkafkaLogster.py [operations/puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/100430 (owner: 10Ottomata) [21:05:13] (03PS1) 10Ottomata: Adding ganglia view for varnishkafka. [operations/puppet] - 10https://gerrit.wikimedia.org/r/100432 [21:06:27] !log deployed Parsoid 3191007575830, see https://www.mediawiki.org/wiki/Parsoid/Deployments [21:06:43] Logged the message, Master [21:07:41] (03CR) 10Ottomata: [C: 032 V: 032] Adding ganglia view for varnishkafka. [operations/puppet] - 10https://gerrit.wikimedia.org/r/100432 (owner: 10Ottomata) [21:09:49] (03PS1) 10Ottomata: Fixing varnishkafka ganglia view name [operations/puppet] - 10https://gerrit.wikimedia.org/r/100433 [21:10:01] (03CR) 10Ottomata: [C: 032 V: 032] Fixing varnishkafka ganglia view name [operations/puppet] - 10https://gerrit.wikimedia.org/r/100433 (owner: 10Ottomata) [21:10:46] (03PS1) 10Dan-nl: beta: gwtoolset-whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100434 [21:15:07] paravoid: I'm still testing the proxy setting- if you'd like to upgrade node in this window then let me know [21:15:26] hey [21:15:46] I can put packages in apt, but are you sure you want to tie the node upgrade with your update? [21:16:26] there is one sub-dependency (jsdom used by html5) that is binary and thus depends on the node version [21:16:34] (03CR) 10Hashar: [C: 032] "will self deploy on beta cluster" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100434 (owner: 10Dan-nl) [21:16:42] (03PS1) 10Ottomata: monitoring.pp - betterin varnishkafka view [operations/puppet] - 10https://gerrit.wikimedia.org/r/100437 [21:16:52] (03Merged) 10jenkins-bot: beta: gwtoolset-whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100434 (owner: 10Dan-nl) [21:17:06] we don't exercise it and I had no issues in rt testing after just upgrading node [21:17:16] but I'd prefer to be around [21:17:22] PROBLEM - MySQL Replication Heartbeat on db73 is CRITICAL: CRIT replication delay 302 seconds [21:17:31] (03PS2) 10Ottomata: monitoring.pp - betterin varnishkafka view [operations/puppet] - 10https://gerrit.wikimedia.org/r/100437 [21:17:44] (03CR) 10Ottomata: [C: 032 V: 032] monitoring.pp - betterin varnishkafka view [operations/puppet] - 10https://gerrit.wikimedia.org/r/100437 (owner: 10Ottomata) [21:18:02] PROBLEM - MySQL Slave Delay on db73 is CRITICAL: CRIT replication delay 322 seconds [21:18:38] paravoid: in general, the best strategy would be a staggered upgrade: take one node down, upgrade the code and node, restart node; then move on to the next node [21:19:13] do you have a way to do this as part of your deployment strategy? [21:19:23] I have no idea how the deployment tool works [21:19:30] afaik no, as any push restarts the server too [21:19:42] git-deploy is a bit annoying [21:20:22] Ryan_Lane could probably disable restarts when pushing the config module though [21:20:56] well, yeah, that's doable [21:21:04] we could also write a staggered restart runner [21:21:11] and allow tin to call it [21:21:48] then you'd deploy, then do a restart after the deployment is done [21:21:51] for the node_modules + node upgrade just not restarting on config change would be enough [21:22:01] the node upgrade needs to be done as root anyway [21:22:25] but most of the time you want restart on config updates [21:22:58] yeah, it would definitely be handy to have a way to restart all parsoids [21:23:17] so, let's just add a service restart runner [21:23:41] that'll do restarts as a batch job, with a default batch amount that's overridable [21:23:50] like a default batch size of 5 [21:24:14] that's incredibly easy to do. give me a sec [21:25:06] k, thx [21:27:02] PROBLEM - MySQL Slave Delay on db73 is CRITICAL: CRIT replication delay 320 seconds [21:27:13] gwicke: has parsoid fixed its init script yet? [21:27:43] Ryan_Lane, we don't plan to fix it really [21:27:48] what? why? [21:27:50] (03CR) 10Legoktm: Create "Draft" namespace on the English Wikipedia (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [21:27:52] better to replace it with upstart [21:27:57] yeah. that's what I mean [21:28:04] when is that going to happen? [21:28:18] this week [21:28:22] PROBLEM - MySQL Replication Heartbeat on db73 is CRITICAL: CRIT replication delay 314 seconds [21:28:22] I'd like to make this a generic deploy.restart call [21:28:53] but that's going to mean calling service.restart, which parsoid doesn't currently support [21:28:56] due to its init script [21:29:05] we are also splitting the php code from the node service [21:29:12] yay [21:29:14] so repos will change [21:29:27] I was confused when I wanted to submit that patch [21:29:29] am currently playing with debianization [21:29:40] npm2debian [21:29:54] using deb rather than git-deploy? [21:29:59] or for third parties? [21:30:18] csteipp: is CentralAuth going soon? [21:30:22] mostly third parties [21:30:26] for now at least [21:30:29] * Ryan_Lane nods [21:30:36] for wikimedia debs will definitely be worse [21:30:41] gwicke: did you set the proxy on the config? [21:30:54] am not so sure- staggered upgrades should work fine with debs [21:31:09] paravoid, not yes- am testing it in betalabs currently [21:31:13] *yet [21:31:19] but you did push the new version [21:31:29] api.php is something like 1000 req/s right now :) [21:31:30] is there an internal api cluster I can use for betalabs testing? [21:31:43] I don't think so [21:31:54] just pointing it to wikipedia-lb.wikimedia.org does not work [21:32:06] how come? [21:32:20]

This domain points to a Wikimedia Foundation server, but is not configured on this server.
That�s all we know.

[21:32:28] gwicke: except you need to create a deb, someone from ops needs to push it to the repo, then you need to update the puppet repo with the specified version [21:32:35] oh that's because Host: would be en.betalabs or something [21:32:37] then someone from ops needs to merge the change [21:33:01] paravoid: 2:30 or 3pm today [21:33:02] it would possibly take two hours for it to run puppet everywhere [21:33:12] I wouldn't recommend a .deb for internal consumption either, but I think this was never the plan [21:33:14] and you'd need to do the same thing to revert [21:33:28] paravoid: gwicke keeps suggesting it as an alternative to using trebuchet [21:33:48] it's fine as an alternative, if you need to deploy once per month or two :P [21:33:49] Ryan_Lane: if the deb was build from our source automatically there would be little reason for ops involvement [21:33:52] indeed [21:34:07] the root stuff can be in a separate repo [21:34:09] gwicke: I just showed you why there would be a need for ops involvement [21:34:24] there's at least two steps that require ops [21:34:36] the only reason I see is to make sure that only ops has merge rights on stuff that runs as root [21:34:38] and realistically, we wouldn't allow debs to be deployed without review [21:34:49] apt has no package filters; every repo is trusted to provide all packages [21:34:52] because debs give root level permissions [21:35:07] using debs for deployment is a bad idea [21:35:13] for lots of reasons [21:35:21] there is no concept of a restricted deb; installation scripts run as root and are not restricted in any way [21:35:33] sure, but those can come from an ops-controlled repo [21:35:37] (ubuntu was playing with a sandboxed deb concept for some time but I think it ultimately failed) [21:35:41] not very hard to do [21:36:09] this means involving us in the hot path of deployment [21:36:13] no [21:36:14] I think it'll be more trouble than it's worth [21:36:29] the root scripts don't change often [21:36:42] you can still build a new deb with new js code [21:36:54] you're kind of missing the point [21:36:54] and the same unchanged root scripts [21:36:59] ops doesn't want to be involved in this [21:37:05] at all [21:37:06] :) [21:37:12] yup, makes sense [21:37:16] there is no need though ;) [21:37:16] using debs requires that [21:37:23] why? [21:37:28] <^d> Ryan_Lane: I heard you like making debs for me ;-) [21:37:40] are you suggesting that we only install things like init scripts and users with the debs? [21:37:43] anyway, paravoid: I think we need to figure out some way to test the proxy change [21:37:50] and you'd still using the deployment system for deploying the js? [21:37:54] PROBLEM - Puppet freshness on labsdb1002 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 03:36:38 PM UTC [21:38:10] ^d: heh. it's annoying every single time :) [21:38:18] Ryan_Lane: no, but the deb can combine init scripts from an ops repo with js code from our repo [21:38:30] building the deb is an automated process [21:38:33] gwicke: right, so we'd need to do something every single time you wanted to deploy [21:38:38] nope [21:38:41] yes [21:38:43] <^d> Ryan_Lane: 2.8 went final. I'm playing around with it and considering upgrading :p [21:38:44] absolutely [21:38:49] <^d> Of course, it'll probably break everything. [21:38:49] every single time [21:38:52] why? [21:39:05] because we'd need to review what was being generated [21:39:11] the js code? [21:39:12] and we'd need to push the deb into the repo [21:39:15] that's not automated [21:39:18] why would you suddenly want to review that? [21:39:26] and how would the deb be installed? [21:39:33] pkg = latest? [21:39:36] via puppet? [21:40:07] ensure => latest has bitten us in the past [21:40:09] it is not my top priority in any case [21:40:29] <^d> mutante: s/the past/the past week/ [21:40:29] the deployment system works. if there's something wrong with it, let's fix that [21:40:31] <^d> :\ [21:40:36] let's not make a debian based deployment system [21:40:37] I don't see a technical reason why it would not work for long-running services with staggered deployment though [21:41:14] with a shitload of effort that could work [21:41:27] and it would still be a non-ideal way to deploy something [21:41:50] prone to failure and slow to revert [21:42:30] <^d> Case in point: every failed gerrit upgrade? There's a long-running service managed by debian package. [21:42:37] not sure- right now we don't have any automated support for updating debs in sync with code [21:42:43] or downgrading for that matter [21:43:06] node_modules and nodejs for example [21:43:33] gwicke: know why? because ops moved away from that model years aho [21:43:33] maybe hiphop and some opcode cache in the future [21:43:35] *ago [21:43:37] because it sucks [21:44:12] I agree that is sucks for PHP deploys [21:44:22] I'm mostly talking about services though [21:44:40] it sucks for anything that needs to happen with any relative frequency [21:44:50] paravoid, without testing I won't enable the proxy URL in prod [21:45:39] gwicke: is there some major downside to the current deployment system in use? [21:45:44] I'm more than willing to fix issues [21:46:11] Ryan_Lane, is there a way to synchronize node_modules upgrades with deb upgrades? [21:46:14] I don't work for wikimedia anymore, so I can't really oppose using debs for this, but it's a bad idea [21:46:33] gwicke: I'm not sure what you mean [21:46:45] (03CR) 10Hashar: "Some random bits. Don't trust me!" (037 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 (owner: 10BryanDavis) [21:46:54] when you install node you need something to occur? [21:46:55] we need new node_modules with a new major node version [21:47:14] where is node_modules? [21:47:38] in Parsoid we are currently lucky that we don't depend on binary modules, but other node services like the collection service depend on binary modules [21:48:00] right now we have a dirty hack of a config repository [21:48:27] but afaik I have no way to tell git-deploy 'this hash has a precondition of installing nodejs version 0.10' [21:48:50] with a deb you'd do that with a dependency [21:49:16] * Ryan_Lane nods [21:49:32] well, that's a good question [21:49:43] the deployment system *could* handle that [21:49:56] it's a question of whether we want to have the deployment system manage packages at all [21:50:14] the other thing that we actually don't need is atomic upgrades [21:50:32] yeah, that's something free you just happen to get [21:51:00] staggered upgrades would be better to catch errors early and reduce perf impact [21:51:53] well, it could be possible to add a "canary" stage [21:53:02] that would be nice, followed by a staggered upgrade over the remaining hosts [21:53:13] restarting a node service takes it down for a few seconds [21:53:22] so you don't want to do that with all hosts at the same time [21:53:54] well, if you did: git deploy start; git deploy canary [21:53:54] PROBLEM - Puppet freshness on db1053 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 03:52:08 PM UTC [21:54:00] it would do a deploy to the canary systems [21:54:08] (03CR) 10Mattflaschen: [C: 04-1] "Hashar is right about the Labs part. It's best for this patch to remove the Labs configs I added (as part of testing this change). When " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [21:54:12] then you could restart the canaries [21:54:26] I guess really in that situation you don't need to do a manual restart [21:54:40] you could let the checkout hook restart it [21:55:00] yes, especially if it is only one system [21:55:05] then later you could do a sync [21:55:36] there was an idea of doing this for mediawiki too [21:55:42] for test.wikipedia.org [21:55:51] the current system is workable in any case, but it is designed more for PHP upgrades than service upgrades [21:55:58] agreed [21:56:16] I'll start by adding a manual restart action [21:56:26] but it's going to do a batched restart of all services [21:56:41] canary actions is going to take some work [21:57:20] because it requires changes in every part of the system [21:57:34] a batched restart and no restart by default would be helpful already [21:57:42] cool. I'll push that in soonish [21:58:14] Ryan_Lane, thanks! [21:58:46] yw [21:59:12] I'm rewriting the frontend right now, so once I have all the basic functionality added I'll add the canary stuff too [22:00:44] gwicke: one hard thing for this will be defining canaries [22:01:17] I'll use a grain, but you'll need to figure out where to add that with puppet. I think the node itself, or a separate role would be a good place [22:01:22] (03CR) 10Dzahn: [C: 032] Correct capitalization of "ShoutWiki". [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/96018 (owner: 10Jack Phoenix) [22:01:40] role::parsoid::canary <-- likely a good place for it [22:02:01] Ryan_Lane, could the deployment system pick a random subset? [22:02:09] say 5% of hosts? [22:02:25] it would use something like deployment::canary { "parsoid": } [22:02:28] (03CR) 10Dzahn: [V: 032] Correct capitalization of "ShoutWiki". [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/96018 (owner: 10Jack Phoenix) [22:02:40] gwicke: I think that would be relatively difficult to do [22:02:54] PROBLEM - Puppet freshness on db1057 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 04:01:22 PM UTC [22:02:54] PROBLEM - Puppet freshness on db1054 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 04:01:16 PM UTC [22:02:57] ok [22:03:12] and dangerous, since you wouldn't necessarily know which ones it was targeting [22:03:21] I guess it would show you through reporting.... [22:03:27] still difficult ,though [22:03:58] how are other systems doing automated canary deployments? [22:05:27] well, there's basically nothing for deployment systems around [22:05:33] that are open source and usable [22:05:51] hrm, I see [22:06:12] different secret sauce everywhere [22:06:26] PROBLEM - Puppet freshness on labsdb1003 is CRITICAL: Last successful Puppet run was Mon 09 Dec 2013 04:04:47 PM UTC [22:07:02] (03PS1) 10Springle: puppet error after recent lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/100480 [22:07:41] yeah [22:07:55] I wonder how heroku handles this [22:08:19] (03CR) 10Springle: [C: 032] puppet error after recent lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/100480 (owner: 10Springle) [22:08:26] well, anyway, we define nodes through puppet [22:08:33] so defining a set of nodes as canary is doable [22:08:45] (and relatively easy) [22:09:26] RECOVERY - Puppet freshness on db1053 is OK: puppet ran at Mon Dec 9 22:09:17 UTC 2013 [22:09:52] most people create their own deployment systems using fabric :D [22:10:02] in the longer term it would be awesome if the deployment system could automatically monitor logs or other stats from the canary systems [22:10:18] and abort if thresholds are exceeded [22:10:37] yes, so, that's definitely my later plans [22:10:45] for a continuous deployment system [22:10:54] so it would need to know which hosts were just upgraded [22:11:01] that's tracked in redis [22:11:03] and which are up etc [22:11:15] I should track service restarts in redis too [22:11:49] the idea would be you push a tag into git, then say "deploy this tag" [22:11:55] and the system would handle the rest [22:12:03] hopefully also including depooling hosts, if necessary [22:12:12] and dealing with thresholds [22:12:16] logs could go into logstash [22:12:16] RECOVERY - Puppet freshness on db1054 is OK: puppet ran at Mon Dec 9 22:12:11 UTC 2013 [22:12:29] and the deployment system could query elasticsearch for errors [22:12:52] *nod* [22:13:03] doing automatic threshold checks on logs would be... difficult :) [22:13:06] but doable [22:13:14] they'd need to be user defined, likely [22:13:19] (03CR) 10Springle: "This change broke puppet on all mysql_multi_instance boxes, specifically the bit removing name => 'mysql' from generic::systemuser." [operations/puppet] - 10https://gerrit.wikimedia.org/r/100357 (owner: 10Matanya) [22:14:01] with hiphop on the horizon it seems likely that we'll move more and more towards long-running processes that take some time to restart [22:14:06] (03PS1) 10Mattflaschen: Test VE on Labs Draft namespace for opt-in users [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100481 [22:14:26] RECOVERY - Puppet freshness on db1057 is OK: puppet ran at Mon Dec 9 22:14:24 UTC 2013 [22:14:38] yep [22:15:02] so atomic upgrades seem to become less and less achievable [22:15:16] RECOVERY - Puppet freshness on labsdb1001 is OK: puppet ran at Mon Dec 9 22:15:10 UTC 2013 [22:15:26] that's not necessarily true [22:15:36] RECOVERY - Puppet freshness on labsdb1002 is OK: puppet ran at Mon Dec 9 22:15:30 UTC 2013 [22:15:38] if you run multiple hphp processes, you can switch between them [22:15:46] RECOVERY - Puppet freshness on labsdb1003 is OK: puppet ran at Mon Dec 9 22:15:45 UTC 2013 [22:15:50] you could do the same with node [22:15:54] (03CR) 10Mattflaschen: "We're also planning to add VE (will only apply for opt-in users currently) in a follow-up patch." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97675 (owner: 10MZMcBride) [22:15:59] maybe, yes [22:16:19] anyway, it doesn't need to be fully atomic [22:16:20] not sure that it is better than avoiding the need for atomic upgrades though [22:16:26] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (200952) [22:16:38] atomicity between all nodes wasn't ever really an objective [22:17:04] yeah, not achievable in any case [22:17:10] it's definitely nice to strive for :) [22:17:26] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [22:17:29] it should just be able to do mass version changes as quickly as possible [22:17:54] my main objective is to provide a good an usable deployment system that's also open source and usable by third parties [22:18:03] because everyone writing a system from scratch sucks [22:18:17] (03CR) 10Ori.livneh: [C: 032] Test VE on Labs Draft namespace for opt-in users [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100481 (owner: 10Mattflaschen) [22:18:23] agreed [22:18:26] (03Merged) 10jenkins-bot: Test VE on Labs Draft namespace for opt-in users [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100481 (owner: 10Mattflaschen) [22:26:02] so [22:26:08] what should we do re: node upgrade? :) [22:27:59] -_- can't do batch calls via runners [22:28:03] well, that's a pain in the ass [22:31:01] (03PS1) 10Ottomata: Putting ganglia aggregators for Analytics cluster in different Rows [operations/puppet] - 10https://gerrit.wikimedia.org/r/100484 [22:31:20] (03CR) 10Ottomata: [C: 032 V: 032] Putting ganglia aggregators for Analytics cluster in different Rows [operations/puppet] - 10https://gerrit.wikimedia.org/r/100484 (owner: 10Ottomata) [22:32:18] (03PS1) 10Ottomata: Also need to change gmetad config for analytics ganglia aggregators [operations/puppet] - 10https://gerrit.wikimedia.org/r/100486 [22:32:32] (03CR) 10Ottomata: [C: 032 V: 032] Also need to change gmetad config for analytics ganglia aggregators [operations/puppet] - 10https://gerrit.wikimedia.org/r/100486 (owner: 10Ottomata) [22:34:43] paravoid, we should be able to upgrade the parsoid cluster nodes without a node_modules upgrade [22:35:46] to be sure it would be good to depool one node, upgrade it, check, and then re-pool it & upgrade the other nodes in a staggered manner if everything is fine [22:37:11] let me upgrade node on betalabs too [22:37:59] (03PS1) 10Faidon Liambotis: parsoid: nodejs ensure s/latest/present/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/100489 [22:39:02] (03CR) 10Faidon Liambotis: [C: 032] parsoid: nodejs ensure s/latest/present/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/100489 (owner: 10Faidon Liambotis) [22:40:31] all good after upgrade in betalabs [22:43:59] !log including new c-ares, libv8 & nodejs 0.10 in apt [22:44:16] Logged the message, Master [22:47:14] gwicke: I'm gonna do wtp1001 first [22:47:46] !log moving bits-lb.esams from amslvs1 to amslvs3 [22:48:02] Logged the message, Mistress of the network gear. [22:49:12] bah. it is possible to run batch via a runner. it's just undocumented [22:49:20] paravoid: k, will test against that [22:51:06] ab -v9 http://wtp1001:8000/enwiki/Foo?oldid=556366262 looks good [22:51:33] PROBLEM - DPKG on wtp1002 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [22:51:49] I haven't done it yet :) [22:52:09] ah, k ;) [22:52:33] RECOVERY - DPKG on wtp1002 is OK: All packages OK [22:53:09] how about now? [22:53:58] looks good [22:56:13] timings are about the same [22:56:22] uhm... [22:56:27] http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&h=wtp1001.eqiad.wmnet&m=cpu_report&s=descending&mc=2&g=cpu_report&c=Parsoid+eqiad [22:57:04] normally it takes some time for LVS to give it some load again [22:57:13] PROBLEM - DPKG on wtp1003 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [22:57:23] PROBLEM - DPKG on wtp1005 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [22:57:33] PROBLEM - DPKG on wtp1002 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [22:57:43] PROBLEM - DPKG on wtp1004 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [22:58:03] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [22:58:13] PROBLEM - DPKG on wtp1006 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [22:58:13] RECOVERY - DPKG on wtp1003 is OK: All packages OK [22:58:17] (i'm upgrading multiple at the same time but not restarting parsoid on all of them together) [22:58:20] (03PS1) 10Dzahn: sync some live hack changes, changes to extinfo, and functions.php that weren't reflected in the package [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/100494 [22:58:33] RECOVERY - DPKG on wtp1002 is OK: All packages OK [22:58:43] RECOVERY - DPKG on wtp1004 is OK: All packages OK [22:58:53] doesn't the node package restart its services on upgrade? [22:58:58] !log csteipp synchronized php-1.23wmf5/extensions/CentralAuth 'Update CentralAuth to master for ops' [22:59:13] Logged the message, Master [22:59:13] paravoid: ^ [22:59:21] yay [22:59:23] RECOVERY - DPKG on wtp1005 is OK: All packages OK [22:59:46] I don't see a huge difference tbh :) [23:00:10] /wiki/Special:CentralAutoLogin/createSession?gu_id=0&type=script&proto=http is first [23:00:13] RECOVERY - DPKG on wtp1006 is OK: All packages OK [23:00:17] on backend requests [23:00:20] Ryan_Lane, node knows nothing about parsoid [23:00:25] ah, ok [23:00:28] /wiki/Special:CentralAutoLogin/checkLoggedIn?type=script&wikiid=enwiki&proto=http is second, /wiki/Special:CentralAutoLogin/checkLoggedIn?type=script&wikiid=eswiki&proto=http is third [23:00:51] (03PS2) 10Dzahn: sync some live hack changes, changes to extinfo, and functions.php that weren't reflected in the package [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/100494 [23:00:54] and api.php is fourth, but that's because gwicke hasn't tweaked the new proxy setting yet ;-) [23:01:43] !log rolling upgrade of nodejs to 0.10 on all wtp* boxes [23:01:45] paravoid, the apiProxyURI setting does work locally btw, but I'm not sure if that is only true because I'm not using vhosts [23:02:00] Logged the message, Master [23:02:10] it definitely does not work against the text varnishes [23:02:41] I suspect it would have worked against squid [23:02:49] * AaronSchulz wonders where ori-l is [23:03:03] if you're trying from betalabs [23:03:08] it won't work with a production api endpoint [23:03:22] because you're probably requesting a Host: that doesn't exist in production? [23:03:52] node is sending a proxy request with a full URL [23:04:03] GET http://domain/path [23:04:05] yes [23:04:29] not sure whether the Host is set, let me try [23:04:49] 1001-1006 done btw [23:04:54] I'll leave it like that for 5' [23:04:59] paravoid: /createSession is first? Really? That should not be happening... [23:06:00] 6392.37 RxURL /w/index.php?title=MediaWiki:Wikiminiatlas.js&action=raw&ctype=text/javascript&smaxage=21600&maxage=86400 [23:06:04] 6133.61 RxURL /wiki/Special:CentralAutoLogin/createSession?gu_id=0&type=script&proto=http [23:06:07] 4772.53 RxURL /w/api.php [23:06:09] on frontends [23:06:12] that's 1/8, per minute [23:06:25] so about 50k/min [23:06:41] That looks right. /createSession is what all anons call. [23:06:49] Sorry, no [23:06:59] Wow, that does not look right.... [23:07:56] /checkLoggedIn for enwiki is immediately after [23:08:02] Yay WikiMiniAtlas ^_^ [23:08:06] paravoid, the host header is set [23:08:25] RoanKattouw: that's frontend [23:08:29] paravoid: Are those Wikiminiatlas requests actually served with Cache-Control headers corresponding to the smaxage and maxage params in the query string? [23:08:49] Right, frontendss [23:09:06] but if you look at backends [23:09:09] I think you might cry [23:09:15] the amount of gadgets... [23:09:22] actually [23:09:24] you have root :) [23:09:29] * AaronSchulz wonders where ori-l is [23:09:33] ^ AaronSchulz, hm? [23:09:42] RoanKattouw: log into e.g. cp1065 [23:09:48] and run "varnishtop -n frontend -i TxURL" [23:10:10] remove the '-n frontend' if you want to see backend (a shard of the cache, typically) [23:10:23] PROBLEM - DPKG on wtp1008 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:10:33] PROBLEM - DPKG on wtp1010 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:10:35] ori-l: IRL is better...more puppet :) [23:10:38] and RxURL instead of TxURL if you want to see client requests [23:10:43] PROBLEM - DPKG on wtp1009 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:10:53] PROBLEM - DPKG on wtp1007 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:11:03] PROBLEM - DPKG on wtp1012 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:11:13] PROBLEM - DPKG on wtp1011 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:11:20] gwicke: how's parsoid looking? [23:11:28] http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&h=wtp1001.eqiad.wmnet&m=cpu_report&s=descending&mc=2&g=cpu_report&c=Parsoid+eqiad [23:11:36] is new node/libv8 really _that_ better? [23:11:42] http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&h=wtp1002.eqiad.wmnet&m=cpu_report&s=descending&mc=2&g=cpu_report&c=Parsoid+eqiad [23:11:55] (03CR) 10Dzahn: [C: 032] sync some live hack changes, changes to extinfo, and functions.php that weren't reflected in the package [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/100494 (owner: 10Dzahn) [23:12:03] RECOVERY - DPKG on wtp1012 is OK: All packages OK [23:12:14] RECOVERY - DPKG on wtp1011 is OK: All packages OK [23:12:23] RECOVERY - DPKG on wtp1008 is OK: All packages OK [23:12:33] RECOVERY - DPKG on wtp1010 is OK: All packages OK [23:12:44] RECOVERY - DPKG on wtp1009 is OK: All packages OK [23:12:53] RECOVERY - DPKG on wtp1007 is OK: All packages OK [23:12:58] stupid check [23:14:09] it's more or less just "dpkg -l | grep -v ^ii" [23:14:12] hehe [23:14:33] eh anything not ii or rc [23:15:13] PROBLEM - DPKG on wtp1017 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:15:23] PROBLEM - DPKG on wtp1013 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:15:33] PROBLEM - DPKG on wtp1018 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:15:43] PROBLEM - DPKG on wtp1015 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:15:43] PROBLEM - DPKG on wtp1016 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:15:44] PROBLEM - DPKG on wtp1014 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:16:43] RECOVERY - DPKG on wtp1014 is OK: All packages OK [23:17:16] man, deploy-info needs to be rewritten something fierce [23:17:23] RECOVERY - DPKG on wtp1013 is OK: All packages OK [23:17:33] RECOVERY - DPKG on wtp1018 is OK: All packages OK [23:17:36] paravoid: https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Parsoid%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1386631027&g=cpu_report&z=large [23:17:39] not much of a change [23:17:43] RECOVERY - DPKG on wtp1015 is OK: All packages OK [23:17:43] RECOVERY - DPKG on wtp1016 is OK: All packages OK [23:17:56] you can tell I wrote it in an hour during the sprint when we were first trying to deploy MW with git-deploy [23:18:13] RECOVERY - DPKG on wtp1017 is OK: All packages OK [23:18:19] mwalker: I'm done for now [23:19:08] paravoid: https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=Parsoid%2520eqiad&tab=m&vn= suggests that LVS balancing does not yet send much traffic to the restarted machines [23:19:29] hm, i think I know what's going on [23:19:33] varnish has kept connections open [23:19:44] pipelining [23:20:11] and those need a while to time out? [23:20:37] while counting towards some LVS stats I guess [23:22:43] PROBLEM - DPKG on wtp1024 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:22:43] PROBLEM - DPKG on wtp1022 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:22:44] PROBLEM - DPKG on wtp1023 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:22:53] PROBLEM - DPKG on wtp1019 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:22:53] PROBLEM - DPKG on wtp1020 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:23:03] PROBLEM - DPKG on wtp1021 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [23:24:03] RECOVERY - DPKG on wtp1021 is OK: All packages OK [23:24:43] RECOVERY - DPKG on wtp1024 is OK: All packages OK [23:24:43] RECOVERY - DPKG on wtp1022 is OK: All packages OK [23:24:43] RECOVERY - DPKG on wtp1023 is OK: All packages OK [23:24:53] RECOVERY - DPKG on wtp1019 is OK: All packages OK [23:24:53] RECOVERY - DPKG on wtp1020 is OK: All packages OK [23:27:04] paravoid, I looked a bit more into the proxy issue: when I set the proxy to localhost but request a page from enwiki, I get a Host of en.wikipedia.org and 'POST http://en.wikipedia.org/w/api.php' [23:27:12] that should be fine [23:27:21] yeah, should [23:27:36] could we depool one of the prod machines and test it? [23:28:04] am not comfortable with pushing it to all machines without any successful testing [23:30:26] sure [23:30:29] let me finish with the node upgrade [23:35:20] gwicke: all 24 parsoid runners are now at node 0.10 [23:35:32] awesome [23:35:33] gwicke: and I've depooled wtp1001 [23:35:45] the first restarted nodes are getting traffic now [23:35:51] (03PS1) 10Lcarr: removing asher's access per his request [operations/puppet] - 10https://gerrit.wikimedia.org/r/100504 [23:35:58] :( [23:36:57] yeah, just saw [23:38:07] (03CR) 10Lcarr: [C: 032] removing asher's access per his request [operations/puppet] - 10https://gerrit.wikimedia.org/r/100504 (owner: 10Lcarr) [23:39:30] paravoid, which is the api cluster LVS name or IP we should use? [23:39:49] api.svc.eqiad.wmnet [23:39:57] k [23:40:25] didn't find it with 'api cluster' or 'application server cluster' on wikitech [23:40:55] !log mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Reverting collection (live) to b8a4d14eaca507022b011b979bfabe58bbc82361 from a4f97e428d32f4e6bf5adf88d628df9e49b4f075 to establish good revision' [23:41:12] Logged the message, Master [23:41:20] site.pp is probably the better place to look [23:42:35] and lvs.pp [23:43:54] csteipp: any luck with /createSession? [23:44:40] I also see on no5 /w/index.php?title=Spezial:Zentrale_automatische_Anmeldung/createSession&gu_id=0&type=script&proto=http so I guess https://gerrit.wikimedia.org/r/#/c/98461/ didn't work [23:45:23] !log mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), now on 488d8daad500f4d9e9b400b6568ba170b54b0a70' [23:45:36] Logged the message, Master [23:46:13] paravoid, since I can't restart parsoid on wtp1001 it would be easier if you could test parsoidConfig.apiProxyURI = 'http://api.svc.eqiad.wmnet:80'; [23:46:36] sure [23:47:27] probably better to use the IP, btw [23:47:40] !log mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), last good; now on afec13ba4aef2f3f79cae0195744656833c5a1a7' [23:47:52] depends on whether that ip might change [23:47:58] Logged the message, Master [23:48:04] it works :) [23:48:23] sweet! [23:48:50] I even tcpdumped to make sure the traffic goes there [23:48:56] can you revert localsettings to the old state so that I can push out the change with git-deploy? [23:49:25] parsoidConfig.apiProxyURI = 'http://10.2.2.22/'; [23:49:28] this is what I added [23:49:32] right after parsoidConfig.parsoidCacheURI = 'http://10.2.2.29/'; [23:49:40] and I just reverted [23:50:04] !log mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), last bad; now on 93282cb9721316ee427c226b8f5c98274e79f4f0' [23:50:20] Logged the message, Master [23:51:09] !log updated Parsoid config to use the API service directly [23:51:23] Logged the message, Master [23:51:40] !log mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), last bad; now on 484305e9ffb19d96155d3957240d74ff92992bb9' [23:51:54] Logged the message, Master [23:51:54] it looks like the implicit node restart is still active [23:53:06] did some VE testing, all is well [23:53:14] !log mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection (live), last bad; now on 734de04a62a669022b3e7883301630f1e7947e78' [23:53:31] Logged the message, Master [23:54:30] paravoid: thanks for your help! [23:54:37] !log bad collection commit is 734de04a62a669022b3e7883301630f1e7947e78 -- now the only thing left is to figure out why! [23:54:53] Logged the message, Master [23:55:34] thank you :-) [23:56:26] paravoid, gwicke: am I reading the scroll back correctly? we now are on Node 0.10 in production? [23:56:27] !log mwalker synchronized php-1.23wmf6/extensions/Collection/ 'Bisecting collection; found bad commit -- restoring to known bad until I figure out the correct way to pin this at the good commits or find the fix' [23:56:36] mwalker: Are you done bisecting things in production? I have a deployment window in 4 minutes :) [23:56:44] Logged the message, Master [23:56:47] mwalker: yes [23:56:53] RoanKattouw: yep yep; just finished breaking things [23:56:56] paravoid: awesome! [23:57:02] Sweet [23:57:04] ori-l: let's upgrade statsd too while we're at it [23:57:14] i.e apt-get dist-upgrade & restart it [23:57:33] and jenkins (for jsduck) would be the other one [23:57:47] anything else I miss? [23:57:55] I guess I could salt a dpkg-query