[00:02:51] Hey... I want to set a "new" configuration var (for the translate ext. on wikidatawiki), where and how should I do this? In InitialiseSettings.php? [00:08:13] Reedy / anyone : last time I deployed sync_dir prompted me for a login on machine spence. This time scap reports rsync "Permission denied" errors rsync'ing to spence. spence is "Miscellaneous pmtpa" and "SSH Nagios" according to wikitech, so I believe it's not serious [00:08:54] Yeah, it's not running mw directly... [00:09:22] Probably should get someone to look at it.. [00:09:43] New patchset: Reedy; "Set $wgTranslateRcFilterDefault for wikidatawiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31354 [00:09:50] Should I enter an RT ticket? [00:10:05] hoo: ^ like taht [00:10:07] Might aswell [00:10:21] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31354 [00:10:33] ah :) [00:12:14] !log reedy synchronized wmf-config/ [00:12:20] Logged the message, Master [00:13:54] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [00:19:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:29:23] !log spage Finished syncing Wikimedia installation... : PostEdit i18n, E3 ACUX updates, fixing Cross site reqest problem in CentralNotice [00:29:29] Logged the message, Master [00:31:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.123 seconds [01:03:41] !log pgehres synchronized php-1.21wmf2/extensions/CentralNotice/ 'Updating CentralNotice, fixing Bug 41632' [01:03:45] Logged the message, Master [01:05:48] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [01:06:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:39] Change abandoned: J?r?mie Roquet; "Dup of I4066878c." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31165 [01:15:51] PROBLEM - Puppet freshness on ms-fe3 is CRITICAL: Puppet has not run in the last 10 hours [01:19:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.029 seconds [01:40:28] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 293 seconds [01:45:08] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [01:45:08] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [01:45:08] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [01:47:04] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 6 seconds [01:53:13] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:17] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 264 seconds [02:07:55] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.025 seconds [02:26:03] !log LocalisationUpdate completed (1.21wmf3) at Fri Nov 2 02:26:03 UTC 2012 [02:47:29] !log LocalisationUpdate completed (1.21wmf2) at Fri Nov 2 02:47:29 UTC 2012 [02:54:05] !log tstarling synchronized php-1.21wmf3/extensions/CentralAuth [02:58:10] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [03:30:53] RECOVERY - Puppet freshness on db51 is OK: puppet ran at Fri Nov 2 03:30:47 UTC 2012 [03:56:13] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 11 seconds [05:59:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:03:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.543 seconds [06:39:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:46] !log some files on wikitech (old backups etc) moved to tridge in /data/wikitech-nov2012 to give us a couple more gb of room [06:40:54] Logged the message, Master [06:50:53] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.033 seconds [07:22:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:39:02] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.027 seconds [08:05:20] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [08:05:20] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [08:05:20] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [08:12:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:15:30] New review: Nikerabbit; "Not really my area, but seems okay and not alter the behavior of tm solr." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/29827 [08:25:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.090 seconds [09:00:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:13] mark: I don't find ports allocated for ms-fe300x (or ms-be300x) anywhere, though I see you have ip addresses in dns. what switch are they on and what subnet should I put them in? [09:07:31] or did I just not look at the right switch? (I checked the ones in the same rack) [09:13:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.549 seconds [09:17:32] nm about the subnet, that's obvious form the ip. but which switch are they on? [09:17:35] *from [09:42:41] i'm at the ceph workshop [09:43:40] ah [09:43:46] hope you get lots of good info [09:43:47] and I want to install ceph on those servers ;) [09:44:13] I was going to do a vanilla base install on one [09:44:20] they have aggregated links [09:45:48] well I tried pinging the mgmt console on ms-fe3001 and got no answer [09:46:07] why do you want to install that? [09:46:28] faidon asked me to do a base install on one of them as prep for when our 720xds come in for swift [09:46:40] but that's not a 720XD [09:46:57] do we not have a bunch of 720xds? [09:46:58] in esams? [09:47:00] sure [09:47:02] ms-be* [09:47:13] bleah [09:47:13] ok well [09:47:20] let me re-ask all those questions about ms-be3* then :-D [09:47:44] (I looked for those too) [09:48:00] that's weird [09:48:03] because it's pinging fine for me :-D [09:48:17] grrrr [09:48:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:33] ms-be3001.mgmt.esams.wmnet right? [09:48:36] sure [09:49:05] --- ms-be3001.mgmt.esams.wmnet ping statistics --- [09:49:05] 10 packets transmitted, 0 received, 100% packet loss, time 9001ms [09:49:08] from bast1001 [09:49:13] isn't there a big pool of water between where you're pinging from and ms-be3001? ;-) [09:50:13] no bastion host in esams, where do you go to get on mgmt then? (shows how seldom I look at those boxes) [09:50:16] from the bastion host in esams [09:50:31] which is? [09:50:36] hooft.esams.wikimedia.org [09:50:40] or any other esams server really! [09:50:46] hooft [09:50:50] * apergos makes a note [09:51:34] ok, so do you mind if I do a base install for ms-be3001 then? no extra puppet, no nothing [09:51:41] yeah that's fine [09:51:50] as long as it's available next week or so [09:51:50] obviously you can reinstall over it if that's easier later [09:51:55] yep, not plannin gto put anything on it [09:52:06] go ahead [09:52:06] thanks [09:52:29] enjoy your conference, sorry for the time waste [09:52:39] np [10:03:05] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.039 seconds [10:15:20] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [10:35:59] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:50:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.038 seconds [11:07:24] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [11:17:18] PROBLEM - Puppet freshness on ms-fe3 is CRITICAL: Puppet has not run in the last 10 hours [11:23:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:33:11] mark: around? [11:36:38] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.011 seconds [11:46:23] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [11:46:23] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [11:46:23] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [12:11:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:51] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.516 seconds [12:50:06] paravoid: hello :-] continuous integration would need the php xdiff extension to be packaged. Should I just assign the bug report to you? [12:50:37] erm [12:50:38] I'd be happy to do so but [12:50:46] you should at least try first to do it yourself :) [12:50:53] haha [12:51:10] debstack does not scale [12:51:26] I mean, I don't mind doing it, but you don't want me to be on the critical path for everything debian-related [12:51:30] yeah, I don't think I'll scale that well [12:53:27] paravoid: I must agree :-] [12:53:58] my main issue is that I lack a simple tutorial [12:54:12] though for the pecl extension there is probably a good one floating around [12:54:34] and of course I can't use reprepo to put the package on apt.wm.org ;-D [12:59:20] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [12:59:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:27] !log temporarily deactivating IPv6 peering with tele2 on cr1-eqiad due to routing issues [13:09:34] Logged the message, Master [13:15:57] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.032 seconds [13:47:49] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:02] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [14:01:04] ppoooor memcached died again [14:01:05] on virt0 :( [14:02:26] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.002 second response time on port 11000 [14:02:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.042 seconds [14:02:32] thanks :D [14:15:17] New patchset: Alex Monk; "(bug 41690) Reclose wikimania2010wiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31379 [14:17:20] New review: Alex Monk; "Reverted in Ie4634d14 per bug 41690" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23287 [14:28:19] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30785 [14:35:58] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:35:58] New patchset: Reedy; "Bah, stupid load order" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31382 [14:36:15] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31382 [14:39:07] !log reedy synchronized wmf-config/ [14:39:16] Logged the message, Master [14:50:03] sbernardin: can you go to storage3 and see if the disk finished wiping plz [14:52:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.108 seconds [14:52:36] cmjohnson1: yes it is done wiping... [14:52:50] New patchset: ArielGlenn; "add ms-be3xxx to netboot" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31384 [14:53:34] New review: Mark Bergsma; "you're missing a 0" [operations/puppet] (production); V: 0 C: -2; - https://gerrit.wikimedia.org/r/31384 [14:53:50] k..thx [14:55:18] New patchset: ArielGlenn; "add ms-be3xxx to netboot" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31384 [15:03:23] sbernardin: can you plug the network (blue) cable into storage3 for me...goes into the 1st ethernet port [15:05:13] cmjohnson1: done.. [15:05:19] cool. thx [15:06:51] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31384 [15:11:53] RECOVERY - Host storage3 is UP: PING OK - Packet loss = 0%, RTA = 0.57 ms [15:23:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:05] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.872 seconds [15:50:58] PROBLEM - Host storage3 is DOWN: PING CRITICAL - Packet loss = 100% [16:12:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:18:17] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31379 [16:19:43] !log maxsem synchronized closed.dblist 'https://gerrit.wikimedia.org/r/#/c/31379/' [16:19:50] Logged the message, Master [16:23:54] !log maxsem synchronized wmf-config/InitialiseSettings.php 'Close wikimania2010wiki' [16:23:57] Logged the message, Master [16:24:04] https://gerrit.wikimedia.org/r/#/q/status:open+project:operations/mediawiki-config,n,z – mine is the oldest, I see [16:25:40] so many changes [16:27:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.031 seconds [16:29:29] Nemo_bis, no, mine's even older:P [16:29:59] MaxSem: but it has a -2! [16:30:01] by myself [16:30:04] :p [16:30:17] if we include abandoned changes I have an older one I think [16:30:32] in principle, nothing preve nts me from deploying it right now [16:31:06] the oldest is https://gerrit.wikimedia.org/r/#/c/6997/ [16:59:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:59:24] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30954 [17:00:19] !log reedy synchronized wmf-config/CommonSettings.php [17:00:25] Logged the message, Master [17:04:35] New patchset: J; "enable TMH encoding on test2wiki not test2" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31403 [17:06:33] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31403 [17:07:15] !log reedy synchronized wmf-config/CommonSettings.php [17:07:15] Logged the message, Master [17:11:39] well the ms-be3001 install is "done" which means no puppet run (I wonder what puppet host can talk to it anyways), only a bare base install with nothing else. didn't see any issues, but [17:12:11] cool! [17:12:14] waiting to hear the "but" :) [17:12:15] that's not surprising since I have borrowed from someone or other, maybe asher, one of these, maybe even two of these, and there was no problem [17:12:51] no swift, no nothing on here as you understand. it's just another dell box [17:13:00] yeah [17:13:04] what about jbod? [17:13:06] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.098 seconds [17:13:07] all is well? [17:13:16] I remember going through the "convert to no raid" for these before on some other box [17:13:23] anyways it's the same deal, you just set that in the raid config [17:13:51] I could I guess partition and mkfs these by hand but I've already done that on one of these [17:13:53] seems pointless to do it again [17:14:24] h310, very standard, very easy [17:14:25] great [17:14:40] I do wish I knew how puppet to esams works though [17:15:00] there's no connection to sockpuppet, so you have to copy keys manually [17:15:14] * apergos makes a note [17:15:16] I noticed there was no connction, yup [17:15:21] stafford goes through brewster's haproxy, but there's puppet.esams.wikimedia.org pointed there [17:15:33] so puppet autodiscovers that [17:16:23] ok [17:16:58] so the new box is due mon or most likely tues [17:17:02] yeah I read my mails [17:17:17] I guess you'll only need a few hours until we give them the okay [17:17:20] to send us the rest [17:17:21] right? [17:17:32] should be [17:17:34] as soon as chris racks it [17:17:40] assuming I'm awake when that happens [17:18:24] this is going to have SSDs too [17:18:24] all we want to do is get a full swift insall on it and make sure that copies to disk don't explode, [17:18:47] not worrying about actually putting it in production on the same day are we? [17:18:57] I'm not [17:19:02] before giving them the go ahead [17:19:04] ok fine [17:19:14] I'm more worried about giving the go-ahead to Dell to start shipping the rest [17:19:14] yeah [17:20:02] hmm I might go see a movie or something radical like that tonight [17:20:10] (skyfall) [17:20:19] need mindless entertainment [17:20:19] haha I didn't have you for a skyfall type [17:20:53] eh not every day. but once in a while you want something goofy, with action, not too serious [17:21:10] yeah same here [17:21:12] and the reviews have been decent anyways [17:21:48] it's been a very stressful couple weeks, so this will be a nice little break [17:22:09] (this is me getting ready to say: afk, mostly not around this weekend but I'll check in :-P) [17:23:12] don't worry! [17:23:31] New patchset: Jgreen; "adjusting udp2log filters for banners" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31408 [17:31:11] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31408 [17:31:51] ok, going to go me mindless, see folks later [17:31:59] have a nice weekend [17:32:16] you too [17:42:33] RobH: whats our update for https://rt.wikimedia.org/Ticket/Display.html?id=3738 ? [17:42:36] woosters: --^ [17:48:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.440 seconds [18:06:30] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [18:06:30] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [18:06:30] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [18:08:38] New patchset: Jgreen; "add fundraising archive to db78 since it's the pmtpa host with space" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31415 [18:09:43] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31415 [18:34:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:38:37] !log aaron synchronized php-1.21wmf2/thumb.php 'deployed 0b7378c966ead76422d03fb0c663fa714ad8a48a' [18:38:47] Logged the message, Master [18:39:58] !log aaron synchronized php-1.21wmf3/thumb.php 'deployed a16d25a721b502f83f6d9354d482f8f2bc08176b' [18:40:03] Logged the message, Master [18:49:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.031 seconds [18:52:13] New patchset: Cmjohnson; "adding storage3 to netboot" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31420 [19:09:47] paravoid: are you going to add a mount point to take the place of /mnt/thumbs? [19:22:57] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:36:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.237 seconds [19:49:26] !log srv266 troubleshooting cpu [19:49:33] Logged the message, Master [20:01:48] RECOVERY - Host srv266 is UP: PING OK - Packet loss = 0%, RTA = 2.04 ms [20:09:46] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31252 [20:11:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:25] paravoid: can you +2 this for me please [20:15:28] https://gerrit.wikimedia.org/r/31420 [20:16:04] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [20:22:21] PROBLEM - Host srv266 is DOWN: PING CRITICAL - Packet loss = 100% [20:24:49] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.891 seconds [20:32:25] binasher: why does your nick start with "bin" btw? [20:33:18] He's executable [20:33:26] heh [20:35:33] mutante: can you look at this https://gerrit.wikimedia.org/r/31420 [20:41:54] Damianz: ++ [20:50:36] New patchset: Dereckson; "(bug 41563) Enable Collection on ur.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31556 [21:00:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:08:18] !log reedy synchronized php-1.21wmf3/extensions/ProofreadPage [21:08:19] Logged the message, Master [21:08:34] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [21:10:32] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30319 [21:14:55] New patchset: Reedy; "Add Ex:MergeUser and Ex:GeoCrumbs to Wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30310 [21:15:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.022 seconds [21:16:33] New patchset: Reedy; "Add Ex:MergeUser and Ex:GeoCrumbs to Wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30310 [21:17:38] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30310 [21:18:30] PROBLEM - Puppet freshness on ms-fe3 is CRITICAL: Puppet has not run in the last 10 hours [21:19:02] New review: Reedy; "Need to remember to create the docroot.." [operations/apache-config] (master) C: 1; - https://gerrit.wikimedia.org/r/30182 [21:21:47] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31420 [21:24:16] !log authdns-update, adding new zone for wikivoyage-old [21:24:21] Logged the message, Master [21:39:16] New patchset: Reedy; "Deprecated: Use of wfGetIP was deprecated in MediaWiki 1.19." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/27404 [21:47:37] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [21:47:37] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [21:47:37] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [21:47:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:24] New patchset: Nemo bis; "(bug 32411) Transwiki import to multilingual wikisource broken" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31562 [21:50:16] New patchset: Nemo bis; "(bug 32411) Transwiki import to multilingual wikisource broken" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31562 [21:51:04] silly kwrite, should never use it [21:51:35] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/30182 [21:55:14] dzahn is doing a graceful restart of all apaches [21:55:29] !log adding apache config for wikivoyage [21:55:29] !log dzahn gracefulled all apaches [21:55:31] Logged the message, Master [21:55:35] Logged the message, Master [21:57:04] New review: Hashar; "few comments." [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/28627 [22:01:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.751 seconds [22:13:29] New patchset: Reedy; "Add Wikivoyage related extensions" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31567 [22:13:40] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31567 [22:20:52] New patchset: Cmjohnson; "adding labsdb1 and 2 to netboot.cfg" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31568 [22:20:57] New patchset: Dereckson; "Collection extension configuration for ur. wikis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31556 [22:25:52] New review: Dereckson; "PS2: ODF as default format" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/31556 [22:28:11] RECOVERY - Host storage3 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [22:32:51] New patchset: Nemo bis; "(bug 40212) Mass update Wiktionary favicons" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31570 [22:32:56] Change abandoned: Cmjohnson; "will not amend" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31568 [22:35:13] New patchset: Hashar; "Unit testing for InitialiseSettings.php (WIP - DO NOT MERGE)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/28627 [22:35:59] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:16] New patchset: Hashar; "experimental tests" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31575 [22:41:50] PROBLEM - Host storage3 is DOWN: PING CRITICAL - Packet loss = 100% [22:47:50] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.384 seconds [22:53:36] New patchset: Hashar; "lookup dblist relative to $wmfConfigDir" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31579 [23:00:26] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [23:04:08] New patchset: Dereckson; "(bug 41712) he.wiki images size configuration" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31580 [23:04:50] New patchset: Hashar; "allow injection of variables in CommonSettings.php" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31582 [23:11:37] New patchset: Hashar; "include/require extensions only once" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31583 [23:12:42] New patchset: Hashar; "lookup dblist relative to $wmfConfigDir" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31579 [23:14:37] New patchset: Hashar; "lookup dblist relative to $wmfConfigDir" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31579 [23:19:25] New patchset: Nemo bis; "(bug 40212) Mass update Wiktionary favicons" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31584 [23:22:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:29:12] binasher: mysqlbinlog has a -v option, which seems like it would make looking at a row based replication log tolerable [23:39:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.020 seconds [23:40:07] New patchset: Nemo bis; "(bug 41717) Update default (language-neutral) Wiktionary logo" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31585 [23:45:03] !log reedy synchronized php-1.21wmf3/extensions/ 'Dark deploy wikivoyage extensions' [23:45:11] Logged the message, Master [23:45:16] hey binasher, if you have a sec, can you confirm the command line of the varnishncsa invocation that you have on bits? i'm getting duplicate events and i worry that it's emitting multiple events per request because we're not filtering by varnish log tag. (shouldn't be an issue with varnishncsa, but....trying to eliminate possibilities.) [23:45:18] AaronSchulz: it's not good enough in 5.1 though [23:46:40] AaronSchulz: https://kb.askmonty.org/en/annotate_rows_log_event/ -- why i want to wait til mariadb [23:50:25] ori-l: 109 32325 1 16 Oct20 ? 2-08:17:17 /usr/bin/varnishncsa -n strontium -w 10.64.21.123:8422 -m RxURL:^/event\.gif -D -P /var/run/varnishncsa/varnishncsa-vanadium.pid -F %q %l %n %t