[00:14:24] PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours [02:04:17] !log LocalisationUpdate completed (1.22wmf1) at Sun Apr 14 02:04:17 UTC 2013 [02:25:53] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [03:36:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [03:38:46] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [03:43:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [03:55:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:02:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.145 second response time [04:17:31] flappy flappy! [05:14:06] PROBLEM - RAID on professor is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:14:16] PROBLEM - Disk space on professor is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:14:36] PROBLEM - profiling collector on professor is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:14:46] PROBLEM - profiler-to-carbon on professor is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:15:06] PROBLEM - DPKG on professor is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:16:06] RECOVERY - Disk space on professor is OK: DISK OK [05:16:26] RECOVERY - profiling collector on professor is OK: PROCS OK: 1 process with command name collector [05:16:36] RECOVERY - profiler-to-carbon on professor is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/udpprofile/sbin/profiler-to-carbon [05:16:56] RECOVERY - DPKG on professor is OK: All packages OK [05:18:46] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [06:12:40] PROBLEM - Puppet freshness on search1015 is CRITICAL: No successful Puppet run in the last 10 hours [06:14:40] PROBLEM - Puppet freshness on search1016 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:08] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:08] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:08] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [08:12:59] PROBLEM - Puppet freshness on gallium is CRITICAL: No successful Puppet run in the last 10 hours [08:42:45] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [10:15:08] PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours [11:58:55] ooo [11:59:06] etherpad.wmflabs.org just died. [12:17:50] works for me odder? [12:26:08] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [12:29:41] Thehelpfulone: yes, it started working again after three or so minutes [12:45:58] odder: yes it's very unstable and also extremely slow for pads with ~700 lines or more [13:39:10] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [13:52:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [14:01:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [14:43:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [14:46:34] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [15:03:34] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [15:19:03] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [15:22:33] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [15:27:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [15:32:33] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [15:52:23] apergos: ping [15:54:34] i hope Vito realizes it's the weekend! [15:54:44] lol [15:55:02] late weekend now :D [15:55:10] not really [15:55:28] anyway, whatchya need? if it's not an apergos only thing [15:55:30] for both me and apergos it's Sunday afternoon [15:55:36] yeah, yeah [15:55:43] noon's not quite here yet [15:55:57] so you can help me :p [15:56:25] maybe, maybe not :) [15:56:33] I need to withelist a certain IP from ratelimit since on tomorrow there will be a Wiki-related event [15:56:39] with a bunch of account creation [15:56:55] *s [15:57:27] yeah, definitely not a good case to ping just one person... [15:57:30] did you file a bug? [15:57:35] what event? [15:57:53] not yet jeremyb_ [15:58:16] I wondered about having my issue resolved without opening a bug :D [15:58:26] well speak... [15:59:53] 159.213.138.5 [16:00:21] it's a GLAM-related event with Tuscany's public libraries [16:00:58] details, details. how many people, what hours, event page [16:01:05] don't make us pull teeth... [16:01:25] will you have sysops on site? [16:01:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:33] bbiab [16:02:06] jeremyb_: ehm, I'm a stewie [16:02:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [16:02:52] the event will be managed by some WMI's guys definitely trustworthy [16:03:08] Vito, one day for this isn't really good enough warning :/ [16:03:13] which wiki? enwiki? [16:03:19] itwiki [16:03:30] I was asked just an hour ago [16:03:46] https://meta.wikimedia.org/wiki/Mass_account_creation [16:05:01] I know Thehelpfulone [16:05:03] I know [16:05:13] and considering the short time I choose the second option [16:06:24] Vito: doesn't piero_tasso have sysop? [16:07:45] nope [16:08:14] and since these people are supposed to create new accounts in the future is better to let them do it by themselves [16:08:28] otherwise I would had already granted him all the necessary rights [16:09:03] Vito, still file the bug for the sysadmin [16:09:12] these things need to be logged somewhere :) [16:09:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:28] tme to remove the line about irc on meta [16:09:38] and Tim might online later this evening (morning AU time) [16:09:41] nope, you file the bug [16:09:46] then poke the admin on IRC ;P [16:10:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [16:13:13] PROBLEM - Puppet freshness on search1015 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [16:15:13] PROBLEM - Puppet freshness on search1016 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:47] Thehelpfulone: it would be amazing to have something like mediawiki:autoblockwhitelist [16:26:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:27:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [16:28:33] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [16:28:55] Vito: as far as I can tell you've still not provided the info i requested... [16:29:08] why u make me pull your teeth out? [16:29:59] uh? [16:30:14] 14 16:00:57 < jeremyb_> details, details. how many people, what hours, event page [16:30:18] 14 16:01:25 < jeremyb_> will you have sysops on site? [16:30:29] you said you're a steward. you didn't say if you'll be there though [16:30:42] I misread what did you write actually [16:30:44] anyway [16:30:51] hours are listed, event page too [16:31:06] where????!????????????????????????????????????????? [16:31:09] there will be an ex-sysop [16:31:12] in the bug [16:31:18] where's the bug? [16:31:31] https://bugzilla.wikimedia.org/47221 [16:32:03] no one mentioned that bug num in here until Thehelpfulone just did [16:32:16] you should have. also, Thehelpfulone's right, your timing sucks [16:32:21] give us some warning [16:32:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:52] Vito, hmm, yes that could be useful [16:33:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [16:34:15] for the non-english speakers: https://en.wiktionary.org/wiki/pull_teeth#English [16:34:33] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [16:34:55] Vito: you *still* haven't said how many people [16:35:01] (3rd time i'm asking) [16:35:44] "higher" is not sufficient [16:36:00] feel free to ask the fourth and then the fifth, I won't answer until I'll get an answer from the guy who will be on site on tomorrow [16:36:12] then guess. 10, 100 or 300? [16:36:41] i'm assuming it's not less than 10 nor more than 300 [16:37:30] yep, though I'm still waiting for his answer [16:37:37] ok... [16:37:39] it seems to have been poorly managed by the Region [16:38:04] but honestly I think it's quite harmful to consider the upper bound of 300 [16:38:43] idk what that means [16:41:53] there's no room for more then 99 people [16:42:03] so 100 is definitely a safe upper limit [16:42:22] ok... [16:43:55] Vito: so, it's 12-19 local time? [16:44:21] yep 12-18+1h of margin [16:44:48] good [16:47:26] ok, will prepare something on the way to [[WP:MEET/NYC]] [16:47:33] should be able to git push from the train [16:47:40] * jeremyb_ heads off [16:50:34] good! [17:09:20] New review: Hashar; "Please check with ops :-D" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59004 [17:43:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.136 second response time [17:45:38] odder: i already patched it [17:46:22] New patchset: Jeremyb; "bug 46686 - raise throttle for itwiki GLAM event" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59074 [17:54:21] jeremyb_: very good. [17:56:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.143 second response time [17:57:56] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [17:57:56] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [17:57:56] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [17:58:43] New review: Thehelpfulone; "Bug number in the commit message is incorrect, but otherwise everything looks fine (do you need to a..." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/59074 [18:01:40] LOL [18:01:51] Thehelpfulone: good point :) [18:02:15] poor review odder ;) [18:02:44] New patchset: Odder; "(bug 47221) Raise throttle for itwiki GLAM event" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59074 [18:02:56] bah [18:03:03] I gerrit-conflicted with you. [18:03:07] Thehelpfulone: I was reviewing the code, not the commit message! [18:03:12] heh [18:03:24] but I was wondering why the bot didn't comment on the bug... [18:03:52] New review: Thehelpfulone; "Thanks." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/59074 [18:04:32] Thehelpfulone: plus, you don't need to add the topic. You can. It even looks better. [18:05:02] But overall, I don't think there is a /requirement/ to add that. [18:13:37] PROBLEM - Puppet freshness on gallium is CRITICAL: No successful Puppet run in the last 10 hours [18:43:08] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [18:52:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.143 second response time [19:58:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:59:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [20:15:19] PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours [20:39:55] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:02:55] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [22:27:05] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [22:32:27] New patchset: Krinkle; "noc: Refactor highlight.php to be simpler and ~~less~~ more secure" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59034 [22:32:28] New patchset: Krinkle; "noc: Add missing entries to createTxtFileSymlinks.sh" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59033 [22:32:52] New review: Krinkle; "Rebased" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59034 [23:35:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:37:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [23:39:15] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours