[00:00:01] New patchset: Pyoungmeister; "bookkeeping for pmtpa dbs" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60361 [00:00:26] New review: Dzahn; "just don't include it in site.pp until it's fixed" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/60359 [00:00:27] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60359 [00:01:04] !log py synchronized wmf-config/db-pmtpa.php 'bookkeeping for pmtpa slaves' [00:01:11] Logged the message, Master [00:01:42] Change merged: Pyoungmeister; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60361 [00:02:22] !log py synchronized wmf-config/db-pmtpa.php 'bookkeeping for pmtpa slaves (this time for real)' [00:02:29] Logged the message, Master [00:02:47] New patchset: Jdlrobson; "Enable CentralNotice on mobile" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60362 [00:03:54] hey Ryan_Lane -- I'm still not able to add Eric to editor-engagement "Failed to add ebernhardson to editor-engagement. This needs user ebernhardson to have the "loginviashell" right. ". It's been more than half an hour, so I don't think it's waiting on a puppet run, but dunno. [00:04:15] there's no puppet run needed for that [00:04:18] it should be immediate [00:05:01] that must not be his username [00:05:19] uuuuggghhhhhh [00:05:20] EBernhardson (WMF) [00:05:27] I fucking hate (WMF) user names [00:05:34] * Susan smiles. [00:05:37] me too, it's like a cattle brand [00:05:40] yes [00:05:45] Some people should start creating (!WMF) usernames [00:05:48] and when they leave they lose all of their contributions [00:06:11] I may add it to the username blacklist [00:06:22] for new creations [00:06:23] Heh. [00:06:27] we shouldn't be encouraging that [00:06:29] It's Philippe's doing. [00:06:38] that's perfectly fine for project wikis [00:06:45] it's total bullshit for developer accounts [00:06:53] Local username blacklists fragment SUL. [00:07:03] we aren't using SUL [00:07:13] * Reedy beats Susan [00:07:13] Okay. [00:07:13] Useless! [00:07:17] Useless! (WMF) [00:07:22] ori-l: anyway, that's your issue :) [00:07:27] when i wanted to vote down a gerrit change for non-technical reasons, i was expecting to be asked to have 2 separate gerrit accounts, one "WMF"-one and one for my private opinion that is never a statement by the foundation :p, hehe [00:07:28] that and a bad error message [00:07:42] i got two turntables and a microphone [00:07:57] hmm Ryan_Lane I seem to be locked out of my wikitech account [00:08:02] * Reedy pets Thehelpfulone [00:08:06] mutante: "I think this is a bad idea". mutante (WMF): "I still think this is a bad idea" [00:08:16] heh [00:08:32] I used a temporary password, I tried to change it, then I get a Incorrect password entered. Please try again. [00:08:32] error :-/ [00:08:50] Reedy: mutante (WMF): "I don't have a strong opinion, please show consensus link" mutante: " i hate this":) [00:08:58] Thehelpfulone: try your old one? [00:09:21] nope, doesn't seem to work [00:10:55] alt-f4 [00:11:06] Thehelpfulone: Have you tried turning it off and on again? [00:11:40] * Thehelpfulone trouts Reedy [00:12:18] I've even hit a captcha now for incorrectly trying to login too many times :P [00:14:06] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 238 seconds [00:14:09] Thehelpfulone: try sending another temporary? [00:14:37] did it accept your temporary at all? [00:14:45] I did, same problem, it allows me to login to the "change your password because you've got a temporary code" screen, but then when you enter a new password it doesn't like it [00:14:47] New patchset: Jdlrobson; "Enable CentralNotice on mobile" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60362 [00:14:49] yes it accepted the temporary one [00:15:02] it wouldn't let you set a permanent? [00:16:23] exactly [00:16:28] hm [00:16:46] PROBLEM - Puppet freshness on gallium is CRITICAL: No successful Puppet run in the last 10 hours [00:17:52] I see the MOD in LDAP [00:18:07] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 23 seconds [00:18:41] TimStarling: ping [00:18:54] hello [00:23:29] New patchset: Dzahn; "remove group.pp from jenkins module to fix duplicate definition. conflicts with systemuser in generic-definitions." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60363 [00:39:03] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [00:49:03] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [00:55:03] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 13 seconds [01:17:14] New review: awjrichards; "(2 comments)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/60362 [01:19:59] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 218 seconds [01:24:59] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 222 seconds [01:28:00] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 12 seconds [02:06:55] !log LocalisationUpdate completed (1.22wmf2) at Tue Apr 23 02:06:54 UTC 2013 [02:07:02] Logged the message, Master [02:09:16] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [02:09:16] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [02:09:17] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [02:10:33] !log LocalisationUpdate completed (1.22wmf1) at Tue Apr 23 02:10:32 UTC 2013 [02:10:40] Logged the message, Master [02:14:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 210 seconds [02:23:09] New review: MZMcBride; "I personally didn't mind 180 days." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/54505 [02:23:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 225 seconds [02:25:16] PROBLEM - Puppet freshness on ms-be9 is CRITICAL: No successful Puppet run in the last 10 hours [02:25:26] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 23 02:25:25 UTC 2013 [02:25:33] Logged the message, Master [02:25:56] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 4 seconds [02:26:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.202 second response time [02:29:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 187 seconds [02:30:56] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 27 seconds [02:31:05] New review: Tim Starling; "Ram, do you still want this?" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/55841 [02:33:36] New review: MZMcBride; "I don't want to file a separate bug for this, but please do annotate the configuration variable in C..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/54505 [02:33:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 208 seconds [02:37:12] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 7 seconds [02:43:16] New patchset: Tim Starling; "Bug: 47293 Further reduce noise in log files" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/59533 [02:43:25] Change merged: Tim Starling; [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/59533 [03:08:12] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 181 seconds [03:10:06] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [03:11:46] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [03:26:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [03:34:06] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 240 seconds [03:35:04] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [03:38:03] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 181 seconds [03:40:03] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [04:13:09] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 181 seconds [04:15:09] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [04:23:09] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 181 seconds [04:29:09] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 12 seconds [04:31:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [04:38:07] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 181 seconds [04:40:07] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds [04:44:08] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 232 seconds [04:45:07] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [05:24:12] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 240 seconds [05:25:12] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [05:38:08] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 181 seconds [05:40:08] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [05:51:41] New review: Hashar; ".Oh yeah puppet is broken on gallium currently because of that. I submitted a similar change at htt..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60363 [05:59:08] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 214 seconds [06:01:48] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:38] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [06:05:08] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [06:33:27] New patchset: ArielGlenn; "attempt to shut up gmetric job queue cronspam" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60371 [06:34:20] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60371 [06:52:40] New patchset: ArielGlenn; "Revert "attempt to shut up gmetric job queue cronspam"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60372 [06:53:46] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60372 [07:20:51] ori-l: have you had time to test it? :) [07:21:08] not yet, but you're right to call me out on it -- i'll do that next [07:21:25] ty :D [07:21:41] give me an hour or so, have to write an e-mail or three. [07:22:02] but i'll have it running tonight, promise. [07:22:17] legoktm: I think you have a legobot job tht is cronspamming [07:22:28] if you are legobot at tools-login or something like that [07:22:30] oh? [07:22:31] yeah [07:22:37] thats me [07:22:39] lemme see [07:22:45] so mail from those instances cron goes to all of ops cause.. just cause [07:23:07] so we get nice updates about $HOME/counter [07:23:27] heh [07:23:36] let me deflect the mail... [07:23:39] thanks [07:24:15] also can you pass the word around to other folks to really check their cron jobs? I think it's not general knowledge althogh I try to let the folks with the noisiest cron jobs know [07:24:33] its probably easiest if you just send a mail to labs-l ? [07:24:57] oh, huh [07:25:02] I"m not on it though [07:25:17] too much mail already... [07:25:59] * * * * * python $HOME/counter.py >> $HOME/counter.log 2>&1 [07:26:21] we shall see :-D [07:27:56] im keeping track of how fast we're creating items, eventually https://www.stathat.com/stats/JDTw will be a nice looking graph [07:28:46] ah hah [07:30:01] legoktm: how do you account for french guys with weird misunderstandings about copyright? :) [07:30:07] in the stats [07:30:15] lol :P [07:30:23] its just incremental [07:30:28] doesnt count for deletes [07:30:40] so why can't you just use dumps then? [07:30:54] i mean besides that the dumps are taking forever [07:31:09] this is instantaneous [07:31:21] and it already works :P [08:01:58] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60325 [08:02:15] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60337 [08:02:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:03:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.141 second response time [08:21:37] New patchset: Nikerabbit; "Optimize even more with optipng" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60375 [08:22:32] New review: Nikerabbit; "There are better alternatives to pngcrush. I use optipng myself. See gerrit 60375." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59638 [08:33:52] legoktm: having a bit of a hard time getting it to work :/ [08:34:02] uhoh whats not working? [08:34:49] it looks like it's entering into a busy loop waiting for events before the connection succeeds [08:35:08] i had to struggle a bit to get to that point, i'll make notes either here if you're around or on the patch [08:35:27] here is fine [08:37:12] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [08:37:13] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [08:40:52] !log maxsem synchronized wmf-config 'Beta changes, no-op for production' [08:41:00] Logged the message, Master [08:41:43] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60375 [08:42:59] New review: MaxSem; ".../bits/DolphinBrowser/image/ios/toolbar-back.png | Bin 523 -> 463 bytes" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60375 [08:46:01] !log maxsem synchronized docroot/ 'https://gerrit.wikimedia.org/r/#/c/60375/' [08:46:08] Logged the message, Master [08:51:51] oh [08:53:14] well, bot.connect() no longer attempts to make a connection [08:53:46] you removed the call to self.server.connect but didn't replace them with anything [08:53:50] ^ legoktm [08:54:13] er, i thought it was called internally [08:54:14] * legoktm checks [08:56:31] New review: Ori.livneh; "(1 comment)" [operations/debs/adminbot] (master) C: -1; - https://gerrit.wikimedia.org/r/60240 [08:57:05] i have to crash, but I think that in order to do this patch set right you need to be able to test [08:57:14] do you want my configuration notes? [08:57:46] if you have mediawiki-vagrant you should be able to get this going quite easily [08:57:47] yeah sure [08:57:52] * legoktm does :D [08:57:59] the version on wikitech is the same one that's in ubuntu precise [08:58:10] so you can simply 'sudo apt-get install python-irclib' [08:58:19] and get the exact depdendency [08:58:39] oh, another issue [08:59:28] not introduced by you: "if 'confarg' in args" will always be true [08:59:41] the *key* is present in args regardless of whether or not the argument was supplied [09:00:02] i think the intent was to do 'if args.confargs is not None:' [09:00:14] ok [09:00:46] so this means that invoking it without any command line params will simply trigger a crash [09:00:55] but you can still work around it by actually specifying the config file on the command line [09:01:14] i'm invoking it like this: python adminlogbot.py --config=/etc/adminbot/foobarlegoktmbot.py [09:01:24] i'll pastebin the content of that file in a moment [09:01:41] you'll also need to 'pip install mwclient' [09:03:22] PROBLEM - Host ms-be9 is DOWN: PING CRITICAL - Packet loss = 100% [09:03:39] this is the content of my /etc/adminbot/foobarlegoktmbot.py : [09:03:40] http://dpaste.de/AwfJ3/raw/ [09:03:59] lol :P [09:04:23] [04:04:17 AM] -NickServ- foobarlegoktmbot is not registered. :( [09:04:44] yes, fortunately (or unfortunately) the bot is not getting quite that far :P [09:05:11] oh, i had to make one other change, sec.. [09:06:22] i'll note it in the change [09:06:30] ok [09:08:00] New review: Ori.livneh; "(1 comment)" [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/60240 [09:08:19] o.O does gerrit not work on opera? [09:08:49] bleh [09:08:51] dunno, i'd imagine MaxS_em would have complained by now [09:09:04] !log [09:09:21] back to firefox :/ [09:09:40] !log restarted all services [09:09:47] Logged the message, Master [09:11:15] you might be able to do something like [09:11:23] def connect(self, *args, **kwargs): [09:11:33] !log [09:11:56] super(logbot, self).connect(*args, **kwargs) [09:12:03] what on earth is 'pp-pdf1'? [09:12:06] !log restarted all services [09:12:12] oy vey. [09:12:14] Logged the message, Master [09:13:52] anwyays i'm going to crash, but feel free to ping me at some point tomorrow if you get stuck and i'll take a look at it with you [09:14:07] ok thanks :) [09:15:40] !log [09:16:13] !log restarted all services [09:16:21] Logged the message, Master [09:19:05] o.O [09:28:16] ori-l-zzz: pediapress [09:28:40] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [09:45:50] RECOVERY - Host ms-be9 is UP: PING OK - Packet loss = 0%, RTA = 26.61 ms [09:52:30] PROBLEM - Host ms-be9 is DOWN: PING CRITICAL - Packet loss = 100% [09:54:21] New patchset: Legoktm; "Convert logbot to use ircbot.SingleServerIRCBot for auto-reconnection." [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/60240 [09:55:54] !log - alow numprocs to be configured via initenv.sh [09:56:02] Logged the message, Master [09:56:27] !log restarted all services [09:56:35] Logged the message, Master [10:00:00] RECOVERY - SSH on ms-be9 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [10:00:10] RECOVERY - Host ms-be9 is UP: PING OK - Packet loss = 0%, RTA = 26.70 ms [10:04:08] New review: Legoktm; "PS3: Fixed the major issue where the bot actually connects to the server by adding a call to ircbot...." [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/60240 [10:04:40] New review: Legoktm; "(2 comments)" [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/60240 [10:09:02] PROBLEM - Puppet freshness on nickel is CRITICAL: No successful Puppet run in the last 10 hours [10:10:08] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [10:17:02] PROBLEM - Puppet freshness on gallium is CRITICAL: No successful Puppet run in the last 10 hours [10:28:23] RECOVERY - NTP on ms-be9 is OK: NTP OK: Offset -0.02109622955 secs [10:29:51] New patchset: Legoktm; "Convert logbot to use ircbot.SingleServerIRCBot for auto-reconnection." [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/60240 [10:41:40] New review: Ori.livneh; "(6 comments)" [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/60240 [10:46:55] ori-l-zzz: you don't sleep much :o [10:47:20] the 'z' is for zorro [10:47:38] hahaha [10:47:51] :) [10:50:55] New patchset: Reedy; "Losslessly compress pngs further..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60380 [10:52:01] Reedy, lol [10:52:09] MaxSem: Moment for size changelogf [10:52:13] the competition has started! [10:52:19] New review: Reedy; "reedy@ubuntu64-web-esxi:~/git/operations/mediawiki-config$ git pull ssh://reedy@gerrit.wikimedia.org..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60380 [10:52:33] docroot/bits/DolphinBrowser/image/icon.png | Bin 8646 -> 6459 bytes [10:52:33] :D [10:52:56] do you commit from Linux? [10:53:00] Yeah [10:53:18] cba with windows for it [10:53:41] Nikerabbit: ^^ [10:54:15] from command line? TortoiseGit is vastly superior to anything else I've seen [10:57:43] Reedy: ? [10:57:54] Nikerabbit: I beat you [10:58:02] See https://gerrit.wikimedia.org/r/60380 [10:58:27] Reedy: what did you do? [10:58:37] Losslessly compressed them [10:59:24] why are we discussing this in #wikimedia-operations anyway? [10:59:24] with what? [11:00:04] Because gerrit-wm reports it in here [11:00:08] pngout [11:02:42] docroot/bits/DolphinBrowser/image/icon.png | Bin 10173 -> 8646 bytes [11:02:42] docroot/bits/DolphinBrowser/image/icon.png | Bin 8646 -> 6459 bytes [11:03:19] ^ Quite a saving [11:03:35] MaxSem: Is there somewhere the DolphinBrowser ones can be upstreamed to? [11:04:16] Reedy, as the matter of fact, the whole directory needs to be upstreamed to /dev/null [11:04:26] Haha. Really? [11:04:37] let's just wait for it to bitrot a bit [11:05:03] it's allegedly still working [11:05:15] you shouldn't touch the files in test/, since unit tests for image handling assert things about the binary content of those files [11:06:06] Jenkins should tell us if tests pass.. [11:06:38] sigh [11:07:05] I just hit Ctrl-X, y, Enter in Notepad:P [11:08:19] should we have a test which checks that images are optimized? :) [11:08:41] ori-l-zzz: Yup, I b0rked the build [11:08:41] gerrit should do it itself [11:18:19] do what? [11:20:51] compress the images [11:23:25] probably not worthwhile [11:23:35] oh, why pretend [11:23:52] the savings are almost certainly imperceptible to the majority of users [11:23:58] ori-l, do you ever sleep, and why?:P [11:24:24] tonight i just can't sleep :/ [11:25:28] it's basically for free, so it's good to do it once in a while and upon committing a lot of images, but i think a casual / occasional approach is sufficient. [11:26:34] if it's really worth doing, it's worth doing it on the edge somehow [11:27:25] * ori-l tries again [11:31:47] he also works on weekends [11:31:51] :) [11:32:39] holidays too [11:40:42] paravoid: hi -:] [11:40:46] hey [11:40:53] my python modules PTS pages have been created by whatever bot \O/ [11:41:50] paravoid: do you get time to fix up puppet on gallium ? There is some duplicate Group['jenkins'] definition caused by systemuser {} [11:42:47] isn't this https://gerrit.wikimedia.org/r/#/c/60024/ ? [11:44:17] paravoid: yeah that one [11:44:38] paravoid: I had a jenkins::user and jenkins::group and was using user {} [11:44:52] then switched to system user{} which does the group declare for us [11:45:11] I would drop systemuser {} and revert back to the building user {} [11:45:25] that's fine I think [11:46:02] build-in [11:46:02] grrg [11:52:02] New patchset: Hashar; "Revert "create jenkins user with systemuser"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60385 [11:52:17] Change abandoned: Hashar; "finally we are going to stick to user {}" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60024 [11:52:38] https://gerrit.wikimedia.org/r/60385 would do it :-D [11:52:58] I could also use a deletion of the python-statsd package currently at apt.wm.o (v1.5.8) that is the wrong package. [11:53:25] and get instead the python-statsd v2.0.1 I have packaged (that is a different upstream module though having the same binary package name) [11:56:32] Change abandoned: Hashar; "Will get the git repo deleted. We are using upstream svn repo." [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/59397 [11:56:45] Change abandoned: Hashar; "Will get the git repo deleted. We are using upstream svn repo." [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55028 [11:57:05] Change abandoned: Hashar; "Will get the git repo deleted. We are using upstream svn repo." [operations/debs/python-voluptuous] (master) - https://gerrit.wikimedia.org/r/59605 [12:09:54] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:54] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:54] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:15] RECOVERY - Puppet freshness on ms-be9 is OK: puppet ran at Tue Apr 23 12:11:13 UTC 2013 [12:20:32] hashar: ah right, I forgot about that [12:20:37] apologies [12:21:46] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60385 [12:27:46] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Tue Apr 23 12:27:38 UTC 2013 [12:34:13] !log reprepro python-statsd 2.0.1-1~wmf1 & voluptuous 0.6.1-1~wmf1 for precise-wikimedia [12:34:16] hashar: ^ [12:34:21] Logged the message, Master [12:35:01] paravoid: you are the best :-) [12:35:07] want me to upgrade gallium? [12:35:12] or are you taking care of it? [12:35:15] PROBLEM - LVS HTTPS IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:15] PROBLEM - LVS HTTPS IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection timed out [12:35:24] PROBLEM - LVS HTTP IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection timed out [12:35:24] PROBLEM - LVS HTTPS IPv6 on wikinews-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:25] PROBLEM - LVS HTTPS IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:25] PROBLEM - LVS HTTP IPv6 on mediawiki-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:25] PROBLEM - LVS HTTPS IPv6 on wikipedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection timed out [12:35:35] paravoid: I will upgrade gallium [12:35:45] I guess you want to look at the IPv6 connectivity issue at eqiad :-] [12:37:08] PROBLEM - LVS HTTP IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:18] RECOVERY - LVS HTTPS IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 95715 bytes in 3.508 second response time [12:37:18] RECOVERY - LVS HTTPS IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 63690 bytes in 6.320 second response time [12:37:28] RECOVERY - LVS HTTP IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 63690 bytes in 1.456 second response time [12:38:18] RECOVERY - LVS HTTPS IPv6 on wikinews-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 63690 bytes in 0.087 second response time [12:38:18] RECOVERY - LVS HTTP IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 63690 bytes in 3.248 second response time [12:38:18] PROBLEM - LVS HTTPS IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection timed out [12:38:27] PROBLEM - LVS HTTP IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:28] PROBLEM - LVS HTTPS IPv6 on wikivoyage-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:40:20] RECOVERY - LVS HTTP IPv6 on mediawiki-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 63690 bytes in 0.027 second response time [12:40:45] paravoid: need to drop out python-voluptuous 0.6.1-4 . That prevents us from installing upstream 0.6.1-1~wmf1 [12:40:48] RECOVERY - LVS HTTPS IPv6 on wikivoyage-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 41720 bytes in 9.402 second response time [12:40:52] paravoid: the statsd one got upgraded [12:43:11] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [12:43:11] PROBLEM - Puppet freshness on ms-be9 is CRITICAL: No successful Puppet run in the last 10 hours [12:43:12] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [12:43:12] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [12:44:48] New patchset: BBlack; "Initial Gerrit push of Work-In-Progress vhtcpd code." [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/60390 [12:46:51] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Tue Apr 23 12:46:46 UTC 2013 [12:48:31] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 198 seconds [12:50:31] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 12 seconds [12:55:01] PROBLEM - Packetloss_Average on analytics1005 is CRITICAL: CRITICAL: packet_loss_average is 8.27482272727 (gt 8.0) [12:59:01] PROBLEM - Packetloss_Average on analytics1004 is CRITICAL: CRITICAL: packet_loss_average is 8.80145209677 (gt 8.0) [13:08:58] RECOVERY - Packetloss_Average on analytics1004 is OK: OK: packet_loss_average is 1.79159968254 [13:09:00] RECOVERY - Packetloss_Average on analytics1005 is OK: OK: packet_loss_average is 0.0 [13:12:08] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [13:24:19] PROBLEM - Host mw27 is DOWN: PING CRITICAL - Packet loss = 100% [13:25:19] RECOVERY - Host mw27 is UP: PING OK - Packet loss = 0%, RTA = 26.56 ms [13:27:18] PROBLEM - Apache HTTP on mw27 is CRITICAL: Connection refused [13:42:28] New patchset: Mark Bergsma; "Revert "decom'ing sq33, fixing role/cache.pp puppet and ganglia.pp puppet ot match standards"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60395 [13:43:26] New patchset: Mark Bergsma; "Revert "decom'ing sq33, fixing role/cache.pp puppet and ganglia.pp puppet ot match standards"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60395 [13:45:23] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60395 [13:51:26] hashar: voluptuous is now fixed on gallium btw [13:52:04] New patchset: Mark Bergsma; "Set dysprosium backend weight at 4x" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60397 [13:52:05] paravoid: thx :) [13:52:54] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60397 [13:53:31] paravoid: I think I have all needed dependencies now. I can proceed with zuul packaging :-) [13:59:26] PROBLEM - Varnish traffic logger on dysprosium is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [14:00:25] RECOVERY - Varnish traffic logger on dysprosium is OK: PROCS OK: 3 processes with command name varnishncsa [14:14:11] PROBLEM - Varnish traffic logger on cp3009 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [14:16:02] New patchset: Mark Bergsma; "Move proxy URL rewriting into a function" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60399 [14:18:56] apergos: around? [14:19:11] yes [14:19:18] wondering about the swift status [14:20:10] ms-be9 installed, but not pooled. I looked at ganglia a bit earlier today and there is still a fair amount of catching up due to ms-be1 being down [14:20:34] once that settles out a bit I'll start adding ms-be9 and hopefully 2 back in [14:20:59] ms-be2 is apparently not ready, it's on my list to ask about [14:21:12] (in theory it was to be made ready yesterday evening our time) [14:25:30] catching up of what? [14:26:00] data it should have [14:26:05] which one? [14:26:07] ms-be1? [14:26:11] yep [14:26:29] what does that matter? [14:26:39]