[00:00:48] lol [00:01:06] if you can't fix the problem, just forget it ever happened ;) [00:01:08] PROBLEM - NTP on search20 is CRITICAL: NTP CRITICAL: No response from NTP server [00:01:18] can you merge that then Reedy and I'll pull [00:01:28] I will just try not to make a mistake next time ;) [00:01:30] I will, just need to tidy up something else first [00:03:17] ok [00:04:59] Ryan_Lane: how do I view the users in an ldap group on gerrit? [00:05:09] you can't on gerrit [00:05:19] bleh [00:05:23] seemingly preilly isn't in WMF [00:05:26] could you fix htat? [00:05:59] Reedy: on any labs instance you can use ldaplist -l group [00:06:38] The group list for wmf is weird.. [00:09:35] Ryan_Lane: http://p.defau.lt/?lyX0Np1knlmP3ObaJiyuYQ [00:10:52] do you need to add a group first? [00:10:56] so group wmf [00:11:33] What? [00:11:36] It already exists [00:12:32] does the ordering matter? so modify-ldap-group wmf --addmembers=preilly [00:14:17] https://labsconsole.wikimedia.org/wiki/Help:Access#Giving_users_Labs_access.2C_if_they_already_have_an_SVN_account [00:14:23] and no, that wouldn't make sense [00:21:22] New patchset: Ryan Lane; "Fixes for device detection" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7142 [00:21:41] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/7142 [00:22:11] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/7142 [00:22:13] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7142 [00:22:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:08] Ryan_Lane: can you run "modify-ldap-group --addmembers=preilly wmf" for me? [00:28:35] Reedy: https://bugzilla.wikimedia.org/show_bug.cgi?id=33753 did you figure out which config bits you need? [00:28:51] no [00:31:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.813 seconds [00:46:21] Reedy: yeah [00:46:21] sec [00:46:41] Reedy: you need to run that via sudo [00:46:43] I ran it, though [00:46:49] Ah [00:46:56] just literally sudo foobar...? [00:47:14] https://labsconsole.wikimedia.org/wiki/Help:Access#Giving_users_Labs_access.2C_if_they_already_have_an_SVN_account [00:47:20] yep [00:47:22] docs said nothing ;) [00:47:34] add-labs-user doesn't need sudo, does it? [00:47:41] umm [00:47:45] I think all of them do [00:47:50] hmm [00:47:54] I thought I'd run that before [00:48:14] preilly: can you do git push from /home/wikipedia/common now? [00:48:23] (i haven't submitted your commit as it was complaining at me) [00:49:11] Reedy: To ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config.git [00:49:12] ! [remote rejected] master -> master (can not update the reference as a fast forward) [00:49:13] error: failed to push some refs to 'ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config.git' [00:50:10] You probably don't have the right to push merge commits then [00:50:15] Run git rebsae origin/master and try again [00:50:18] *rebase [00:50:22] It's not a merge commit [00:50:39] If he pulled after committing the pull might have created a merge commit locally [00:50:51] Computers suck [00:51:24] Reedy, RoanKattouw: git rebase origin/master [00:51:25] Current branch master is up to date. [00:52:00] Reedy, RoanKattouw: still ! [remote rejected] master -> master (can not update the reference as a fast forward) [00:52:31] I get ! [remote rejected] master -> master (invalid committer) [00:53:00] if (ctl.canUpdate()) { [00:54:04] RefControl ctl = projectControl.controlForRef(cmd.getRefName()); [00:54:05] if (ctl.canUpdate()) { [00:54:09] to be more complete [00:54:14] preilly: Could you pastebin your git log --graph for me please? [00:54:21] so it's probably permissions related right? [00:54:50] I just attempted to fix the permissions [00:54:55] Could you try again? [00:55:07] Though really the typical perms error is 'prohibited by Gerrit' [00:55:14] RoanKattouw: http://pastebin.mozilla.org/1628983 [00:55:38] RoanKattouw: now it's ! [remote rejected] master -> master (invalid committer) [00:55:50] hmm [00:55:59] Is the email you used for hte commit the one that's set on your gerrit account? [00:57:14] Ah [00:57:17] You used your full name [00:57:24] in gerrit your display name is preilly [00:57:44] Reedy: it's fixed To ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config.git [00:57:45] d4c704f..1604a07 master -> master [00:57:53] yay [00:58:26] Lol [00:58:34] I can direct push to that repo, but I can't review [00:58:34] Code Review: [00:58:35] +1 Looks good to me, but someone else must approve [00:58:35] 0 No score [00:58:35] -1 There's a problem with this change, please improve [00:58:49] Reedy, RoanKattouw: thanks, for the help [00:59:44] RoanKattouw: do you have rights to update https://gerrit.wikimedia.org/r/#/admin/projects/operations/mediawiki-config,access to give wmf and wmf-deployment? [01:00:33] else Ryan_Lane ^ [01:00:52] Sure [01:01:26] done [01:03:39] Nothing has changed.. [01:04:01] beat me to it [01:04:11] RoanKattouw: it needs to be added to submit [01:04:15] and verified [01:04:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:04:49] I thought we didn't want wmf to have this [01:05:00] why does wmf need it? shouldn't it only be wmf-deployment? [01:05:11] and why does push have anything specified? [01:05:17] by default everyone can push to everything [01:05:22] lol [01:05:30] * Reedy shrugs [01:05:40] ok [01:05:42] changed [01:05:43] look now [01:05:59] New review: Reedy; "(no comment)" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/7133 [01:06:00] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/7133 [01:06:01] changed again [01:06:01] Yup, thanks [01:07:28] Krinkle-away: doesn't gitweb already handle this? [01:07:36] or does this just link to gitweb? [01:07:56] the mwSnapshots [01:08:01] Ryan_Lane: snapshots ? [01:08:04] Ryan_Lane: It doesn't use gitweb [01:08:07] I know gitweb has it [01:08:23] why not just link to that? [01:08:24] I thought I mentioned in the mail why I didn't use it [01:08:28] oh [01:08:31] the problem is that those are generated on the fly and often time out [01:08:33] very alos [01:08:39] very slow, not cached [01:08:42] ah [01:08:43] right [01:08:57] Need to get the checkouts for ED updated to have git in it [01:08:57] these are hourly generated for the HEAD of each public branch [01:09:03] ED ? [01:09:09] extension distributor [01:09:11] extension distributor [01:09:13] Right [01:09:15] fuck ED [01:09:18] I hate ED [01:09:27] I wonder what ED looks like now [01:09:34] apparently git clone is too hard for people [01:10:15] Reedy: For the regular john doe with one-click XAMP install or web host with FTP server, using the command line at all is too hard [01:10:23] let a lone installing Git for Windows and what not [01:10:43] Reedy: how do I abandon my changes? [01:10:52] I agree thought, that those people likely won't need nighties though, and we provide .tar for releases [01:10:53] it's giving me the stash or commit mesage again [01:11:09] git checkout changedfile [01:11:10] Thehelpfulone: On the gerrit change page, "Reject patch set #" or "Abandon change" [01:11:21] I mean within git Krinkle on my computer [01:11:39] Thehelpfulone: are you sure you mean "abandon" ? [01:11:46] Or do you mean something else [01:11:52] Oh, reset uncommitted changes ? [01:12:04] I mean overwrite my changed file with the one on the git repo [01:12:11] yeah, git checkout does that :) [01:12:25] which checks out that file from the last committed version of that file [01:12:43] now I've got merge conflict message [01:13:00] I did git checkout wmf-config/InitialiseSettings.php [01:13:05] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.019 seconds [01:13:09] and then which command ? [01:13:15] git pull [01:13:30] Thehelpfulone: have you already made changes ? [01:13:34] check git status [01:13:37] I did make changes yeah [01:13:41] git pull origin master --rebase [01:13:44] but I want to discard my changes [01:13:47] oh [01:14:03] Thehelpfulone: To undo a committed change, "git checkout" doesn't' do that [01:14:06] Thehelpfulone: If you want to completely discard your changes and reset your working tree to the master state, run git reset --hard origin/master [01:14:12] ^ [01:14:14] ok [01:14:33] what would I do if I just wanted to do it for one file? [01:14:52] Thehelpfulone: Also to make this easier nex time, make it a habbit to never edit in 'master' but make a local branch [01:14:55] I've done what you said and it worked Roan [01:15:05] how do I do that? :) [01:15:18] git branch? [01:15:21] git checkout -b myfancynewfeature master [01:15:24] so whenever you start working, checkout master, git reset --hard origin/master, git pull origin master; then git checkout -b my-fix-foobar [01:15:36] That will create a new branch called 'myfancynewfeature' based off 'master', and switch to it [01:15:41] yep [01:16:02] If you then get a clash with the origin, you can simple pull origin master and then rebase locally, without getting merge conflicts [01:16:57] ok [01:17:07] I'll have a go with that on my next shell request :) [01:17:39] git is actually quite good! it should help reduce these shell backlogs when devs just have to review it [01:18:06] a bug like this https://bugzilla.wikimedia.org/show_bug.cgi?id=22939 to run a script [01:18:12] The easy ones usually get done [01:18:21] I guess mortals can't do that? [01:18:37] anyone with shell access can [01:18:47] We actually need to triage and make a list of all these scripts that need running [01:19:11] screen foreachwiki scriptToRun.php [01:19:12] wait [01:19:27] hmm [01:19:28] I can't run that anyways ;) [01:19:31] seems cleanupimages is quick [01:19:42] * Reedy leaves it running [01:19:44] https://bugzilla.wikimedia.org/show_bug.cgi?id=29062 - did Priyanka put that on a cronjob? [01:20:07] Probably not [01:20:12] see comment 7 https://bugzilla.wikimedia.org/show_bug.cgi?id=29062#c7 [01:20:16] You can do that in the ops/puppet repo [01:20:35] hell [01:20:40] cleanupImages is quick on commons [01:20:54] 2000+ images a second [01:20:58] you have to run that for each wiki? all 800 and something on them? [01:21:13] there's no nice global way of doing it? [01:21:13] no, we have a script that runs it on each wiki in turn [01:21:19] ah :) [01:21:28] so literally screen foreachwiki cleanupImages.php [01:21:32] and wait [01:23:45] running cleanupTitles also now [01:41:26] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 236 seconds [01:44:17] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds [01:45:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:50] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.685 seconds [02:02:44] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [02:26:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:56] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.737 seconds [03:18:47] PROBLEM - swift-container-auditor on ms-be4 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [03:24:22] New patchset: Hashar; "Enable show update markers on dewiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/7152 [03:25:01] New review: Hashar; "(no comment)" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/7152 [03:25:05] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/7152 [03:34:20] RECOVERY - swift-container-auditor on ms-be4 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [05:08:47] PROBLEM - swift-container-auditor on ms-be5 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [05:32:47] RECOVERY - swift-container-auditor on ms-be5 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [05:52:17] PROBLEM - Host knsq18 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:17] PROBLEM - Host hooft is DOWN: PING CRITICAL - Packet loss = 100% [05:52:17] PROBLEM - Host knsq20 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:17] PROBLEM - Host knsq21 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:17] PROBLEM - Host knsq23 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:17] PROBLEM - Host knsq16 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:18] PROBLEM - Host knsq17 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:18] PROBLEM - Host knsq22 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:19] PROBLEM - Host knsq24 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:19] PROBLEM - Host knsq19 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:20] PROBLEM - Host knsq26 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:26] PROBLEM - Host knsq28 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:26] PROBLEM - Host knsq27 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:35] PROBLEM - Host maerlant is DOWN: PING CRITICAL - Packet loss = 100% [05:52:35] PROBLEM - Host mediawiki-lb.esams.wikimedia.org is DOWN: PING CRITICAL - Packet loss = 100% [05:52:44] PROBLEM - Host ms6 is DOWN: PING CRITICAL - Packet loss = 100% [05:52:53] PROBLEM - BGP status on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, sessions up: 21, down: 4, shutdown: 1BRPeering with AS1257 not established - The + flag cannot be used with the sub-query features described below.BRPeering with AS24115 not established - ASN-EQIX-MLPEBRPeering with AS3292 not established - TDCBRPeering with AS4565 not established - The + flag cannot be used with the sub-query features described below.BR [05:53:02] RECOVERY - Host knsq20 is UP: PING OK - Packet loss = 0%, RTA = 118.42 ms [05:53:02] RECOVERY - Host knsq22 is UP: PING OK - Packet loss = 0%, RTA = 118.15 ms [05:53:02] RECOVERY - Host ms6 is UP: PING OK - Packet loss = 0%, RTA = 119.78 ms [05:53:02] RECOVERY - Host hooft is UP: PING OK - Packet loss = 0%, RTA = 118.41 ms [05:53:02] RECOVERY - Host knsq18 is UP: PING OK - Packet loss = 0%, RTA = 118.43 ms [05:53:02] RECOVERY - Host knsq19 is UP: PING OK - Packet loss = 0%, RTA = 119.27 ms [05:53:03] RECOVERY - Host knsq28 is UP: PING OK - Packet loss = 0%, RTA = 119.79 ms [05:53:11] RECOVERY - Host knsq26 is UP: PING OK - Packet loss = 0%, RTA = 118.51 ms [05:53:11] RECOVERY - Host mediawiki-lb.esams.wikimedia.org is UP: PING OK - Packet loss = 0%, RTA = 119.37 ms [05:53:12] RECOVERY - Host knsq17 is UP: PING OK - Packet loss = 0%, RTA = 119.26 ms [05:53:12] RECOVERY - Host knsq16 is UP: PING OK - Packet loss = 0%, RTA = 118.31 ms [05:53:12] RECOVERY - Host knsq23 is UP: PING OK - Packet loss = 0%, RTA = 118.47 ms [05:53:29] RECOVERY - Host knsq27 is UP: PING OK - Packet loss = 0%, RTA = 118.76 ms [05:53:39] RECOVERY - Host knsq21 is UP: PING OK - Packet loss = 0%, RTA = 120.12 ms [05:53:39] RECOVERY - Host knsq24 is UP: PING OK - Packet loss = 0%, RTA = 118.97 ms [05:54:41] RECOVERY - Host maerlant is UP: PING OK - Packet loss = 0%, RTA = 120.29 ms [06:04:16] hmmm [06:05:59] my thoughts exactly [06:06:08] cr2-eqiad didn't recover? [06:12:41] PROBLEM - swift-container-auditor on ms-be5 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [06:35:29] RECOVERY - swift-container-auditor on ms-be5 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [06:59:53] PROBLEM - BGP status on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, sessions up: 15, down: 10, shutdown: 1BRPeering with AS7385 not established - The + flag cannot be used with the sub-query features described below.BRPeering with AS2603 not established - NORDUNETBRPeering with AS2711 not established - The + flag cannot be used with the sub-query features described below.BRPeering with AS1257 not established - The + flag c [08:20:22] PROBLEM - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is CRITICAL: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z CRITICAL - *2600* [08:26:04] RECOVERY - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is OK: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z OK - 2325 [08:26:22] RECOVERY - BGP status on cr2-eqiad is OK: OK: host 208.80.154.197, sessions up: 8, down: 0, shutdown: 18 [08:30:16] PROBLEM - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is CRITICAL: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z CRITICAL - *2563* [08:38:40] RECOVERY - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is OK: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z OK - 2388 [09:10:08] PROBLEM - Puppet freshness on search20 is CRITICAL: Puppet has not run in the last 10 hours [09:10:08] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: Puppet has not run in the last 10 hours [09:29:38] PROBLEM - Host knsq26 is DOWN: PING CRITICAL - Packet loss = 100% [09:30:41] RECOVERY - Host knsq26 is UP: PING OK - Packet loss = 0%, RTA = 108.86 ms [10:17:53] PROBLEM - Host knsq29 is DOWN: PING CRITICAL - Packet loss = 100% [10:19:23] RECOVERY - Host knsq29 is UP: PING OK - Packet loss = 0%, RTA = 109.56 ms [10:33:53] PROBLEM - Host knsq24 is DOWN: PING CRITICAL - Packet loss = 100% [10:35:05] RECOVERY - Host knsq24 is UP: PING OK - Packet loss = 0%, RTA = 108.65 ms [10:35:14] PROBLEM - NTP on knsq24 is CRITICAL: NTP CRITICAL: Offset unknown [10:39:35] RECOVERY - NTP on knsq24 is OK: NTP OK: Offset -0.0367449522 secs [10:49:29] PROBLEM - Host knsq16 is DOWN: PING CRITICAL - Packet loss = 100% [10:50:50] RECOVERY - Host knsq16 is UP: PING OK - Packet loss = 0%, RTA = 108.83 ms [11:58:09] PROBLEM - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is CRITICAL: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z CRITICAL - *2588* [12:04:00] RECOVERY - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is OK: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z OK - 2325 [12:04:00] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [12:12:24] PROBLEM - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is CRITICAL: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z CRITICAL - *2563* [12:15:06] RECOVERY - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is OK: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z OK - 2263 [12:23:39] PROBLEM - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is CRITICAL: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z CRITICAL - *2588* [12:29:21] RECOVERY - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is OK: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z OK - 2350 [12:55:26] PROBLEM - Host knsq18 is DOWN: CRITICAL - Host Unreachable (91.198.174.28) [12:56:29] RECOVERY - Host knsq18 is UP: PING OK - Packet loss = 0%, RTA = 108.79 ms [13:06:02] New patchset: Dzahn; "add logrotate config for lighttpd on install-server (RT-2753)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7167 [13:06:19] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/7167 [13:07:32] New patchset: Dzahn; "add logrotate config for lighttpd on install-server (RT-2753)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7167 [13:07:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/7167 [13:09:33] New review: Dzahn; "config file as on brewster, just changed "weekly" to "size 100M" and fixed tabs." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/7167 [13:20:56] PROBLEM - Backend Squid HTTP on knsq19 is CRITICAL: Connection refused [13:21:23] PROBLEM - Frontend Squid HTTP on knsq19 is CRITICAL: Connection refused [13:28:42] mutante: were you going to post about getting chapter mw's to upgrade? [13:33:03] mutante: if you haven't yet, I will... Just let me know if you want me to [13:47:12] hexmode: i'll do it. i was going to post on the chapter specific mailing lists, sounds good? [13:47:44] mutante: yes, that sounds fabulous. Could you CC me? [13:47:50] sure [13:51:02] mutante: so that we can keep a record, I'll update http://etherpad.wikimedia.org/OsJfzQH80T with whatever emails you send (or pointers to them). [13:52:08] hexmode: alright [13:54:34] RECOVERY - Backend Squid HTTP on knsq19 is OK: HTTP OK HTTP/1.0 200 OK - 632 bytes in 0.222 seconds [13:59:22] RECOVERY - Frontend Squid HTTP on knsq19 is OK: HTTP OK HTTP/1.0 200 OK - 653 bytes in 0.218 seconds [14:07:00] New patchset: Ottomata; "site.pp - Adding Fabian on stat1.wikimedia.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7070 [14:07:22] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/7070 [14:25:28] New patchset: Jgreen; "adding logmover account to hume so we can store fundraising impression logs there while storage3 is dead" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7169 [14:25:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/7169 [14:26:16] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/7169 [14:26:18] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7169 [14:28:46] PROBLEM - Host knsq20 is DOWN: PING CRITICAL - Packet loss = 100% [14:31:37] RECOVERY - Host knsq20 is UP: PING OK - Packet loss = 0%, RTA = 108.97 ms [14:34:55] PROBLEM - Frontend Squid HTTP on knsq20 is CRITICAL: Connection refused [14:35:13] PROBLEM - Backend Squid HTTP on knsq20 is CRITICAL: Connection refused [14:38:52] RECOVERY - Backend Squid HTTP on knsq20 is OK: HTTP OK HTTP/1.0 200 OK - 632 bytes in 0.218 seconds [14:40:58] RECOVERY - Frontend Squid HTTP on knsq20 is OK: HTTP OK HTTP/1.0 200 OK - 788 bytes in 0.219 seconds [14:46:58] PROBLEM - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is CRITICAL: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z CRITICAL - *2638* [14:48:23] mutante: got the emails, updated the etherpad. Let me know if there is any other place you think I should work on. But targetting chapters is a great strategic move. ty [14:52:45] look at 'im go! \o/ [14:56:43] PROBLEM - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is CRITICAL: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z CRITICAL - *2600* [14:58:13] RECOVERY - ps1-b5-sdtpa-infeed-load-tower-A-phase-Z on ps1-b5-sdtpa is OK: ps1-b5-sdtpa-infeed-load-tower-A-phase-Z OK - 2138 [15:07:28] hiya mark, you around? [15:08:55] wondering about installing precise on stat1 [15:10:22] PROBLEM - Backend Squid HTTP on knsq22 is CRITICAL: Connection refused [15:11:38] yes [15:11:43] PROBLEM - Frontend Squid HTTP on knsq22 is CRITICAL: Connection refused [15:11:43] RECOVERY - Backend Squid HTTP on knsq22 is OK: HTTP OK HTTP/1.0 200 OK - 632 bytes in 0.218 seconds [15:14:25] RECOVERY - Frontend Squid HTTP on knsq22 is OK: HTTP OK HTTP/1.0 200 OK - 788 bytes in 0.219 seconds [15:15:00] New patchset: Jgreen; "redo the way puppet manages logmover account which is used to move around fundraising proxy logs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7174 [15:15:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/7174 [15:15:45] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/7174 [15:15:47] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7174 [15:19:02] New patchset: Jgreen; "remove logmover from admins.pp, now installed by systemuser instead" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7176 [15:19:20] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/7176 [15:19:39] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/7176 [15:23:16] PROBLEM - Host knsq21 is DOWN: PING CRITICAL - Packet loss = 100% [15:23:38] New patchset: Jgreen; "remove logmover from admins.pp, now installed by systemuser instead fixed missing comma in site.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7176 [15:23:56] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/7176 [15:24:19] RECOVERY - Host knsq21 is UP: PING OK - Packet loss = 0%, RTA = 109.08 ms [15:24:46] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/7176 [15:24:49] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/7176 [15:31:39] oop, mark, sorry, just saw you responded [15:31:51] yeah, so, I heaarrrrd that precise is ready for prod machines [15:32:00] and, if we are going to install on stat1, we should do sooner rather than later [15:32:38] and to do that, we need a temporary places to store 206G [15:32:43] til it is reinstalled [15:32:50] possible? [15:44:13] yes [15:44:28] ideally we would use the netapps for that, but they're not in service at the moment :( [15:45:39] is there an alternative? [15:47:35] yeah [15:47:40] just for during the reinstall, right? [15:47:45] yeah [15:47:59] just to hold it somewhere until we can copy it back after [15:51:51] that shouldn't be hard [15:57:45] ^demon, are you around? [16:05:09] mark, if you let me know how to copy the stuff there, I can do start that now [16:05:14] and then you guys can reinstall anything asap [16:06:01] ottomata: how about we copy that data before the install, that's gonna be a whole lot easier ;) [16:07:02] when is the install? [16:07:09] we can schedule it [16:07:12] mutante: where do you have the chapter wikis on wikistats? [16:07:41] ok [16:07:56] when can we schedule it for…or, how do I go about doing that? [16:08:07] drdee, we need to make sure Erik Z and Andre E are ready [16:08:10] and know that this is happening [16:08:11] can do tomorrow or so [16:08:20] just put it in the ticket and get confirmation from us ;) [16:08:49] ok [16:08:56] will make sure this is ready with drdee, and then put in RT ticket [16:09:01] and will include info about saving /a [16:09:09] excellent [16:09:16] hexmode: "wmspecials" but that is exactly the table where version links are broken momentarily [16:09:38] thanks mark! [16:09:41] mutante: kk tyfti [16:14:34] LeslieCarr: so port 14/1 is tagged [16:14:49] and unless you changed it, cr1-eqiad is not using vlans, so expects it untagged... [16:16:04] yeah … now curious if i had changed it, looking through rollback versions ... [16:16:34] well just make 14/1 untagged I'd say, don't really wanna be switching vlans across that wave anyway ;) [16:16:40] ahha yeah rollback 8 [16:16:49] i had switched both to vlans [16:16:49] ok [16:18:03] and we're gonna replace this by a WDM setup anyway [16:18:28] let's talk about that after this is done :) [16:18:31] yep [16:18:38] mutante: wm.fr is running drupal!!!! [16:18:40] lame [16:19:22] yea, there are more :p [16:19:24] New patchset: Thehelpfulone; "adding flood group to itwikisource per bug 36600" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/7180 [16:19:31] ok, have the deactivated config on cr2-pmtpa now [16:19:39] * mark checks [16:19:42] second time's a charm? [16:19:43] (deactivated interface config, that is) [16:20:30] hexmode: and wikimedia.org.ar requires login for version info [16:20:48] ok [16:20:50] * hexmode sighs [16:22:00] so, how about i deactivate ospf on cr1-eqiad and we try moving it ? [16:22:09] deactivate ospf?! [16:22:17] just that interface you mean? [16:22:24] just that one [16:22:24] yes [16:22:27] ah ok ;) [16:22:28] sorry to give you a heart attack [16:22:29] hehehe [16:22:32] mutante: from page source on wm.o.ar: [16:22:44] yeah let's do that [16:23:01] don't forget ospf3 [16:23:10] hexmode: ah, heh:) that's a bug "leaks version info when admin didnt want to" .hehe [16:23:33] ah yeah, i now see some of the appeal of isis and not needing two protocols :) [16:23:44] yes [16:23:49] couldn't do isis because of foundry [16:23:51] but that's now gone [16:25:50]