[00:00:36] !log gallium - restarting Apache, enabling new ServerAlias [00:00:45] Logged the message, Master [00:07:38] New patchset: Pyoungmeister; "test hack coredb_mysql module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36677 [00:09:12] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36677 [00:13:16] New patchset: Pyoungmeister; "Revert "test hack coredb_mysql module"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36680 [00:13:21] New patchset: Asher; "db68 was missing from hostsByName" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/36681 [00:13:22] New patchset: Ryan Lane; "Add grains for realm, site and cluster" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36682 [00:13:50] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36682 [00:14:16] Change merged: Asher; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/36681 [00:14:27] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36680 [00:15:10] !log asher synchronized wmf-config/db.php 'adding db68 to s7' [00:15:18] Logged the message, Master [00:18:29] New review: Dzahn; "this breaks because of librsvg Ubuntu packages vs. our own. can be fixed manually by removing librsv..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36583 [00:28:55] New patchset: Ryan Lane; "Get repourls by site before iterating repos" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36685 [00:29:15] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36685 [00:30:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:46] New patchset: Dzahn; "add librsvg2-2 package to gallium (integration)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36688 [00:35:49] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36688 [00:43:35] !log pgehres synchronized php-1.21wmf5/extensions/CentralNotice/ 'Updating CentralNotice to master. Adding API for CN logs' [00:43:43] Logged the message, Master [00:44:40] Ryan_Lane: stop trying to hax0r tin [00:44:45] heh [00:44:53] I fucked up the permissions on the sudoers file I created :) [00:46:01] :) [00:46:48] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.515 seconds [00:46:57] PROBLEM - MySQL Slave Delay on db1013 is CRITICAL: CRIT replication delay 312 seconds [00:47:31] !log pgehres synchronized php-1.21wmf4/extensions/CentralNotice/ 'Updating CentralNotice to master. Adding API for CN logs' [00:47:39] Logged the message, Master [01:00:45] !log gallium - removing multiarch support (32bit on 64bit systems) which causes issues with librsvg package [01:00:54] Logged the message, Master [01:01:35] ah.. there is the culprit /Stage[main]/Misc::Contint::Android::Sdk/Package[ia32-libs] [01:03:27] New review: Dzahn; "this and removing multiarch support (/etc/dpkg/dpkg.cfg.d/multiarch) fixes the issue with librsvg. w..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36688 [01:04:09] New patchset: Ryan Lane; "Add deployment pillars to deployment servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36691 [01:05:34] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36691 [01:05:51] RECOVERY - MySQL Slave Delay on db1013 is OK: OK replication delay 0 seconds [01:13:12] New patchset: Ryan Lane; "Ensure the join sets a value" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36693 [01:13:35] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36693 [01:17:54] pgehres gave E3 permission to sync-dir E3Experiments, starting it now [01:18:16] heh, I only said I wasn't doing anything anymore :-p [01:19:27] improved account creation will ensure happy new users who donate more money. It's the circle of life [01:19:48] except that we don't show banners to logged-in users :-p [01:19:56] it means less money!!! [01:20:06] actually spagewmf -- do you have any retention statistics on all the users we've pushed to sign up? [01:22:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:23:14] mwalker, DarTar would be the one. We haven't encouraged account creation yet (just made it better and given people surveys and improved Community Portal after signup). EE's ArticleFeedbackv5 has a call to action to create an account, as I recall it doesn't generate a lot of account creations. [01:30:36] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [01:31:07] !log spage synchronized php-1.21wmf5/extensions/E3Experiments 'E3 backports to wmf5' [01:31:17] Logged the message, Master [01:38:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.033 seconds [01:47:33] New patchset: Ryan Lane; "Fix repourl pulling" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36696 [01:48:52] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36696 [02:11:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:29] !log LocalisationUpdate completed (1.21wmf5) at Tue Dec 4 02:23:29 UTC 2012 [02:23:39] Logged the message, Master [02:24:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.097 seconds [02:44:31] !log LocalisationUpdate completed (1.21wmf4) at Tue Dec 4 02:44:31 UTC 2012 [02:44:41] Logged the message, Master [03:30:32] New patchset: Ryan Lane; "Various git deploy fixes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36699 [03:34:03] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36699 [03:41:20] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [03:48:53] New patchset: Ryan Lane; "Ensure the repo exists and not just the repodir" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36701 [03:54:01] New patchset: Ryan Lane; "Ensure the repo exists and not just the repodir" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36701 [03:54:42] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36701 [04:10:16] New patchset: Ryan Lane; "Move slots out of common" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36702 [04:10:39] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36702 [04:29:14] New patchset: Ryan Lane; "Allow tin to run deployment runners" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36704 [04:29:32] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36704 [04:40:51] New patchset: Ryan Lane; "Set pillar roots for production" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36706 [04:41:07] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36706 [04:52:12] New patchset: Ryan Lane; "Make salt master erb properly output runner_dirs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36707 [04:53:24] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36707 [05:06:06] New patchset: Ryan Lane; "Compatibility changes for deployment runner" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36708 [05:06:24] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36708 [05:08:37] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [05:08:37] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [05:25:45] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [05:25:45] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [06:27:27] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [06:41:17] New patchset: Ryan Lane; "Whitespace" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36709 [06:41:36] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36709 [07:49:44] New patchset: ArielGlenn; "expand description of pages-logging.xml, fixes bug #42618" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/36712 [09:24:40] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [09:28:06] !log nikerabbit synchronized php-1.21wmf5/extensions/Narayam/ 'i18n deploy' [09:28:21] Logged the message, Master [09:28:46] 10.28 <+logmsgbot> !log nikerabbit synchronized php-1.21wmf5/extensions/Narayam/ 'i18n deploy' [09:29:05] can I ignore this: [09:29:06] snapshot1002: rsync: mkdir "/apache/common-local/php-1.21wmf5/extensions/Narayam" failed: No such file or directory (2) [09:29:09] snapshot1002: rsync error: error in file IO (code 11) at main.c(605) [Receiver=3.0.9] [09:34:56] yes [09:35:12] it's in the process of being upgraded [09:35:17] thanks for checking [09:38:46] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [09:40:00] New patchset: ArielGlenn; "ms-be3 gets new setup as 720xd with ssds" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36720 [09:42:00] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36720 [09:43:11] whose runner change in puppet? [09:43:13] - ret = sorted(set(keys['minions_pre']) - set(minions)) [09:43:13] + ret = sorted(set(keys['minions_pre']) - set(minions)) [09:45:20] I guess this is a whitespace change [09:46:43] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [10:50:19] New patchset: Matthias Mullie; "Enable AFTv5 for dewiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34964 [10:51:30] New review: Jarry1250; "Thanks for the pointer Daniel. This is not my area of expertise; hopefully one of the others can act..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36583 [11:32:02] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [12:17:22] Change abandoned: Hashar; "no time to work on that" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31575 [12:17:26] Change abandoned: Hashar; "no time to work on that" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31579 [12:17:31] Change abandoned: Hashar; "no time to work on that" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31582 [12:17:38] Change abandoned: Hashar; "no time to work on that" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31583 [13:42:35] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [14:08:42] New review: preilly; "Thanks." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36709 [14:26:19] New patchset: Mark Bergsma; "Don't software raid ms-fe300x, they have hw RAID" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36764 [14:26:45] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36764 [14:44:59] !log Installed ms-fe3001 [14:45:09] Logged the message, Master [15:09:21] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [15:09:21] PROBLEM - Puppet freshness on sockpuppet is CRITICAL: Puppet has not run in the last 10 hours [15:09:21] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [15:17:50] mark: can you check rt3687 and verify that the switch ports are enabled. I am not able to pxe boot..."media test failure" [15:18:00] ok [15:19:43] cmjohnson1: they look fine [15:20:04] hrmm..ok [15:26:27] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [15:26:27] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [15:42:59] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.027 seconds [15:54:32] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [15:56:11] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [15:58:11] !log authdns update...adding pc1001-1003 [15:58:20] Logged the message, Master [16:13:24] sbernardin: did you set the mgmt ip on ms-be3 [16:14:15] cmjohnson1: yes I did...that was done yesterday [16:14:40] ok..i can't ssh into it...did you fix the idrac7 w/temp license? [16:15:11] cmjohnson1: yes...I was able to log into idrac7 to verify [16:16:40] hmm [16:23:06] sbernardin: can you check the mgmt cable [16:23:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:28:44] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [16:40:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.019 seconds [16:46:57] New patchset: Dereckson; "(bug 42511) Namespaces configuration for ru.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/35663 [16:48:37] !log es1004 replacing bad hdd slot 1 (rt3954) [16:48:46] Logged the message, Master [16:49:45] New review: Dereckson; "PS2: community requested a new alias, T -> NS_TEMPLATE." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/35663 [17:13:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:24:34] apergos: mgmt is working now on ms-be3 [17:24:39] yay [17:24:49] New patchset: Andrew Bogott; "Move the puppetdoc websites out of /" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36785 [17:24:52] what was the problem, out of curiosity? [17:25:16] New patchset: Dereckson; "(bug 42690) Namespaces configuration for ro.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/36787 [17:25:26] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36785 [17:25:53] apergos: from what I gather...there was a conflict with the old c2100 still cabled...steve disconnected and rebooted the new be3 [17:27:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.432 seconds [17:28:02] huh, ok [17:28:08] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [17:30:25] well I'm on it now so that's the good thing [17:30:50] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [17:34:26] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [17:35:08] really? is that server still being a bleeper? [17:35:56] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.047 second response time [17:40:50] !log cp1031 powering down for fan assembly/main board swap rt3614 [17:40:58] Logged the message, Master [17:43:29] New patchset: ArielGlenn; "update mac for ms-be3 (new host, old name)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36790 [17:43:36] !log reedy synchronized php-1.21wmf5/extensions/GoogleNewsSitemap/ [17:43:44] Logged the message, Master [17:44:04] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36790 [18:00:30] New patchset: Jgreen; "overhauled fundraising db dumper" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36792 [18:00:58] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36792 [18:01:17] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:34] New patchset: Jgreen; "deprecate fundraising dump config files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36794 [18:05:16] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36794 [18:09:50] RECOVERY - Host ms-be3 is UP: PING OK - Packet loss = 0%, RTA = 0.46 ms [18:13:08] PROBLEM - swift-container-auditor on ms-be3 is CRITICAL: Connection refused by host [18:13:26] PROBLEM - SSH on ms-be3 is CRITICAL: Connection refused [18:13:26] PROBLEM - swift-object-auditor on ms-be3 is CRITICAL: Connection refused by host [18:13:35] PROBLEM - swift-container-server on ms-be3 is CRITICAL: Connection refused by host [18:13:35] PROBLEM - swift-account-reaper on ms-be3 is CRITICAL: Connection refused by host [18:13:35] PROBLEM - swift-object-replicator on ms-be3 is CRITICAL: Connection refused by host [18:13:44] PROBLEM - swift-container-replicator on ms-be3 is CRITICAL: Connection refused by host [18:13:44] PROBLEM - swift-container-updater on ms-be3 is CRITICAL: Connection refused by host [18:13:44] PROBLEM - swift-account-server on ms-be3 is CRITICAL: Connection refused by host [18:14:20] PROBLEM - swift-object-server on ms-be3 is CRITICAL: Connection refused by host [18:14:29] PROBLEM - swift-object-updater on ms-be3 is CRITICAL: Connection refused by host [18:14:29] PROBLEM - swift-account-replicator on ms-be3 is CRITICAL: Connection refused by host [18:14:47] PROBLEM - swift-account-auditor on ms-be3 is CRITICAL: Connection refused by host [18:19:14] New patchset: Andrew Bogott; "Remove the broken 'puppetsource' site." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36796 [18:19:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.022 seconds [18:20:12] !log reedy synchronized php-1.21wmf5/extensions/GoogleNewsSitemap [18:20:20] Logged the message, Master [18:20:34] New review: Andrew Bogott; "Reviewers --" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/36796 [18:34:26] PROBLEM - NTP on ms-be3 is CRITICAL: NTP CRITICAL: No response from NTP server [18:39:24] mutante / notpeter - ready for working on icinga ? [18:47:13] !log taking down bast1001 for reimaging [18:47:21] Logged the message, notpeter [18:51:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:59] PROBLEM - SSH on bast1001 is CRITICAL: Connection refused [18:57:23] PROBLEM - Host ms-be3 is DOWN: PING CRITICAL - Packet loss = 100% [18:59:18] cmjohnson1: you on site today by any chance? [18:59:36] yes in eqiad [19:00:00] I'm not getting concole output from bast1001, and this might hamper my upgrade efforts [19:00:05] would you be willing to investigate? [19:00:20] yep i will take a look [19:00:54] thanks! [19:01:29] or, if you don't have much time, can you just plug in a keyboard/monitor and press "ok" to whatever disk partitioning related bullshit is up on the screen? :) [19:04:22] werrr [19:04:38] * apergos is glad they took a copy of their (tiny) home dir earlier today [19:05:26] apergos: shouldn't affect homdirs [19:05:33] has nfs [19:05:52] I thought bast1001 does not use nfs home for /home [19:06:11] it has the class nfs::netapp::home::othersite [19:06:12] in puppet [19:06:22] so, unless that's extremely misleadingly named.... [19:06:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.881 seconds [19:06:44] well my home dir there has much different stuff than the one on fenari [19:06:48] huh [19:06:54] ok [19:07:34] it mounts it but not as /home iirc [19:08:02] RECOVERY - SSH on bast1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:08:18] ah! gotcha [19:08:41] well, then I'm extra glad that you backed stuff up [19:14:20] PROBLEM - NTP on bast1001 is CRITICAL: NTP CRITICAL: No response from NTP server [19:18:14] PROBLEM - Host bast1001 is DOWN: PING CRITICAL - Packet loss = 100% [19:20:38] RECOVERY - Host ms-be3 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [19:21:37] notpeter: you should be good to go...make sure you using console com2 not connect com2 [19:22:35] RECOVERY - Host bast1001 is UP: PING OK - Packet loss = 0%, RTA = 26.55 ms [19:24:11] still no output :/ [19:24:17] Imean, the host is back up [19:24:24] so this can totally wait until later [19:24:34] but I just ge ta blinky cursor [19:24:50] !log upgraded Bugzilla to 4.2.4 [19:24:59] Logged the message, Master [19:25:44] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [19:26:48] notpeter: odd..can you close the console [19:26:57] i will tinker with it some more if that's okay? [19:28:06] yep [19:28:11] sorry [19:28:14] logged out of terminal now [19:30:31] k..thx [19:31:39] andre__: thanks for the new bugzilla :) [19:31:44] + whoever did the ops stuff :-] [19:32:50] hashar, mutante did [19:35:29] PROBLEM - Host bast1001 is DOWN: PING CRITICAL - Packet loss = 100% [19:36:25] !log committing to pfw1-eqiad - has occasionally lost a few packets in the past [19:36:33] Logged the message, Mistress of the network gear. [19:37:18] mutante: thanks for the new bugzilla :-] [19:37:55] New patchset: Matthias Mullie; "Enable AFTv5 for dewiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34964 [19:38:06] Change merged: Matthias Mullie; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34964 [19:38:27] hashar: hey, wanna join the talk with Jarry on other channel [19:38:41] mutante: which chan ? [19:38:42] hashar: i am glad it works:) [19:39:01] ah mutante #mediawiki I guess [19:39:05] hashar: yes [19:39:20] or ..actually.. lets get him over here [19:39:41] RECOVERY - SSH on ms-be3 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:39:50] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [19:39:50] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:55] hi [19:40:08] hashar: so... the situation on gallium is like this: [19:40:20] i could fix the librsvg issue... but there was a price to it [19:40:25] it broke something else [19:40:29] Jarry1250: we follow up in this channel. it is quieter [19:40:37] which is: /Stage[main]/Misc::Contint::Android::Sdk/Package[ia32-libs] [19:40:43] mutante: Hi, so, as I say, I'm really quite clueless on these issues, but it seems to me that forcing puppet to keep the version manually placed there would work? [19:40:50] the issue comnes from having "multiarch" enabled [19:40:57] mutante: I think that package is required by the Sun jdk [19:41:08] or maybe the Android sdk [19:41:26] Jarry1250: maybe we need to use APT pinning via puppet ..yep [19:41:41] hashar: so, do we need 32bit stuff on gallium [19:41:51] because Android SDK is not avail as 64bit or something [19:42:11] or could we get rid of the 32bit stuff .. ia32* are compatibility packages [19:42:43] if i delete this file: /etc/dpkg/dpkg.cfg.d/multiarch it disables that [19:42:50] Re: priorities, Android SDK wins every time. [19:43:02] and then puppet will also install the correct librsvg packages... our packages... as opposed to the Ubuntu packages [19:43:39] apparently we need the ia32-libs to actually build the Android packages [19:43:52] ok, there goes the easiest solution then [19:44:28] actually it is already being told to prefer Wikimedia packages over Ubuntu packages.. hrmmm [19:44:31] $ file platform-tools/aapt [19:44:31] platform-tools/aapt: ELF 32-bit LSB executable, Intel 80386, v [19:44:33] in apt preferences [19:45:11] * andre__ bbl [19:46:44] RECOVERY - Host bast1001 is UP: PING OK - Packet loss = 0%, RTA = 26.70 ms [19:46:45] yeah.. let me look at it again in a little while [19:46:49] ahh [19:46:51] mutante: http://andrzejgrzesik.info/2012/08/02/ubuntu-12-04-android-sdk-fun/ [19:47:31] notpeter: i don't know yet...need to look into more. the cfg is correct [19:47:47] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [19:47:59] mutante: his solution is: apt-get --no-install-recomends install ia32-libs-multiarch [19:48:12] mutante: honestly I have no idea what is wrong. Might try on labs with a Precise instance [19:48:16] New patchset: Dereckson; "(bug 42693) Enable Article Feedback Tool on de.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/36810 [19:49:49] !log mlitn synchronized php-1.21wmf4/extensions/ArticleFeedbackv5 'desc' [19:49:57] Logged the message, Master [19:50:12] !log mlitn synchronized php-1.21wmf5/extensions/ArticleFeedbackv5 'desc' [19:50:21] Logged the message, Master [19:50:54] hashar: Removing librsvg2-bin ... [19:50:54] Removing librsvg2-2 ... [19:50:56] :/ [19:50:58] !log mlitn synchronized wmf-config/InitialiseSettings.php [19:51:06] Logged the message, Master [19:51:23] !log mlitn synchronized wmf-config/CommonSettings.php [19:51:31] Logged the message, Master [19:51:37] mutante: :( [19:51:40] hashar: oh wait..just let me try a little more [19:51:56] Change merged: Matthias Mullie; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/36810 [19:52:32] !log mlitn synchronized wmf-config/InitialiseSettings.php [19:52:40] Logged the message, Master [19:53:36] New patchset: Cmjohnson; "Replaced main board for cp1031, updating NIC1 MAC" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36811 [19:53:57] aus der Funktion „ArticleFeedbackv5Fetch::run“. Die Datenbank meldete den Fehler „1146: Table 'dewiki.aft_article_feedback' doesn't exist (10.0.6.54)“. [19:54:48] robh: can you +2 https://gerrit.wikimedia.org/r/36811 [19:54:56] cmjohnson1: ok, cool. thanks [19:55:24] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36811 [19:55:37] cmjohnson1: done [19:55:42] thx [19:56:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.031 seconds [19:57:40] could someone please create the AFTv5 tables for dewiki? [19:57:49] dewiki is broken [19:59:32] wait, someone deployed an extension without deploying the schema? [19:59:39] seems so [19:59:45] ask Matthis Mulle [19:59:56] he deployed AFTv5 for dewiki a few minutes ago [20:00:04] is he here? [20:00:22] don't know his IRC nick [20:01:12] he's mlitn, just asked him to come in [20:01:18] thanks [20:01:30] hi [20:01:53] mlitn: you just deployed aftv5 to dewiki? [20:02:04] still busy ;) [20:02:14] ? [20:02:38] mlitn: what the fuck are you doing? [20:02:59] mlitn: this seems urgent; de.wikipedia is broken. see https://bugzilla.wikimedia.org/show_bug.cgi?id=42693 . did you deploy an extension without deploying the schema? [20:03:51] sumanah: it's only peculiar pages like the watchlist, all is fine reading and editing [20:03:53] mlitn: when you deploy, you need to be on irc [20:03:56] every time [20:04:13] <^demon> Ok, let's talk about that later. [20:04:16] probably the 11k pages that are in the aftv5 lottery as well [20:04:26] <^demon> New tables either need creating, or we need to shut off Aftv5. [20:04:28] <^demon> I don't care which. [20:04:32] winning the lottery has never been so.. win! [20:05:30] how about we turn off the extension for now [20:05:32] mlitn: fyi for the future, the person deploying an extension for the first time generally deploys the whole thing [20:05:33] ^demon: extensions/ArticleFeedbackv5/sql/ArticleFeedbackv5.sql is what needs to be executed [20:05:38] and then deal with this once we don't have 11k broken articles [20:05:45] se4598: we have to create your MySQL tables and all will be fine again [20:06:00] <^demon> mlitn: Yes, I'm well aware of that. But unless someone's doing it right now, we need to turn the extension back off. [20:06:00] mlitn: as in - you run the create table .sql file before enabling [20:06:03] se4598: (your = those required by the extension) [20:06:55] ok, but seriously, we've been talking about this for 5 minutes now. why is the broken code still live. [20:07:17] <^demon> I'm syncing. [20:07:18] !log demon synchronized wmf-config/InitialiseSettings.php 'Fix dewiki breakage until tables are created and everyone's bikeshedding over the proper way to deploy' [20:07:26] Logged the message, Master [20:07:26] ok, but table-creating was mentioned in the commit-msg, lot of complains at dewp :) [20:07:39] <^demon> dewiki fixed. [20:07:46] ^demon: thanks :) [20:07:49] on the other hand, aftv5 is a less offensive product if its db tables are dropped.. [20:08:02] just ran sql [20:09:04] tables are created [20:10:37] so - I'm sorry; I was in the middle of getting tables deployed [20:10:42] obviously should have run that first [20:11:53] mlitn: once you're done with the immediate fixing, I think it would be good to talk about whether there were missing instructions somewhere like at http://wikitech.wikimedia.org/view/How_to_deploy_code [20:11:54] you should also be on irc and communicative before, during, and after any deploy, especially if it has gone wrong [20:13:12] http://wikitech.wikimedia.org/view/How_to_deploy_code mentions schema changes and links to the appropriate doc [20:13:44] it also talks about understanding prerequisites before deploying [20:14:34] I will add the tip in the == Basic common sense == section. It lists the things to be careful of. [20:14:58] sumanah: I'm sure it's all listed on there (though I've got something to make up for and will re-read it tomorrow), it's more of a "I got used to deploying stuff with less impact, that I did not check it as thoroughly as I obviously should have" [20:15:28] Yeah. [20:17:40] wangatlargo: your name makes me think you're chilling out in key largo [20:18:50] mlitn: congrats on your first site outage, it's a badge of honor (not sarcasm) [20:19:08] <^demon> We really should've made t-shirts :\ [20:19:27] create a barnstar for it [20:20:08] I've been talking up the ti-shirt idea for years [20:20:09] rofl - can't say I'm feeling proud ;) [20:20:12] but there's no rt ticket so... [20:20:21] mlitn: and if you think there's anyone else in your shoes (or in the shoes you figuratively were wearing 2 hours ago), might be worth telling them your story as a Confession [20:20:51] sumanah: makes sense ;) [20:21:07] mlitn: "mea culpa and I should have read this doc better: My True Story" [20:21:55] ^demon or LeslieCarr > could you add me in the Editor group? http://wikitech.wikimedia.org/index.php?title=Special%3AUserRights&user=Dereckson [20:22:06] * sumanah has to go to lunch [20:22:13] <^demon> Dereckson: Doned. [20:22:21] Thank you. [20:27:39] so, now that all tables are up, is it ok to re-enable aft on dewiki? [20:28:39] mlitn: sure [20:29:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:31:36] * mlitn crosses fingers [20:31:45] !log mlitn synchronized wmf-config/InitialiseSettings.php 're-enable aftv5 on dewiki after having created missing tables' [20:31:53] Logged the message, Master [20:32:23] uhoh [20:32:25] who broke enwiki ? [20:32:26] http://en.wikipedia.org/wiki/Shooting_an_apple_off_one%27s_child%27s_head [20:32:51] LeslieCarr: you really think this kind of jokes work? [20:33:00] no, not a joke [20:33:22] oh force reload worked [20:33:36] I was about to say [20:33:42] both logged in and logged out looked ok for me [20:33:48] crazy [20:33:51] oh well [20:33:59] maybe one varnish cache has the old css ? [20:34:33] ugh [20:39:18] RoanKattouw_away: https://gerrit.wikimedia.org/r/#/c/2141/ [20:39:32] paravoid: https://gerrit.wikimedia.org/r/#/c/15874/ [20:39:36] (i'm looking at old changes [20:40:21] <^demon> LeslieCarr: Not old, but if you're looking for easy merges... https://gerrit.wikimedia.org/r/#/c/35615/ [20:40:23] <^demon> :) [20:40:38] checking out now [20:41:15] ^demon - why the mixing of tabs and spaces ? https://gerrit.wikimedia.org/r/#/c/35615/1/files/gerrit/hooks/hookhelper_test.py [20:41:36] <^demon> Should all be spaces now, I was removing tabs. [20:42:00] there's lots of tabs [20:42:04] look at all the tabs! [20:42:18] <^demon> I don't see them on the right side of the diff. [20:42:26] <^demon> I see spaces in increments of 4. [20:42:34] really ? [20:43:00] <^demon> My diff options are: ignore whitespace all, whitespace errors and show tabs on [20:43:29] <^demon> If you don't have Show Tabs on, it's kind of hard to tell the difference between spaces and tabs. [20:43:53] i have show tabs, ignore whitespace none, whitespace errors [20:44:14] weird, if i update ignore whitespace it appears to be tabs [20:44:28] <^demon> Bah, stupid diff preferences. [20:44:38] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.032 seconds [20:44:38] but when i switch to ignore whitespace none it goes back to tabsa [20:44:47] <^demon> If you really want to verify, you could pull the patch and grep for ^\t [20:44:53] <^demon> I promise I changed all the tabs -> spaces. [20:44:55] that was what i was just about to say [20:44:56] :) [20:44:58] do [20:45:35] yep, when looking at it after pulling it's correct [20:45:40] why does gerrit hate me ? [20:46:11] New review: Lcarr; "FYI on gerrit preferences - ignore whitespace NONE this appears to still have tabs. Pulled the patc..." [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/35615 [20:46:12] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/35615 [20:46:42] wow, sockpuppet has a lot of patches to be merged and ironically has not run puppet in forever [20:47:53]