[00:46:06] New patchset: Catrope; "Fix the target of the node_modules symlink" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49188 [00:46:16] Ryan_Lane: Quick typofix ---^^ [00:46:38] New review: Ryan Lane; "Patch Set 1: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/49188 [00:46:46] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49188 [00:50:19] RoanKattouw: ok. I've sync'd the module out [00:50:23] RoanKattouw: it should work now [00:50:38] Thanks [00:50:42] yw [00:50:50] I synced the wrong module anyways, which brought down the service, so I've already fixed it with dsh [00:50:58] heh [00:51:05] (It's associated with Parsoid, not config as I thought) [00:51:09] * Ryan_Lane nods [00:53:01] I'll sabotage a Tampa machine and test it [00:53:07] cool [00:56:21] drdee: hey [00:59:52] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [01:04:44] New review: Faidon; "Patch Set 2: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/49074 [01:06:57] New patchset: Ori.livneh; "Explicitly add Schema NS to test2" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/49191 [01:07:48] New review: Ori.livneh; "Patch Set 1: Code-Review+2" [operations/mediawiki-config] (master) C: 2; - https://gerrit.wikimedia.org/r/49191 [01:07:55] New review: Ori.livneh; "Patch Set 1: Verified+2" [operations/mediawiki-config] (master); V: 2 - https://gerrit.wikimedia.org/r/49191 [01:07:56] Change merged: Ori.livneh; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/49191 [01:09:58] !log olivneh synchronized wmf-config/CommonSettings.php 'Enabling Schema NS on test2wiki' [01:09:59] Logged the message, Master [01:12:12] New review: MZMcBride; "Patch Set 1:" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/49191 [01:19:24] !log deploying squid changes for s/knams/esams/ for knsq* squids (see git log for rationale) [01:19:28] Logged the message, Master [01:21:01] RECOVERY - Backend Squid HTTP on knsq24 is OK: HTTP OK HTTP/1.0 200 OK - 1420 bytes in 0.239 seconds [01:21:33] up after 159d [01:21:58] hm [01:22:01] and others are down dammit [01:30:55] RECOVERY - MySQL disk space on neon is OK: DISK OK [01:31:01] LeslieCarr: what's the deal with the neon nagios alerts? seems to have been happening for 18+ hours [01:31:06] ^^ [01:31:18] couldn't have timed that better [01:36:44] New patchset: Catrope; "Puppetize the /var/lib/parsoid/Parsoid symlink" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49193 [01:37:07] * RoanKattouw looks around for this week's duty monkey [01:37:20] Boo, notpeter isn't on IRC [01:47:39] New patchset: Jeremyb; "redirect wikimediafoundation.info -> wmfwiki" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49194 [01:48:54] New patchset: Jeremyb; "redirect wikimediafoundation.info -> wmfwiki" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49194 [01:53:03] New patchset: Jeremyb; "redirect wikimediafoundation.info -> wmfwiki" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49194 [02:07:25] jeremyb_, if you'd like to do https://rt.wikimedia.org/Ticket/Display.html?id=4240 too? [02:07:46] Thehelpfulone: i'm in the middle of a rebase... [02:07:52] ah, finished [02:07:57] heh ok [02:08:58] New patchset: Jeremyb; "Linkify RT a little more liberally." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49196 [02:09:00] idk if that will actually work... [02:09:22] i guess i should also log in to gerrit [02:30:05] !log LocalisationUpdate completed (1.21wmf9) at Fri Feb 15 02:30:05 UTC 2013 [02:30:09] Logged the message, Master [02:35:40] New patchset: Jeremyb; "redirects for wikiartpedia.{biz,co,info,me,mobi,net}" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49197 [02:35:46] Thehelpfulone [02:36:53] i guess it's pretty late there actually :) [02:46:17] New review: Jeremyb; "Patch Set 1:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49197 [03:07:24] New patchset: Jeremyb; "redirect wikimediafoundation.info -> wmfwiki" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49194 [03:09:36] New review: Jeremyb; "Patch Set 4:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49194 [03:54:49] New patchset: Jeremyb; "fix https://bug-attachment.wikimedia.org/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49200 [04:18:26] New patchset: Jeremyb; "fix https://bug-attachment.wikimedia.org/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49200 [04:21:29] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 200 seconds [04:30:20] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 184 seconds [04:31:23] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 198 seconds [04:38:40] PROBLEM - Puppet freshness on stafford is CRITICAL: Puppet has not run in the last 10 hours [05:28:46] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [05:59:49] RECOVERY - MySQL disk space on neon is OK: DISK OK [06:05:40] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [06:30:43] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [06:42:47] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 187 seconds [06:42:56] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 181 seconds [06:48:02] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [06:48:11] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [06:55:51] New review: Ori.livneh; "Patch Set 2: Code-Review-1" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/49200 [07:26:13] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [07:26:31] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [07:30:43] Aaron|home: around? [07:33:10] barely [07:33:28] man those headache pills seemed to work [07:34:27] Echo is throwing exceptions; https://gerrit.wikimedia.org/r/#/c/49088/ would fix it [07:34:56] it was merged, the submodule ref updated, and a pull on fenari, but no sync, it seems [07:36:57] trace: http://dpaste.de/zJEbr/raw/ [07:39:21] I'd be comfortable syncing it -- it's on testwiki at the moment. would there be anyone around? apergos? [07:39:43] I'm here [07:39:49] what are we syncing? [07:40:04] https://gerrit.wikimedia.org/r/#/c/49088/, fixes http://dpaste.de/zJEbr/raw/ [07:40:28] merged and on testwiki already but someone forgot to sync, I think. [07:41:28] how widely is the notification stuff used? [07:41:53] Nemo_bis got it just trying to add a message to a talk page, I think. [07:41:55] I see that code on srv230 and fenari [07:42:10] what's srv230? [07:43:33] some random appserver [07:43:49] why is it in decommissioning.pp? [07:43:54] anyways... [07:44:22] go ahead I guess, like you say it's already merged [07:46:34] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [07:46:42] Clearly we need an auto-scap bot to run every few hours. Keep everyone on their toes. [07:47:43] Susan: like chaos monkey? [07:47:59] Something like that. [07:50:28] i think auto-scap isn't the answer [07:51:03] some combination of something like chaos monkey and also git-deploy instead of scap is better [07:51:49] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [07:53:34] apergos / Aaron|home - sorry, not going to after all. it appears to have occurred only once and i can't determine the state of the code on mw1181 to my satisfaction [07:53:52] so going to report it and leave it for now [07:54:03] as long as the next scapper doesn't push it unknowingly [07:54:06] all righty [07:57:07] * apergos autoscaps Susan [07:57:20] :D [07:58:08] apergos: well, s/he would, and a revert could make things worse if the code was half-deployed somehow (confused by aaron seeing it on srv230) [07:58:33] so if you're here, I will sync, if that's ok. [07:58:39] :-D :-D [07:58:43] PROBLEM - Puppet freshness on virt0 is CRITICAL: Puppet has not run in the last 10 hours [07:58:59] i take that to be "yes" [07:59:13] can you stick around for a bit in case it needs to be undone? [07:59:19] if so, then it's a "yes" [07:59:28] yes, of course [07:59:33] then go ahead [07:59:46] PROBLEM - Puppet freshness on labstore3 is CRITICAL: Puppet has not run in the last 10 hours [08:02:01] !log olivneh synchronized php-1.21wmf9/extensions/Echo 'Syncing Iab077a77aa75a6ff0cf03acecca38403595dda93 to fix exception 640c60c4' [08:02:04] Logged the message, Master [08:02:23] Now there you have a nice, human-readable message [08:02:31] I should have removed the words and kept just the hashes [08:02:32] :) [08:02:51] At least one can grep the IRC logs to have context ^^ [08:17:10] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 181 seconds [08:17:37] RECOVERY - MySQL disk space on neon is OK: DISK OK [08:18:49] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 223 seconds [08:20:37] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 5 seconds [08:20:37] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 2 seconds [08:38:39] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 183 seconds [08:39:24] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 181 seconds [08:41:12] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 5 seconds [08:42:15] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [08:49:43] bock [09:07:54] PROBLEM - Puppet freshness on mw1003 is CRITICAL: Puppet has not run in the last 10 hours [09:25:57] New review: Hashar; "Patch Set 1: Code-Review+1" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/49133 [09:26:44] Susan: I might help you to edit the foundation wiki :-] [09:27:28] Susan: http://wikimediafoundation.org/wiki/Staff_and_contractors list me wearing an invisible cloak . I have uploaded yesterday a pic of me taken by Victor last year when I was in SF. I have uploaded it on commons : https://commons.wikimedia.org/wiki/File:Antoine_Musso-3500.jpg [09:32:02] hashar, done: https://wikimediafoundation.org/w/index.php?title=Template:Staff_and_contractors&diff=87693&oldid=87675 [09:32:48] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 181 seconds [09:32:57] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 183 seconds [09:33:13] MaxSem: thanks! [09:33:29] Susan: zoned by MaxSem :-] [09:43:36] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 2 seconds [09:43:45] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 214 seconds [09:44:03] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 227 seconds [09:45:15] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [09:46:10] hashar: Do you have an account there? [09:46:16] If not, I'd be happy to make one for you. [09:47:30] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 10 seconds [09:48:21] Susan: I don't have any account on the WMF wiki [09:48:25] Susan: not sure I need any [09:48:32] Fair enough. [09:48:53] there is already tons of helpful people on there ;-] That is the first time I actually have to edit something [09:49:00] the quality of that wiki is just too high [09:49:00] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 2 seconds [09:49:42] Hah, it's a mess. [09:49:52] Everything is quickly outdated due to the fishbowl-ness of the wiki. [09:50:07] We're working on opening it up, but it's a very slow process. [09:52:57] New patchset: Jeremyb; "make logo protorel instead hardcoding HTTP" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49215 [10:02:19] New patchset: Ori.livneh; "(RT 2490) HTTPS on bug-attachment.wikimedia.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49200 [10:04:09] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [10:04:13] New review: Ori.livneh; "Patch Set 3: Code-Review-1" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/49200 [10:15:28] New review: Ori.livneh; "Patch Set 1:" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/49191 [10:29:26] hashar, Victor usually likes to add photos he's taken to his user page too: https://commons.wikimedia.org/wiki/User:Vgrigas [10:34:42] New patchset: Hashar; "Jenkins module created out of contint manifests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47664 [10:36:06] New review: Hashar; "Patch Set 6:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47664 [10:36:21] RECOVERY - MySQL disk space on neon is OK: DISK OK [10:36:21] I hate you puppet [10:39:50] New review: Hashar; "Patch Set 6:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47664 [10:55:29] New patchset: Hashar; "Jenkins module created out of contint manifests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47664 [10:58:09] hashar: speaking of which, I noticed the nice gallery but I'm prety sure one or two of the team members without a photo do have a good one on Commons [11:02:33] New patchset: Hashar; "Jenkins module created out of contint manifests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47664 [11:03:06] New review: Hashar; "Patch Set 8:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47664 [11:03:47] New review: Silke Meyer; "Patch Set 3: Code-Review-1" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/48979 [11:10:37] Nemo_bis: I have updated the platform engineering team section https://www.mediawiki.org/wiki/Wikimedia_Platform_Engineering#Team [11:10:49] Nemo_bis: feel free to add pics for people missing. [11:13:17] New patchset: Jalexander; "planet config updates . add 2 update 2 de,en,it" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49220 [11:16:07] out for lunch [11:16:25] New patchset: Hashar; "Jenkins module created out of contint manifests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47664 [11:21:59] New review: Silke Meyer; "Patch Set 3:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48979 [11:25:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:39] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.860 seconds [11:48:28] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [11:55:38] is apache-graceful-all in puppet somewhere? looks like not.. [11:55:52] we need to fix it to also work on eqiad Apaches [11:58:05] more precise, it is /usr/bin/apache-sanity-check which is used by graceful-all [12:00:12] ah.. "it's in the wikimedia-task-appserver Debian package" [12:07:10] New patchset: Dzahn; "apache-sanity-check needs to use new regex to also work on eqiad Apaches see details in RT-4449" [operations/debs/wikimedia-task-appserver] (master) - https://gerrit.wikimedia.org/r/49231 [12:15:30] New patchset: Silke Meyer; "Inline documentation for Wikidata's puppet files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49232 [12:16:49] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 184 seconds [12:18:29] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [12:19:58] RECOVERY - MySQL disk space on neon is OK: DISK OK [12:27:19] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 186 seconds [12:28:32] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 215 seconds [12:58:49] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [12:59:16] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [13:04:02] re [13:08:40] New patchset: Hashar; "Jenkins module created out of contint manifests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47664 [13:09:21] New review: Hashar; "Patch Set 10:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47664 [13:20:28] New review: Hashar; "Patch Set 10: Verified+1" [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/47664 [13:20:36] New patchset: Hashar; "contint::website regroups apache + basic files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47742 [13:36:01] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 185 seconds [13:37:40] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 233 seconds [13:46:47] New patchset: Hashar; "contint::website regroups apache + basic files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47742 [13:46:47] New patchset: Hashar; "Jenkins module created out of contint manifests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47664 [13:50:07] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [13:50:07] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [13:51:41] New patchset: Hashar; "contint::website regroups apache + basic files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47742 [14:02:42] mutante: wanna review yet another beta hack change ? :-D [14:03:13] New review: Dzahn; "Patch Set 4: Code-Review+2" [operations/apache-config] (master) C: 2; - https://gerrit.wikimedia.org/r/49194 [14:04:57] hashar: i can take a look in a little while.. when i [14:05:04] ok :-] [14:05:06] m done with another thing [14:05:18] btw, apache-config repo does not have Jenkins, right [14:05:43] i mean, need to manually hit Verified.. [14:06:14] New review: Dzahn; "Patch Set 4: Verified+2" [operations/apache-config] (master); V: 2 - https://gerrit.wikimedia.org/r/49194 [14:06:15] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49194 [14:08:57] mw1085: rsync: mkdir "/usr/local/apache/conf" failed: No such file or directory (2) [14:09:16] {{somakeit}} [14:09:22] xD [14:09:26] mutante: yeah there are not linting test yet for our apache-config [14:11:30] sync-apache does not sync properly [14:15:44] <^demon> !log [gerrit] updated replication plugin to 20c6fc7e (1.1-SNAPSHOT) which is back in line with upstream [14:15:48] Logged the message, Master [14:23:43] ah, srv233/234 are in decom and not in Nagios, but still running ..that explains [14:24:50] dzahn is doing a graceful restart of all apaches [14:25:29] !log dzahn gracefulled all apaches [14:25:30] Logged the message, Master [14:28:41] !log gracefulling Apaches in eqiad using dsh [14:28:42] Logged the message, Master [14:31:34] !log wikimediafoundation.info activated to redirect .org [14:31:36] Logged the message, Master [14:32:44] how many of these alias domains do we have?:) [14:32:58] too many ?:) [14:33:07] waiting for .museum for GLAM :) [14:33:31] hehe, not for real, but it would match [14:35:21] mutante, know anything about Nagios plugins? [14:35:57] yep, which one? [14:36:14] there are several different packages, plugins-basic, -standard, -extra, -bla [14:36:38] wikimediafoundation.info works for you too, right? [14:36:55] I was wondering if you can review https://gerrit.wikimedia.org/r/47111 [14:37:14] "This domain points to a Wikimedia Foundation server, but is not configured on this server." [14:38:03] hmm sure its not browser cache? [14:38:08] it did that before the change [14:38:21] I didn't visit it before [14:38:27] hrmm [14:38:47] hrm, force-reload did it:) [14:38:52] weird [14:38:53] looks like that is more about Solr [14:38:58] ah, cool [14:40:04] PROBLEM - Puppet freshness on stafford is CRITICAL: Puppet has not run in the last 10 hours [14:40:06] i am probably not a good reviewer for Python, but the return values look good, 0/1/2 for OK/WARN/CRIT [14:40:18] but the Solr part.. ehmm.. [14:41:16] since no op knows much about Solr, you all will have to believe me:P [14:42:16] i think in this case i can merge it to just try if it works, it doesnt break anything, worst case is a broken check we don't have yet anyways [14:42:59] I'm not sure how to actually register it with Nagios though [14:43:03] it won't show up in Nagios [14:43:11] by just adding this one file, yea [14:43:53] i have done that part before.. that shouldn't be a big issue as long as the check works [14:44:37] New review: Dzahn; "Patch Set 4: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/47111 [14:44:47] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47111 [14:46:00] MaxSem: have a host:port this should work on? [14:46:26] eh, i see 8983 as port [14:48:05] check_solr -r -a 400:600 -t 5 [14:48:24] it uses the right port bydefault [14:49:32] ValueError: need more than 0 values to unpack [14:50:15] line 190 [14:50:49] on which host? [14:51:30] search1001 [14:52:09] ugh, I don't have access to it [14:52:31] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [14:52:49] trying on fenari [14:53:08] ah, solr1001 ! [14:53:13] not search1001 [14:54:34] unable to open database file [14:55:03] where did you expect the sqlite db? [14:55:18] let me commit a manifest for it [14:55:29] db_conn = sqlite3.connect('/var/lib/check_solr/checks.sqlite3') [14:56:24] we don't have that on neon or spence [14:56:40] where neon is Icinga and spence is Nagios [14:58:17] ok, committing a followup [14:58:49] spence has sqlite the package installed [14:58:57] neon does not, just libsqlite [15:00:25] mutante: if you have finished with the apache config, would you mind reviewing my beta hack please? https://gerrit.wikimedia.org/r/#/c/47398/2 [15:00:41] hashar: yes, it's your turn [15:03:13] hashar: yea, that makes sense, Apache waits for the sync to finish before it starts.. ack [15:03:26] \O/ [15:03:31] that is a bit of a hack [15:04:09] New review: Dzahn; "Patch Set 2: Code-Review+2" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/47398 [15:04:17] New review: Mark Bergsma; "Patch Set 5: Code-Review+1" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/47067 [15:04:21] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47398 [15:04:23] using /bin/true , yea [15:04:55] credits go to platinides :-] [15:05:20] going to test out an apache linter [15:05:30] cool [15:05:33] gsed -i s%/etc/apache2/wmf/%% * [15:05:34] httpd -t -d "$WORKSPACE" -f all.conf [15:05:36] oh man [15:05:38] apache2 [15:07:46] hashar: remnant.conf :p [15:08:33] mutante, I'm lost: Synaptic shows that I have no SQLite library for Python, yet the script works [15:10:42] MaxSem: on spence, where sqlite is installed i get this: [15:10:47] AttributeError: _ElementInterface instance has no attribute 'iter' [15:11:04] ha [15:11:05] New patchset: Hashar; "Jenkins job validation (DO NOT SUBMIT)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49243 [15:11:27] looks like a no n-compatible ElementTree [15:11:33] MaxSem: and there is just no /var/lib/check_solr/ where it looks for the db [15:11:38] that doesn't work as expected sniff [15:11:44] you would have to create that via puppet [15:11:50] 15:11:08 Syntax error on line 3 of /var/lib/jenkins/jobs/operations-apache-config-lint/workspace/www.wikipedia.conf: [15:11:50] 15:11:08 Invalid command 'Redirect', perhaps misspelled or defined by a module not included in the server configuration [15:11:52] yeah, doing it now [15:11:56] ok, cool [15:12:09] but not sure what packages are needed [15:12:34] spence has both, sqlite and sqlite3 [15:12:42] i suppose just sqlite3 is good enough [15:12:47] sqlite is 2.x [15:13:18] mutante, the error you posted is not related to sqlite [15:13:20] icinga doesnt have either yet, so that means it probably wasnt puppetized [15:13:42] no, it's not [15:13:51] but that's what i get where sqlite is installed [15:14:27] and where it's not installed i get the "unable to open database file" [15:15:57] heh, I tried to uninstall python-libxml2 and apt told me that it would involve uninstalling ubuntu-desktop:P [15:16:52] grmblb [15:16:58] WE do not have ack-grep in production [15:17:00] seriously [15:17:02] :D [15:17:20] hashar, and not even MC:] [15:19:16] hashar: does it not get that mod_alias is installed? [15:19:24] yeah that is the issue [15:19:30] heh [15:19:31] /usr/sbin/apache2 -t -d "$PWD" -C 'Include /etc/apache2/mods-enabled/*.load' -C 'Include /etc/apache2/mods-enabled/*.conf' -f all.conf [15:19:41] that would hack it to load the global conf [15:19:41] yes, alias.conf [15:19:59] aha [15:20:10] ok, brb [15:20:20] New review: Hashar; "Patch Set 1:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49243 [15:21:54] New review: Hashar; "Patch Set 1:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49243 [15:22:13] RECOVERY - MySQL disk space on neon is OK: DISK OK [15:23:04] New review: Hashar; "Patch Set 1:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49243 [15:23:18] bah [15:23:19] zuul ignores me [15:25:23] mutante, what python version does neon have? [15:25:27] New review: Hashar; "Patch Set 1:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49243 [15:27:53] New review: Hashar; "Patch Set 1:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49243 [15:29:45] * MaxSem wonders if the script can be fixed by using specifically python2.7 [15:30:16] MaxSem: 2.7.3 [15:30:22] weird [15:30:29] meh [15:31:32] MaxSem: once it has that directory, /var/lib/check_solr it creates the db [15:31:43] but then TypeError: object of type 'NoneType' has no len() [15:35:10] be back after food [15:35:31] mutante: notice: /Stage[main]/Applicationserver::Config::Apache/Exec[Fake sync apache wmf config on beta]/returns: executed successfully [15:35:33] \O/ [15:35:33] thx [15:36:04] New review: Hashar; "Patch Set 2:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47398 [15:37:04] heh, that error is indeed caused by python2.6 [15:37:05] mutante: that was bug https://bugzilla.wikimedia.org/show_bug.cgi?id=38996 [15:39:50] New review: Hashar; "Patch Set 1:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49243 [15:39:56] :-((( [15:43:09] New review: Hashar; "Patch Set 1:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49243 [15:43:20] 2013-02-15 15:43:09,219 INFO zuul.Jenkins: Launch job operations-apache-config-lint for change with dependent changes [] [15:43:21] \O/ [15:43:49] 15:43:13 Invalid command 'ExpiresActive', perhaps misspelled or defined by a module not included in the server configuration [15:43:51] good luck [15:47:50] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 185 seconds [15:48:08] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 184 seconds [15:52:06] New patchset: MaxSem; "Solr monitoring fixes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49250 [15:52:17] mutante, ^^^ [15:53:15] New review: Hashar; "Patch Set 1:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49243 [15:54:09] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49074 [15:55:04] New review: Hashar; "Patch Set 1:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49243 [15:59:26] apache linter is on [16:00:46] !log Enabled a linter for operations/apache-config . Any apache typo in operations/apache-config will publicly write your name on the typo wall of shame. See {{gerrit|49251}} for implementation details and https://integration.mediawiki.org/ci/job/operations-apache-config-lint/ for some build examples. [16:00:48] Logged the message, Master [16:03:38] * hashar dance [16:03:55] Change abandoned: Hashar; "(no reason)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49243 [16:04:38] New review: Hashar; "Patch Set 2:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/24407 [16:05:11] New review: Hashar; "Patch Set 4:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/47088 [16:05:53] New review: Hashar; "Patch Set 1:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49197 [16:06:00] New review: Hashar; "Patch Set 2:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49069 [16:06:08] New review: Hashar; "Patch Set 1:" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/48868 [16:06:38] https://gerrit.wikimedia.org/r/#/q/project:operations/apache-config+is:open,n,z [16:06:40] all +1 ed [16:06:42] \|O/ [16:07:11] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [16:10:43] New patchset: Jgreen; "building db29 for RT #4518" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49254 [16:11:51] * Jeff_Green is confused by apparent gerrit UI changes--where did the [review side-by-side] etc go? [16:15:23] Jeff_Green: they are no more buttons but links [16:15:33] Jeff_Green: just below the list of files for the patchset [16:15:39] ahh i see them now. thx [16:15:43] :D [16:15:45] congrats [16:15:48] lol [16:15:49] New patchset: Ottomata; "Fixing hdfs::sync rsync command" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49255 [16:16:02] New review: Jgreen; "Patch Set 1: Verified+2 Code-Review+2" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/49254 [16:16:03] New review: Ottomata; "Patch Set 1: Verified+2 Code-Review+2" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/49255 [16:16:08] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49254 [16:16:15] mark: so I guess no varnish is going to be deployed on a friday evening ? :-] [16:16:15] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49255 [16:16:44] ah, Jeff_Green [16:16:53] did we just collide? [16:16:59] we merged at the same time I think [16:17:01] ja [16:17:04] db29 ok to merge? [16:17:07] yes [16:17:10] k doing [16:17:11] i just merged yours actually [16:17:13]