[00:08:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 00:07:58 UTC 2013 [00:09:01] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 00:08:58 UTC 2013 [00:10:01] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 00:10:40 UTC 2013 [00:11:01] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:31] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 00:11:22 UTC 2013 [00:12:01] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 00:11:56 UTC 2013 [00:13:01] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 00:14:48 UTC 2013 [00:15:01] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:17:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [00:24:18] New patchset: coren; "Tool Labs: Implement mail relay" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63399 [00:27:38] New patchset: coren; "Tool Labs: Implement mail relay" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63399 [00:29:38] New patchset: coren; "Tool Labs: Implement mail relay" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63399 [00:31:56] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63399 [00:48:41] New patchset: coren; "Tool Labs: Mail works, redirecting /var/mail" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63400 [00:50:21] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63400 [00:50:29] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [01:04:35] New patchset: coren; "Tool Labs: Mail config for other instances" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63401 [01:05:24] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63401 [01:54:10] New patchset: coren; "Tool Labs: Make mailrelay actually listen for mail" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63403 [01:56:27] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63403 [01:57:52] New patchset: coren; "Tool Labs: minor fix" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63404 [01:58:40] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63404 [02:02:23] I have been /seriously/ spamming gerrit these past couple days. :-) [02:06:03] !log LocalisationUpdate completed (1.22wmf3) at Mon May 13 02:06:03 UTC 2013 [02:06:12] Logged the message, Master [02:23:00] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [02:24:00] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [02:35:49] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon May 13 02:35:49 UTC 2013 [02:35:58] Logged the message, Master [03:19:44] TimStarling: I hope we won't break the cluster by deleting 7000 items on Wikidata [03:20:01] unlikely [03:20:14] as long as you do it in a single thread [03:20:17] * Jasper_Deng was afraid of a bigdelete-style disaster [03:56:31] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [03:56:31] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [03:56:31] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [04:07:58] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 04:07:56 UTC 2013 [04:08:08] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:08] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 04:09:03 UTC 2013 [04:09:08] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:08] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 04:10:03 UTC 2013 [04:10:08] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:08] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 04:10:58 UTC 2013 [04:11:08] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:48] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 04:11:46 UTC 2013 [04:12:08] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:38] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 04:12:27 UTC 2013 [04:13:08] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:08] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 04:13:02 UTC 2013 [04:14:08] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:58] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 04:14:49 UTC 2013 [04:15:08] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:10:39] good morning, could someone merge a few long pending changes please? [06:10:41] https://gerrit.wikimedia.org/r/#/q/owner:%22Yurik+%253Cyurik%2540wikimedia.org%253E%22+project:operations/puppet+status:open,n,z [06:11:45] the XFF spoofing is the most useful for testing, and the opera_mini -- very important for our ZERO users [06:30:18] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 06:30:12 UTC 2013 [06:30:38] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:31:08] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 06:30:58 UTC 2013 [06:31:38] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:31:48] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 06:31:38 UTC 2013 [06:32:38] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:07:50] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 08:07:49 UTC 2013 [08:08:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:10] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 08:09:01 UTC 2013 [08:09:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:20] PROBLEM - Packetloss_Average on analytics1003 is CRITICAL: CRITICAL: packet_loss_average is 15.8485892481 (gt 8.0) [08:14:21] PROBLEM - Packetloss_Average on analytics1006 is CRITICAL: CRITICAL: packet_loss_average is 15.0038864179 (gt 8.0) [08:14:30] PROBLEM - Packetloss_Average on analytics1004 is CRITICAL: CRITICAL: packet_loss_average is 14.9541200813 (gt 8.0) [08:14:50] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 08:14:48 UTC 2013 [08:15:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:18:20] RECOVERY - Packetloss_Average on analytics1003 is OK: OK: packet_loss_average is 0.54060637931 [08:18:21] RECOVERY - Packetloss_Average on analytics1006 is OK: OK: packet_loss_average is 1.50710939655 [08:18:21] PROBLEM - Packetloss_Average on oxygen is CRITICAL: CRITICAL: packet_loss_average is 13.8904496899 (gt 8.0) [08:18:30] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 14.2149421667 (gt 8.0) [08:18:31] PROBLEM - Packetloss_Average on analytics1008 is CRITICAL: CRITICAL: packet_loss_average is 15.2724807812 (gt 8.0) [08:18:31] RECOVERY - Packetloss_Average on analytics1004 is OK: OK: packet_loss_average is 0.934467580645 [08:32:52] RECOVERY - LVS HTTPS IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 90578 bytes in 8.456 second response time [08:32:52] PROBLEM - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:33:50] RECOVERY - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 76619 bytes in 3.960 second response time [08:34:20] RECOVERY - LVS HTTP IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 69025 bytes in 0.448 second response time [08:34:30] PROBLEM - Backend Squid HTTP on amssq43 is CRITICAL: Connection refused [08:35:08] RECOVERY - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 53741 bytes in 3.196 second response time [08:35:08] RECOVERY - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 69025 bytes in 5.270 second response time [08:35:08] PROBLEM - LVS HTTP IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.186 second response time [08:35:08] PROBLEM - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:08] PROBLEM - LVS HTTP IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:18] RECOVERY - LVS HTTP IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 90769 bytes in 3.514 second response time [08:35:28] PROBLEM - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:39] RECOVERY - LVS HTTP IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 69025 bytes in 0.667 second response time [08:35:45] wth is going on [08:35:48] PROBLEM - LVS HTTP IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:48] PROBLEM - LVS HTTPS IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:49] PROBLEM - LVS HTTPS IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:58] PROBLEM - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:08] PROBLEM - Puppet freshness on db26 is CRITICAL: No successful Puppet run in the last 10 hours [08:36:18] RECOVERY - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 40906 bytes in 0.722 second response time [08:36:18] RECOVERY - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 68567 bytes in 0.701 second response time [08:36:18] RECOVERY - LVS HTTP IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 69025 bytes in 0.353 second response time [08:36:18] RECOVERY - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 69456 bytes in 0.757 second response time [08:36:28] RECOVERY - LVS HTTP IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 69025 bytes in 0.430 second response time [08:36:28] RECOVERY - LVS HTTP IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 69025 bytes in 0.459 second response time [08:36:28] RECOVERY - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 47094 bytes in 0.951 second response time [08:36:28] RECOVERY - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 69025 bytes in 0.434 second response time [08:36:28] RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 49986 bytes in 0.631 second response time [08:52:28] RECOVERY - Backend Squid HTTP on amssq43 is OK: HTTP OK: HTTP/1.0 200 OK - 1421 bytes in 0.182 second response time [09:16:14] New patchset: Nemo bis; "(bug 48397) Add localised/v2 logos for Wikipedias without one III" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63422 [09:17:25] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [09:17:25] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [09:28:11] New review: Dereckson; "We're May 13, not the 21 ..." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/63424 [09:33:01] New review: Dereckson; "...and I should avoid to read -"21" when it's -"12" +"21"." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63424 [09:51:03] New patchset: Dereckson; "New setting: wgCascadingRestrictionLevels" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63425 [09:56:14] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [10:21:42] New patchset: Hashar; "beta protoproxy" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63431 [10:50:57] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:19] New review: Matmarex; "(1 comment)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63425 [10:53:03] New review: Odder; "All files are protected on Commons against uploads and moves." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/63422 [11:21:31] New review: ArielGlenn; "What do you think about checking for all missing commands and providing a list of them all at once, ..." [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/63390 [11:53:00] Change abandoned: Hashar; "replaced by https://gerrit.wikimedia.org/r/#/c/63431/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62582 [11:57:00] New patchset: Hashar; "protoproxy: mobile + beta support" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63431 [12:07:59] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 12:07:57 UTC 2013 [12:08:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 12:09:04 UTC 2013 [12:09:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 12:10:04 UTC 2013 [12:10:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 12:11:00 UTC 2013 [12:11:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:49] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 12:11:48 UTC 2013 [12:12:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:39] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 12:12:30 UTC 2013 [12:13:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 12:13:05 UTC 2013 [12:14:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:59] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 12:14:53 UTC 2013 [12:15:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:23:09] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [12:24:09] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [12:39:47] New patchset: Hashar; "protoproxy: mobile + beta support" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63431 [12:45:25] New review: Hashar; "* made proxy_configuration the main entry point (it is now init.pp)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63431 [12:48:53] go hashar [12:49:01] we'll need to do some protoproxy work soon for ulsfo [12:49:06] so we'll be able to help then [12:57:46] PROBLEM - Host cp3003 is DOWN: PING CRITICAL - Packet loss = 100% [13:00:00] mark: reviewing that with apergos :-] [13:00:22] mark: and I have discovered the very helpful create_resource() [13:01:09] that calls a define on each elements of a hash passing the hash key as name and the hash values as arguments to the define [13:01:28] please use 4 char spaces [13:01:32] :-D [13:01:34] indent i mean [13:01:37] *cough* [13:01:43] ? [13:02:01] ah we had this discussion earlier, puppet-lint etc [13:02:16] I also prefer the 4 char space indents [13:02:26] *invites paravoid to the tab / space madness* [13:03:14] I find 2 not nearly as readable [13:03:17] i don't mind spaces, but not 2 [13:03:40] I don't care between tabs and spaces as long as we pick one [13:03:47] we picked one [13:03:53] then you guys all started making it horribly inconsistent [13:03:58] "you guys" [13:06:09] so 4 spaces? [13:06:11] and: # vim:sw=4:ts=4:et: [13:06:21] either 4 spaces or tabs [13:06:46] I compromise here! [13:06:49] New patchset: Hashar; "protoproxy: mobile + beta support" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63431 [13:06:52] :-D [13:07:02] done :] [13:07:14] :) [13:07:51] ah [13:08:02] ideally I would not be lazy and make puppet-lint to recognize tabs [13:08:16] ideally we would pick something [13:09:06] at some point I thought I saw in one of these here channels, "either tabs or spaces as long as it's consistent within a given file" [13:10:27] yes [13:27:52] !log - update mwlib to 0.15.8 [13:28:05] !log - update mwlib.rl to 0.14.3 [13:28:13] !log - remove PIL [13:28:25] !log restarted all services [13:29:33] RECOVERY - Host cp3003 is UP: PING OK - Packet loss = 0%, RTA = 89.28 ms [13:32:23] PROBLEM - SSH on cp3003 is CRITICAL: Connection refused [13:36:10] PROBLEM - Host cp3003 is DOWN: PING CRITICAL - Packet loss = 100% [13:41:20] RECOVERY - Host cp3003 is UP: PING OK - Packet loss = 0%, RTA = 89.33 ms [13:48:00] PROBLEM - Host cp3003 is DOWN: PING CRITICAL - Packet loss = 100% [13:55:20] RECOVERY - SSH on cp3003 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:55:30] RECOVERY - Host cp3003 is UP: PING OK - Packet loss = 0%, RTA = 89.58 ms [13:57:00] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [13:57:00] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [13:57:00] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [14:02:00] RECOVERY - Varnish HTCP daemon on cp3003 is OK: PROCS OK: 1 process with UID = 997 (varnishhtcpd), args varnishhtcpd worker [14:02:20] RECOVERY - Varnish traffic logger on cp3003 is OK: PROCS OK: 2 processes with command name varnishncsa [14:04:10] RECOVERY - Varnish HTTP upload-frontend on cp3003 is OK: HTTP OK: HTTP/1.1 200 OK - 671 bytes in 0.175 second response time [14:06:51] PROBLEM - Host cp3003 is DOWN: PING CRITICAL - Packet loss = 100% [14:08:11] RECOVERY - Varnish HTTP upload-backend on cp3003 is OK: HTTP OK: HTTP/1.1 200 OK - 632 bytes in 0.175 second response time [14:08:11] RECOVERY - Host cp3003 is UP: PING OK - Packet loss = 0%, RTA = 90.41 ms [14:13:15] disconnection ftw [14:15:51] New patchset: Odder; "(bug 48379) Create custom logos for new private wikis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63468 [14:28:55] New patchset: Hashar; "protoproxy: mobile + beta support" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63431 [14:34:40] New patchset: Odder; "(bug 48404) Update fdcwiki and ombudsmenwiki favicons" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63473 [14:37:34] New patchset: Hashar; "contint: proxy only enabled on wikimedia.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63391 [14:40:59] New patchset: Hashar; "protoproxy: mobile + beta support" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63431 [14:42:01] !log gallium: deleted an ip route blacklist [14:42:09] pff [14:42:16] more bot disappeared nice [14:43:04] annnnd [14:43:06] novaadmin blocked me [14:43:26] New patchset: Odder; "(bug 47287) Configure $wgBabelCategoryNames for ca.wiktionary" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63474 [14:46:58] New patchset: Odder; "(bug 47287) Configure $wgBabelCategoryNames for ca.wiktionary" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63474 [14:49:40] New review: Dereckson; "(1 comment)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/63425 [14:50:51] New patchset: Dereckson; "New setting: wgCascadingRestrictionLevels" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63425 [14:51:06] New review: Dereckson; "PS2: fixed whitespace issue" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63425 [14:55:35] New review: Faidon; "No objections on performing purges but a note on the "should not happen": there are indeed discrepan..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63396 [14:56:11] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63391 [14:57:05] New review: Ottomata; "How many different packet loss metrics & alerts will this create? There are occasions where packet ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63220 [15:03:23] !log reedy synchronized php-1.22wmf2/cache/l10n/ [15:03:45] New review: Faidon; "Setting X-Carrier to X-CS is about the same as setting X-Carrier to "deprecated" or whatever dummy v..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/62867 [15:06:27] New review: Faidon; "I think you care more than we do (opera_mini is basically for Zero), but are you sure you should tru..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63077 [15:17:31] apergos: thanks for updating the Tin wikitech page! [15:21:59] greg-g: you're welcome [15:22:29] I'm still waiting to find out what tin is actually supposed to be used for besides deployments; when/if I have a bit more info I'll add it [15:24:18] <^demon> I dunno of any other planned usages for it other than deployment host. [15:24:33] well e.g. [15:24:42] are people supposed o be able to run sql/mysql on it? [15:25:05] what about noc? will that move from fenari there? (with some proxy on a public ip in front of it [15:25:06] ) [15:25:18] <^demon> For the first question, I'd assume so. [15:25:43] <^demon> About noc, I dunno. I brought it up on the gerrit change but got no response. [15:25:47] so it's that sort of thing I want to get clear about, I left rob a message since he seems to be in that loop [15:26:09] it is eary for him yt [15:26:10] yet [15:26:13] early [15:26:25] also it is late for chocolate for me [15:26:49] <^demon> I suppose it'd be sane to keep noc served from tin, but yeah we'd need a proxy or somesuch. [15:27:10] so /hme is mounted over there I guess [15:27:17] which would make it a natural location, yep [15:27:59] we don't necessary need noc to be on tin, do we? [15:28:05] also, could we puppetize all those files ? [15:28:13] graphite? is on fenari? [15:28:23] well it needs to be somewhere with /home [15:28:38] <^demon> LeslieCarr: Puppetize the *dblist and wmf config? [15:28:41] <^demon> No. [15:28:45] yeah and the interwikicache.cdb [15:28:46] n [15:28:51] ot so much [15:29:08] ok [15:29:19] i was thinking the pybal files [15:29:37] <^demon> That sounds reasonable yeah...I was thinking like MW config and such. [15:29:40] I wuld love the pybal files to not be "go to fenari/tin and update them in /home" [15:29:42] <^demon> And got scared :) [15:29:46] that would make me very happy [15:30:13] heh ^demon I was thinking the same thing [15:33:27] <^demon> Oh, am I correct in remembering you use Cyanogen? [15:33:55] uh [15:34:19] no, I used to have a rooted android but I don't even have a data plan righ tnow so meh [15:34:28] I just use the phone as a phone [15:34:52] <^demon> I took the brave leap this weekend and flashed CM 10.1 to my primary phone. [15:35:02] what's the new feature list? [15:35:15] <^demon> Android 4.2.2 :D [15:35:19] (bear in mind it's been at leasy 2 years now since I looked at it) [15:35:33] 4.2.2! ok I am running... [15:36:16] 2.2 :-D [15:36:54] <^demon> My phone had 4.1.1, with tons of crappy Samsung & TMo bloatware. [15:37:04] ah [15:37:26] wel I have an lg, not a lot of memory and dunno if it could take the updates [15:38:41] <^demon> https://en.wikipedia.org/wiki/List_of_devices_supported_by_CyanogenMod :) [15:39:02] eh I meant the more recent anroid releases [15:39:29] <^demon> What phone model do you have>? [15:39:50] RobH, you there? [15:40:07] I have the optimus one [15:40:13] p500 or 509 or something [15:40:18] ottomata: sup? [15:40:25] p509 [15:40:48] yoyo, i'm trying to find our previous conversations about stat1002….I'm checking on it for the first time now…I don't think it has enough disk space, but I thougth we talked about this so I must be remembering something wrong [15:41:08] checking on disks now, but /a there is 1.5TB, on stat1 it is 6.5TB [15:41:21] 7.2 that's the best I can do = android 2.3 [15:41:33] <^demon> apergos: Aww, listed as "unofficial" right now :( [15:41:58] ottomata: i recall them as well, i assigned stat1002 to a misc box, and it wasnt big enough [15:41:59] yeah not really worth doing [15:42:03] so we setup stat1002 on a db class machine [15:42:05] !log reedy synchronized php-1.22wmf4/ [15:42:05] which is what its on now [15:42:12] New review: Umherirrender; "Would be nice when this or I2ea15dc0 get merged this week, to get the module namespaces as review na..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62985 [15:42:29] That took some time [15:42:30] so, stat1: [15:42:31] root@stat1:~# pvscan [15:42:32] PV /dev/sda3 VG stat1 lvm2 [9.09 TiB / 1.65 TiB free] [15:42:39] stat1002: [15:42:40] root@stat1002:~# pvscan [15:42:40] PV /dev/sda6 VG tank lvm2 [1.59 TiB / 211.71 GiB free] [15:42:52] ottomata: 1521.3 GB is the size of the data partition [15:42:56] (for the mount) [15:43:03] aye [15:43:12] and /a on stat1: [15:43:14] if you want bigger than that, we have to purchase a machine specifically for you guys. [15:43:15] /dev/mapper/stat1-a 6.4T 3.0T 3.1T 50% /a [15:43:23] we're using 3TB there right now [15:43:24] i just assigned you the machine with most space that i had free [15:43:28] ayeee, err, hm [15:43:33] which means it has to be budgeted and purchased. [15:43:38] !log reedy synchronized w [15:43:56] ottomata: So yea, if you need more space, we can do it, but nothing we own can do it. [15:43:56] drdee ^^ [15:44:22] So it'll require you guys putting in a ticket and knowing we have to have ct/mark find/approve budget and purchase [15:44:34] which should be fine, just a bit more steps [15:44:41] we can also just reopen the old ticket if easier [15:44:42] aye [15:44:48] !log reedy synchronized docroot [15:45:25] New patchset: Reedy; "Add 1.22wmf4 symlinks" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63478 [15:45:38] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63478 [15:45:41] hrmm [15:45:51] thse look like identical machines, i wonder if the raid is just not set right [15:45:53] lemme see [15:46:23] yeah, I'm confused because I remember talking about this because of that first machine we tried to use [15:46:29] I went and looked at how much we needed [15:46:33] and then we went and found this machine [15:46:40] ottomata: AHHH [15:46:41] so I'm surprised that there's not enough space [15:46:42] i see the issue [15:46:45] stat1 is raid6 [15:46:48] Reedy: Have you cut wmf4 yet? [15:46:48] stat1002 os raid10 [15:46:48] ahhhh! [15:46:52] that is why [15:46:57] that was done by partman? [15:46:59] ottomata: So we can redo the raid on stat1002 [15:47:01] nope, hardware [15:47:02] yeah [15:47:05] hmm, ok [15:47:05] yeah [15:47:10] we can wipe it if we need to [15:47:14] I started puppetizing it [15:47:16] James_F: Yus [15:47:18] redoing the raid will wipe the data [15:47:21] and was in the process of rsyncing data from stat1 over [15:47:22] thats fine [15:47:26] its all still on stat1 [15:47:32] ottomata: can i handle this for you later this afternoon and you'll have it back tomorrow ? [15:47:33] 72 minutes ago apparently [15:47:37] Reedy: Argh. Urgent additional fix for VE; should we cherry-pick or do you want to do it? [15:47:46] ottomata: atleast no new machines are needed =] [15:47:51] yay! [15:47:53] yeah much better [15:47:59] yeah we can reinstall or whatever is needed for this thing [15:48:08] i'll get its raid identical to stat1 and reinstall it later today [15:48:13] ok cool, tahnk sso much [15:48:25] yeah, no hurries on this one, we're just doing this as we can anyway [15:48:26] no worries, sorry it was deployed wrong the first time, i should have noticed [15:48:32] lemme know when it is back up [15:48:36] no probs, thank you! [15:48:46] quite welcome =] [16:00:11] New review: Yurik; "Faidon, this patch does not remove any ranges, only adds them. The reason for the shorter list is th..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63077 [16:01:22] hi paravoid, you there? got a sec to discuss kafka deb stuff? [16:01:24] paravoid, opera mini list is bigger, not smaller - I ran it through CIDR optimizer [16:01:58] hehe, paravoid is in demand :) [16:07:59] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 16:07:55 UTC 2013 [16:08:06] !log reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf4 and rebuild l10n cache [16:08:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 16:09:05 UTC 2013 [16:09:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 16:10:04 UTC 2013 [16:10:39] RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Mon May 13 16:10:30 UTC 2013 [16:10:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 16:11:00 UTC 2013 [16:11:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:11] dear ops, please merge this, our Zero users are getting large red banner instead of proper ones, causing problems and calls to tech support. https://gerrit.wikimedia.org/r/#/c/63077/ [16:12:39] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 16:12:34 UTC 2013 [16:12:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 16:13:07 UTC 2013 [16:13:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Mon May 13 16:15:00 UTC 2013 [16:15:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:15] !log authdns-update to add barium.frack.eqiad.wnnet [16:22:40] New patchset: Mark Bergsma; "Add AFTv5 archive feedback cron job" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62602 [16:23:06] New patchset: Andrew Bogott; "Added manifest for rt4 running with Apache." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63213 [16:23:56] New patchset: Mark Bergsma; "Readd cp3003" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63481 [16:24:58] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63481 [16:29:33] mark, poke ^^ [16:34:24] PROBLEM - RAID on ms-be6 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:35:56] PROBLEM - RAID on ms-be6 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:37:12] PROBLEM - Apache HTTP on mw1157 is CRITICAL: Connection timed out [16:37:42] PROBLEM - Apache HTTP on mw1159 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:42] PROBLEM - Apache HTTP on mw1155 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:42] PROBLEM - Apache HTTP on mw1160 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:42] PROBLEM - Apache HTTP on mw1156 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:50] !log reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf4 and rebuild l10n cache [16:37:57] swift cluster is unhappy [16:38:42] PROBLEM - Apache HTTP on mw1158 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:42] PROBLEM - Apache HTTP on mw1154 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:42] PROBLEM - RAID on ms-be8 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:39:09] ugh [16:39:09] Just a bit [16:39:33] RECOVERY - Apache HTTP on mw1158 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.055 second response time [16:39:33] RECOVERY - Apache HTTP on mw1154 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.037 second response time [16:39:52] PROBLEM - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:00] jobrunners, imagescallers and apackes all not exactly loving it [16:40:02] RECOVERY - Apache HTTP on mw1157 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.051 second response time [16:40:24] uh, what just happened? [16:40:31] looking [16:40:32] RECOVERY - Apache HTTP on mw1160 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.049 second response time [16:40:33] RECOVERY - Apache HTTP on mw1155 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.050 second response time [16:40:33] RECOVERY - Apache HTTP on mw1156 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.052 second response time [16:40:33] I don't know [16:40:35] kk [16:40:42] RECOVERY - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 68683 bytes in 0.170 second response time [16:40:58] note that I pooled cp3003 earlier [16:41:21] !log reedy synchronized php-1.22wmf4/extensions/VisualEditor/ [16:41:24] it's doing around 600 misses/s to eqiad varnish atm, so that shouldn't be a problem, but be aware of it [16:41:34] load is starting to drop on the swift boxes fwiw [16:41:34] RECOVERY - Apache HTTP on mw1159 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.048 second response time [17:56:06] New review: Sanja pavlovic; "That sounds like a good idea. Honestly, I'm little messed up today, so I won't have time to deal wit..." [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/63390 [17:56:08] paravoid, i see your point, agree, will postpone that patch until begining of july. And the last pending patch -- https://gerrit.wikimedia.org/r/#/c/62103/ -- the testing it allows will be a great help to our whole team, hope I addressed your concorns in the patch reply [17:56:48] Thanks mark! [17:58:55] !log reedy synchronized php-1.22wmf4/extensions/VisualEditor/ [18:00:18]