[00:00:05] well i have to go fight with the philippines and then dinner [00:00:15] have fun with graphite :) [00:00:19] that is a quip for sure [00:00:20] thanks [00:00:40] Reedy: you can configure arbitrary long retention for arbitrary resolution of data [00:01:09] set it to infinity! [00:01:10] like 'keep data at one-minute resolution for 1 day, then at 10-minute resolution for a week, then at an hourly resolution for 5 years' [00:01:18] and in fact you *have* to declare these rules [00:01:40] if they were hard-coded, i'd say "ok, i guess i can live with that" and move on with my life [00:02:40] Use either 7 or 42 [00:02:55] good idea [00:03:17] or 9001 in you need a really big value [00:04:05] ∃ n ∈ N: n > 9000 [00:04:24] iff you say so [00:04:48] actual graphite config options: [00:05:12] 'ENABLE_MANHOLE' [00:05:36] 'PICKLE_RECEIVER_PORT' [00:05:42] Where is the man sized hole? [00:06:26] it's that pickle receiver port over there [00:06:45] i like this one: [00:06:48] USE_FLOW_CONTROL = True [00:06:53] what happens if you set that to false?! [00:07:08] you'll flood downstream [00:07:23] haha [00:07:30] did you look this up? :P [00:29:26] haha, noope [00:31:01] ori-l: Reedy's hair is soo big because it already contains all this knowledge [01:08:48] (03CR) 10TTO: [C: 04-1] "shellpolicy, see bug" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92070 (owner: 10Vogone) [01:50:47] (03CR) 10Peachey88: [C: 04-1] "Commit message needs to be updated to match our guidelines." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92070 (owner: 10Vogone) [02:06:57] !log LocalisationUpdate completed (1.22wmf22) at Sun Oct 27 02:06:57 UTC 2013 [02:07:25] Logged the message, Master [02:12:09] !log LocalisationUpdate completed (1.23wmf1) at Sun Oct 27 02:12:09 UTC 2013 [02:12:23] Logged the message, Master [02:15:56] (03CR) 10TTO: "@Peachey88: you know you can edit the commit message yourself? Click the little icon next to "permalink"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92070 (owner: 10Vogone) [02:26:02] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Oct 27 02:26:02 UTC 2013 [02:26:18] Logged the message, Master [02:28:19] (03CR) 10Peachey88: "I'm more than aware, But I'm also one for encouraging people to fix their own mistakes." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92070 (owner: 10Vogone) [06:47:14] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:48:14] RECOVERY - DPKG on snapshot3 is OK: All packages OK [06:55:14] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:58:24] PROBLEM - Swift HTTP backend on ms-fe1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:04] PROBLEM - RAID on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:59:55] RECOVERY - RAID on snapshot3 is OK: OK: no RAID installed [07:00:14] RECOVERY - Swift HTTP backend on ms-fe1003 is OK: HTTP OK: HTTP/1.1 200 OK - 343 bytes in 0.006 second response time [07:01:14] RECOVERY - DPKG on snapshot3 is OK: All packages OK [07:14:04] PROBLEM - RAID on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:14] PROBLEM - Disk space on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:14] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:37] apergos: snapshot3 ^ [07:15:13] ignore, thanks [07:15:14] RECOVERY - Disk space on snapshot3 is OK: DISK OK [07:15:14] RECOVERY - DPKG on snapshot3 is OK: All packages OK [07:18:14] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:19:14] PROBLEM - Disk space on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:21:14] RECOVERY - Disk space on snapshot3 is OK: DISK OK [07:24:54] * p858snake|l looks at apergos with https://bugzilla.wikimedia.org/show_bug.cgi?id=56211 [07:25:13] someone might want to do a shell patch to turn echo off [07:25:14] PROBLEM - Disk space on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:25:19] (or create the required dbs) [07:25:45] * apergos looks at p858snake|l with 'it's sunday morning, no way am I doing work unless the site falls over' [07:26:00] hah [07:26:11] Don't make me take your coffee hostage [07:26:17] (still trrying to kick this cold/cough that has been hanging on for days and days) [07:26:19] is that kannada? [07:26:21] I don't have coffee [07:26:31] apergos: tea? [07:26:41] non caffeinated [07:26:54] yes, i drink herbal mostly [07:26:54] RECOVERY - RAID on snapshot3 is OK: OK: no RAID installed [07:26:57] oh god, someone else like me [07:27:04] RECOVERY - Disk space on snapshot3 is OK: DISK OK [07:27:05] but i drink insane amounts of coke [07:27:14] RECOVERY - DPKG on snapshot3 is OK: All packages OK [07:27:49] p858snake|l: seems not entirely broken [07:28:17] I don't do soda either [07:30:30] * apergos goes in search of morning comfort food [07:40:14] PROBLEM - Disk space on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:14] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:41:04] PROBLEM - RAID on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:41:04] PROBLEM - SSH on snapshot3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:17:54] RECOVERY - RAID on snapshot3 is OK: OK: no RAID installed [08:17:54] RECOVERY - SSH on snapshot3 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [08:18:04] RECOVERY - Disk space on snapshot3 is OK: DISK OK [08:18:04] RECOVERY - DPKG on snapshot3 is OK: All packages OK [13:52:20] (03PS1) 10Vogone: Added import source for 'wikidata' [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92178 [15:03:31] Could somebody look up "[4672fc92] 2013-10-27 14:18:21: ..." in the exception log for me? [15:12:04] bawolff: on a sunday? good luck. :P [15:12:16] Thought it couldn't hurt to ask [15:12:26] but yeah.... [15:12:38] (but if anybody is actually around, i have a few pretty critical patches to review) [15:12:46] bawolff: wanna? :P [15:12:56] * bawolff wouldn't have this problem if the stupid exceptions were public like they used to be :P [15:13:06] MatmaRex: No promises, but which patches [15:14:00] bawolff: depends on what you're comfortable with – the topmost 3 of https://gerrit.wikimedia.org/r/#/q/owner:%22Bartosz+Dziewo%25C5%2584ski+%253Cmatma.rex%2540gmail.com%253E%22,n,z , mostly [15:14:55] Do you have any non-front end ones? [15:15:24] sure. https://gerrit.wikimedia.org/r/84315 [15:16:08] (but out of those three only the ext.echo.alert is actually frontend :) ) [15:21:25] * bawolff takes a look at the wl_notification one [15:22:49] yay [15:27:28] bawolff: (i was wondering if i should add release notes to it?) [15:28:04] Honestly, it seems pretty random these days what does and does not get release notes [15:30:43] lol, you used a real unicode ellipsis in a code comment? That's going hard core on the fancy unicode glyphs [15:32:45] heh [15:32:51] i enjoy unicode glyphs [15:33:06] i have …, „” and – on my keyboard [15:33:18] („” are typographically correct Polish quotation marks) [15:43:14] PROBLEM - Host mw27 is DOWN: PING CRITICAL - Packet loss = 100% [15:44:04] RECOVERY - Host mw27 is UP: PING OK - Packet loss = 0%, RTA = 26.52 ms [16:09:22] https://gerrit.wikimedia.org/r/#/q/owner:%22Bartosz+Dziewo%25C5%2584ski+%253Cmatma.rex%2540gmail.com%253E%22+status:open,n,z [16:09:53] https://gerrit.wikimedia.org/r/#/c/53029/ [16:09:56] What's the status of that? [16:10:03] I'm in the wrong channel. [16:30:15] Elsie: the status of that is 'nobody cares' :) [16:30:30] (i don't care much either) [17:06:57] !log Created echo related tables for kawiktionary on 10.64.16.18 [17:07:13] Logged the message, Master [17:07:29] !log Created echo related tables for ndswiktionary on 10.64.16.18 [17:07:45] Logged the message, Master [17:10:43] Reedy: If you're not too busy, any chance you could look up "[4672fc92] 2013-10-27 14:18:21: Fatal exception of type MWException" in the exception log? [17:11:28] nah, he's just fixing some fatals caused by missing tables. :D [17:19:11] !log Creating echo_notification, echo_event and echo_email_batch IF NOT EXISTS for all echo related wikis on 10.64.16.18 [17:19:27] Logged the message, Master [17:19:32] lols. :D [17:19:45] Wheee [17:19:49] Strip markers bawolff [17:19:50] 2013-10-27 14:18:21 mw1050 commonswiki: [4672fc92] /w/index.php?title=Commons:Project_scope&action=edit§ion=13 Exception from line 77 of /usr/local/apache/common-local/php-1.22wmf22/includes/parser/StripState.php: Invalid marker:UNIQ7fcfb94d3c323dfe-h-0--QIN [17:19:56] Thanks :) [17:19:58] Want a full stack trace? [17:20:08] Ah [17:20:12] Translate related I think [17:20:28] sure [17:20:45] hmm, h is for header if I recall [17:22:50] http://p.defau.lt/?u18LN3OwzyT6NKZw5QW4UA [17:23:07] ERROR 1049 (42000): Unknown database 'hewikivoyage' [17:23:22] Brilliant [17:23:33] ERROR 1049 (42000): Unknown database 'iegcomwiki' [17:25:05] * Reedy sighs [17:25:43] Same for loginwik [17:25:51] i [17:29:34] test2wiki [17:29:34] ERROR 1066 (42000) at line 1: Not unique table/alias: 'echo_notification' [17:29:35] heh [17:30:58] I'm slightly surprised there hasn't been more complains [17:31:58] 'default' => false, [17:31:58] 'medium' => 'extension1', [17:31:58] 'large' => 'extension1', [17:33:34] Blergh. [17:34:53] Of course, that's also somewhat useless [17:34:55] Wikis get bigger [17:35:10] We hope. [17:41:30] https://github.com/wikimedia/operations-mediawiki-config/commit/0ee8c95ab6cd0ad764982881d775a23f3f698b97#diff-32285bf4316ad70244851ab826ae3f07 [17:41:32] BLAH [18:51:04] PROBLEM - swift-account-reaper on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:51:04] PROBLEM - swift-container-auditor on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:51:04] PROBLEM - DPKG on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:51:14] PROBLEM - swift-object-replicator on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:51:14] PROBLEM - swift-container-server on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:51:14] PROBLEM - RAID on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:51:14] PROBLEM - swift-object-auditor on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:51:24] PROBLEM - swift-container-replicator on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:51:24] PROBLEM - swift-account-auditor on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:51:24] PROBLEM - swift-account-replicator on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:51:24] PROBLEM - swift-container-updater on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:51:24] PROBLEM - SSH on ms-be1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:34] PROBLEM - swift-account-server on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:51:44] PROBLEM - swift-object-updater on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:52:04] PROBLEM - swift-object-server on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:03:14] PROBLEM - NTP on ms-be1001 is CRITICAL: NTP CRITICAL: No response from NTP server [20:01:52] some stuff is still hitting pmtpa. (e.g. second level domains are not geo balanced apparently) [20:02:07] is it ok to move all traffic for a country back to pmtpa? [20:02:12] will something break? [20:10:17] lol [20:10:31] Tampa isn't so useable... [20:11:03] what wiki's? Leslie noticed wikipedia.org (no sub domain) is pointing at Tampa [20:11:20] right. that's what i meant by second level [20:11:43] all wikipedias at least. [22:16:04] PROBLEM - Disk space on ms-be1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.