[00:00:34] <JohnLewis>	 mutante: yeah... :p
[00:00:59] <mutante>	 JohnLewis: we are getting closer since you found the newer version :)
[00:01:19] <JohnLewis>	 yeah, its harder for me though as I don't have a schema at all, you do :p (kinda)
[00:01:35] <mutante>	 just one thing to skip as opposed to like 3 or 4
[00:02:02] <mutante>	 well.. we just gotta know what we could leak when not doing  "delete_deleted_comments" 
[00:03:14] <mutante>	 there's the "isprivate" field, checking what that has
[00:04:16] <JohnLewis>	 isprivate would be a 0 / 1 field.
[00:04:19] <JohnLewis>	 *should be
[00:04:28] <mutante>	 good news, there are none that are 1
[00:04:37] <mutante>	 select count(bug_id) from longdescs where isprivate=0;
[00:04:42] <mutante>	 390980 
[00:04:51] <mutante>	  where isprivate!=0;
[00:04:53] <mutante>	 0
[00:04:57] <JohnLewis>	 sweet
[00:06:18] <chasemp>	 be careful guys there is some weirdness where a ticket can be private, but not have private comments
[00:06:31] <chasemp>	 though they are private by extension it's not notated
[00:06:44] <wikibugs>	 7Blocked-on-Operations, 6operations, 10Citoid, 6Scrum-of-Scrums, 6Services: Zotero not running in production - https://phabricator.wikimedia.org/T76308#1055831 (10GWicke) Also worth considering, especially on jessie: https://packages.debian.org/sid/xulrunner-24.0
[00:07:07] <andrewbogott>	 Is git review hanging for everyone or just me?
[00:07:22] <chasemp>	 andrewbogott: was hanging for me big time a bit ago
[00:07:28] <chasemp>	 eventually finished
[00:07:29] <JohnLewis>	 chasemp: private tickets are being dealt with separately
[00:07:40] <andrewbogott>	 chasemp: 1 minute?  10 minutes?
[00:07:42] <andrewbogott>	 90?
[00:07:47] <chasemp>	 okay not 90
[00:07:57] <chasemp>	 JohnLewis: no worries just a thought as I lived through bz pain
[00:08:00] <^demon|away>	 lol git review
[00:08:11] <andrewbogott>	 I can’t imagine what it’s doing.  I fetched a minute ago, so it’s not that...
[00:08:41] <^demon|away>	 It has a --verbose mode or something where it spits out what it's doing
[00:09:21] <andrewbogott>	 so it does!
[00:09:28] <andrewbogott>	 ' Running: git remote update gerrit'
[00:10:01] <andrewbogott>	 I kind of expected it to just have a line item “sleeping: just to fuck with you”
[00:10:20] <andrewbogott>	 Well, I guess I’ll go get my laundry and see if it’s still ‘git remove update gerrit’ when I get back
[00:11:29] <bblack>	 I'm getting hung up on a git pull too
[00:12:26] <bblack>	 the ssh session it rides over itself is fine (checked with ssh debug options), it's the git part where it does upload-pack or whatever that's taking forever
[00:12:40] <andrewbogott>	 it’s a race!
[00:13:44] <^demon|away>	 Well that'd explain sluggishness
[00:14:21] <bblack>	 the machine doesn't seem busy
[00:14:30] <^demon|away>	 https://phabricator.wikimedia.org/P319
[00:14:34] <mutante>	 wow, i'm finding bugs now that have been imported TO old Bugzilla
[00:14:36] <mutante>	 Migrated from sf.net bugtracker
[00:14:41] <mutante>	 Originally opened 2002-07-18 08:48
[00:15:35] <bblack>	 I notice gerrit is running w/ -Xmx20g but has 33.2g VM mapped one way or another.  it could be that it needs a restart due to some kind of resource issue
[00:17:21] <bblack>	 I've never debugged our gerrit before.  is it normal and reasonably-nondestructive to restart the service daemon?
[00:17:55] <^demon|away>	 Nope, which is why it's good to check the queue before restarting.
[00:18:07] <^demon|away>	 there's no safe "drain all existing work and then restart" option
[00:18:21] <bblack>	 I cancelled my pull
[00:18:42] <mutante>	 yet, if you merge a config change you have to restart it too
[00:18:50] <bblack>	 what's up with the 907M for jar files and such in /var/lib/gerrit2/review_site/tmp
[00:19:41] <^demon|away>	 gerrit caching stuff and not clearing it, likely
[00:20:04] <^demon|away>	 Those all look like plugin jars
[00:20:39] <bblack>	 wow the storage setup on that box is... interesting :p
[00:21:08] <^demon|away>	 Yeah, it's rather odd
[00:21:22] <bblack>	 the standard two disks are setup with 2x partitions each for the normal linux md-raid mirroring with /dev/md0 as rootfs and /dev/md1 as the rest
[00:21:34] <mutante>	 JohnLewis: "longdescs:  The meat of bugzilla -- here is where all user comments are stored!
[00:21:37] <mutante>	 You've only got 2^24 bytes per comment (it's a mediumtext field), so speak
[00:21:40] <mutante>	 sparingly -- that's only the amount of space the Old Testament from the Bible
[00:21:43] <mutante>	 would take"
[00:21:48] <bblack>	 and then /dev/md1 is a PV for LVM, where it's the only PV in the only VG on the box, which only has one volume, which is the gerrit fs (and doesn't even use all the space avail)
[00:22:05] <bblack>	 why not just put the gerrit fs on /dev/md1 and kill LVM? :p
[00:22:08] <JohnLewis>	 mutante: kay :p
[00:22:32] <^demon|away>	 bblack: Honestly, I don't know why it's setup that way.
[00:22:57] <mutante>	 JohnLewis: seems like this is the closest we get to a schema http://www.bugzilla.org/docs/2.18/html/dbdoc.html
[00:23:01] <bblack>	 I'm gonna log out and pretend I never logged in there, it will make my weekend better :p
[00:23:21] <JohnLewis>	 2.18? jeez that 2 major releases away
[00:23:51] <^demon|away>	 bblack: I wouldn't touch it so late on a friday either :)
[00:24:08] <mutante>	 JohnLewis: and doesn't mention the "tag" field 
[00:24:49] <JohnLewis>	 mutante: which confused me
[00:25:07] <mutante>	 yea, they stopped writing dbdoc.html later .. meh
[00:25:28] <mutante>	 Badzilla
[00:25:35] <bblack>	 oh I guess there was a swap LV in that VG too, but still, those could be /dev/md[012]
[00:25:57] <bblack>	 anyways, I'm not touching shit.  I'm pretty much done for the day, gonna go start binging my way through some Dexter on netflix :)
[00:26:44] <^demon|away>	 I'm going to make $adult_beverage and start my weekend as well
[00:30:14] <mutante>	 JohnLewis: there's actually some interesting discussion i found where they discussed if comments should ever be editable/deletable or not. like ""has several undesirable effects. The rationale for decisions often gets lost by people “tidying up”. You can’t be certain that you are reading what someone else read before they said what they said."
[00:30:22] <mutante>	 that's actually true for phab
[00:30:47] <mutante>	 maybe the "tag" thing is new instead of old
[00:31:19] <JohnLewis>	 but its not in the db schema for longdescs
[00:31:21] <andrewbogott>	 ok!  I have picked up my laundry and am back from a pleasant stroll.  And yet...
[00:31:23] <andrewbogott>	 “git remote update gerrit"
[00:31:25] <JohnLewis>	 well table for it
[00:32:44] <mutante>	 andrewbogott: works for me
[00:32:50] <mutante>	 also, git pull origin/production
[00:33:05] <mutante>	 cancel and try again?
[00:33:24] <mutante>	 (without the / , sry)
[00:34:04] * andrewbogott has cancelled and tried again many times
[00:34:07] <andrewbogott>	 but I will, once more!
[00:34:45] <mutante>	 real	0m5.179s
[00:34:53] <mutante>	 for git remote update gerrit
[00:38:12] <andrewbogott>	 90 minutes ago I thought, ‘I’ll just push this demo patch in gerrit so I can link to it from the bug’
[00:38:20] <andrewbogott>	 What has happened since then?  I have no idea
[00:39:07] <mutante>	 JohnLewis: "DESTROY(DBI::db=HASH(0x25f3b70))= ( undef ) [1 items] during global destruction" gotta love 
[00:39:10] <andrewbogott>	 bblack: you didn’t actually restart gerrit, right?  Just looked at it and backed away, hands up?
[00:40:24] <mutante>	 i dont think he did, he said touched nothing
[00:41:20] <andrewbogott>	 Hm, this time ‘git fetch origin’ returned 406.
[00:41:27] <andrewbogott>	 I’m not sure I’ve ever seen a 406 before
[00:42:40] <grrrit-wm>	 (03PS1) 10Dzahn: fix some indentation and test if gerrit works [dns] - 10https://gerrit.wikimedia.org/r/192040 
[00:42:45] <mutante>	 andrewbogott: ^ i can use it just normal
[00:42:50] <mutante>	 wouldn't have noticed it
[00:42:54] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] fix some indentation and test if gerrit works [dns] - 10https://gerrit.wikimedia.org/r/192040 (owner: 10Dzahn)
[00:43:18] <mutante>	 you happen to be on comcast?
[00:43:35] <andrewbogott>	 nah, att
[00:43:51] <andrewbogott>	 DSL has that retro charm that I can’t resist
[00:44:42] <mutante>	 does it only happen on the puppet repo? how about another one
[00:44:51] <mutante>	 that was dns up there
[00:46:22] <andrewbogott>	 just puppet, it seems
[00:46:49] <Bsadowski1>	 Why the hell is Wikipedia loading so slow
[00:47:08] <Bsadowski1>	 https://en.wikipedia.org/wiki/Wikipedia:Long_term_abuse/MascotGuy is taking forever to load in Wikipedia
[00:47:10] <Bsadowski1>	 er
[00:47:14] <Bsadowski1>	 Firefox
[00:47:15] <Bsadowski1>	 :P
[00:47:37] <Bsadowski1>	 If you go to https://en.wikipedia.org/wiki/User:MascotGuy and click on that link, it takes a long time
[00:47:42] <Bsadowski1>	 for me at least
[00:50:56] <wikibugs>	 6operations, 6Phabricator, 10Wikimedia-Bugzilla: Sanitise a Bugzilla database dump - https://phabricator.wikimedia.org/T85141#1055958 (10Dzahn) made this modification:  https://phabricator.wikimedia.org/transactions/detail/PHID-XACT-PSTE-laovvu2r7b335db/  to work around the missing "tag" field, supposing it...
[00:51:16] <andrewbogott>	 well, that’s not helping
[00:54:01] <mutante>	 andrewbogott: and if you do it from a different host, not at home..a labs instance or so
[00:54:13] <mutante>	 Bsadowski1: can't confirm it's slow for me
[00:54:58] <Bsadowski1>	 Well https://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=startup&only=scripts&skin=monobook&* took 3 seconds to load
[00:55:53] <Bsadowski1>	 And https://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=ext.centralNotice.bannerChoiceData[...] took 18 seconds
[00:56:11] <Bsadowski1>	 Does that help some? :P
[00:56:18] <andrewbogott>	 yeah, it works for me on a labs box.  So I think it’s probably chugging along on my local machine, just VERY slowly
[00:56:21] <Bsadowski1>	 sorry 28
[00:56:25] <Bsadowski1>	 not 18
[00:56:33] <mutante>	 Bsadowski1: do you happen to be on comcast?
[00:56:42] <Bsadowski1>	 nope
[00:56:44] <Bsadowski1>	 AT&T
[00:56:50] <mutante>	 heh, like andrewbogott 
[00:56:55] <mutante>	 and both things are fast for me
[00:56:57] <Bsadowski1>	 And it's failing to load at all
[00:57:05] <Bsadowski1>	 The actual page I mean
[00:57:22] <andrewbogott>	 hm, weird that it was comcast yesterday and att torday
[00:57:49] <Bsadowski1>	 Forbidding scripts loads the page fine
[00:58:01] <Bsadowski1>	 I have NoScript
[00:58:17] <Bsadowski1>	 But if I allow "wikimedia.org"....
[00:58:35] <Bsadowski1>	 nothing
[00:58:37] <Bsadowski1>	 :/
[00:58:43] <Bsadowski1>	 It must be due to bits
[01:00:14] <mutante>	 i also have noscript but wikimedia.org on the whitelist
[01:17:40] <wikibugs>	 6operations, 6Phabricator, 10Wikimedia-Bugzilla: Sanitise a Bugzilla database dump - https://phabricator.wikimedia.org/T85141#1056039 (10Dzahn) file:///home/dzahn/wmf/bz_schema.sql
[01:21:07] <wikibugs>	 6operations, 6Phabricator, 10Wikimedia-Bugzilla: Sanitise a Bugzilla database dump - https://phabricator.wikimedia.org/T85141#1056054 (10Dzahn) {F44606}
[01:25:26] <grrrit-wm>	 (03PS2) 10BryanDavis: logstash: Ship logs via syslog udp datagrams [mediawiki-config] - 10https://gerrit.wikimedia.org/r/191259 (https://phabricator.wikimedia.org/T88732) 
[01:25:48] <grrrit-wm>	 (03CR) 10BryanDavis: "> There is needless duplication of handlers." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/191259 (https://phabricator.wikimedia.org/T88732) (owner: 10BryanDavis)
[01:27:42] <bd808>	 ori: ^ I hope that addresses some of you concerns
[02:01:11] <wikibugs>	 6operations, 6Phabricator, 10Wikimedia-Bugzilla: Sanitise a Bugzilla database dump - https://phabricator.wikimedia.org/T85141#1056081 (10JohnLewis) Leaving my ideas here quickly before I go for @dzahn (and public reasons):  Get group_id of the security group and then using that id, associate it with bug_id...
[02:15:27] <logmsgbot>	 !log l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 02s)
[02:15:35] <morebots>	 Logged the message, Master
[02:16:34] <logmsgbot>	 !log LocalisationUpdate completed (1.25wmf17) at 2015-02-21 02:15:31+00:00
[02:16:38] <morebots>	 Logged the message, Master
[02:17:11] <logmsgbot>	 !log l10nupdate Synchronized php-1.25wmf18/cache/l10n: (no message) (duration: 00m 01s)
[02:17:15] <morebots>	 Logged the message, Master
[02:18:18] <logmsgbot>	 !log LocalisationUpdate completed (1.25wmf18) at 2015-02-21 02:17:15+00:00
[02:18:22] <morebots>	 Logged the message, Master
[02:53:06] <wikibugs>	 6operations, 10Continuous-Integration, 5Patch-For-Review: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#1056111 (10scfc) Could the `operations-apache-config-lint` job be completely removed until it is enabled again?
[03:07:24] <icinga-wm>	 PROBLEM - Router interfaces on cr1-eqiad is CRITICAL: CRITICAL: host 208.80.154.196, interfaces up: 220, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-4/2/0: down - Core:  cr1-codfw:xe-5/2/1 (Telia, IC-307235) (#2648) [10Gbps wave]BR  
[03:22:25] <icinga-wm>	 PROBLEM - puppet last run on ms-be2014 is CRITICAL: CRITICAL: Puppet has 1 failures  
[03:22:54] <icinga-wm>	 PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: Puppet has 1 failures  
[03:40:05] <icinga-wm>	 RECOVERY - puppet last run on mw1246 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures  
[03:40:44] <icinga-wm>	 RECOVERY - puppet last run on ms-be2014 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures  
[03:54:13] <wikibugs>	 6operations, 10Parsoid, 6Services, 5Patch-For-Review, 7service-runner: Create a standard service template / init / logging / package setup - https://phabricator.wikimedia.org/T88585#1056162 (10GWicke) Now updated merged & slightly improved at https://github.com/wikimedia/service-template-node.  I think...
[04:11:44] <grrrit-wm>	 (03CR) 10GWicke: "@Alex, either way when a box is moved some information about it needs to be changed. Forgetting to update the rack/row information for any" [puppet] - 10https://gerrit.wikimedia.org/r/167645 (https://phabricator.wikimedia.org/T84518) (owner: 10Alexandros Kosiaris)
[04:32:44] <Bsadowski1>	 Could the slowdown for me be due to HHVM?
[04:49:21] <ori>	 Bsadowski1: <http://gdash.wikimedia.org/dashboards/frontend/> doesn't show anything unusual
[04:49:48] <ori>	 so I think it might just be you. Is the response time the same when you log out and try to load the page as an anon?
[04:53:49] <Krinkle>	 ori: Hm.. "nav start to nav end for desktop sites last month" looks interesting
[04:54:05] <Krinkle>	 what happened there 2 weeks ago?
[04:55:37] <ori>	 dunno
[05:08:28] <wikibugs>	 6operations: wish to introduce our fleece blankets/bathrobes factory - https://phabricator.wikimedia.org/T90325#1056199 (10emailbot)
[06:09:43] <grrrit-wm>	 (03CR) 10Ori.livneh: "A separate request should not be necessary for counting image views -- you could just count the image requests, no?" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/190821 (https://phabricator.wikimedia.org/T89088) (owner: 10Gilles)
[06:28:34] <icinga-wm>	 PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:28:34] <icinga-wm>	 PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: puppet fail  
[06:28:44] <icinga-wm>	 PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: puppet fail  
[06:28:44] <icinga-wm>	 PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: puppet fail  
[06:29:05] <icinga-wm>	 PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:14] <icinga-wm>	 PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:14] <icinga-wm>	 PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:24] <icinga-wm>	 PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:24] <icinga-wm>	 PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:24] <icinga-wm>	 PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:24] <icinga-wm>	 PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:34] <icinga-wm>	 PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:34] <icinga-wm>	 PROBLEM - puppet last run on elastic1030 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:43] <icinga-wm>	 PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:30:13] <icinga-wm>	 PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:45:54] <icinga-wm>	 RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures  
[06:46:54] <icinga-wm>	 RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures  
[06:47:04] <icinga-wm>	 RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures  
[06:47:34] <icinga-wm>	 RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures  
[06:47:44] <icinga-wm>	 RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures  
[06:47:54] <icinga-wm>	 RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures  
[06:48:04] <icinga-wm>	 RECOVERY - puppet last run on elastic1030 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures  
[06:48:04] <icinga-wm>	 RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures  
[06:48:05] <icinga-wm>	 RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures  
[06:48:24] <icinga-wm>	 RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures  
[06:48:24] <icinga-wm>	 RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures  
[06:48:44] <icinga-wm>	 RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures  
[06:48:44] <icinga-wm>	 RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures  
[06:48:44] <icinga-wm>	 RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures  
[06:48:54] <icinga-wm>	 RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures  
[06:50:24] <icinga-wm>	 RECOVERY - Router interfaces on cr1-eqiad is OK: OK: host 208.80.154.196, interfaces up: 222, down: 0, dormant: 0, excluded: 0, unused: 0  
[06:51:53] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Feb 21 06:50:49 UTC 2015 (duration 50m 48s)
[06:52:01] <morebots>	 Logged the message, Master
[06:52:10] <wikibugs>	 6operations, 10Beta-Cluster, 6Release-Engineering: Beta cluster unable to create thumbnail for WebM video - https://phabricator.wikimedia.org/T90332#1056292 (10Krinkle) 3NEW
[07:52:54] <wikibugs>	 6operations, 10Beta-Cluster, 7Performance: Need a way to simulate replication lag to test replag issues - https://phabricator.wikimedia.org/T40945#1056310 (10Nemo_bis)
[08:16:31] <wikibugs>	 6operations, 6Phabricator, 10Wikimedia-Bugzilla, 5Patch-For-Review: Create a static HTML version of Bugzilla - https://phabricator.wikimedia.org/T85140#1056325 (10Nemo_bis) > To get rid of Bugzilla  This is not going to happen in this decade anyway.  FYI, static HTML dumps made for archival purposes shoul...
[10:28:33] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: "Actually, it quite often doesn't, albeit only on a documentation level (aka racktables). Adding one extra step in the procedure of moving " [puppet] - 10https://gerrit.wikimedia.org/r/167645 (https://phabricator.wikimedia.org/T84518) (owner: 10Alexandros Kosiaris)
[11:54:54] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0]  
[11:54:54] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0]  
[12:06:55] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0]  
[12:06:55] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]  
[12:31:59] <grrrit-wm>	 (03PS2) 10Mjbmr: Unifying talk namespaces for fawikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190232 (https://phabricator.wikimedia.org/T90340) 
[13:18:40] <wikibugs>	 6operations, 10Citoid: Configure citoid to use outbound proxy - https://phabricator.wikimedia.org/T89875#1047771 (10Mvolz)
[16:29:42] <hoo>	 Is runJobs.log incomplete as well now
[16:29:45] <hoo>	 ?
[16:32:24] <icinga-wm>	 PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: Puppet has 1 failures  
[16:35:56] <grrrit-wm>	 (03PS1) 10Hoo man: Revert "Limit runJobs output to warning and higher severity" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/192083 
[16:49:44] <icinga-wm>	 RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures  
[16:54:19] <MaxSem>	 hoo, what do you need these logs for?
[16:55:03] <hoo>	 Seeing when jobs ran and whether they ran at all
[21:48:57] <grrrit-wm>	 (03PS1) 10Odder: Enable recent changes patrolling on Wikispecies [mediawiki-config] - 10https://gerrit.wikimedia.org/r/192103 (https://phabricator.wikimedia.org/T89147)