[00:00:04] 20after4: Respected human, time to deploy Phabricator Upgrade (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170602T0000). Please do the needful. [00:00:22] all done jouncebot [00:00:30] jouncebot: take a break for the weekend [00:00:34] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:27] jouncebot: :P [00:10:43] oh nice, the sidebar in commits is a nice touch (see the right side of the page: https://phabricator.wikimedia.org/rPHAB678062f1826fa70dcf2fc994d6393eb306f262e7 ) [00:11:06] also, floating header as you scroll down [00:28:34] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [01:17:06] (03PS4) 10Jforrester: Beta Features: Update last-big-change-plus-six-month dates in comments [mediawiki-config] - 10https://gerrit.wikimedia.org/r/354731 [01:17:27] (03PS2) 10Jforrester: Cleanup ORES config: Drop wgOresExtensionStatus (default), alphasort [mediawiki-config] - 10https://gerrit.wikimedia.org/r/354732 [01:18:25] (03PS2) 10Jforrester: Enable TimedMediaHandler's new video player Beta Feature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/354390 (https://phabricator.wikimedia.org/T148103) [01:27:14] PROBLEM - Apache HTTP on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:24] PROBLEM - Nginx local proxy to apache on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:28:14] RECOVERY - Apache HTTP on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 614 bytes in 7.174 second response time [01:30:14] RECOVERY - Nginx local proxy to apache on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.218 second response time [01:31:14] PROBLEM - Apache HTTP on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:04] RECOVERY - Apache HTTP on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.296 second response time [01:39:14] PROBLEM - Apache HTTP on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:39:24] PROBLEM - Nginx local proxy to apache on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:40:04] RECOVERY - Apache HTTP on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.250 second response time [01:40:14] RECOVERY - Nginx local proxy to apache on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 614 bytes in 0.349 second response time [01:55:14] PROBLEM - Apache HTTP on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:24] PROBLEM - Nginx local proxy to apache on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:04] RECOVERY - Apache HTTP on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 612 bytes in 0.165 second response time [01:56:14] RECOVERY - Nginx local proxy to apache on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.206 second response time [02:07:44] PROBLEM - HHVM rendering on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:08:34] RECOVERY - HHVM rendering on mw1198 is OK: HTTP OK: HTTP/1.1 200 OK - 78443 bytes in 0.653 second response time [02:17:44] PROBLEM - HHVM rendering on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:18:34] RECOVERY - HHVM rendering on mw1198 is OK: HTTP OK: HTTP/1.1 200 OK - 78563 bytes in 0.336 second response time [02:36:24] PROBLEM - Apache HTTP on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:36:24] PROBLEM - Nginx local proxy to apache on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:36:44] PROBLEM - HHVM rendering on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:37:14] RECOVERY - Apache HTTP on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.204 second response time [02:37:14] RECOVERY - Nginx local proxy to apache on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 614 bytes in 0.272 second response time [02:37:34] RECOVERY - HHVM rendering on mw1198 is OK: HTTP OK: HTTP/1.1 200 OK - 78531 bytes in 0.973 second response time [02:40:03] what's up with mw1198? [02:44:15] looks like maybe it just caught a string of ApiQueryContributors requests that were all really heavy on db requests. [02:46:41] !log Loadavg on mw1198 very high (44+) and nginx/hhvm checks flapping [02:46:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:52:46] ACKNOWLEDGEMENT - HP RAID on db2049 is CRITICAL: CRITICAL: Slot 0: OK: 1I:1:2, 1I:1:3, 1I:1:4, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:9, 1I:1:10, 1I:1:11, 1I:1:12 - Failed: 1I:1:1 - Controller: OK - Battery/Capacitor: OK nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T166853 [04:19:14] PROBLEM - mailman I/O stats on fermium is CRITICAL: CRITICAL - I/O stats: Transfers/Sec=499.10 Read Requests/Sec=520.20 Write Requests/Sec=5.80 KBytes Read/Sec=37918.40 KBytes_Written/Sec=41.60 [04:27:14] RECOVERY - mailman I/O stats on fermium is OK: OK - I/O stats: Transfers/Sec=98.80 Read Requests/Sec=0.90 Write Requests/Sec=2.70 KBytes Read/Sec=12.00 KBytes_Written/Sec=64.00 [04:42:06] !log removed some old scap revs for the Analytics refinery on stat1002 to free space (git fat jars replicating after each deployment, known issue) [04:42:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:03:07] PROBLEM - MariaDB Slave Lag: s5 on db1049 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 315.49 seconds [05:06:15] (03CR) 10Muehlenhoff: [C: 032] Gerrit: Set ulimit's in gerrit.service [debs/gerrit] - 10https://gerrit.wikimedia.org/r/356480 (https://phabricator.wikimedia.org/T158946) (owner: 10Paladox) [05:10:57] I don't understand how db1049 is supposed to be a vslow host and it's commented out in the s5 list [05:11:00] db1049 issues? [05:12:26] apparently it is depooled [05:12:39] then what is the vslow host? and why is there [05:13:10] that is wrong [05:13:29] let me blame that file [05:13:57] I63ce9c1ff37c9c070b9b47414d2f47d2e5b1095f well it's this [05:14:11] or 6ae4548bce04f06bce238f591e61b86b689186a8 if you like [05:14:20] but why is it paging then [05:15:18] I have no idea [05:15:26] but it is pooled [05:15:32] it is not depooled right [05:15:47] and this is probably an outage [05:15:52] are dumps running there? because wikidatawiki is running right now [05:15:58] * apergos goes to look at tendril [05:17:45] yes it is being used as a dumps host [05:17:46] I think, for starters, we should pool that host [05:17:48] so [05:17:52] +1 [05:18:39] unless you find a SAL reason not to [05:18:45] ah good point [05:19:27] it's not mentioned in sal at all [05:20:03] let me see if there's anything in the irc logs from this channel [05:21:28] nope nothing [05:21:46] checked form date of the gerrit change through yesterday [05:22:02] (03PS1) 10Jcrespo: mariadb: pool db1049 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356785 [05:22:16] can you please check ^ [05:22:58] looks ok to me [05:23:34] (03CR) 10Jcrespo: [C: 032] mariadb: pool db1049 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356785 (owner: 10Jcrespo) [05:24:40] wait [05:24:42] argh [05:25:11] what? [05:25:32] nm [05:25:34] fine fine [05:25:37] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Emergency pool of db1049 (duration: 00m 48s) [05:25:45] brain stoppage there for a minute [05:25:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:25:46] (03CR) 10jenkins-bot: mariadb: pool db1049 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356785 (owner: 10Jcrespo) [05:25:56] (03PS1) 10Muehlenhoff: Bump changelog for new build [debs/gerrit] - 10https://gerrit.wikimedia.org/r/356786 [05:28:25] I am not sure if it is getting better [05:29:51] it might not but if only vslow and dumps is going there, it shouldn't matter right? [05:30:24] (03PS29) 10Elukey: role::zookeeper: refactor to multiple profiles [puppet] - 10https://gerrit.wikimedia.org/r/354449 (https://phabricator.wikimedia.org/T114815) [05:30:32] it is now, I think [05:30:48] no, going up again [05:31:34] let me see if there is too much load on s5 [05:33:04] PROBLEM - Host elastic2033 is DOWN: PING CRITICAL - Packet loss = 100% [05:34:03] BrokenRedirectsPage and SpecialGadgetsUsage seem to be the real time sinks over there atm but that could be standard, I don't know normal vslow host query behavior [05:34:04] RECOVERY - Host elastic2033 is UP: PING OK - Packet loss = 0%, RTA = 0.20 ms [05:35:43] I think we could have an edit problem: https://grafana.wikimedia.org/dashboard/db/edit-count?refresh=5m&orgId=1 [05:37:07] RECOVERY - MariaDB Slave Lag: s5 on db1049 is OK: OK slave_sql_lag Replication lag: 4.30 seconds [05:37:47] do we have here someone with admin rights on wikidata? [05:37:57] because we need to block a bot [05:38:58] could ask in stewards [05:40:04] join #wikidata [05:40:11] (03PS30) 10Elukey: role::zookeeper: refactor to multiple profiles [puppet] - 10https://gerrit.wikimedia.org/r/354449 (https://phabricator.wikimedia.org/T114815) [05:40:12] grrr still half asleep [05:40:31] I thought db1049 was depooled [05:40:33] because it had old hardware [05:41:58] https://gerrit.wikimedia.org/r/#/c/356027/ this says it's supposedly pooled [05:42:07] except it was only partially? [05:42:32] Ah shit [05:42:35] Missing the "1" [05:42:36] Damn it [05:42:38] jynus, I am asking in the wikidata channel, no idea if anyone's around [05:42:45] 1113 edits per minute [05:42:57] well also the comment at the beginning [05:43:11] anyways, jy nus fixed that up [05:43:18] We can pool db1070 for vslow if needed [05:44:16] jynus: which bot is it btw? (so far no responses) [05:44:26] https://www.wikidata.org/w/index.php?title=Special:Contributions/ValterVBot&offset=20170602053909&limit=500&target=ValterVBot [05:44:47] daaaannnggg [05:44:56] :o [05:46:15] where did you ask? [05:46:24] the wikidata channel [05:46:32] if there were admins around they'd be there [05:47:16] otherwise gotta find a staffer with some rights who is awake, or ask for steward intervention (worst choice) [05:49:11] (03CR) 10Elukey: "From https://puppet-compiler.wmflabs.org/6649/ it looks good. The only real change is on druid1001 but for the best, since we are correcti" [puppet] - 10https://gerrit.wikimedia.org/r/354449 (https://phabricator.wikimedia.org/T114815) (owner: 10Elukey) [05:54:10] https://www.wikidata.org/wiki/Special:Contributions/MisterSynergy admin and currently active but not responding in irc [05:55:52] in the meantime db1049 is now looking ok as far as slow queries, according to tendril [05:57:05] I will revert the innodb and binlog options I applied to catch up [05:57:21] it was only during the bot edits that it lagged [05:57:51] ok [05:59:34] We can use db1070, it is not being used for anything else and it was the original vslow one [06:00:22] do we still want the bot blocked, jynus? [06:00:24] I have a response [06:00:29] yes [06:00:41] if it starts again, it will happen again [06:01:38] based on https://www.mediawiki.org/wiki/API:Etiquette [06:02:11] I have passed it on and also invited them here if they have questions [06:02:22] hello MisterSynergy [06:02:24] Good morning :-) [06:02:26] jynus ^^ [06:02:34] hi, there [06:03:14] jynus is one of our dbas and can explain exactly the impact [06:03:24] so ValterVBot is running too fast, I heard [06:03:42] the bot account edited at a rate of almost 20 edits per second [06:04:02] creating more edits than all other users toghether on all wikis [06:04:23] that is not something that wikidata is right no ready to handle [06:04:27] *now [06:04:53] is there any chart or so which indicates the load? [06:04:55] user should be warned to do edits serialized, as the page api suggets [06:04:58] yes [06:05:16] https://grafana.wikimedia.org/dashboard/db/edit-count?refresh=5m&orgId=1 [06:05:29] see those growths at the end [06:06:03] ~1500 edits per minute [06:06:06] okay, edit rate is high, but is it a really a problem? [06:06:13] yes [06:06:23] the api says that edits should be done serially [06:07:00] it is not as much the rate but how much it waited for every edit [06:07:12] it caused slowdown for other users [06:07:25] do we have a chart for that as well? :-) [06:08:29] I am sorry, do you really need such a justification for a temporary block of a bot account? [06:08:40] it is a bot, not a user [06:08:51] just want to know, I'm quite new in admin business [06:09:04] but will do; you're User:Jynus, correct? [06:09:12] bots are just blocked unceremonisally [06:09:20] can be unblocked later [06:09:32] it is not a user we are affecting [06:10:08] we just need the user to notice it so he or she slows down the bot [06:10:51] yes, I am JCrespo (WMF) / Jynus [06:11:01] do you want me to do a request on wiki? [06:12:26] https://www.wikidata.org/w/index.php?title=Topic:Tropyz2fx7j3vf9x&action=history [06:12:54] should be done [06:13:23] I can also contact the user, no need for that [06:15:26] Thank you, MisterSynergy ! [06:16:18] I left him a note on his talk page [06:16:26] he was active half an hour ago [06:16:35] yeah, I saw [06:16:36] refered him to you [06:16:38] thank you for the help [06:16:41] no problem [06:16:44] thank you again [06:17:50] I think he was going for a new run, so the block was needed [06:19:22] ValterVB is online, just thanked me for my not on his talk page [06:19:31] I guess he will look into this now [06:19:36] nice [06:20:09] note this is not inteded as a permanent ban [06:20:26] just a temporary measure intil he/she tunes down the bot [06:20:56] yes of course [06:20:57] and I prefer an active member of the community involved [06:26:02] Re: perfomance impact- you can see the impact here: https://grafana.wikimedia.org/dashboard/db/mysql-replication-lag?panelId=5&fullscreen&orgId=1&from=1496375183013&to=1496383626978 [06:26:09] if there were admins around they'd be there [06:26:19] jynus: ^ you can also ping me [06:26:23] fyi [06:26:46] JD|cloud: thanks, very good to know [06:27:05] (note the graph is logarithmic) [06:27:14] thanks, very clear now :-) [06:27:30] I guess ValterVB should see it as well ;-) [06:28:38] the bots are supposed to check for lag [06:33:55] <+snitch> ValterVB unblocked User:ValterVBot: Stopped the BOT, I ask to developer for some solution; https://www.wikidata.org/wiki/Special:Log/unblock [06:34:00] MisterSynergy ^ [06:35:14] https://www.wikidata.org/wiki/User_talk:JCrespo_(WMF)#ValterVBot_problem [06:35:37] jynus: this is your talk page [06:35:51] yeah, saw it [06:36:01] k, thnx [06:45:20] to clarify, there was some problems on our side, we are not blaming the user at all - but the way edits were done magnified the issue [06:47:59] mhh, did we lose the phab bot? [06:48:57] (03CR) 10Muehlenhoff: [C: 032] Bump changelog for new build [debs/gerrit] - 10https://gerrit.wikimedia.org/r/356786 (owner: 10Muehlenhoff) [06:56:49] (03PS1) 10Alexandros Kosiaris: lvs: Remove all bgp keywords from configuration [puppet] - 10https://gerrit.wikimedia.org/r/356790 [07:02:07] !log starting fleet wide PCC for gerrit change 356030. Should take a while to complete [07:02:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:06:04] (03PS1) 10Muehlenhoff: Fix distribution line in changelog [debs/gerrit] - 10https://gerrit.wikimedia.org/r/356791 [07:06:46] (03CR) 10Muehlenhoff: [V: 032 C: 032] Fix distribution line in changelog [debs/gerrit] - 10https://gerrit.wikimedia.org/r/356791 (owner: 10Muehlenhoff) [07:14:04] PROBLEM - HHVM rendering on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:14:54] RECOVERY - HHVM rendering on mw1198 is OK: HTTP OK: HTTP/1.1 200 OK - 78580 bytes in 2.859 second response time [07:29:19] (03PS1) 10Alexandros Kosiaris: calico: Supploy a calicoctl.cfg file [puppet] - 10https://gerrit.wikimedia.org/r/356793 [07:30:00] (03PS1) 10Marostegui: db-eqiad.php: Repool db1059 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356794 (https://phabricator.wikimedia.org/T166206) [07:36:07] !log Deploy alter table on s4 - labsdb1009 - T166206 [07:36:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:36:16] T166206: Convert unique keys into primary keys for some wiki tables on s4 - https://phabricator.wikimedia.org/T166206 [07:37:33] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Repool db1059 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356794 (https://phabricator.wikimedia.org/T166206) (owner: 10Marostegui) [07:38:50] (03Merged) 10jenkins-bot: db-eqiad.php: Repool db1059 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356794 (https://phabricator.wikimedia.org/T166206) (owner: 10Marostegui) [07:38:59] (03CR) 10jenkins-bot: db-eqiad.php: Repool db1059 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356794 (https://phabricator.wikimedia.org/T166206) (owner: 10Marostegui) [07:39:37] (03CR) 10Alexandros Kosiaris: [C: 032] calico: Supploy a calicoctl.cfg file [puppet] - 10https://gerrit.wikimedia.org/r/356793 (owner: 10Alexandros Kosiaris) [07:39:45] (03PS2) 10Alexandros Kosiaris: calico: Supploy a calicoctl.cfg file [puppet] - 10https://gerrit.wikimedia.org/r/356793 [07:39:50] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] calico: Supploy a calicoctl.cfg file [puppet] - 10https://gerrit.wikimedia.org/r/356793 (owner: 10Alexandros Kosiaris) [07:43:12] !log marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1059 - T166206 (duration: 00m 39s) [07:43:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:43:21] T166206: Convert unique keys into primary keys for some wiki tables on s4 - https://phabricator.wikimedia.org/T166206 [07:45:23] !log uploaded gerrit 2.13.8+git1-wmf4 to apt.wikimedia.org [07:45:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:45:34] (03CR) 10Marostegui: [C: 031] mariadb: Add tokudb support for analytics eventlogging nodes [puppet] - 10https://gerrit.wikimedia.org/r/356648 (owner: 10Jcrespo) [07:47:56] !log Resume alter table on db1047 enwiki.revision - T166452 [07:48:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:48:05] T166452: db1047 has been restarted - needs another restart - https://phabricator.wikimedia.org/T166452 [08:21:31] (03PS1) 10Gehel: logstash - curator connects only to localhost [puppet] - 10https://gerrit.wikimedia.org/r/356797 [08:32:22] (03CR) 10Alexandros Kosiaris: [C: 04-1] "LGTM, but -1 until the thresholds are figured out in T153170" [puppet] - 10https://gerrit.wikimedia.org/r/356449 (https://phabricator.wikimedia.org/T153170) (owner: 10Phedenskog) [08:33:02] (03CR) 10Alexandros Kosiaris: [C: 031] base: export puppet agent stats to prometheus [puppet] - 10https://gerrit.wikimedia.org/r/354457 (owner: 10Filippo Giunchedi) [08:34:23] akosiaris: nice, thanks ^ I'll merge next week -- don't want to jinx it on fri [08:35:12] :-) [08:43:20] (03CR) 10Filippo Giunchedi: [C: 031] "> > Are we doing temperature monitoring using both IPMI and" [puppet] - 10https://gerrit.wikimedia.org/r/356567 (https://phabricator.wikimedia.org/T125205) (owner: 10Ema) [08:48:56] (03CR) 10Phedenskog: "If you want I can just set them even higher for now to avoid alerting, would be nice to get this in?" [puppet] - 10https://gerrit.wikimedia.org/r/356449 (https://phabricator.wikimedia.org/T153170) (owner: 10Phedenskog) [08:50:26] (03CR) 10Alexandros Kosiaris: [C: 04-1] "Yes that would work too. Let me know when you are ready and I 'll merge" [puppet] - 10https://gerrit.wikimedia.org/r/356449 (https://phabricator.wikimedia.org/T153170) (owner: 10Phedenskog) [08:53:38] (03CR) 10Phedenskog: "Cool, it is fixed now. The median value for the last 10h need to be either 50% or 20% higher than last week. I'll update the task in phabr" [puppet] - 10https://gerrit.wikimedia.org/r/356449 (https://phabricator.wikimedia.org/T153170) (owner: 10Phedenskog) [08:56:28] (03Abandoned) 10Alexandros Kosiaris: Increase TTL for etcd client records [dns] - 10https://gerrit.wikimedia.org/r/350225 (https://phabricator.wikimedia.org/T159687) (owner: 10Alexandros Kosiaris) [08:56:44] (03Abandoned) 10Alexandros Kosiaris: Switch conftool etcd records to codfw [dns] - 10https://gerrit.wikimedia.org/r/350216 (https://phabricator.wikimedia.org/T159687) (owner: 10Alexandros Kosiaris) [08:56:48] (03CR) 10Phedenskog: "I've updated the alerts a couple of days ago to keep the limit higher and no false alerts since then so this seems ready to go = we have a" [puppet] - 10https://gerrit.wikimedia.org/r/356382 (https://phabricator.wikimedia.org/T153169) (owner: 10Gilles) [08:58:48] (03CR) 10Alexandros Kosiaris: [C: 032] Add Navigation Timing alerts to Icinga [puppet] - 10https://gerrit.wikimedia.org/r/356382 (https://phabricator.wikimedia.org/T153169) (owner: 10Gilles) [08:58:53] (03PS2) 10Alexandros Kosiaris: Add Navigation Timing alerts to Icinga [puppet] - 10https://gerrit.wikimedia.org/r/356382 (https://phabricator.wikimedia.org/T153169) (owner: 10Gilles) [08:58:56] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] Add Navigation Timing alerts to Icinga [puppet] - 10https://gerrit.wikimedia.org/r/356382 (https://phabricator.wikimedia.org/T153169) (owner: 10Gilles) [09:02:54] (03PS4) 10Alexandros Kosiaris: Add Save Timing alerts to Icinga [puppet] - 10https://gerrit.wikimedia.org/r/356449 (https://phabricator.wikimedia.org/T153170) (owner: 10Phedenskog) [09:03:37] (03CR) 10Alexandros Kosiaris: [C: 032] Add Save Timing alerts to Icinga [puppet] - 10https://gerrit.wikimedia.org/r/356449 (https://phabricator.wikimedia.org/T153170) (owner: 10Phedenskog) [09:03:43] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] Add Save Timing alerts to Icinga [puppet] - 10https://gerrit.wikimedia.org/r/356449 (https://phabricator.wikimedia.org/T153170) (owner: 10Phedenskog) [09:12:27] !log Deploy alter table s3 - labsdb1003 - T166278 [09:12:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:12:37] T166278: Unify revision table on s3 - https://phabricator.wikimedia.org/T166278 [09:15:32] (03PS2) 10Gehel: logstash - cleanup dead code [puppet] - 10https://gerrit.wikimedia.org/r/356064 (https://phabricator.wikimedia.org/T166154) [09:17:03] (03CR) 10Gehel: [C: 032] logstash - cleanup dead code [puppet] - 10https://gerrit.wikimedia.org/r/356064 (https://phabricator.wikimedia.org/T166154) (owner: 10Gehel) [09:18:55] !log Deploy alter table s3 - db1015 - T166278 [09:19:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:19:02] T166278: Unify revision table on s3 - https://phabricator.wikimedia.org/T166278 [09:21:41] (03CR) 10Hashar: [C: 031] "I have passed this change through the puppet compiler with hosts:" [puppet] - 10https://gerrit.wikimedia.org/r/354186 (https://phabricator.wikimedia.org/T129148) (owner: 10Thcipriani) [09:26:12] (03CR) 10Jcrespo: [C: 031] mariadb: Add tokudb support for analytics eventlogging nodes [puppet] - 10https://gerrit.wikimedia.org/r/356648 (owner: 10Jcrespo) [09:28:28] (03PS3) 10Jcrespo: mariadb: Allow full reimage of db2041,38,37,35,44 (still on trusty) [puppet] - 10https://gerrit.wikimedia.org/r/356387 [09:29:08] PROBLEM - Citoid LVS codfw on citoid.svc.codfw.wmnet is CRITICAL: /api (Zotero alive) timed out before a response was received: /api (Scrapes sample page) timed out before a response was received [09:29:09] PROBLEM - citoid endpoints health on scb2002 is CRITICAL: /api (Zotero alive) timed out before a response was received: /api (Scrapes sample page) timed out before a response was received [09:29:09] PROBLEM - citoid endpoints health on scb2005 is CRITICAL: /api (Zotero alive) timed out before a response was received: /api (Scrapes sample page) timed out before a response was received [09:29:09] PROBLEM - citoid endpoints health on scb2001 is CRITICAL: /api (Zotero alive) timed out before a response was received: /api (Scrapes sample page) timed out before a response was received [09:29:09] PROBLEM - citoid endpoints health on scb2003 is CRITICAL: /api (Zotero alive) timed out before a response was received: /api (Scrapes sample page) timed out before a response was received [09:29:09] PROBLEM - citoid endpoints health on scb2006 is CRITICAL: /api (Zotero alive) timed out before a response was received: /api (Scrapes sample page) timed out before a response was received [09:29:09] PROBLEM - citoid endpoints health on scb2004 is CRITICAL: /api (Zotero alive) timed out before a response was received: /api (Scrapes sample page) timed out before a response was received [09:29:11] (03PS4) 10Jcrespo: mariadb: Allow full reimage of db2041,38,37,35 (still on trusty) [puppet] - 10https://gerrit.wikimedia.org/r/356387 [09:29:31] (03PS5) 10Jcrespo: mariadb: Allow full reimage of db2041,38,37,35 (still on trusty) [puppet] - 10https://gerrit.wikimedia.org/r/356387 [09:29:43] damn zotero [09:29:58] RECOVERY - Citoid LVS codfw on citoid.svc.codfw.wmnet is OK: All endpoints are healthy [09:29:58] RECOVERY - citoid endpoints health on scb2006 is OK: All endpoints are healthy [09:29:58] RECOVERY - citoid endpoints health on scb2002 is OK: All endpoints are healthy [09:29:58] RECOVERY - citoid endpoints health on scb2001 is OK: All endpoints are healthy [09:29:59] RECOVERY - citoid endpoints health on scb2005 is OK: All endpoints are healthy [09:29:59] RECOVERY - citoid endpoints health on scb2003 is OK: All endpoints are healthy [09:29:59] RECOVERY - citoid endpoints health on scb2004 is OK: All endpoints are healthy [09:31:23] (03CR) 10Jcrespo: "This RC, like all other accepts cherrypicks + amends" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356584 (https://phabricator.wikimedia.org/T166345) (owner: 10Jcrespo) [09:33:14] (03CR) 10Jcrespo: [C: 032] mariadb: Allow full reimage of db2041,38,37,35 (still on trusty) [puppet] - 10https://gerrit.wikimedia.org/r/356387 (owner: 10Jcrespo) [09:38:03] (03PS2) 10Alexandros Kosiaris: motd::script: Don't use validate_re on an integer [puppet] - 10https://gerrit.wikimedia.org/r/356483 [09:41:54] (03CR) 10Alexandros Kosiaris: "PCC at https://puppet-compiler.wmflabs.org/6651/" [puppet] - 10https://gerrit.wikimedia.org/r/356030 (https://phabricator.wikimedia.org/T166372) (owner: 10Faidon Liambotis) [09:49:43] !log stopping db2041 to prepare it for reimage [09:49:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:17:33] (03CR) 10DCausse: [C: 031] logstash - curator connects only to localhost [puppet] - 10https://gerrit.wikimedia.org/r/356797 (owner: 10Gehel) [10:17:55] (03PS2) 10Gehel: logstash - curator connects only to localhost [puppet] - 10https://gerrit.wikimedia.org/r/356797 [10:21:06] (03PS1) 10Ema: admin: add ema's Yubikey [puppet] - 10https://gerrit.wikimedia.org/r/356808 [10:23:58] (03CR) 10Gehel: [C: 032] logstash - curator connects only to localhost [puppet] - 10https://gerrit.wikimedia.org/r/356797 (owner: 10Gehel) [10:31:00] (03PS3) 10DCausse: [wikitech] Increase weight on Tool and Nova Resource ns [mediawiki-config] - 10https://gerrit.wikimedia.org/r/354474 (https://phabricator.wikimedia.org/T165725) [10:32:20] (03CR) 10DCausse: [C: 04-1] "Maybe not needed as the doc has been refactored in the meantime (pages moved from Nova Resource to Portal)" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/354474 (https://phabricator.wikimedia.org/T165725) (owner: 10DCausse) [10:33:52] (03PS6) 10Filippo Giunchedi: swift: introduce storage policies [puppet] - 10https://gerrit.wikimedia.org/r/353878 (https://phabricator.wikimedia.org/T151648) [10:33:54] (03PS2) 10Filippo Giunchedi: swift: introduce container-reconciler [puppet] - 10https://gerrit.wikimedia.org/r/356198 (https://phabricator.wikimedia.org/T151648) [10:33:56] (03PS1) 10Filippo Giunchedi: swift: make swift-dispersion-stats policy-aware [puppet] - 10https://gerrit.wikimedia.org/r/356810 (https://phabricator.wikimedia.org/T151648) [10:35:46] (03CR) 10Filippo Giunchedi: [C: 032] swift: make swift-dispersion-stats policy-aware [puppet] - 10https://gerrit.wikimedia.org/r/356810 (https://phabricator.wikimedia.org/T151648) (owner: 10Filippo Giunchedi) [10:58:23] (03PS1) 10Nemo bis: [Planet Wikimedia] Add some hackathon-related blogs [puppet] - 10https://gerrit.wikimedia.org/r/356813 [11:05:27] (03PS2) 10Nemo bis: [Planet Wikimedia] Add some hackathon-related blogs [puppet] - 10https://gerrit.wikimedia.org/r/356813 [11:16:46] (03PS1) 10Alexandros Kosiaris: Refactor facts exporting to better cleanup facts [puppet] - 10https://gerrit.wikimedia.org/r/356814 [11:20:09] (03PS2) 10Alexandros Kosiaris: Refactor facts exporting to better cleanup facts [puppet] - 10https://gerrit.wikimedia.org/r/356814 [11:44:36] (03PS1) 10BBlack: maps: coalesce DNS to upload IPs [dns] - 10https://gerrit.wikimedia.org/r/356819 (https://phabricator.wikimedia.org/T164608) [11:46:31] (03CR) 10BBlack: [C: 032] maps: coalesce DNS to upload IPs [dns] - 10https://gerrit.wikimedia.org/r/356819 (https://phabricator.wikimedia.org/T164608) (owner: 10BBlack) [11:56:46] (03PS1) 10Muehlenhoff: Add debug repository for stretch onwards [puppet] - 10https://gerrit.wikimedia.org/r/356822 (https://phabricator.wikimedia.org/T164819) [11:57:37] (03CR) 10Faidon Liambotis: [C: 04-1] Refactor facts exporting to better cleanup facts (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/356814 (owner: 10Alexandros Kosiaris) [11:58:05] moritzm: that won't work :) [12:00:17] (03PS2) 10Ema: Re-enable temperature monitoring via NRPE [puppet] - 10https://gerrit.wikimedia.org/r/356567 (https://phabricator.wikimedia.org/T125205) [12:00:25] (03CR) 10Ema: [V: 032 C: 032] Re-enable temperature monitoring via NRPE [puppet] - 10https://gerrit.wikimedia.org/r/356567 (https://phabricator.wikimedia.org/T125205) (owner: 10Ema) [12:02:23] (03CR) 10Faidon Liambotis: [C: 04-1] "See inline of why this won't work as-is. I'll investigate whether we can/should mirror debian-debug. I'm not sure if enabling debug for al" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/356822 (https://phabricator.wikimedia.org/T164819) (owner: 10Muehlenhoff) [12:04:23] ah, damn. completely forgot about the proxy setting... [12:05:21] (03PS2) 10Ema: admin: add ema's Yubikey [puppet] - 10https://gerrit.wikimedia.org/r/356808 [12:05:29] (03CR) 10Ema: [V: 032 C: 032] admin: add ema's Yubikey [puppet] - 10https://gerrit.wikimedia.org/r/356808 (owner: 10Ema) [12:11:11] !log restarting Jenkins to upgrade the logstash plugin [12:11:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:24:08] (03Draft1) 10Paladox: Gerrit: Set gc.auto and gc.autopacklimit to 0 in ~/.gitconfig [puppet] - 10https://gerrit.wikimedia.org/r/356824 (https://phabricator.wikimedia.org/T151676) [12:24:13] (03Draft2) 10Paladox: Gerrit: Set gc.auto and gc.autopacklimit to 0 in ~/.gitconfig [puppet] - 10https://gerrit.wikimedia.org/r/356824 (https://phabricator.wikimedia.org/T151676) [12:39:37] PROBLEM - IPMI Temperature on cp1049 is CRITICAL: Sensor Type(s) Temperature Status: Critical [System Board 1 Inlet Temp = Warning, System Board 1 Inlet Temp = Critical, System Board 1 Inlet Temp = Warning] [12:39:37] PROBLEM - IPMI Temperature on ms-be2028 is CRITICAL: Sensor Type(s) Temperature Status: Critical [System Board 12 29-LOM = Critical] [12:41:07] PROBLEM - IPMI Temperature on aqs1004 is CRITICAL: Sensor Type(s) Temperature Status: Critical [System Board 2 26-LOM = Critical, System Board 2 26-LOM = Critical] [12:41:47] PROBLEM - IPMI Temperature on ms-be2014 is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:42:27] PROBLEM - IPMI Temperature on wtp1010 is CRITICAL: Sensor Type(s) Temperature Status: Critical [System Board Inlet Temp = Warning, System Board Inlet Temp = Critical, System Board Inlet Temp = Warning, System Board Inlet Temp = Warning, System Board Inlet Temp = Critical, System Board Inlet Temp = Warning, System Board Inlet Temp = Warning, System Board Inlet Temp = Critical, System Board Inlet Temp = Warning, System Board Inlet Tem [12:47:32] (03PS10) 10Paladox: Upgrade gerrit to 2.14.1 (DO NOT MERGE) [debs/gerrit] - 10https://gerrit.wikimedia.org/r/350440 [12:50:24] (03PS11) 10Paladox: Upgrade gerrit to 2.14.1 (DO NOT MERGE) [debs/gerrit] - 10https://gerrit.wikimedia.org/r/350440 [12:54:21] (03PS3) 10BBlack: LVS: new redundancy layout for new eqiad+ulsfo hosts [puppet] - 10https://gerrit.wikimedia.org/r/356605 (https://phabricator.wikimedia.org/T150256) [12:56:01] (03CR) 10BBlack: [C: 032] LVS: new redundancy layout for new eqiad+ulsfo hosts [puppet] - 10https://gerrit.wikimedia.org/r/356605 (https://phabricator.wikimedia.org/T150256) (owner: 10BBlack) [13:09:05] (03PS3) 10Alexandros Kosiaris: Refactor facts exporting to better cleanup facts [puppet] - 10https://gerrit.wikimedia.org/r/356814 [13:09:21] PROBLEM - IPMI Temperature on labsdb1001 is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:09:31] PROBLEM - IPMI Temperature on labsdb1011 is CRITICAL: Sensor Type(s) Temperature Status: Critical [System Board 12 29-LOM = Critical] [13:09:51] PROBLEM - IPMI Temperature on labsdb1003 is CRITICAL: Sensor Type(s) Temperature Status: Critical [Processor 1 P1_TEMP_SENS = Critical, Processor 1 P1_TEMP_SENS = Warning, Processor 1 P1_TEMP_SENS = Critical, Processor 1 P1_TEMP_SENS = Warning, Processor 1 P1_TEMP_SENS = Critical, Processor 1 P1_TEMP_SENS = Warning, Processor 1 P1_TEMP_SENS = Critical, Processor 1 P1_TEMP_SENS = Warning, Processor 1 P1_TEMP_SENS = Critical, Processo [13:10:21] PROBLEM - IPMI Temperature on db2049 is CRITICAL: Sensor Type(s) Temperature Status: Critical [Power Unit 2 18-VR P2 = Critical, Power Unit 2 18-VR P2 = Critical] [13:12:23] (03PS2) 10Andrew Bogott: puppet: disable stringified facts in Labs as well [puppet] - 10https://gerrit.wikimedia.org/r/356644 (owner: 10Faidon Liambotis) [13:14:52] (03CR) 10Andrew Bogott: [C: 032] puppet: disable stringified facts in Labs as well [puppet] - 10https://gerrit.wikimedia.org/r/356644 (owner: 10Faidon Liambotis) [13:23:51] RECOVERY - puppet last run on labtestvirt2003 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:26:14] (03PS4) 10Paladox: Gerrit: Increase packedGitOpenFiles to 6000 [puppet] - 10https://gerrit.wikimedia.org/r/356586 [13:28:34] !log restart elastic2003 to reload logging configuration [13:28:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:40:51] PROBLEM - IPMI Temperature on ocg1002 is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:42:51] PROBLEM - IPMI Temperature on sodium is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:46:32] (03PS1) 10BBlack: LVS refactor: service IPs and sparing out lvs101[12] [puppet] - 10https://gerrit.wikimedia.org/r/356833 (https://phabricator.wikimedia.org/T150256) [13:47:31] (03CR) 10jerkins-bot: [V: 04-1] LVS refactor: service IPs and sparing out lvs101[12] [puppet] - 10https://gerrit.wikimedia.org/r/356833 (https://phabricator.wikimedia.org/T150256) (owner: 10BBlack) [13:49:42] (03PS2) 10BBlack: LVS refactor: service IPs and sparing out lvs101[12] [puppet] - 10https://gerrit.wikimedia.org/r/356833 (https://phabricator.wikimedia.org/T150256) [13:51:12] (03CR) 10jerkins-bot: [V: 04-1] LVS refactor: service IPs and sparing out lvs101[12] [puppet] - 10https://gerrit.wikimedia.org/r/356833 (https://phabricator.wikimedia.org/T150256) (owner: 10BBlack) [13:52:25] (03PS3) 10BBlack: LVS refactor: service IPs and sparing out lvs101[12] [puppet] - 10https://gerrit.wikimedia.org/r/356833 (https://phabricator.wikimedia.org/T150256) [13:52:28] jenkins is so negative [13:52:37] it never trumpets my eventual V+2 :P [14:20:21] (03PS1) 10Hashar: (DO NOT SUBMIT) jenkins: hack for custom keystore [puppet] - 10https://gerrit.wikimedia.org/r/356838 (https://phabricator.wikimedia.org/T78705) [14:21:36] (03PS3) 10Alexandros Kosiaris: motd::script: Don't use validate_re on an integer [puppet] - 10https://gerrit.wikimedia.org/r/356483 [14:22:09] (03Abandoned) 10Hashar: (DO NOT SUBMIT) jenkins: hack for custom keystore [puppet] - 10https://gerrit.wikimedia.org/r/356838 (https://phabricator.wikimedia.org/T78705) (owner: 10Hashar) [14:24:31] PROBLEM - puppet last run on db1101 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:25:46] !log mobrovac@tin Started deploy [restbase/deploy@4b14527]: Add the extract_html property to the summary end point for T165017 [14:25:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:25:55] T165017: Setup RESTBase HTML extract endpoint - https://phabricator.wikimedia.org/T165017 [14:27:08] (03CR) 10Filippo Giunchedi: "LGTM, just nits really" (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/356814 (owner: 10Alexandros Kosiaris) [14:30:58] (03PS1) 10Ori.livneh: Capture messages on 'autoloader' debug log channel [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356841 [14:31:14] o_O [14:31:15] an ori! [14:32:10] :) [14:32:29] !log mobrovac@tin Finished deploy [restbase/deploy@4b14527]: Add the extract_html property to the summary end point for T165017 (duration: 06m 43s) [14:32:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:32:38] T165017: Setup RESTBase HTML extract endpoint - https://phabricator.wikimedia.org/T165017 [14:33:19] (03CR) 10BBlack: [C: 032] LVS refactor: service IPs and sparing out lvs101[12] [puppet] - 10https://gerrit.wikimedia.org/r/356833 (https://phabricator.wikimedia.org/T150256) (owner: 10BBlack) [14:36:07] I am going to check db1101, it seems to have puppet issues [14:38:31] RECOVERY - puppet last run on db1101 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:39:43] (03PS3) 10Faidon Liambotis: Remove str2bool from is_virtual facts [puppet] - 10https://gerrit.wikimedia.org/r/356031 (https://phabricator.wikimedia.org/T166372) [14:39:57] (03CR) 10Faidon Liambotis: [C: 032] "Looks good: http://puppet-compiler.wmflabs.org/6660/" [puppet] - 10https://gerrit.wikimedia.org/r/356031 (https://phabricator.wikimedia.org/T166372) (owner: 10Faidon Liambotis) [14:42:44] (03CR) 10Ori.livneh: [C: 031] "LGTM. I don't typically carry my key with me, so please find someone else to merge." [puppet] - 10https://gerrit.wikimedia.org/r/349352 (owner: 10Krinkle) [14:45:21] (03PS3) 10Faidon Liambotis: raid: switch from stringified fact to array [puppet] - 10https://gerrit.wikimedia.org/r/356030 (https://phabricator.wikimedia.org/T166372) [14:45:33] (03CR) 10Faidon Liambotis: [C: 032] "Looks good: http://puppet-compiler.wmflabs.org/6660/" [puppet] - 10https://gerrit.wikimedia.org/r/356030 (https://phabricator.wikimedia.org/T166372) (owner: 10Faidon Liambotis) [14:46:21] (03CR) 10Faidon Liambotis: [V: 032 C: 032] "Not going to wait for another 5 minutes" [puppet] - 10https://gerrit.wikimedia.org/r/356030 (https://phabricator.wikimedia.org/T166372) (owner: 10Faidon Liambotis) [14:49:42] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:51:12] that lvs2001 error seems to be a race between the two puppetmasters [14:51:21] i.e. the fact was deployed before the code was [14:52:42] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:01:57] !log restarting ircecho on tegment [15:02:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:12:51] PROBLEM - IPMI Temperature on sodium is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:12:59] (03PS3) 10Faidon Liambotis: Do not confine LLDP fact to physical/non-VMs [puppet] - 10https://gerrit.wikimedia.org/r/354108 [15:14:15] (03PS1) 10BryanDavis: ircecho: notify service on config change [puppet] - 10https://gerrit.wikimedia.org/r/356852 [15:25:59] (03CR) 10Jcrespo: [C: 031] "I think the logic is sane, the service is not so critical -we should just deploy and revert if it creates unintended effects." [puppet] - 10https://gerrit.wikimedia.org/r/356852 (owner: 10BryanDavis) [16:02:52] (03PS1) 10BryanDavis: planet: add Wikikmedia Performance Team blog feed [puppet] - 10https://gerrit.wikimedia.org/r/356859 [16:03:31] !log start wmf-auto-reimage of lvs1011, lvs1012 [16:03:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:16:15] !log gerrit2001: gerrit updated to 2.13.8+git1-wmf.4 [16:16:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:42:54] PROBLEM - IPMI Temperature on sodium is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:50:32] (03CR) 10Chad: [C: 031] Gerrit: Set gc.auto and gc.autopacklimit to 0 in ~/.gitconfig [puppet] - 10https://gerrit.wikimedia.org/r/356824 (https://phabricator.wikimedia.org/T151676) (owner: 10Paladox) [16:50:47] (03CR) 10Chad: [C: 031] Gerrit: Increase packedGitOpenFiles to 6000 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/356586 (owner: 10Paladox) [16:51:20] (03CR) 10Chad: [C: 031] Gerrit: Increase packedGitOpenFiles to 6000 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/356586 (owner: 10Paladox) [16:55:05] I guess we've lost a bot [16:55:24] the phab outputs aren't coming into here anymore [16:56:27] wikibugs is still here, just not saying Phab stuff :\ [16:56:30] That's no bueno [17:01:51] !log starting wmf-auto-reimage on lvs1007-10 [17:01:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:04:14] PROBLEM - salt-minion processes on puppetmaster1001 is CRITICAL: PROCS CRITICAL: 5 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [17:09:43] I guess it needs restarting [17:16:05] (03Abandoned) 10Dzahn: gerrit: switch to base::service_unit, import sysvinit script to puppet [puppet] - 10https://gerrit.wikimedia.org/r/356516 (owner: 10Dzahn) [17:16:24] (03Abandoned) 10Dzahn: stop using exim4::ganglia [puppet] - 10https://gerrit.wikimedia.org/r/355741 (owner: 10Dzahn) [17:16:59] (03Abandoned) 10Dzahn: phabricator: avoid root@wm.org mail alias in labs [puppet] - 10https://gerrit.wikimedia.org/r/355640 (owner: 10Dzahn) [17:19:24] (03PS2) 10Dzahn: Drop gerrit2001.yaml only includes temp admin permissions [puppet] - 10https://gerrit.wikimedia.org/r/356765 (owner: 10Chad) [17:24:14] RECOVERY - salt-minion processes on puppetmaster1001 is OK: PROCS OK: 4 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [17:24:42] (03CR) 10Dzahn: [C: 032] Drop gerrit2001.yaml only includes temp admin permissions [puppet] - 10https://gerrit.wikimedia.org/r/356765 (owner: 10Chad) [17:25:29] (03Abandoned) 10Dzahn: gerrit: import systemd unit file from deb to puppet [puppet] - 10https://gerrit.wikimedia.org/r/356517 (owner: 10Dzahn) [17:26:55] (03CR) 10Dzahn: "1) Put it in the package, no reason (and way more confusing) to have it in puppet." [puppet] - 10https://gerrit.wikimedia.org/r/356516 (owner: 10Dzahn) [17:27:23] (03PS3) 10Dzahn: [Planet Wikimedia] Add some hackathon-related blogs [puppet] - 10https://gerrit.wikimedia.org/r/356813 (owner: 10Nemo bis) [17:28:29] (03CR) 10Dzahn: "had a dependency on using base::service_unit which apparently i am not supposed to use in this case" [puppet] - 10https://gerrit.wikimedia.org/r/356517 (owner: 10Dzahn) [17:29:16] (03Restored) 10Chad: gerrit: switch to base::service_unit, import sysvinit script to puppet [puppet] - 10https://gerrit.wikimedia.org/r/356516 (owner: 10Dzahn) [17:30:36] (03PS4) 10Dzahn: [Planet Wikimedia] Add some hackathon-related blogs, add Greek planet [puppet] - 10https://gerrit.wikimedia.org/r/356813 (owner: 10Nemo bis) [17:34:40] (03PS1) 10Mobrovac: Set the User-Agent header field when doing requests [software/service-checker] - 10https://gerrit.wikimedia.org/r/356870 [17:36:27] (03CR) 10Dzahn: [C: 032] [Planet Wikimedia] Add some hackathon-related blogs, add Greek planet [puppet] - 10https://gerrit.wikimedia.org/r/356813 (owner: 10Nemo bis) [18:12:54] PROBLEM - IPMI Temperature on sodium is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:16:34] PROBLEM - Nginx local proxy to apache on mw1202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.161 second response time [18:16:37] PROBLEM - HHVM rendering on mw1202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.073 second response time [18:16:57] (03PS1) 10Andrew Bogott: Make the ORD wikitech-static the official wikitech-static. [dns] - 10https://gerrit.wikimedia.org/r/356874 (https://phabricator.wikimedia.org/T164271) [18:17:34] RECOVERY - Nginx local proxy to apache on mw1202 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.187 second response time [18:17:35] RECOVERY - HHVM rendering on mw1202 is OK: HTTP OK: HTTP/1.1 200 OK - 78677 bytes in 1.446 second response time [18:28:21] !log mobrovac@tin Started deploy [restbase/deploy@4b14527]: h [18:28:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:29:01] !log mobrovac@tin Started deploy [restbase/deploy@4b14527]: (no justification provided) [18:29:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:29:43] !log mobrovac@tin Finished deploy [restbase/deploy@4b14527]: (no justification provided) (duration: 00m 41s) [18:29:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:48:02] (03PS2) 10Dzahn: add admin group releasers-mediawiki to mwreleases1001 [puppet] - 10https://gerrit.wikimedia.org/r/356425 (https://phabricator.wikimedia.org/T164030) [18:49:51] (03CR) 10Dzahn: [C: 032] "existing group on new host that is still being setup" [puppet] - 10https://gerrit.wikimedia.org/r/356425 (https://phabricator.wikimedia.org/T164030) (owner: 10Dzahn) [18:57:53] (03PS2) 10Dzahn: planet: add Wikikmedia Performance Team blog feed [puppet] - 10https://gerrit.wikimedia.org/r/356859 (owner: 10BryanDavis) [18:58:32] (03CR) 10Dzahn: [C: 032] planet: add Wikikmedia Performance Team blog feed [puppet] - 10https://gerrit.wikimedia.org/r/356859 (owner: 10BryanDavis) [19:04:06] (03PS1) 10Bmansurov: Enable ElectronPdf on all projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356881 (https://phabricator.wikimedia.org/T165954) [19:07:51] (03PS2) 10Dzahn: ircecho: notify service on config change [puppet] - 10https://gerrit.wikimedia.org/r/356852 (owner: 10BryanDavis) [19:40:54] PROBLEM - IPMI Temperature on ocg1002 is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:42:54] PROBLEM - IPMI Temperature on sodium is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:06:24] (03PS1) 10Hashar: kibana: allow any arbitrary setting [puppet] - 10https://gerrit.wikimedia.org/r/356900 [20:07:16] (03CR) 10Hashar: "Absolutely untested. Would at least need to puppet compile it on the hosts that run kibana." [puppet] - 10https://gerrit.wikimedia.org/r/356900 (owner: 10Hashar) [20:11:31] (03CR) 10Smalyshev: [C: 031] flake8 fixes for E305 [puppet] - 10https://gerrit.wikimedia.org/r/356234 (owner: 10BryanDavis) [20:25:32] (03CR) 10Pmiazga: Enable ElectronPdf on all projects (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356881 (https://phabricator.wikimedia.org/T165954) (owner: 10Bmansurov) [20:31:10] (03PS2) 10Bmansurov: Enable ElectronPdf on all projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356881 (https://phabricator.wikimedia.org/T165954) [20:36:58] (03PS1) 10Dzahn: wikitech-static: lower TTL to 5M [dns] - 10https://gerrit.wikimedia.org/r/356930 (https://phabricator.wikimedia.org/T164271) [20:37:06] (03CR) 10Pmiazga: [C: 031] "Looks good" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356881 (https://phabricator.wikimedia.org/T165954) (owner: 10Bmansurov) [20:38:34] (03CR) 10Dzahn: [C: 032] wikitech-static: lower TTL to 5M [dns] - 10https://gerrit.wikimedia.org/r/356930 (https://phabricator.wikimedia.org/T164271) (owner: 10Dzahn) [20:40:54] PROBLEM - IPMI Temperature on ocg1002 is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:41:06] (03PS2) 10Dzahn: Make the ORD wikitech-static the official wikitech-static. [dns] - 10https://gerrit.wikimedia.org/r/356874 (https://phabricator.wikimedia.org/T164271) (owner: 10Andrew Bogott) [20:44:58] (03PS1) 10Dzahn: status.wm.org: lower TTL to 5M [dns] - 10https://gerrit.wikimedia.org/r/356936 (https://phabricator.wikimedia.org/T164271) [20:46:48] (03CR) 10Dzahn: [C: 032] status.wm.org: lower TTL to 5M [dns] - 10https://gerrit.wikimedia.org/r/356936 (https://phabricator.wikimedia.org/T164271) (owner: 10Dzahn) [21:01:54] PROBLEM - Host lvs1007 is DOWN: PING CRITICAL - Packet loss = 100% [21:02:04] PROBLEM - Host lvs1008 is DOWN: PING CRITICAL - Packet loss = 100% [21:02:44] PROBLEM - Check systemd state on lvs1010 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [21:06:11] ^ not worrying about that because i know brandon is installing lvs1007-9 [21:08:00] mutante: why the TTL changes? [21:08:35] (03PS1) 10Dzahn: add el.planet.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/356952 [21:09:21] Zppix: if for some reason i have to revert and it breaks i can revert in 5 minutes instead of 1 hour [21:10:37] mutante: oh makes sense :) thanks [21:12:54] PROBLEM - IPMI Temperature on sodium is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:13:03] (03PS4) 10Paladox: contint: Only install libmysqlclient-dev if on trusty or jessie [puppet] - 10https://gerrit.wikimedia.org/r/356246 (https://phabricator.wikimedia.org/T166611) [21:13:24] (03PS12) 10Paladox: jenkins: Install java 8 on stretch and greater [puppet] - 10https://gerrit.wikimedia.org/r/356243 (https://phabricator.wikimedia.org/T166611) [21:13:40] (03PS4) 10Paladox: contint: Only install java 7 on trusty and jessie [puppet] - 10https://gerrit.wikimedia.org/r/356241 (https://phabricator.wikimedia.org/T166611) [21:16:28] (03PS5) 10Paladox: contint: Only install java 7 on trusty and jessie [puppet] - 10https://gerrit.wikimedia.org/r/356241 (https://phabricator.wikimedia.org/T166611) [21:16:33] (03CR) 10Paladox: contint: Only install java 7 on trusty and jessie (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/356241 (https://phabricator.wikimedia.org/T166611) (owner: 10Paladox) [21:17:52] (03PS13) 10Paladox: contint: skip hhvm experimental pin on Trusty [puppet] - 10https://gerrit.wikimedia.org/r/353964 (https://phabricator.wikimedia.org/T165462) [21:22:54] PROBLEM - puppet last run on mw1301 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:23:16] (03CR) 10Dzahn: [C: 032] add el.planet.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/356952 (owner: 10Dzahn) [21:32:04] PROBLEM - Check Varnish expiry mailbox lag on cp1074 is CRITICAL: CRITICAL: expiry mailbox lag is 2035736 [21:42:32] (03CR) 10Jdlrobson: [C: 031] Enable ElectronPdf on all projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356881 (https://phabricator.wikimedia.org/T165954) (owner: 10Bmansurov) [21:50:54] RECOVERY - puppet last run on mw1301 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [21:51:44] (03PS3) 10Paladox: Gerrit: Set gc.auto and gc.autopacklimit to 0 in ~/.gitconfig [puppet] - 10https://gerrit.wikimedia.org/r/356824 (https://phabricator.wikimedia.org/T151676) [21:59:44] PROBLEM - puppet last run on puppetmaster2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:00:48] (03PS5) 10Dzahn: Gerrit: Increase packedGitOpenFiles to 6000 [puppet] - 10https://gerrit.wikimedia.org/r/356586 (owner: 10Paladox) [22:01:58] (03PS6) 10Dzahn: Gerrit: Increase packedGitOpenFiles to 6000 [puppet] - 10https://gerrit.wikimedia.org/r/356586 (owner: 10Paladox) [22:02:18] (03CR) 10Dzahn: [C: 032] Gerrit: Increase packedGitOpenFiles to 6000 [puppet] - 10https://gerrit.wikimedia.org/r/356586 (owner: 10Paladox) [22:02:26] thanks mutante ^^ :) [22:03:16] (03CR) 10Dzahn: [V: 032 C: 032] Gerrit: Increase packedGitOpenFiles to 6000 [puppet] - 10https://gerrit.wikimedia.org/r/356586 (owner: 10Paladox) [22:03:40] (03PS4) 10Paladox: Gerrit: Set gc.auto and gc.autopacklimit to 0 in ~/.gitconfig [puppet] - 10https://gerrit.wikimedia.org/r/356824 (https://phabricator.wikimedia.org/T151676) [22:15:47] (03CR) 10Dzahn: [C: 032] "https://groups.google.com/forum/#!topic/repo-discuss/lVR37Pm4G3c" [puppet] - 10https://gerrit.wikimedia.org/r/356824 (https://phabricator.wikimedia.org/T151676) (owner: 10Paladox) [22:16:10] (03CR) 10Dzahn: [C: 032] "this is to make sure GC _stays off_ in the next version" [puppet] - 10https://gerrit.wikimedia.org/r/356824 (https://phabricator.wikimedia.org/T151676) (owner: 10Paladox) [22:19:34] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[git_pull_analytics/limn-language-data],Exec[git_pull_analytics/limn-flow-data],Exec[git_pull_geowiki-scripts],Exec[git_pull_analytics/discovery-stats] [22:20:44] PROBLEM - puppet last run on labsdb1010 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_operations/mediawiki-config] [22:22:44] PROBLEM - puppet last run on labsdb1011 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_operations/mediawiki-config] [22:23:38] Cannot load SSH keys for suchabot [22:23:42] git review is saying "Permission denied (publickey)" [22:23:43] grrrr [22:24:08] same issue? [22:24:10] yea, i'm already looking [22:24:13] thanks! [22:24:30] i merged something else and it caused a restart and now there is an issue about that missing pubkey for the bot ??! [22:24:31] gerrit seems to be down [22:25:15] i'm trying to fix it [22:25:24] PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_operations/mediawiki-config] [22:25:37] mutante i found the fix. [22:25:38] [gc] [22:25:39] auto = 0 [22:26:05] PROBLEM - puppet last run on kafka2003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_mediawiki/event-schemas] [22:26:06] same for that other one [22:26:13] no, /var/lib/gerrit2/.gitconfig invalid org.eclipse.jgit.errors.ConfigInvalidException: Cannot read file /var/lib/gerrit2/.gitconfig [22:26:19] Yep [22:26:27] [gc] auto=0 fixes it [22:26:37] tested locally and managed to reproduce it [22:27:37] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_operations/software/xhprof],Exec[git_pull_operations/software/xhgui] [22:27:44] https://phabricator.wikimedia.org/P5539 [22:29:09] live fix applied, restarted [22:29:14] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_operations/mediawiki-config] [22:29:25] kaldari: SMalyshev : should be back [22:29:43] mutante: indeed it is, thanks! [22:29:44] RECOVERY - puppet last run on puppetmaster2001 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:30:11] pheeew, ok :) [22:30:32] that was a little scary with the "file not found" for the whole repo [22:30:44] PROBLEM - puppet last run on kafka1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_mediawiki/event-schemas] [22:30:45] (03Draft1) 10Paladox: Gerrit: Fix wrong syntax in ~/.gitconfig [puppet] - 10https://gerrit.wikimedia.org/r/356969 [22:30:48] (03PS2) 10Paladox: Gerrit: Fix wrong syntax in ~/.gitconfig [puppet] - 10https://gerrit.wikimedia.org/r/356969 [22:40:19] (03PS3) 10Paladox: Gerrit: Fix wrong syntax in ~/.gitconfig [puppet] - 10https://gerrit.wikimedia.org/r/356969 [22:41:30] mutante ^^ probaly want to merge that before puppet overides the file you updated to fix gerrit. [22:42:21] yes, but disabling puppet is the first thing i did when it broke [22:42:33] ah i see [22:42:44] (03CR) 10Dzahn: [C: 032] Gerrit: Fix wrong syntax in ~/.gitconfig [puppet] - 10https://gerrit.wikimedia.org/r/356969 (owner: 10Paladox) [22:42:54] PROBLEM - IPMI Temperature on sodium is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:43:05] (03CR) 10Dzahn: [C: 032] "yes, the wrong syntax brought gerrit down and the new syntax here fix it when applying manual fix" [puppet] - 10https://gerrit.wikimedia.org/r/356969 (owner: 10Paladox) [22:43:30] thanks [22:47:44] RECOVERY - puppet last run on labsdb1010 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [22:47:44] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [22:51:44] RECOVERY - puppet last run on labsdb1011 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [22:52:04] RECOVERY - Check Varnish expiry mailbox lag on cp1074 is OK: OK: expiry mailbox lag is 0 [22:53:04] PROBLEM - High lag on wdqs2001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [1800.0] [22:53:24] RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [22:55:14] RECOVERY - puppet last run on kafka2003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:55:24] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [22:56:14] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [22:58:44] RECOVERY - puppet last run on kafka1003 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:07:04] RECOVERY - High lag on wdqs2001 is OK: OK: Less than 30.00% above the threshold [600.0] [23:12:10] (03PS3) 10Dzahn: Make the ORD wikitech-static the official wikitech-static. [dns] - 10https://gerrit.wikimedia.org/r/356874 (https://phabricator.wikimedia.org/T164271) (owner: 10Andrew Bogott) [23:14:15] !log maintenance on status.wikimedia.org and wikitech-static.wikimedia.org [23:14:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:15:16] (03CR) 10Dzahn: [C: 032] Make the ORD wikitech-static the official wikitech-static. [dns] - 10https://gerrit.wikimedia.org/r/356874 (https://phabricator.wikimedia.org/T164271) (owner: 10Andrew Bogott) [23:18:08] !log wikitech-static-ord: installed package upgrades, installed vim, removing "ord" from Apache config after DNS change .. [23:18:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:20:00] !log wikitech-static (iad): adjust Apache config to use wikitech-static-iad [23:20:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:22:54] !log wikitech-static-ord copied Lets-Encrypt intermediate certs from /usr/local/share/ca-certificates on old server [23:23:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:36:00] would you guys oppose to me cleaning up throttle.php and removing really old throttle rules? [23:36:34] sounds ok to me if they are all expired anyways [23:37:09] ok [23:40:54] PROBLEM - IPMI Temperature on ocg1002 is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:41:30] detail": "Provided agreement URL [https://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf] does not match current agreement URL [https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf]", [23:41:37] lol, that was unexpected.. ok [23:41:54] PROBLEM - IPMI Temperature on ms-be2014 is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:42:41] (03Draft2) 10Zppix: Remove old expired throttle rules to cleanup throttle.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356972 [23:44:19] if you get a moment (i know your busy) can you check ^ [23:45:31] !log wikitech-static-iad: create new cert for "iad" hostname, using acme-setup/acme-tiny: /usr/local/sbin# acme-setup -i "wikitech-static-iad" -s "wikitech-static-iad.wikimedia.org" ; python acme_tiny.py --account-key /etc/acme/acct/acct.key --csr /etc/acme/csr/wikitech-static-iad.pem --acme-dir /var/acme/challenge/ > /etc/acme/cert/wikitech-static-iad-signed.csr ; had to hack acme_tiny.py [23:45:37] to adjust URL to agreement PDF, or "Provided agreement URL does not match current agreement URL" [23:45:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:45:40] meh [23:46:10] got to love multiline :P [23:46:42] !log wikitech-static-iad: edited acme_tiny.py to adjust URL to agreement PDF, to fix ""Provided agreement URL [https://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf] does not match current agreement URL[https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf]" [23:46:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:48:23] (03CR) 10Dzahn: [C: 031] "both "to" dates have passed" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356972 (owner: 10Zppix) [23:48:41] Zppix: lgtm, though i wouldn't call "end of May" "really old" on June 2nd :) [23:49:04] mutante: 1 or to of those are rules i added lol [23:49:15] 2* [23:49:21] ah, ok [23:50:42] im not going to bother with swat scheduling it its so minor ill let it get merged when it gets merged :P [23:51:39] (03CR) 10Chad: [C: 032] Remove old expired throttle rules to cleanup throttle.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356972 (owner: 10Zppix) [23:52:25] Zppix: i'm not sure if it works that way, heh, but you can see [23:52:32] mutante: ^^ [23:52:38] (03Merged) 10jenkins-bot: Remove old expired throttle rules to cleanup throttle.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356972 (owner: 10Zppix) [23:52:48] heh, ok [23:52:50] (03CR) 10jenkins-bot: Remove old expired throttle rules to cleanup throttle.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/356972 (owner: 10Zppix) [23:53:04] i wish that always happened when i say that stuff [23:53:24] same :) [23:53:37] !log demon@tin Synchronized wmf-config/throttle.php: pruning some old throttle exceptions (duration: 00m 40s) [23:53:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:54:01] Ok and now I start my weekend, it's beer o'clock [23:54:02] ty RainbowSprinkles [23:55:18] RainbowSprinkles: word [23:55:19] finally :) i got the cert thing working too [23:55:20] https://wikitech-static-iad.wikimedia.org/ [23:55:33] 5 minutes before beer'o'clock indeed [23:56:05] wikitech-static is now in Chicago [23:56:10] i never liked rackspace [23:56:24] eh, except the logo is gone, heh [23:56:54] no, it just didn't have one [23:57:34] im going to semi-afk have a good weekend guys [23:59:55] bye Zppix, thx