[00:00:29] same on cobalt. done. thanks for the contribution to removing apache module [00:01:26] your welcome :) [00:02:56] https://gerrit.wikimedia.org/r/#/q/topic:apache-httpd+(status:open+OR+status:merged) [00:04:29] there are the "simplap" and "simplelamp" classes for Cloud VPS.. should do them as well.. if they are used [00:04:36] simplelap [00:16:09] (03Draft1) 10Paladox: racktables: apache -> httpd module [puppet] - 10https://gerrit.wikimedia.org/r/409478 [00:16:11] (03PS2) 10Paladox: racktables: apache -> httpd module [puppet] - 10https://gerrit.wikimedia.org/r/409478 [00:17:41] (03PS3) 10Paladox: racktables: apache -> httpd module [puppet] - 10https://gerrit.wikimedia.org/r/409478 [00:30:30] (03CR) 10Dzahn: [C: 04-1] "it's good but all the "misc apps" need to be done at once in a single change or we get duplicates like in http://puppet-compiler.wmflabs.o" [puppet] - 10https://gerrit.wikimedia.org/r/409478 (owner: 10Paladox) [00:31:58] (03Draft1) 10Paladox: RequestTracker: apache -> httpd module [puppet] - 10https://gerrit.wikimedia.org/r/409479 [00:32:01] (03PS2) 10Paladox: RequestTracker: apache -> httpd module [puppet] - 10https://gerrit.wikimedia.org/r/409479 [00:36:50] (03CR) 10Dzahn: [C: 032] "http://puppet-compiler.wmflabs.org/9925/ununpentium.wikimedia.org/" [puppet] - 10https://gerrit.wikimedia.org/r/409479 (owner: 10Paladox) [00:36:57] thanks :) [00:39:24] (03CR) 10Huji: [C: 031] Log accessing private abusefilter details [mediawiki-config] - 10https://gerrit.wikimedia.org/r/409445 (https://phabricator.wikimedia.org/T160357) (owner: 10MarcoAurelio) [00:39:30] yep, that was also no-op, thanks. now going afk for a bit [00:40:34] thanks :) [00:40:41] * paladox is also going to go too [00:40:49] getting very late (almost 1am) [00:43:05] (03PS1) 10Dzahn: lists: apache -> httpd module [puppet] - 10https://gerrit.wikimedia.org/r/409480 [01:10:07] (03CR) 10Chad: [C: 032] Updating gitiles to stable-2.14's head [software/gerrit] - 10https://gerrit.wikimedia.org/r/409442 (owner: 10Chad) [01:10:08] (03CR) 10Chad: [V: 032 C: 032] Updating gitiles to stable-2.14's head [software/gerrit] - 10https://gerrit.wikimedia.org/r/409442 (owner: 10Chad) [01:10:40] !log demon@tin Started deploy [gerrit/gerrit@5d5193e]: one last gitiles rebuild before the weekend [01:10:49] !log demon@tin Finished deploy [gerrit/gerrit@5d5193e]: one last gitiles rebuild before the weekend (duration: 00m 10s) [01:10:52] paladox: ^^^ [01:10:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:11:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:21:29] ACKNOWLEDGEMENT - Host dns5001.mgmt is DOWN: PING CRITICAL - Packet loss = 100% daniel_zahn https://phabricator.wikimedia.org/T186902 [01:25:51] no_justification: thanks :) [01:26:02] We can merge the puppet changes now [01:26:06] mutante: ^^ [02:02:30] (03PS4) 10Dzahn: icinga: add notification type to SMS content and other improvements [puppet] - 10https://gerrit.wikimedia.org/r/406535 (https://phabricator.wikimedia.org/T185862) [02:04:06] (03PS5) 10Dzahn: icinga: add notification type to SMS content and other improvements [puppet] - 10https://gerrit.wikimedia.org/r/406535 (https://phabricator.wikimedia.org/T185862) [02:06:54] (03PS2) 10Paladox: Gerrit: Set cookie path to / [puppet] - 10https://gerrit.wikimedia.org/r/409216 (owner: 10Chad) [02:07:39] (03CR) 10Paladox: [C: 031] "This can be done now, as we have installed the updated gitiles plugin." [puppet] - 10https://gerrit.wikimedia.org/r/409216 (owner: 10Chad) [02:07:49] (03PS3) 10Paladox: Gerrit: Proxy gitiles through gerrit.wikimedia.org/g/ [puppet] - 10https://gerrit.wikimedia.org/r/409211 (https://phabricator.wikimedia.org/T184116) (owner: 10Chad) [02:08:12] (03CR) 10Paladox: [C: 031] "This works locally." [puppet] - 10https://gerrit.wikimedia.org/r/409211 (https://phabricator.wikimedia.org/T184116) (owner: 10Chad) [02:08:34] (03PS4) 10Paladox: Gerrit: Set gerrit.baseUrl in gitiles.config [puppet] - 10https://gerrit.wikimedia.org/r/409385 [02:10:42] (03CR) 10Paladox: [C: 031] "@Dzahn, hi wondering if you can review these Serbia’s of changed please?" [puppet] - 10https://gerrit.wikimedia.org/r/409216 (owner: 10Chad) [03:26:28] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 801.91 seconds [03:55:29] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 128.08 seconds [04:19:38] PROBLEM - Host conf1002.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [04:24:48] RECOVERY - Host conf1002.mgmt is UP: PING OK - Packet loss = 0%, RTA = 2.75 ms [06:07:32] (03CR) 10Krinkle: "To follow-up, I'd recommend renaming just one variable or one group of variables. For example the ones set by MWRealm (wmfDatacenter and w" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/392184 (https://phabricator.wikimedia.org/T45956) (owner: 10TerraCodes) [06:11:04] (03CR) 10Krinkle: "Hm.. [nocanon] means the backend will perceive as if the browser is not on /g/ but on the longer form url. That should work fine as long a" [puppet] - 10https://gerrit.wikimedia.org/r/409211 (https://phabricator.wikimedia.org/T184116) (owner: 10Chad) [07:10:39] 10Operations, 10DBA, 10Performance-Team, 10Patch-For-Review, 10codfw-rollout: [RFC] improve parsercache replication and sharding handling - https://phabricator.wikimedia.org/T133523#3960171 (10Krinkle) [07:12:46] 10Operations, 10DBA, 10Performance-Team, 10Patch-For-Review, 10codfw-rollout: [RFC] improve parsercache replication and sharding handling - https://phabricator.wikimedia.org/T133523#2234475 (10Krinkle) @jcrespo What's the current status of this task? Is this intended to be a general request for comments... [08:05:28] PROBLEM - Host bohrium is DOWN: PING CRITICAL - Packet loss = 58%, RTA = 9564.62 ms [08:05:48] PROBLEM - Host neon is DOWN: PING CRITICAL - Packet loss = 100% [08:05:58] PROBLEM - Host actinium is DOWN: PING CRITICAL - Packet loss = 100% [08:06:29] PROBLEM - Host etcd1002 is DOWN: PING CRITICAL - Packet loss = 100% [08:06:39] PROBLEM - Host etcd1005 is DOWN: PING CRITICAL - Packet loss = 100% [08:06:39] PROBLEM - Host kubestagetcd1003 is DOWN: PING CRITICAL - Packet loss = 100% [08:06:39] PROBLEM - Host install1002 is DOWN: PING CRITICAL - Packet loss = 100% [08:06:39] PROBLEM - Host boron is DOWN: PING CRITICAL - Packet loss = 100% [08:06:39] PROBLEM - Host rutherfordium is DOWN: PING CRITICAL - Packet loss = 100% [08:06:48] PROBLEM - Host etcd1004 is DOWN: PING CRITICAL - Packet loss = 100% [08:06:58] PROBLEM - Host netmon1003 is DOWN: PING CRITICAL - Packet loss = 100% [08:07:38] PROBLEM - SSH on ganeti1008 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:48] RECOVERY - Host rutherfordium is UP: PING OK - Packet loss = 0%, RTA = 0.77 ms [08:09:58] PROBLEM - Request latencies on argon is CRITICAL: CRITICAL - apiserver_request_latencies is 654095 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:09:58] RECOVERY - Host netmon1003 is UP: PING OK - Packet loss = 0%, RTA = 2.10 ms [08:09:58] RECOVERY - Host etcd1002 is UP: PING OK - Packet loss = 0%, RTA = 2.04 ms [08:09:58] RECOVERY - Host neon is UP: PING OK - Packet loss = 0%, RTA = 1.99 ms [08:09:58] RECOVERY - Host kubestagetcd1003 is UP: PING OK - Packet loss = 0%, RTA = 1.97 ms [08:09:58] RECOVERY - Host etcd1005 is UP: PING OK - Packet loss = 0%, RTA = 1.98 ms [08:10:08] PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?panelId=3&fullscreen&orgId=1&var-site=All&var-cache_type=misc&var-status_type=5 [08:10:10] RECOVERY - Host actinium is UP: PING OK - Packet loss = 0%, RTA = 2.28 ms [08:10:18] RECOVERY - Host etcd1004 is UP: PING OK - Packet loss = 0%, RTA = 0.71 ms [08:10:28] PROBLEM - Request latencies on chlorine is CRITICAL: CRITICAL - apiserver_request_latencies is 510700 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:10:38] RECOVERY - SSH on ganeti1008 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [08:10:48] RECOVERY - Host bohrium is UP: PING OK - Packet loss = 0%, RTA = 0.98 ms [08:10:48] RECOVERY - Host boron is UP: PING OK - Packet loss = 0%, RTA = 0.51 ms [08:10:49] PROBLEM - etcd request latencies on argon is CRITICAL: CRITICAL - etcd_request_latencies is 5682919 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:11:08] PROBLEM - etcd request latencies on chlorine is CRITICAL: CRITICAL - etcd_request_latencies is 619435 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:11:48] RECOVERY - Host install1002 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms [08:14:08] RECOVERY - etcd request latencies on chlorine is OK: OK - etcd_request_latencies is 2931 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:14:09] PROBLEM - Request latencies on neon is CRITICAL: CRITICAL - apiserver_request_latencies is 802490 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:14:28] RECOVERY - Request latencies on chlorine is OK: OK - apiserver_request_latencies is 4461 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:14:49] PROBLEM - etcd request latencies on neon is CRITICAL: CRITICAL - etcd_request_latencies is 594854 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:15:09] RECOVERY - Request latencies on neon is OK: OK - apiserver_request_latencies is 5916 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:15:48] RECOVERY - etcd request latencies on argon is OK: OK - etcd_request_latencies is 4332 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:15:49] RECOVERY - etcd request latencies on neon is OK: OK - etcd_request_latencies is 3833 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:16:00] RECOVERY - Request latencies on argon is OK: OK - apiserver_request_latencies is 7051 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:22:08] RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?panelId=3&fullscreen&orgId=1&var-site=All&var-cache_type=misc&var-status_type=5 [08:25:09] PROBLEM - Request latencies on neon is CRITICAL: CRITICAL - apiserver_request_latencies is 1178835 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:25:58] PROBLEM - etcd request latencies on neon is CRITICAL: CRITICAL - etcd_request_latencies is 3367929 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:31:18] RECOVERY - Request latencies on neon is OK: OK - apiserver_request_latencies is 5674 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [08:31:59] RECOVERY - etcd request latencies on neon is OK: OK - etcd_request_latencies is 3653 https://grafana.wikimedia.org/dashboard/db/kubernetes-api [14:46:37] (03Draft2) 10Tulsi Bhagat: Update the sitename of newiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/409557 (https://phabricator.wikimedia.org/T186952) [14:48:11] (03CR) 10Jayprakash12345: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/409557 (https://phabricator.wikimedia.org/T186952) (owner: 10Tulsi Bhagat) [14:50:21] (03CR) 10Jayprakash12345: [C: 031] Update the sitename of newiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/409557 (https://phabricator.wikimedia.org/T186952) (owner: 10Tulsi Bhagat) [15:46:29] PROBLEM - HHVM rendering on mw2143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:28] RECOVERY - HHVM rendering on mw2143 is OK: HTTP OK: HTTP/1.1 200 OK - 76168 bytes in 0.305 second response time [16:13:49] PROBLEM - Router interfaces on cr1-eqsin is CRITICAL: CRITICAL: host 103.102.166.129, interfaces up: 63, down: 2, dormant: 0, excluded: 0, unused: 0 [16:51:11] !log legoktm@tin Synchronized php-1.31.0-wmf.20/includes/specials/SpecialLog.php: SpecialLog: Fix results when no offender is specified - T186950 (duration: 00m 57s) [16:51:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:51:27] T186950: Wikimedia wikis have empty logs for suppression - https://phabricator.wikimedia.org/T186950 [16:59:21] legoktm: Nice work. We don’t deserve you. [16:59:48] * legoktm hugs James_F [19:26:04] (03PS1) 10Ladsgroup: Add federation-related configs for clients [mediawiki-config] - 10https://gerrit.wikimedia.org/r/409622 (https://phabricator.wikimedia.org/T186955) [19:49:00] (03PS1) 10Urbanecm: Change logos for sdwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/409625 (https://phabricator.wikimedia.org/T185865) [19:51:23] (03Abandoned) 10Urbanecm: New throttle rule, clean obsolete rules [mediawiki-config] - 10https://gerrit.wikimedia.org/r/406820 (https://phabricator.wikimedia.org/T186002) (owner: 10Urbanecm)