[00:00:31] (03Restored) 10Krinkle: hieradata: add redis stretch deployment-prep instances [puppet] - 10https://gerrit.wikimedia.org/r/386869 (https://phabricator.wikimedia.org/T179371) (owner: 10Filippo Giunchedi) [00:01:43] (03Restored) 10Krinkle: interface: IPAddr.new() requires an address family [puppet] - 10https://gerrit.wikimedia.org/r/336840 (owner: 10Hashar) [00:13:58] (03CR) 10Smalyshev: "> Patch Set 8:" [puppet] - 10https://gerrit.wikimedia.org/r/378355 (https://phabricator.wikimedia.org/T198356) (owner: 10Smalyshev) [00:14:45] (03CR) 10Smalyshev: "> Patch Set 8:" [puppet] - 10https://gerrit.wikimedia.org/r/378355 (https://phabricator.wikimedia.org/T198356) (owner: 10Smalyshev) [00:17:30] 10Operations, 10MediaWiki-Maintenance-scripts: sql notadbname gives a poor error message - https://phabricator.wikimedia.org/T199008 (10Reedy) [00:19:11] (03CR) 10Smalyshev: "> Patch Set 8:" [puppet] - 10https://gerrit.wikimedia.org/r/378355 (https://phabricator.wikimedia.org/T198356) (owner: 10Smalyshev) [00:20:16] (03PS9) 10Smalyshev: Generate daily diffs for categories RDF [puppet] - 10https://gerrit.wikimedia.org/r/378355 (https://phabricator.wikimedia.org/T198356) [00:21:21] 10Operations, 10DBA, 10Regression: MySQL prompt missing trailing space on mwmaint1001 - https://phabricator.wikimedia.org/T195636 (10Reedy) [00:23:23] 10Operations, 10DBA: sql prompts differ between mwmaint1001 and deploy1001 - https://phabricator.wikimedia.org/T199009 (10Reedy) [00:23:36] 10Operations, 10DBA: SQL autocomplete broken on mwmaint1001 - https://phabricator.wikimedia.org/T199010 (10Reedy) [00:24:14] 10Operations, 10MediaWiki-Maintenance-scripts: sql enwik gives a poor error message when db doesn't exist - https://phabricator.wikimedia.org/T199008 (10Reedy) [00:36:42] (03PS7) 10Krinkle: webperf: Get graphite_host for coal::processor from Hiera [puppet] - 10https://gerrit.wikimedia.org/r/442900 (https://phabricator.wikimedia.org/T195314) [00:36:44] (03PS9) 10Krinkle: webperf: Move site vars to profile class params (set from Hiera) [puppet] - 10https://gerrit.wikimedia.org/r/443739 (https://phabricator.wikimedia.org/T195314) [00:36:46] (03PS7) 10Krinkle: webperf: Rename webperf profiles for clarity [puppet] - 10https://gerrit.wikimedia.org/r/443752 (https://phabricator.wikimedia.org/T195314) [00:36:48] (03PS6) 10Krinkle: webperf: Rename role::xenon to profile::webperf::xenon [puppet] - 10https://gerrit.wikimedia.org/r/443757 (https://phabricator.wikimedia.org/T195312) [00:36:50] (03PS6) 10Krinkle: mediawiki: Change xenon interval for Beta Cluster from 10min to 30s [puppet] - 10https://gerrit.wikimedia.org/r/443762 [00:36:52] (03PS4) 10Krinkle: webperf: Enable xenondata_host on perfsite in Beta Cluster [puppet] - 10https://gerrit.wikimedia.org/r/443764 (https://phabricator.wikimedia.org/T195312) [00:47:44] (03CR) 10Alex Monk: "Removed from deployment-puppetmaster03" [puppet] - 10https://gerrit.wikimedia.org/r/386869 (https://phabricator.wikimedia.org/T179371) (owner: 10Filippo Giunchedi) [00:48:03] Krinkle, ^ please re-abandon [02:05:02] PROBLEM - Host mr1-eqsin.oob is DOWN: PING CRITICAL - Packet loss = 100% [02:10:21] RECOVERY - Host mr1-eqsin.oob is UP: PING OK - Packet loss = 0%, RTA = 227.16 ms [02:48:26] (03PS1) 10Krinkle: webperf: Split Redis from the rest of the arclamp profile [puppet] - 10https://gerrit.wikimedia.org/r/444331 (https://phabricator.wikimedia.org/T195312) [02:49:44] (03CR) 10jerkins-bot: [V: 04-1] webperf: Split Redis from the rest of the arclamp profile [puppet] - 10https://gerrit.wikimedia.org/r/444331 (https://phabricator.wikimedia.org/T195312) (owner: 10Krinkle) [02:49:56] (03CR) 10Krinkle: beta: update references to redis servers (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/438012 (owner: 10Giuseppe Lavagetto) [02:50:09] (03Abandoned) 10Krinkle: hieradata: add redis stretch deployment-prep instances [puppet] - 10https://gerrit.wikimedia.org/r/386869 (https://phabricator.wikimedia.org/T179371) (owner: 10Filippo Giunchedi) [02:51:32] (03PS2) 10Krinkle: webperf: Split Redis from the rest of the arclamp profile [puppet] - 10https://gerrit.wikimedia.org/r/444331 (https://phabricator.wikimedia.org/T195312) [02:58:38] davidwbarratt: just realised: another option is maybe use CentralNotice which would mean no coding at all [03:10:48] (03CR) 10Krinkle: "Clean diff - https://puppet-compiler.wmflabs.org/compiler02/11719/mwlog1001.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/444331 (https://phabricator.wikimedia.org/T195312) (owner: 10Krinkle) [03:11:34] (03PS3) 10Krinkle: webperf: Split Redis from the rest of the arclamp profile [puppet] - 10https://gerrit.wikimedia.org/r/444331 (https://phabricator.wikimedia.org/T195312) [03:14:09] (03CR) 10Krinkle: [C: 04-1] "Needs Hiera/Yaml for deployment-prep as well, but not in common.yaml because it needs to differ between beta's flourine and beta's webperf" [puppet] - 10https://gerrit.wikimedia.org/r/444331 (https://phabricator.wikimedia.org/T195312) (owner: 10Krinkle) [03:23:57] (03PS4) 10Krinkle: webperf: Split Redis from the rest of the arclamp profile [puppet] - 10https://gerrit.wikimedia.org/r/444331 (https://phabricator.wikimedia.org/T195312) [03:27:51] PROBLEM - toolschecker: tools homepage -admin tool- on tools.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 20 seconds [03:48:48] MaxSem hmm, it doesn't look like CentralNotice lets you target it to a specific special page [03:50:23] davidwbarratt: yeah it doesn't run on special pages [03:53:11] RECOVERY - toolschecker: tools homepage -admin tool- on tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1015 bytes in 0.025 second response time [04:25:42] PROBLEM - Device not healthy -SMART- on db1069 is CRITICAL: cluster=mysql device=megaraid,0 instance=db1069:9100 job=node site=eqiad https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=db1069&var-datasource=eqiad%2520prometheus%252Fops [05:04:53] PROBLEM - High load average on labstore1003 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [24.0] https://grafana.wikimedia.org/dashboard/db/labs-monitoring [05:09:22] RECOVERY - High load average on labstore1003 is OK: OK: Less than 50.00% above the threshold [16.0] https://grafana.wikimedia.org/dashboard/db/labs-monitoring [05:22:42] PROBLEM - Maps tiles generation on einsteinium is CRITICAL: CRITICAL: 90.21% of data under the critical threshold [5.0] https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=8&fullscreen&orgId=1 [05:53:21] 10Operations, 10DBA: sql prompts differ between mwmaint1001 and deploy1001 - https://phabricator.wikimedia.org/T199009 (10Marostegui) For some reason they have different prompt options which explains this. deploy1000: ``` prompt \N [\d]> ``` maint1001: ``` prompt... [05:56:49] 10Operations, 10DBA: SQL autocomplete broken on mwmaint1001 - https://phabricator.wikimedia.org/T199010 (10Marostegui) This is because of: T199009#4405061 mwmaint1001 ``` auto-rehash FALSE ``` deploy1001 ``` auto-rehash TRUE ``` [05:58:04] 10Operations, 10DBA: sql config differ between mwmaint1001 and deploy1001 - https://phabricator.wikimedia.org/T199009 (10Marostegui) [05:59:21] 10Operations, 10DBA: sql config differ between mwmaint1001 and deploy1001 - https://phabricator.wikimedia.org/T199009 (10Marostegui) This is also showing different auto-rehashing default options meaning that autocompletion isn't working as specified here: T199010#4405063 [05:59:58] 10Operations, 10DBA: SQL autocomplete broken on mwmaint1001 - https://phabricator.wikimedia.org/T199010 (10Marostegui) [06:00:00] 10Operations, 10DBA: sql config differ between mwmaint1001 and deploy1001 - https://phabricator.wikimedia.org/T199009 (10Marostegui) [06:10:12] 10Operations, 10DBA: sql config differ between mwmaint1001 and deploy1001 - https://phabricator.wikimedia.org/T199009 (10Marostegui) [06:10:14] 10Operations, 10DBA: sql config differ between mwmaint1001 and deploy1001 - https://phabricator.wikimedia.org/T199009 (10Marostegui) And this is also causing T195636 which can be solved by adding a white space at the end: ``` wikiadmin@10.64.32.115(enwiki)>select now(); +---------------------+ | now()... [06:10:52] 10Operations, 10DBA, 10Regression: MySQL prompt missing trailing space on mwmaint1001 - https://phabricator.wikimedia.org/T195636 (10Marostegui) This is because of: T199009#4405088 let's follow up on that task so we can group all the issues there instead of spreading them into different tasks, which can be e... [06:11:05] 10Operations, 10DBA: sql config differ between mwmaint1001 and deploy1001 - https://phabricator.wikimedia.org/T199009 (10Marostegui) [06:11:07] 10Operations, 10DBA, 10Regression: MySQL prompt missing trailing space on mwmaint1001 - https://phabricator.wikimedia.org/T195636 (10Marostegui) [06:13:59] 10Operations, 10DBA: sql config differs between mwmaint1001 and deploy1001 - https://phabricator.wikimedia.org/T199009 (10Marostegui) [06:27:43] (03PS1) 10Marostegui: mariadb: Enable auto-rehash [puppet] - 10https://gerrit.wikimedia.org/r/444332 [06:30:01] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/profile.d/bash_autologout.sh] [06:31:48] (03CR) 10Marostegui: "https://puppet-compiler.wmflabs.org/compiler02/11721/" [puppet] - 10https://gerrit.wikimedia.org/r/444332 (owner: 10Marostegui) [06:31:52] PROBLEM - puppet last run on labvirt1017 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/nova/policy.json],File[/usr/local/bin/apt2xml] [06:32:47] (03PS2) 10Marostegui: mariadb: Enable auto-rehash [puppet] - 10https://gerrit.wikimedia.org/r/444332 (https://phabricator.wikimedia.org/T199009) [06:36:53] RECOVERY - puppet last run on labvirt1017 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:55:21] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:58:04] (03PS1) 10Marostegui: mariadb/config.pp: Add a space at the end [puppet] - 10https://gerrit.wikimedia.org/r/444333 [07:13:13] (03CR) 10Marostegui: "https://puppet-compiler.wmflabs.org/compiler02/11725/" [puppet] - 10https://gerrit.wikimedia.org/r/444333 (owner: 10Marostegui) [07:14:02] (03PS2) 10Marostegui: mariadb/config.pp: Parse final space at prompt [puppet] - 10https://gerrit.wikimedia.org/r/444333 (https://phabricator.wikimedia.org/T199009) [07:29:30] (03CR) 10Elukey: "Does it also need a cleanup in role::restbase::alerts ?" [puppet] - 10https://gerrit.wikimedia.org/r/444247 (https://phabricator.wikimedia.org/T186567) (owner: 10Mobrovac) [07:38:31] PROBLEM - SSH on scb2005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:38:41] PROBLEM - graphoid endpoints health on scb2005 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [07:38:51] PROBLEM - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is CRITICAL: /{domain}/v1/page/references/{title}{/revision}{/tid} (Get references of a test page) timed out before a response was received: /{domain}/v1/page/mobile-sections/{title}{/revision}{/tid} (retrieve test page via mobile-sections) timed out before a response was received: /_info (retrieve service info) timed out before a response was received: /{domain}/v1/p [07:38:51] revision}{/tid} (Get media in test page) timed out before a response was received [07:39:01] PROBLEM - apertium apy on scb2005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:39:12] PROBLEM - eventstreams on scb2005 is CRITICAL: connect to address 10.192.0.34 and port 8092: Connection refused [07:39:31] RECOVERY - SSH on scb2005 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [07:39:41] RECOVERY - graphoid endpoints health on scb2005 is OK: All endpoints are healthy [07:39:52] RECOVERY - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is OK: All endpoints are healthy [07:39:52] RECOVERY - apertium apy on scb2005 is OK: HTTP OK: HTTP/1.1 200 OK - 5996 bytes in 0.074 second response time [07:40:22] RECOVERY - eventstreams on scb2005 is OK: HTTP OK: HTTP/1.1 200 OK - 1066 bytes in 0.104 second response time [07:41:41] PROBLEM - puppet last run on scb2005 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 5 minutes ago with 18 failures. Failed resources (up to 3 shown): Exec[ip addr add 2620:0:860:101:10:192:0:34/64 dev eth0],Exec[absent_ensure_members],Exec[ops_ensure_members],Exec[wikidev_ensure_members] [08:07:11] RECOVERY - puppet last run on scb2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:37:20] (03PS8) 1020after4: phabricator: refactor preamble.php to separate unrelated functionality. [puppet] - 10https://gerrit.wikimedia.org/r/443665 [08:38:50] (03CR) 1020after4: phabricator: refactor preamble.php to separate unrelated functionality. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/443665 (owner: 1020after4) [09:51:53] hello! [11:17:44] (03CR) 10Reedy: [C: 031] mariadb: Enable auto-rehash [puppet] - 10https://gerrit.wikimedia.org/r/444332 (https://phabricator.wikimedia.org/T199009) (owner: 10Marostegui) [11:18:02] (03CR) 10Reedy: [C: 031] mariadb/config.pp: Parse final space at prompt [puppet] - 10https://gerrit.wikimedia.org/r/444333 (https://phabricator.wikimedia.org/T199009) (owner: 10Marostegui) [11:27:41] PROBLEM - Host lvs2001 is DOWN: PING CRITICAL - Packet loss = 100% [12:43:32] PROBLEM - puppet last run on labcontrol1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/gen_fingerprints] [12:48:33] RECOVERY - puppet last run on labcontrol1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:02:25] <_joe_> !log rebooting lvs2001, panic logs from the ethernet card module [13:02:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:04:44] <_joe_> uhm [13:07:41] RECOVERY - Host lvs2001 is UP: PING WARNING - Packet loss = 54%, RTA = 36.44 ms [14:50:27] PROBLEM - MegaRAID on db1059 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) [14:50:28] ACKNOWLEDGEMENT - MegaRAID on db1059 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T199019 [14:50:36] 10Operations, 10ops-eqiad: Degraded RAID on db1059 - https://phabricator.wikimedia.org/T199019 (10ops-monitoring-bot) [14:55:29] 10Operations, 10ops-eqiad: Degraded RAID on db1059 - https://phabricator.wikimedia.org/T199019 (10Marostegui) 05Open>03declined Nothing to do here, this host is scheduled for decommission T196606 [19:09:56] (03PS1) 10Tacsipacsi: Enable TemplateStyles on huwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444378 (https://phabricator.wikimedia.org/T198725) [19:13:24] (03PS2) 10Tacsipacsi: Enable TemplateStyles on huwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444378 (https://phabricator.wikimedia.org/T198725) [20:54:07] PROBLEM - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is alerting: 70% GET drop in 30min alert. [20:55:07] RECOVERY - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is not alerting. [21:51:37] PROBLEM - nutcracker process on scb2004 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [21:51:47] PROBLEM - cpjobqueue endpoints health on scb2004 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [21:51:48] PROBLEM - pdfrender on scb2004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:48] PROBLEM - Check size of conntrack table on scb2004 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [21:51:58] PROBLEM - SSH on scb2004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:27] PROBLEM - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is CRITICAL: /{domain}/v1/page/summary/{title}{/revision}{/tid} (Get summary for test page) timed out before a response was received: /{domain}/v1/page/most-read/{year}/{month}/{day} (retrieve most-read articles for date with no data (with aggregated=true)) timed out before a response was received: /{domain}/v1/page/metadata/{title}{/revision}{/tid} (retrieve extended [21:52:27] article on English Wikipedia) timed out before a response was received: /{domain}/v1/page/news (get In the News content) timed out before a response was received [21:52:27] PROBLEM - apertium apy on scb2004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:37] PROBLEM - eventstreams on scb2004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:53:07] PROBLEM - recommendation_api endpoints health on scb2004 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [21:53:08] PROBLEM - citoid endpoints health on scb2004 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [21:53:28] RECOVERY - eventstreams on scb2004 is OK: HTTP OK: HTTP/1.1 200 OK - 1066 bytes in 0.163 second response time [21:53:47] RECOVERY - nutcracker process on scb2004 is OK: PROCS OK: 1 process with UID = 113 (nutcracker), command name nutcracker [21:53:48] RECOVERY - cpjobqueue endpoints health on scb2004 is OK: All endpoints are healthy [21:53:57] RECOVERY - pdfrender on scb2004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.074 second response time [21:53:57] RECOVERY - Check size of conntrack table on scb2004 is OK: OK: nf_conntrack is 9 % full [21:54:07] RECOVERY - recommendation_api endpoints health on scb2004 is OK: All endpoints are healthy [21:54:07] RECOVERY - SSH on scb2004 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [21:54:08] RECOVERY - citoid endpoints health on scb2004 is OK: All endpoints are healthy [21:54:28] RECOVERY - apertium apy on scb2004 is OK: HTTP OK: HTTP/1.1 200 OK - 5996 bytes in 0.074 second response time [21:54:28] RECOVERY - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is OK: All endpoints are healthy [23:08:09] (03PS2) 10Krinkle: Set $wgMediaViewer* vars directly (1/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443880 [23:08:36] (03PS2) 10Krinkle: Set $wgMediaViewer* vars directly (2/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443881 [23:08:46] (03PS2) 10Krinkle: Set $wgMediaViewer* vars directly (3/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443882 [23:08:51] (03CR) 10Krinkle: [C: 032] Set $wgMediaViewer* vars directly (1/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443880 (owner: 10Krinkle) [23:12:55] (03Merged) 10jenkins-bot: Set $wgMediaViewer* vars directly (1/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443880 (owner: 10Krinkle) [23:13:12] (03CR) 10jenkins-bot: Set $wgMediaViewer* vars directly (1/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443880 (owner: 10Krinkle) [23:13:44] * Krinkle staging on deploy1001/mwdebug1002 [23:16:46] (03CR) 10Krinkle: [C: 032] Set $wgMediaViewer* vars directly (2/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443881 (owner: 10Krinkle) [23:17:59] (03Merged) 10jenkins-bot: Set $wgMediaViewer* vars directly (2/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443881 (owner: 10Krinkle) [23:18:11] (03CR) 10jenkins-bot: Set $wgMediaViewer* vars directly (2/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443881 (owner: 10Krinkle) [23:18:14] !log krinkle@deploy1001 Synchronized wmf-config/CommonSettings.php: I1b42c260b5ee3 (duration: 01m 00s) [23:18:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:21:50] (03CR) 10Krinkle: [C: 032] Set $wgMediaViewer* vars directly (3/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443882 (owner: 10Krinkle) [23:23:28] (03Merged) 10jenkins-bot: Set $wgMediaViewer* vars directly (3/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443882 (owner: 10Krinkle) [23:23:40] (03CR) 10jenkins-bot: Set $wgMediaViewer* vars directly (3/3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443882 (owner: 10Krinkle) [23:23:42] !log krinkle@deploy1001 Synchronized wmf-config/InitialiseSettings.php: I001f68a1b39 (duration: 00m 53s) [23:23:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:25:50] !log krinkle@deploy1001 Synchronized wmf-config/CommonSettings.php: Icbbccf495899 (duration: 00m 50s) [23:25:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log