[03:29:41] <icinga-wm>	 PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 935.48 seconds
[03:35:21] <icinga-wm>	 PROBLEM - puppet last run on cp4028 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIP2-City.mmdb.gz],File[/usr/share/GeoIP/GeoIP2-City.mmdb.test]
[04:01:40] <icinga-wm>	 RECOVERY - puppet last run on cp4028 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[04:11:11] <icinga-wm>	 RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 12.81 seconds
[06:29:30] <icinga-wm>	 PROBLEM - puppet last run on mw1319 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/php/7.0/fpm/php.ini]
[06:55:41] <icinga-wm>	 RECOVERY - puppet last run on mw1319 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[09:31:00] <icinga-wm>	 PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1968 bytes in 0.097 second response time
[09:36:10] <icinga-wm>	 RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1948 bytes in 0.091 second response time
[10:10:48] <wikibugs_>	 (03PS2) 10Giuseppe Lavagetto: profile::mediawiki::mcrouter_wancache: add ssl, proxy support [puppet] - 10https://gerrit.wikimedia.org/r/431737 (https://phabricator.wikimedia.org/T192370)
[10:10:50] <wikibugs_>	 (03PS2) 10Giuseppe Lavagetto: puppet_ecdsacert: allow IP-based SANs [puppet] - 10https://gerrit.wikimedia.org/r/431738 (https://phabricator.wikimedia.org/T192370)
[10:11:31] <wikibugs_>	 (03CR) 10jerkins-bot: [V: 04-1] profile::mediawiki::mcrouter_wancache: add ssl, proxy support [puppet] - 10https://gerrit.wikimedia.org/r/431737 (https://phabricator.wikimedia.org/T192370) (owner: 10Giuseppe Lavagetto)
[14:09:50] <icinga-wm>	 PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1975 bytes in 0.182 second response time
[14:15:10] <icinga-wm>	 RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1958 bytes in 0.117 second response time
[14:45:51] <wikibugs_>	 10Operations, 10Wikimedia-Mailing-lists: Mass subscription attempt to a mailing list from same domain (aol and yahoo) - https://phabricator.wikimedia.org/T194597#4203034 (10Delphine_wmf)
[17:03:49] <wikibugs_>	 10Operations, 10Wikimedia-Mailing-lists: Mass subscription attempt to a mailing list from same domain (aol and yahoo) - https://phabricator.wikimedia.org/T194597#4203107 (10Aklapper) This is being discussed on the //List admins// mailing list and in non-public T194032.
[18:05:40] <icinga-wm>	 PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1972 bytes in 0.093 second response time
[18:10:50] <icinga-wm>	 RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1943 bytes in 0.110 second response time
[18:33:00] <wikibugs_>	 (03PS1) 10Hoo man: Wikidata: Always have 4 change dispatchers running [puppet] - 10https://gerrit.wikimedia.org/r/432778 (https://phabricator.wikimedia.org/T194602)
[19:03:00] <icinga-wm>	 PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1971 bytes in 0.103 second response time
[19:24:00] <icinga-wm>	 RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1973 bytes in 0.107 second response time
[20:09:52] <bawolff>	 !log Deployed patch for T194605
[20:09:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:10:17] <bawolff>	 !log Reset some botpasswords associated with  T194204
[21:10:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[22:46:40] <wikibugs_>	 10Operations, 10Wikimedia-Mailing-lists: Subscription request spam on mailing list (wikimedia-us-ia-board) - https://phabricator.wikimedia.org/T194612#4203377 (10Lixxx235)
[23:11:53] <wikibugs_>	 10Operations, 10Wikimedia-Mailing-lists: Subscription request spam on mailing list (wikimedia-us-ia-board) - https://phabricator.wikimedia.org/T194612#4203407 (10Aklapper)
[23:11:57] <wikibugs_>	 10Operations, 10Wikimedia-Mailing-lists: Mass subscription attempt to a mailing list from same domain (aol and yahoo) - https://phabricator.wikimedia.org/T194597#4203410 (10Aklapper)
[23:28:06] <wikibugs_>	 10Operations, 10Wikimedia-Mailing-lists: Mass subscription attempt to a mailing list from same domain (aol and yahoo) - https://phabricator.wikimedia.org/T194597#4203414 (10Delphine_wmf)