[00:03:59] !log running sync-common on mw1191 [00:15:41] PROBLEM - HTTPS on antimony is CRITICAL: SSL_CERT CRITICAL svn.wikimedia.org: certificate will expire on Jan 31 10:53:05 2015 GMT [00:56:06] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 1 failures [01:08:39] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [02:49:41] PROBLEM - HTTPS on antimony is CRITICAL: SSL_CERT CRITICAL svn.wikimedia.org: certificate will expire on Jan 31 10:53:05 2015 GMT [02:51:51] pff [02:52:33] Does svn.wikimedia.org serve anything private? [03:17:21] PROBLEM - HTTPS on antimony is CRITICAL: SSL_CERT CRITICAL svn.wikimedia.org: certificate will expire on Jan 31 10:53:05 2015 GMT [03:40:47] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: Puppet has 1 failures [03:41:22] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: Puppet has 1 failures [03:41:37] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Puppet has 1 failures [03:45:20] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: Puppet has 1 failures [03:51:30] RECOVERY - puppet last run on mw1248 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:53:43] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:54:35] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:58:05] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: Puppet has 1 failures [03:58:14] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:01:35] PROBLEM - HTTPS on antimony is CRITICAL: SSL_CERT CRITICAL svn.wikimedia.org: certificate will expire on Jan 31 10:53:05 2015 GMT [04:13:28] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:31:54] (03PS6) 10BryanDavis: logstash: Parse apache syslog messages [puppet] - 10https://gerrit.wikimedia.org/r/179480 [04:33:00] (03CR) 10BryanDavis: "Use "message_prefix" as suggested by Ori." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/179480 (owner: 10BryanDavis) [04:39:47] PROBLEM - HTTPS_m.wiktionary.org on amssq55 is CRITICAL: SSL_CERT CRITICAL: Error: [04:40:07] PROBLEM - HTTPS_m.wikibooks.org on cp3013 is CRITICAL: SSL_CERT CRITICAL: Error: verify depth is 6 [04:40:07] PROBLEM - HTTPS_m.mediawiki.org on cp3020 is CRITICAL: SSL_CERT CRITICAL: Error: [04:40:07] PROBLEM - HTTPS_m.wikidata.org on amssq43 is CRITICAL: SSL_CERT CRITICAL: Error: [04:40:07] PROBLEM - HTTPS_mediawiki.org on amssq50 is CRITICAL: SSL_CERT CRITICAL: Error: [04:40:07] PROBLEM - HTTPS_wikivoyage.org on cp3010 is CRITICAL: SSL_CERT CRITICAL: Error: [04:40:07] PROBLEM - HTTPS_wiktionary.org on cp4015 is CRITICAL: SSL_CERT CRITICAL: Error: [04:43:43] RECOVERY - HTTPS_m.wikipedia.org on cp4007 is OK: SSL_CERT OK - X.509 certificate for *.m.wikipedia.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:16:02 2015 GMT (expires in 337 days) [04:43:44] RECOVERY - HTTPS_mediawiki.org on cp4004 is OK: SSL_CERT OK - X.509 certificate for *.mediawiki.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:51:02 2015 GMT (expires in 337 days) [04:43:44] RECOVERY - HTTPS_wikibooks.org on cp1056 is OK: SSL_CERT OK - X.509 certificate for *.wikibooks.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:21:03 2015 GMT (expires in 337 days) [04:43:44] RECOVERY - HTTPS_wikivoyage.org on amssq46 is OK: SSL_CERT OK - X.509 certificate for *.wikivoyage.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:41:09 2015 GMT (expires in 337 days) [04:43:44] RECOVERY - HTTPS_m.wiktionary.org on cp4012 is OK: SSL_CERT OK - X.509 certificate for *.m.wiktionary.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:46:07 2015 GMT (expires in 337 days) [04:43:46] RECOVERY - HTTPS_mediawiki.org on amssq50 is OK: SSL_CERT OK - X.509 certificate for *.mediawiki.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:51:02 2015 GMT (expires in 337 days) [04:44:14] PROBLEM - HTTPS on antimony is CRITICAL: SSL_CERT CRITICAL svn.wikimedia.org: certificate will expire on Jan 31 10:53:05 2015 GMT [04:47:25] RECOVERY - HTTPS_m.wiktionary.org on cp3006 is OK: SSL_CERT OK - X.509 certificate for *.m.wiktionary.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:46:07 2015 GMT (expires in 337 days) [04:47:28] RECOVERY - HTTPS_m.mediawiki.org on cp3020 is OK: SSL_CERT OK - X.509 certificate for *.m.mediawiki.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:51:04 2015 GMT (expires in 337 days) [04:47:30] RECOVERY - HTTPS_wiktionary.org on cp4015 is OK: SSL_CERT OK - X.509 certificate for *.wiktionary.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:46:05 2015 GMT (expires in 337 days) [04:47:43] RECOVERY - HTTPS_m.wiktionary.org on amssq55 is OK: SSL_CERT OK - X.509 certificate for *.m.wiktionary.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:46:07 2015 GMT (expires in 337 days) [04:47:43] RECOVERY - HTTPS_m.wikibooks.org on cp3013 is OK: SSL_CERT OK - X.509 certificate for *.m.wikibooks.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:21:05 2015 GMT (expires in 337 days) [04:47:43] RECOVERY - HTTPS_wikiquote.org on cp4006 is OK: SSL_CERT OK - X.509 certificate for *.wikiquote.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:36:05 2015 GMT (expires in 337 days) [04:47:43] RECOVERY - HTTPS_m.wikidata.org on amssq43 is OK: SSL_CERT OK - X.509 certificate for *.m.wikidata.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:26:04 2015 GMT (expires in 337 days) [04:47:43] RECOVERY - HTTPS_wikivoyage.org on cp3010 is OK: SSL_CERT OK - X.509 certificate for *.wikivoyage.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:41:09 2015 GMT (expires in 337 days) [04:47:44] RECOVERY - HTTPS_wikiversity.org on cp4014 is OK: SSL_CERT OK - X.509 certificate for *.wikiversity.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:41:04 2015 GMT (expires in 337 days) [04:47:44] RECOVERY - HTTPS_wikibooks.org on cp1061 is OK: SSL_CERT OK - X.509 certificate for *.wikibooks.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:21:03 2015 GMT (expires in 337 days) [04:47:45] RECOVERY - HTTPS_mediawiki.org on cp4014 is OK: SSL_CERT OK - X.509 certificate for *.mediawiki.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:51:02 2015 GMT (expires in 337 days) [04:47:45] RECOVERY - HTTPS_unified on cp1049 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:06:02 2015 GMT (expires in 337 days) [04:47:46] RECOVERY - HTTPS_wikipedia.org on amssq50 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:06:02 2015 GMT (expires in 337 days) [04:47:46] RECOVERY - HTTPS_mediawiki.org on amssq41 is OK: SSL_CERT OK - X.509 certificate for *.mediawiki.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:51:02 2015 GMT (expires in 337 days) [04:47:47] RECOVERY - HTTPS_wikidata.org on amssq55 is OK: SSL_CERT OK - X.509 certificate for *.wikidata.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:26:02 2015 GMT (expires in 337 days) [04:47:47] RECOVERY - HTTPS_zero.wikipedia.org on cp3021 is OK: SSL_CERT OK - X.509 certificate for *.zero.wikipedia.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:16:05 2015 GMT (expires in 337 days) [04:47:48] RECOVERY - HTTPS_wiktionary.org on amssq51 is OK: SSL_CERT OK - X.509 certificate for *.wiktionary.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:46:05 2015 GMT (expires in 337 days) [04:47:48] RECOVERY - HTTPS_wikidata.org on cp4004 is OK: SSL_CERT OK - X.509 certificate for *.wikidata.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:26:02 2015 GMT (expires in 337 days) [04:47:49] RECOVERY - HTTPS_m.wikivoyage.org on cp1049 is OK: SSL_CERT OK - X.509 certificate for *.m.wikivoyage.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:46:02 2015 GMT (expires in 337 days) [04:47:49] RECOVERY - HTTPS_m.mediawiki.org on cp4020 is OK: SSL_CERT OK - X.509 certificate for *.m.mediawiki.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:51:04 2015 GMT (expires in 337 days) [04:47:50] RECOVERY - HTTPS_wikibooks.org on amssq59 is OK: SSL_CERT OK - X.509 certificate for *.wikibooks.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:21:03 2015 GMT (expires in 337 days) [04:47:50] RECOVERY - HTTPS_unified on amssq53 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from GlobalSign Organization Validation CA - SHA256 - G2 valid until Nov 22 18:06:02 2015 GMT (expires in 337 days) [06:34:46] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:03] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:27] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:49] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:08] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:25] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:41] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:56] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:38:02] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:38:12] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:38:13] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [06:39:04] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:38] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:45:52] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:46:17] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:46:19] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:46:51] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:10] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:47:14] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:55] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:48:34] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:38] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:49:14] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:49:50] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:59:09] (03PS1) 10Gergő Tisza: Add -labs settings for Score [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181358 [08:13:56] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: puppet fail [08:32:40] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [10:12:28] !log springle Synchronized wmf-config/db-eqiad.php: raise db1072 load (duration: 00m 06s) [11:53:24] looks like morebots has been gone for more than a day [14:12:47] !log Cleaning the mess Zuul code base is https://phabricator.wikimedia.org/T84917 . Updated master/labs branches and tagged it wmf-deploy-20141221-1 [14:12:58] hashar: there is no morebots [14:13:04] what the fuck [14:14:39] it's holidaying [14:16:17] morebots: hi [14:16:17] I am a logbot running on tools-exec-14. [14:16:17] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [14:16:17] To log a message, type !log . [14:16:21] !log Cleaning the mess Zuul code base is https://phabricator.wikimedia.org/T84917 . Updated master/labs branches and tagged it wmf-deploy-20141221-1 [14:16:27] Logged the message, Master [14:16:46] !log Restarted moretbots on tools labs following some doc at https://wikitech.wikimedia.org/wiki/Morebots [14:16:49] Logged the message, Master [14:16:51] Nemo_bis: MatmaRex it is back \O/ [14:17:54] people were doing some !log-ging yesterday that was probably missed [14:18:24] !log Upgrading Zuul merger and server on gallium wmf-deploy-20141030-1 to wmf-deploy-20141221-1 [14:18:26] Logged the message, Master [15:39:28] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: puppet fail [15:54:34] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:39:29] James_F|Away: about by any chance ? [19:31:46] (03PS1) 10Giuseppe Lavagetto: admin: revoke ori's key temporarily [puppet] - 10https://gerrit.wikimedia.org/r/181375 [19:32:47] (03CR) 10Giuseppe Lavagetto: [C: 032] admin: revoke ori's key temporarily [puppet] - 10https://gerrit.wikimedia.org/r/181375 (owner: 10Giuseppe Lavagetto)