From anandb at ripe.net Mon Jun 11 11:51:54 2012 From: anandb at ripe.net (Anand Buddhdev) Date: Mon, 11 Jun 2012 11:51:54 +0200 Subject: [dns-wg] Graphs of DNSKEY queries at K-root Message-ID: <4FD5BFBA.7040502@ripe.net> Dear colleagues, During the DNS Working Group session at RIPE 64, Olaf Kolkman asked the RIPE NCC to provide graphs of DNSKEY queries arriving at K-root. These graphs, especially the yearly one, are a good indicator about the level of DNSSEC validation happening for the root zone. We have enabled automatic generation of these graphs. They are available at: http://k.root-servers.org/statistics/ROOT/dnskey_queries.html If you have any questions, please email . Regards, Anand Buddhdev RIPE NCC From robachevsky at isoc.org Tue Jun 12 15:58:33 2012 From: robachevsky at isoc.org (Andrei Robachevsky) Date: Tue, 12 Jun 2012 15:58:33 +0200 Subject: [dns-wg] Graphs of DNSKEY queries at K-root In-Reply-To: <4FD5BFBA.7040502@ripe.net> References: <4FD5BFBA.7040502@ripe.net> Message-ID: <4FD74B09.5000905@isoc.org> Hi Anand, Thank you for publishing this. Assuming these are all valid queries (i.e. not belonging to the 98% of malformed queries root servers usually get), what fraction of the total valid queries does this constitute? Would the actual DNSSEC penetration rate be different from this number (e.g. due to possible differences in caching, etc.)? Thanks, Andrei Anand Buddhdev wrote on 11/06/2012 11:51: > Dear colleagues, > > During the DNS Working Group session at RIPE 64, Olaf Kolkman asked the > RIPE NCC to provide graphs of DNSKEY queries arriving at K-root. These > graphs, especially the yearly one, are a good indicator about the level > of DNSSEC validation happening for the root zone. > > We have enabled automatic generation of these graphs. They are available at: > http://k.root-servers.org/statistics/ROOT/dnskey_queries.html > > If you have any questions, please email . > > Regards, > > Anand Buddhdev > RIPE NCC > -- Andrei Robachevsky Technology Program Manager Internet Society www.isoc.org From no-reply at ripe.net Thu Jun 14 13:26:27 2012 From: no-reply at ripe.net (Romeo Zwart) Date: Thu, 14 Jun 2012 13:26:27 +0200 Subject: [dns-wg] Update 14 June (11:22 UTC): Reverse DNS Services Outage Message-ID: <4FD9CA63.5060500@ripe.net> Dear colleagues, We'd like to update you on the reverse DNS service outage situation. As of 10:45 UTC on Thursday, 14 June 2012, all reverse DNS zones we publish and that are not on the list below are correct and reflect the state of 13:30 UTC yesterday, 13 June. If you continue to see problems after this time, please contactdns-help at ripe.net with details of what responses you are getting. Please consider (negative) caching TTLs and restart your caching servers if possible. We've posted this update online at: https://www.ripe.net/internet-coordination/news/announcements/update-14-june-11-22-utc-reverse-dns-services-outage =============== Outage Details =============== We know that some zones not mentioned in the list of the previous update (20:45 UTC, 13 June) were not correctly propagated to some authoritative servers after the roll-back last night. We do not know the extent of this at the present time. However we are sure that as of 10:45 UTC today all authoritative servers answer correctly with the state of 13:30 UTC 13 June for all zones but the few on the list in the previous update. We are working hard to restore the zones listed. We are busy adding all the updates since 13:30 UTC 13 June to all zones. We will report on how this work is progressing. The zones known to be still affected are: 0.4.1.0.0.2.ip6.arpa 185.in-addr.arpa 4.1.1.0.0.2.ip6.arpa 5.1.0.0.2.ip6.arpa 6.1.1.0.0.2.ip6.arpa 7.0.1.0.0.2.ip6.arpa 7.1.1.0.0.2.ip6.arpa 7.4.1.0.0.2.ip6.arpa 8.0.1.0.0.2.ip6.arpa a.0.1.0.0.2.ip6.arpa a.1.1.0.0.2.ip6.arpa a.4.1.0.0.2.ip6.arpa b.0.1.0.0.2.ip6.arpa b.1.1.0.0.2.ip6.arpa b.4.1.0.0.2.ip6.arpa We will publish a full report once we've restored all the zones and have had sufficient time to fully analyse what went wrong. We have been working hard over the past 18 hours to resolve this problem and we apologise for the considerable inconvenience. Kind regards, Romeo Zwart GII Services Manager RIPE NCC From romeo.zwart at ripe.net Thu Jun 14 13:42:00 2012 From: romeo.zwart at ripe.net (Romeo Zwart) Date: Thu, 14 Jun 2012 13:42:00 +0200 Subject: [dns-wg] [staff] [ncc-announce] Update 14 June (11:22 UTC): Reverse DNS Services Outage In-Reply-To: <4FD9CA63.5060500@ripe.net> References: <4FD9CA63.5060500@ripe.net> Message-ID: <4FD9CE08.4000000@ripe.net> Please note that the contact email address mentioned was incorrect. The correct email address is dns-help at ripe.net. Apologies for the cross posting. Best regards, Romeo On 12/06/14 13:26 , Romeo Zwart wrote: > Dear colleagues, > > We'd like to update you on the reverse DNS service outage situation. > > As of 10:45 UTC on Thursday, 14 June 2012, all reverse DNS zones we > publish and that are not on the list below are correct and reflect the > state of 13:30 UTC yesterday, 13 June. If you continue to see problems > after this time, please contactdns-help at ripe.net with details of what > responses you are getting. Please consider (negative) caching TTLs and > restart your caching servers if possible. > > We've posted this update online at: > https://www.ripe.net/internet-coordination/news/announcements/update-14-june-11-22-utc-reverse-dns-services-outage > > > =============== > Outage Details > =============== > > We know that some zones not mentioned in the list of the previous update > (20:45 UTC, 13 June) were not correctly propagated to some authoritative > servers after the roll-back last night. We do not know the extent of > this at the present time. However we are sure that as of 10:45 UTC today > all authoritative servers answer correctly with the state of 13:30 UTC > 13 June for all zones but the few on the list in the previous update. > > We are working hard to restore the zones listed. We are busy adding all > the updates since 13:30 UTC 13 June to all zones. We will report on how > this work is progressing. > > The zones known to be still affected are: > > 0.4.1.0.0.2.ip6.arpa > 185.in-addr.arpa > 4.1.1.0.0.2.ip6.arpa > 5.1.0.0.2.ip6.arpa > 6.1.1.0.0.2.ip6.arpa > 7.0.1.0.0.2.ip6.arpa > 7.1.1.0.0.2.ip6.arpa > 7.4.1.0.0.2.ip6.arpa > 8.0.1.0.0.2.ip6.arpa > a.0.1.0.0.2.ip6.arpa > a.1.1.0.0.2.ip6.arpa > a.4.1.0.0.2.ip6.arpa > b.0.1.0.0.2.ip6.arpa > b.1.1.0.0.2.ip6.arpa > b.4.1.0.0.2.ip6.arpa > > We will publish a full report once we've restored all the zones and have > had sufficient time to fully analyse what went wrong. > > We have been working hard over the past 18 hours to resolve this problem > and we apologise for the considerable inconvenience. > > Kind regards, > > Romeo Zwart > GII Services Manager > RIPE NCC > > > > From no-reply at ripe.net Thu Jun 14 15:35:17 2012 From: no-reply at ripe.net (Romeo Zwart) Date: Thu, 14 Jun 2012 15:35:17 +0200 Subject: [dns-wg] Update 4: Reverse DNS Services Outage Message-ID: <4FD9E895.5080400@ripe.net> Dear colleagues, We'd like to update you on the reverse DNS outage situation: As of 13:00 UTC on Thursday, 14 June 2012 the zones listed in the previous update have been restored to the state of 13:30 UTC, Wednesday, 13 June 2012. So all zones we maintain are now available and zones in the list are no longer special. There may be a very few ERX zones maintained by other RIRs that are still not fully restored. The update has been posted online at: https://www.ripe.net/lir-services/service-announcements/reverse-dns-services-issues All updates since 13:30 UTC yesterday are still pending and are not reflected in the zones. However, no updates were lost. Any updates you will submit or you have already submitted will eventually be reflected in the zones once we complete our restoration procedure. ============= Next actions ============= We will now create up-to-date zones that include all the updates since yesterday 13:30 UTC. We will also recreate all ERX zones with current data. We expect to complete this by this evening. Once that is complete, we will re-enable regular provisioning and everything will be back to normal. If you experience any abnormal responses beyond the residual inconsistencies described here, please let us know atdns-help at ripe.net with as much detail as possible. Kind regards, Romeo Zwart GII Services Manager RIPE NCC -------------- next part -------------- An HTML attachment was scrubbed... URL: From no-reply at ripe.net Thu Jun 14 23:42:39 2012 From: no-reply at ripe.net (Romeo Zwart) Date: Thu, 14 Jun 2012 23:42:39 +0200 Subject: [dns-wg] Update: Reverse DNS Outage (as of 20:00 UTC, 14 June) Message-ID: <4FDA5ACF.1050803@ripe.net> Dear colleagues, As of 20:00 UTC, Thursday 14 June 2012, all regular zone information is again complete and up-to-date. However, we are still processing some imported ERX zones from the APNIC region. If you are still experiencing any abnormal responses, other than related to the above mentioned ERX zone, you can contact us atdns-help at ripe.net. Provide as much detail as possible. This update is available online at: http://www.ripe.net/lir-services/service-announcements/reverse-dns-services-issues Kind regards, Romeo Zwart GII Services Manager RIPE NCC From dfleuriot at hi-media.com Fri Jun 15 00:13:46 2012 From: dfleuriot at hi-media.com (Damien FLEURIOT) Date: Fri, 15 Jun 2012 00:13:46 +0200 Subject: [dns-wg] [ncc-announce] Update: Reverse DNS Outage (as of 20:00 UTC, 14 June) In-Reply-To: <4FDA5ACF.1050803@ripe.net> References: <4FDA5ACF.1050803@ripe.net> Message-ID: <4FDA621A.1070305@hi-media.com> Hi Ripe staff, FYI there's a typo again in the email address. Thanks for the updates and your work on the issue. On 6/14/12 11:42 PM, Romeo Zwart wrote: > Dear colleagues, > > As of 20:00 UTC, Thursday 14 June 2012, all regular zone information > is again complete and up-to-date. However, we are still processing > some imported ERX zones from the APNIC region. > > If you are still experiencing any abnormal responses, other than > related to the above mentioned ERX zone, you can contact us > atdns-help at ripe.net. Provide as much detail as possible. > > This update is available online at: > http://www.ripe.net/lir-services/service-announcements/reverse-dns-services-issues > > > Kind regards, > > Romeo Zwart > GII Services Manager > RIPE NCC > > -------------- next part -------------- A non-text attachment was scrubbed... Name: dfleuriot.vcf Type: text/x-vcard Size: 321 bytes Desc: not available URL: From no-reply at ripe.net Fri Jun 15 16:19:06 2012 From: no-reply at ripe.net (Serge Radovcic) Date: Fri, 15 Jun 2012 16:19:06 +0200 Subject: [dns-wg] Update on Reverse DNS Outage - Resolved Message-ID: <4FDB445A.80307@ripe.net> Dear colleagues, As of Friday, 15 June 2012 at 07:30 UTC, reverse DNS operations are fully back to normal. The service update has been posted at: http://www.ripe.net/lir-services/service-announcements/reverse-dns-services-issues However, if you are still experiencing abnormal reverse DNS responses, please email us at . ======================== Summary Report and Incident Analysis ======================== We are analysing the incident and preparing a summary report on the facts, cause and effects of the outage. We will send this report out early next week. We will follow up with a detailed incident analysis. ============== Service Updates ============== All RIPE NCC-related service announcements will be posted on and published at: http://www.ripe.net/lir-services/service-announcements We will also alert our Twitter and Facebook followers that an update has been posted. Social media updates are posted and maintained during regular business hours. Please note that the RIPE NCC does not use the mailing list to post service announcements or updates. =================== Reporting Procedures Review =================== We strive to provide you with accurate updates on outages. We understand that for incidents of this nature, timely and regular updates to our stakeholders is of the utmost importance. We are analysing feedback from the incident and using it to improve our communication procedures. Kind regards, Serge Radovcic Chief Communications Officer RIPE NCC From matthaeus.wander at uni-due.de Mon Jun 18 15:30:24 2012 From: matthaeus.wander at uni-due.de (=?ISO-8859-1?Q?Matth=E4us_Wander?=) Date: Mon, 18 Jun 2012 15:30:24 +0200 Subject: [dns-wg] Graphs of DNSKEY queries at K-root In-Reply-To: <4FD74B09.5000905@isoc.org> References: <4FD5BFBA.7040502@ripe.net> <4FD74B09.5000905@isoc.org> Message-ID: <4FDF2D70.2050702@uni-due.de> Hi, Am 12.06.2012 15:58, schrieb Andrei Robachevsky: > Assuming these are all valid queries (i.e. not belonging to the 98% of > malformed > queries root servers usually get), what fraction of the total valid > queries does this > constitute? > > Would the actual DNSSEC penetration rate be different from this number > (e.g. due to possible differences in caching, etc.)? A validating resolver should query the root DNSKEY about once per day (TTL/2) and a non-validating resolver not at all. With 1 q/s this would make an estimate of at most 86k validating resolvers for K, minus extra or malformed queries. The fraction of malformed queries is probably not that large as validation seems to be disabled by default on most systems (one must willfully enable validation without noticing that resolution is broken). This number is a nice validation indicator but does not say anything about the actual number of DNSSEC-enabled queries. The number of queries with the DNSSEC OK flag set [1] is neither suitable, as it indicates all DNSSEC-capable resolvers, not just the DNSSEC-enabled ones. Kind regards, Matt [1] http://k.root-servers.org/statistics/ROOT/dnssec.html From pk at DENIC.DE Mon Jun 18 17:19:25 2012 From: pk at DENIC.DE (Peter Koch) Date: Mon, 18 Jun 2012 17:19:25 +0200 Subject: [dns-wg] Graphs of DNSKEY queries at K-root In-Reply-To: <4FD5BFBA.7040502@ripe.net> References: <4FD5BFBA.7040502@ripe.net> Message-ID: <20120618151925.GA11135@x27.adm.denic.de> Anand, On Mon, Jun 11, 2012 at 11:51:54AM +0200, Anand Buddhdev wrote: > We have enabled automatic generation of these graphs. They are available at: > http://k.root-servers.org/statistics/ROOT/dnskey_queries.html this is a good start, but since the graph looks like a DSC excerpt, can we assume that these are queries for any QNAME or only QNAME="."? Also, to get some intel about validation deployment, would it be possible to factor in the number of sources, as well? Thanks, Peter From robachevsky at isoc.org Tue Jun 19 17:22:44 2012 From: robachevsky at isoc.org (Andrei Robachevsky) Date: Tue, 19 Jun 2012 17:22:44 +0200 Subject: [dns-wg] Graphs of DNSKEY queries at K-root In-Reply-To: <4FDF2D70.2050702@uni-due.de> References: <4FD5BFBA.7040502@ripe.net> <4FD74B09.5000905@isoc.org> <4FDF2D70.2050702@uni-due.de> Message-ID: <4FE09944.3040803@isoc.org> Matth?us Wander wrote on 18/06/2012 15:30: > Hi, > > Am 12.06.2012 15:58, schrieb Andrei Robachevsky: >> Assuming these are all valid queries (i.e. not belonging to the 98% of >> malformed >> queries root servers usually get), what fraction of the total valid >> queries does this >> constitute? >> >> Would the actual DNSSEC penetration rate be different from this number >> (e.g. due to possible differences in caching, etc.)? > > A validating resolver should query the root DNSKEY about once per day > (TTL/2) and a non-validating resolver not at all. With 1 q/s this would > make an estimate of at most 86k validating resolvers for K, minus extra > or malformed queries. The fraction of malformed queries is probably not > that large as validation seems to be disabled by default on most systems > (one must willfully enable validation without noticing that resolution > is broken). > > This number is a nice validation indicator but does not say anything > about the actual number of DNSSEC-enabled queries. The number of queries > with the DNSSEC OK flag set [1] is neither suitable, as it indicates all > DNSSEC-capable resolvers, not just the DNSSEC-enabled ones. > Right. There was an interesting paper at SATIN 2011 (http://conferences.npl.co.uk/satin/papers/satin2011-Gudmundsson.pdf) by ?lafur Gudmundsson and Steve Crocker, outlining a methodology for determining dnssec deployment, if RIPE NCC have interest and resources for more data mining. Andrei From daniel.karrenberg at ripe.net Tue Jun 19 17:26:55 2012 From: daniel.karrenberg at ripe.net (Daniel Karrenberg) Date: Tue, 19 Jun 2012 17:26:55 +0200 Subject: [dns-wg] rDNS Incident: The Facts Message-ID: <63A83744-BA08-4283-AC56-6B65919A4AC2@ripe.net> We have just published the time-line of events that led to the rDNS service issues last week: https://labs.ripe.net/Members/dfk/timeline-of-reverse-dns-events I realise that our operational performance was not up to the standard you expect from us and I apologise again for the considerable inconvenience we have caused some of you. Another report will follow once we have fully analysed what happened. Daniel