Reverse DNS Services Issues
Initial summary of the incident is now available on RIPE Labs:
Update 15 June 2012 (13:30 UTC)
Final update on the RIPE NCC rDNS event starting Wednesday 13 June 2012.
As of Friday, 15 June 2012 at 7:30 UTC, we are fully back to normal reverse DNS operations.
A full account of all relevant facts in this incident will be available, after thorough investigation, within the near future.
However, if you are still experiencing abnormal reverse DNS responses, you can contact us at firstname.lastname@example.org.
Update 14 June 2012 (21:30 UTC)
As of 20:00 UTC, Thursday 14 June 2012, all regular zone information is again complete and up-to-date. However, we are still processing some imported ERX zones from the APNIC and ARIN region.
If you are still experiencing any abnormal responses, other than related to the above mentioned ERX zones, you can contact us at email@example.com. Please provide as much detail as possible.
Update 14 June 2012 (13:35 UTC)
As of 13:00 UTC on Thursday, 14 June 2012 the zones listed in the previous update have been restored to the state of 13:30 UTC, Wednesday, 13 June 2012. So all zones we maintain are now available and zones in the list are no longer special. There may be a very few ERX zones maintained by other RIRs that are still not fully restored.
The update has been posted online at:
All updates since 13:30 UTC yesterday are still pending and are not reflected in the zones. However, no updates were lost. Any updates you will submit or you have already submitted will eventually be reflected in the zones once we complete our restoration procedure.
Next actions: We will now create up-to-date zones that include all the updates since yesterday 13:30 UTC. We will also recreate all ERX zones with current data. We expect to complete this by this evening. Once that is complete, we will re-enable regular provisioning and everything will be back to normal.
If you experience any abnormal responses beyond the residual inconsistencies described here, please let us know at firstname.lastname@example.org with as much detail as possible.
Update 14 June 2012 (11:22 UTC)
As of 10:45 UTC on Thursday, 14 June 2012, all reverse DNS zones we publish and that are not on the list in the previous update are correct and reflect the state of 13:30 UTC yesterday, 13 June. If you continue to see problems after this time, please contact email@example.com with details of what responses you are getting. Please consider (negative) caching TTLs and restart your caching servers if possible.
We know that some zones not mentioned in the list of the previous update (20:45 UTC June 13th) were not correctly propagated to some authoritative servers after the roll-back last night. We do not know the extent of this at the present time. However we are sure that as of 10:45 UTC today all authoritative servers answer correctly with the state of 13:30 UTC 13 June for all zones but the few on the list in the previous update.
We are working hard to restore the zones listed. We are busy adding all the updates since 13:30 UTC 13 June to all zones. We will report on how this work is progressing.
We will publish a full report once we've restored all the zones and have had sufficient time to fully analyse what went wrong.
We have been working hard over the past 18 hours to resolve this problem and we apologise for the considerable inconvenience.
The zones known to be still affected are:
Update 13 June 2012 (20:45 UTC)
Currently the vast majority of reverse zones has been restored. For a limited number of zones restoring the zone will take longer than expected, therefore these zones may still return empty responses.
The zones known still to be affected are listed below:
Update 13 June 2012 (18:50 UTC)
Between 15:00 UTC and 18:00 UTC on 13 June 2012, there was breakage in reverse DNS services. Queries may have resulted in empty results. The problems have been fixed and should propagate out in the coming hours.