From gponikie at akamai.com Tue Dec 11 12:46:03 2018 From: gponikie at akamai.com (Ponikierski, Grzegorz) Date: Tue, 11 Dec 2018 11:46:03 +0000 Subject: [atlas] API is sluggish Message-ID: <502BB88A-94C9-4EB6-B538-FD6276919295@akamai.com> Hi all! Did you notice that for last few days API is super slow to response. It takes something like few minutes to get reply. Do you observe any issue with Atlas infrastructure? What details I should provide to help with troubleshoot? Regards, Grzegorz -------------- next part -------------- An HTML attachment was scrubbed... URL: From jterbeest at ripe.net Tue Dec 11 13:13:38 2018 From: jterbeest at ripe.net (Johan Ter Beest) Date: Tue, 11 Dec 2018 13:13:38 +0100 Subject: [atlas] API is sluggish In-Reply-To: <502BB88A-94C9-4EB6-B538-FD6276919295@akamai.com> References: <502BB88A-94C9-4EB6-B538-FD6276919295@akamai.com> Message-ID: <4DA7E2C0-9822-4494-B1A3-32273CF3E079@ripe.net> Hi Gregorz, > On 11 Dec 2018, at 12:46, Ponikierski, Grzegorz wrote: > > Hi all! > > Did you notice that for last few days API is super slow to response. It takes something like few minutes to get reply. Do you observe any issue with Atlas infrastructure? What details I should provide to help with troubleshoot? We are not noticing anything as far as I know. What calls are you making that are slow? I can dig a little deeper into the logs then. Kind regards, Johan ter Beest RIPE Atlas Team > > Regards, > Grzegorz -------------- next part -------------- An HTML attachment was scrubbed... URL: From gponikie at akamai.com Tue Dec 11 13:54:55 2018 From: gponikie at akamai.com (Ponikierski, Grzegorz) Date: Tue, 11 Dec 2018 12:54:55 +0000 Subject: [atlas] API is sluggish In-Reply-To: <4DA7E2C0-9822-4494-B1A3-32273CF3E079@ripe.net> References: <502BB88A-94C9-4EB6-B538-FD6276919295@akamai.com> <4DA7E2C0-9822-4494-B1A3-32273CF3E079@ripe.net> Message-ID: Hi Jonan, Sorry for false alarm. It turns out it's our internal connectivity issues. Anyway, thank you for your attention and great tool which is Atlas :D Regards, Grzegorz From: Johan Ter Beest Date: Tuesday 2018-12-11 at 13:13 To: "Ponikierski, Grzegorz" Cc: "ripe-atlas at ripe.net" Subject: Re: [atlas] API is sluggish Hi Gregorz, On 11 Dec 2018, at 12:46, Ponikierski, Grzegorz > wrote: Hi all! Did you notice that for last few days API is super slow to response. It takes something like few minutes to get reply. Do you observe any issue with Atlas infrastructure? What details I should provide to help with troubleshoot? We are not noticing anything as far as I know. What calls are you making that are slow? I can dig a little deeper into the logs then. Kind regards, Johan ter Beest RIPE Atlas Team Regards, Grzegorz -------------- next part -------------- An HTML attachment was scrubbed... URL: From jterbeest at ripe.net Tue Dec 11 14:01:26 2018 From: jterbeest at ripe.net (Johan Ter Beest) Date: Tue, 11 Dec 2018 14:01:26 +0100 Subject: [atlas] API is sluggish In-Reply-To: References: <502BB88A-94C9-4EB6-B538-FD6276919295@akamai.com> <4DA7E2C0-9822-4494-B1A3-32273CF3E079@ripe.net> Message-ID: Hi Gregorz, > On 11 Dec 2018, at 13:54, Ponikierski, Grzegorz wrote: > > Hi Jonan, > > Sorry for false alarm. It turns out it's our internal connectivity issues. Anyway, thank you for your attention and great tool which is Atlas :D No problem, glad you found the problem, that?s all we want ;) Cheers, Johan > > Regards, > Grzegorz > > From: Johan Ter Beest > Date: Tuesday 2018-12-11 at 13:13 > To: "Ponikierski, Grzegorz" > Cc: "ripe-atlas at ripe.net" > Subject: Re: [atlas] API is sluggish > > Hi Gregorz, > > > > On 11 Dec 2018, at 12:46, Ponikierski, Grzegorz > wrote: > > Hi all! > > Did you notice that for last few days API is super slow to response. It takes something like few minutes to get reply. Do you observe any issue with Atlas infrastructure? What details I should provide to help with troubleshoot? > > > We are not noticing anything as far as I know. What calls are you making that are slow? > > I can dig a little deeper into the logs then. > > Kind regards, > Johan ter Beest > RIPE Atlas Team > > > > > Regards, > Grzegorz -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.muller at sidn.nl Fri Dec 14 15:03:12 2018 From: moritz.muller at sidn.nl (Moritz Muller) Date: Fri, 14 Dec 2018 14:03:12 +0000 Subject: [atlas] Long response times using one-off measurements with old probes Message-ID: Hi, I would like to point you to a discussion on the DNS-OARC mailing list: https://lists.dns-oarc.net/pipermail/dns-operations/2018-December/018196.html One of the operators at .ca noticed that old probes report way higher response times using DNS CHAOS queries (300% or more) (see also [3]). After some digging, we came to the conclusion that this has to do with the fact that he is carrying out ?one-off? measurements. When carrying out the same query for a longer time, we cannot observe this delay. The issues seems to be that one-off measurements are scheduled at probes using a different library than measurements that run for a longer period (?eooqd? instead of ?eperd?). Some small additional delays have been documented before on the RIPE Atlas website and in research papers [1, 2], but the big delay with one off measurements was new to me. Also to others? Is our assumption correct that the scheduler is the culprit? Moritz [1] https://dl.acm.org/citation.cfm?doid=2805789.2805796 [2] https://clarinet.u-strasbg.fr/~pelsser/publications/Holterbach-ripe-atlas-sharing-imc2015.pdf [3] https://atlas.ripe.net/measurements/18086197/#!probes -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: Message signed with OpenPGP URL: From philip.homburg at ripe.net Fri Dec 14 15:08:55 2018 From: philip.homburg at ripe.net (Philip Homburg) Date: Fri, 14 Dec 2018 15:08:55 +0100 Subject: [atlas] Long response times using one-off measurements with old probes In-Reply-To: References: Message-ID: On 2018/12/14 15:03 , Moritz Muller wrote: > Some small additional delays have been documented before on the RIPE Atlas website and in research papers [1, 2], but the big delay with one off measurements was new to me. > > Also to others? Is our assumption correct that the scheduler is the culprit? So far my investigation shows that the measurement actually takes that long. Why is not clear at the moment, but it doesn't seem to be anything external to the probe. It also doesn't seem to be interference from other measurements. Philip -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From moritz.muller at sidn.nl Mon Dec 17 11:46:16 2018 From: moritz.muller at sidn.nl (Moritz Muller) Date: Mon, 17 Dec 2018 10:46:16 +0000 Subject: [atlas] Long response times using one-off measurements with old probes In-Reply-To: References: Message-ID: <09AB501A-61A0-4DC1-BF08-9841DD467881@sidn.nl> Not sure if I understood you right. So you think that it doesn?t have anything to do with the probes themselves, but the round trip time is actually that long? Moritz > On 14 Dec 2018, at 15:08, Philip Homburg wrote: > > On 2018/12/14 15:03 , Moritz Muller wrote: >> Some small additional delays have been documented before on the RIPE Atlas website and in research papers [1, 2], but the big delay with one off measurements was new to me. >> >> Also to others? Is our assumption correct that the scheduler is the culprit? > > So far my investigation shows that the measurement actually takes that > long. > > Why is not clear at the moment, but it doesn't seem to be anything > external to the probe. It also doesn't seem to be interference from > other measurements. > > Philip > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: Message signed with OpenPGP URL: From philip.homburg at ripe.net Mon Dec 17 11:56:00 2018 From: philip.homburg at ripe.net (Philip Homburg) Date: Mon, 17 Dec 2018 11:56:00 +0100 Subject: [atlas] Long response times using one-off measurements with old probes In-Reply-To: <09AB501A-61A0-4DC1-BF08-9841DD467881@sidn.nl> References: <09AB501A-61A0-4DC1-BF08-9841DD467881@sidn.nl> Message-ID: <0b26ff76-c28c-c622-f4af-95a24dd1c4fd@ripe.net> On 2018/12/17 11:46 , Moritz Muller wrote: > Not sure if I understood you right. > So you think that it doesn?t have anything to do with the probes themselves, but the round trip time is actually that long? No. The extra time is caused by the measurement code. The actual round trip time is only part of the time that is measured. It is not clear what the code is doing during that time or why that time is included in the reported time. Philip -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From petr.spacek at nic.cz Mon Dec 17 18:40:51 2018 From: petr.spacek at nic.cz (=?UTF-8?B?UGV0ciDFoHBhxI1law==?=) Date: Mon, 17 Dec 2018 18:40:51 +0100 Subject: [atlas] testing DNS flag day compatibility Message-ID: <1cdc59e7-d7a7-41ad-baa0-6bb2c372be07@nic.cz> Hello everyone, this is follow-up from RIPE 77 hallway discussion, sorry for delay. We are looking for ways to test DNS flag day [1] compatibility from client networks. Objective is to test hypothesis that most breakage happens on authoritative side of DNS. In other words, we would like to test that DNS recursive infrastructure and client networks do not significantly influence compatibility. That would help to provide precise information for network operators who will have to deal with DNS flag day. Problem here is that RIPE Atlas does not allow to send all types of queries [2] required for full test. It was discussed at length that Atlas team has its reasons for not sending random blobs to random IP addresses, which is understood. Question here is: Can we find a middle ground to allow greater variety of valid DNS queries without forcing Atlas team to reimplement everything? My notes from meeting mention two approaches for further dicussion: a) User provides command line arguments for well-known tool dig, which gets executed in controlled environment ("as part of RIPE Atlas infrastructure") and generates query packet/blob. This blob generated by dig is then used as payload so use cannot ship anything but syntactically valid DNS packet. b) User provides blob for payload, which is then analyzed by packet parser of choice (BIND/ldns/Knot DNS/all of them). The payload can be sent out only if packet parsers do not find out any problem/blob is syntactically valid. These two approaches can also be combined to guard again quirks in either component. c) What do you think? Is there a way to allow greater flexibility to Atlas DNS? [1] https://dnsflagday.net/ [2] https://gitlab.isc.org/isc-projects/DNS-Compliance-Testing/blob/master/genreport.c#L216 -- Petr ?pa?ek @ CZ.NIC From ljsong at biigroup.cn Tue Dec 18 06:58:30 2018 From: ljsong at biigroup.cn (=?gb2312?B?RGF2ZXkgU29uZyjLzsHWvaEp?=) Date: Tue, 18 Dec 2018 13:58:30 +0800 Subject: [atlas] Data in the Internet Maps Message-ID: <001501d49696$b4627840$1d2768c0$@cn>+98854D18259BBC89 Hi Atlas people, I?m writing to ask about the Data repo in the intnernet maps. For example I would like to study the DNS root RTT statistics in this page : https://atlas.ripe.net/results/maps/rtt-fixed/ Where and How should I get the access to this data. Best regards, Davey -------------- next part -------------- An HTML attachment was scrubbed... URL: From jterbeest at ripe.net Tue Dec 18 11:16:17 2018 From: jterbeest at ripe.net (Johan Ter Beest) Date: Tue, 18 Dec 2018 11:16:17 +0100 Subject: [atlas] Data in the Internet Maps In-Reply-To: <001501d49696$b4627840$1d2768c0$@cn> References: <001501d49696$b4627840$1d2768c0$@cn> Message-ID: <0A986C50-74B9-4880-82DF-F5D9F0EE1229@ripe.net> Hi Davey, > On 18 Dec 2018, at 06:58, Davey Song(???) wrote: > > Hi Atlas people, > > > > I?m writing to ask about the Data repo in the intnernet maps. For example I would like to study the DNS root RTT statistics in this page : > https://atlas.ripe.net/results/maps/rtt-fixed/ > > Where and How should I get the access to this data. These maps are based on the built-in measurements that are running on all the probes. You can go to any probe and then look at the built-in tab to see the measurement IDs for these built-in measurements. You can then go to the measurement page for the desired measurement and download the results for the timeframe that you want. For instance: https://atlas.ripe.net/measurements/1009/#!download This is the built-in Ping measurement towards a.root Remember that the results for these measurements are huge so be careful in selecting your timeframes. Let me know if you need any more information. Cheers, Johan ter Beest RIPE Atlas Team > > Best regards, > Davey -------------- next part -------------- An HTML attachment was scrubbed... URL: From danny at danysek.cz Tue Dec 18 14:09:47 2018 From: danny at danysek.cz (Daniel Suchy) Date: Tue, 18 Dec 2018 14:09:47 +0100 Subject: [atlas] testing DNS flag day compatibility In-Reply-To: <1cdc59e7-d7a7-41ad-baa0-6bb2c372be07@nic.cz> References: <1cdc59e7-d7a7-41ad-baa0-6bb2c372be07@nic.cz> Message-ID: <0f6df283-9e49-aff7-8864-b6d2cd893294@danysek.cz> Hello, I think there should be specified, which tests/options are really *necesary* for this compability testing related to the DNS flag day. >From operator perspective, you just need to know, if your implementation will have problem or if it's OK... and I think many details reported by [2] will not be even understood by normal users. >From a quick look, you're missing ability to set some bits (flags) and other options in query packet. Majority of tests in linked source code are using SOA, some other common types in query, which are already included in options available, some aren't - but they're quite exotic query types and probably not widely used - so are these really needed? I don't think allowing "simply" anything (as you're proposing in [a] or [b] below) is a good apporach. Some options (ignoretc, for example) will not be even understood by current `dig` implementations, that's another problem. And there's always some risk of malicious use and "open" Atlas network may be misused. So I prefer to stay restrictive in terms of queries allowed over Atlas network. Daniel On 12/17/18 6:40 PM, Petr ?pa?ek wrote: > Hello everyone, > > this is follow-up from RIPE 77 hallway discussion, sorry for delay. > > We are looking for ways to test DNS flag day [1] compatibility from > client networks. Objective is to test hypothesis that most breakage > happens on authoritative side of DNS. In other words, we would like to > test that DNS recursive infrastructure and client networks do not > significantly influence compatibility. > > That would help to provide precise information for network operators who > will have to deal with DNS flag day. > > > Problem here is that RIPE Atlas does not allow to send all types of > queries [2] required for full test. It was discussed at length that > Atlas team has its reasons for not sending random blobs to random IP > addresses, which is understood. > > Question here is: > Can we find a middle ground to allow greater variety of valid DNS > queries without forcing Atlas team to reimplement everything? > > > My notes from meeting mention two approaches for further dicussion: > > a) User provides command line arguments for well-known tool dig, which > gets executed in controlled environment ("as part of RIPE Atlas > infrastructure") and generates query packet/blob. This blob generated by > dig is then used as payload so use cannot ship anything but > syntactically valid DNS packet. > > > b) User provides blob for payload, which is then analyzed by packet > parser of choice (BIND/ldns/Knot DNS/all of them). The payload can be > sent out only if packet parsers do not find out any problem/blob is > syntactically valid. > > These two approaches can also be combined to guard again quirks in > either component. > > > c) > > > What do you think? Is there a way to allow greater flexibility to Atlas DNS? > > > [1] https://dnsflagday.net/ > [2] > https://gitlab.isc.org/isc-projects/DNS-Compliance-Testing/blob/master/genreport.c#L216 > From bortzmeyer at nic.fr Tue Dec 18 15:18:04 2018 From: bortzmeyer at nic.fr (Stephane Bortzmeyer) Date: Tue, 18 Dec 2018 15:18:04 +0100 Subject: [atlas] Data in the Internet Maps In-Reply-To: <001501d49696$b4627840$1d2768c0$@cn> References: <001501d49696$b4627840$1d2768c0$@cn> Message-ID: <20181218141803.piolgplelmwvemfu@nic.fr> On Tue, Dec 18, 2018 at 01:58:30PM +0800, Davey Song wrote a message of 117 lines which said: > It works for me, I receive messages. From petr.spacek at nic.cz Wed Dec 19 10:33:48 2018 From: petr.spacek at nic.cz (=?UTF-8?B?UGV0ciDFoHBhxI1law==?=) Date: Wed, 19 Dec 2018 10:33:48 +0100 Subject: [atlas] testing DNS flag day compatibility In-Reply-To: <0f6df283-9e49-aff7-8864-b6d2cd893294@danysek.cz> References: <1cdc59e7-d7a7-41ad-baa0-6bb2c372be07@nic.cz> <0f6df283-9e49-aff7-8864-b6d2cd893294@danysek.cz> Message-ID: Hello Daniel and others, On 18. 12. 18 14:09, Daniel Suchy wrote: > Hello, > I think there should be specified, which tests/options are really > *necesary* for this compability testing related to the DNS flag day. > From operator perspective, you just need to know, if your implementation > will have problem or if it's OK... and I think many details reported by > [2] will not be even understood by normal users. > > From a quick look, you're missing ability to set some bits (flags) and > other options in query packet. Majority of tests in linked source code > are using SOA, some other common types in query, which are already > included in options available, some aren't - but they're quite exotic > query types and probably not widely used - so are these really needed? > > I don't think allowing "simply" anything (as you're proposing in [a] or > [b] below) is a good apporach. Some options (ignoretc, for example) will > not be even understood by current `dig` implementations, that's another > problem. And there's always some risk of malicious use and "open" Atlas > network may be misused. So I prefer to stay restrictive in terms of > queries allowed over Atlas network. I remember from RIPE 77 meeting that there are strong opinions on limiting what can be done and that there are reasons for that. Purpose of my e-mail is to find out if there is a middle ground. Does your answer mean "it is not going to happen, go away" or is there a room for negotiation? I can provide detailed argumentation if you are willing to negotiate. Petr ?pa?ek @ CZ.NIC > > Daniel > > On 12/17/18 6:40 PM, Petr ?pa?ek wrote: >> Hello everyone, >> >> this is follow-up from RIPE 77 hallway discussion, sorry for delay. >> >> We are looking for ways to test DNS flag day [1] compatibility from >> client networks. Objective is to test hypothesis that most breakage >> happens on authoritative side of DNS. In other words, we would like to >> test that DNS recursive infrastructure and client networks do not >> significantly influence compatibility. >> >> That would help to provide precise information for network operators who >> will have to deal with DNS flag day. >> >> >> Problem here is that RIPE Atlas does not allow to send all types of >> queries [2] required for full test. It was discussed at length that >> Atlas team has its reasons for not sending random blobs to random IP >> addresses, which is understood. >> >> Question here is: >> Can we find a middle ground to allow greater variety of valid DNS >> queries without forcing Atlas team to reimplement everything? >> >> >> My notes from meeting mention two approaches for further dicussion: >> >> a) User provides command line arguments for well-known tool dig, which >> gets executed in controlled environment ("as part of RIPE Atlas >> infrastructure") and generates query packet/blob. This blob generated by >> dig is then used as payload so use cannot ship anything but >> syntactically valid DNS packet. >> >> >> b) User provides blob for payload, which is then analyzed by packet >> parser of choice (BIND/ldns/Knot DNS/all of them). The payload can be >> sent out only if packet parsers do not find out any problem/blob is >> syntactically valid. >> >> These two approaches can also be combined to guard again quirks in >> either component. >> >> >> c) >> >> >> What do you think? Is there a way to allow greater flexibility to Atlas DNS? >> >> >> [1] https://dnsflagday.net/ >> [2] >> https://gitlab.isc.org/isc-projects/DNS-Compliance-Testing/blob/master/genreport.c#L216 From ljsong at biigroup.cn Wed Dec 19 11:11:11 2018 From: ljsong at biigroup.cn (=?UTF-8?B?RGF2ZXkgU29uZyjlrovmnpflgaUp?=) Date: Wed, 19 Dec 2018 18:11:11 +0800 Subject: [atlas] =?utf-8?b?562U5aSNOiAgRGF0YSBpbiB0aGUgSW50ZXJuZXQgTWFw?= =?utf-8?q?s?= In-Reply-To: <0A986C50-74B9-4880-82DF-F5D9F0EE1229@ripe.net> References: <001501d49696$b4627840$1d2768c0$@cn> <0A986C50-74B9-4880-82DF-F5D9F0EE1229@ripe.net> Message-ID: <00bf01d49783$2b5a38b0$820eaa10$@cn>+567E52F8A641CA73 Hi Johan, Thanks for you information. It works follow you advice to find these built-in measurement. But I need more probes (maybe all of them, or large set) and RTT data for just one day. It is impossible for me to do it manually. Is there any tool for me to select a group of probes or measurements in the rtt-fixed map? Davey ???: Johan Ter Beest [mailto:jterbeest at ripe.net] ????: 2018?12?18? 18:16 ???: "Davey Song(???)" ??: ripe-atlas at ripe.net ??: Re: [atlas] Data in the Internet Maps Hi Davey, On 18 Dec 2018, at 06:58, Davey Song(???) wrote: Hi Atlas people, I?m writing to ask about the Data repo in the intnernet maps. For example I would like to study the DNS root RTT statistics in this page : https://atlas.ripe.net/results/maps/rtt-fixed/ Where and How should I get the access to this data. These maps are based on the built-in measurements that are running on all the probes. You can go to any probe and then look at the built-in tab to see the measurement IDs for these built-in measurements. You can then go to the measurement page for the desired measurement and download the results for the timeframe that you want. For instance: https://atlas.ripe.net/measurements/1009/#!download This is the built-in Ping measurement towards a.root Remember that the results for these measurements are huge so be careful in selecting your timeframes. Let me know if you need any more information. Cheers, Johan ter Beest RIPE Atlas Team Best regards, Davey -------------- next part -------------- An HTML attachment was scrubbed... URL: From ljsong at biigroup.cn Wed Dec 19 11:13:28 2018 From: ljsong at biigroup.cn (=?gb2312?B?RGF2ZXkgU29uZyjLzsHWvaEp?=) Date: Wed, 19 Dec 2018 18:13:28 +0800 Subject: [atlas] =?gb2312?b?tPC4tDogIERhdGEgaW4gdGhlIEludGVybmV0IE1hcHM=?= In-Reply-To: <20181218141803.piolgplelmwvemfu@nic.fr> References: <001501d49696$b4627840$1d2768c0$@cn> <20181218141803.piolgplelmwvemfu@nic.fr> Message-ID: <00c401d49783$7d2ba7f0$7782f7d0$@cn>+2C5839A6A8848175 Thanks. I can receive and respond. It weird that I can receive mail from this mailing list now only after I sent a mail. Was my mail account treated as a dead one? : ) Davey > -----????----- > ???: Stephane Bortzmeyer [mailto:bortzmeyer at nic.fr] > ????: 2018?12?18? 22:18 > ???: Davey Song > ??: ripe-atlas at ripe.net > ??: Re: [atlas] Data in the Internet Maps > > On Tue, Dec 18, 2018 at 01:58:30PM +0800, Davey Song > wrote a message of 117 lines which said: > > > > works.> > > It works for me, I receive messages. From danny at danysek.cz Wed Dec 19 11:29:11 2018 From: danny at danysek.cz (Daniel Suchy) Date: Wed, 19 Dec 2018 11:29:11 +0100 Subject: [atlas] testing DNS flag day compatibility In-Reply-To: References: <1cdc59e7-d7a7-41ad-baa0-6bb2c372be07@nic.cz> <0f6df283-9e49-aff7-8864-b6d2cd893294@danysek.cz> Message-ID: Hello, On 12/19/18 10:33 AM, Petr ?pa?ek wrote: > On 18. 12. 18 14:09, Daniel Suchy wrote: > I remember from RIPE 77 meeting that there are strong opinions on > limiting what can be done and that there are reasons for that. Purpose > of my e-mail is to find out if there is a middle ground. > > Does your answer mean "it is not going to happen, go away" > or is there a room for negotiation? In my previous email I tried ask you to more precisely specify, what tests are really *necesary* (important) for DNS flag-day compability testing. I'm missing this information from you :-) I think if you reduce (and explain) your needs, there's space for discussion. In general, proposed test is useful in my oppinion - but you're asking for more than you really need for that purpose, I think. With regards, Daniel From petr.spacek at nic.cz Wed Dec 19 12:43:12 2018 From: petr.spacek at nic.cz (=?UTF-8?B?UGV0ciDFoHBhxI1law==?=) Date: Wed, 19 Dec 2018 12:43:12 +0100 Subject: [atlas] testing DNS flag day compatibility In-Reply-To: References: <1cdc59e7-d7a7-41ad-baa0-6bb2c372be07@nic.cz> <0f6df283-9e49-aff7-8864-b6d2cd893294@danysek.cz> Message-ID: Hello once again, I'm glad you are willing to consider it. Here we go: On 18. 12. 18 14:09, Daniel Suchy wrote: > Hello, > I think there should be specified, which tests/options are really > *necesary* for this compability testing related to the DNS flag day. > From operator perspective, you just need to know, if your implementation > will have problem or if it's OK... and I think many details reported by > [2] will not be even understood by normal users. Let me clarify that [2] is low-level tool with many tests, and all of them are used for DNS flag day testing (see below). Normal users are supposed to use form [1] which does post-processing of results from [2] and transforms it into green/yellow/orange/red signal with more human-friendly description. The important difference here is that some test in [2] have non-binary results whilst DNS flag day [1] is concerned only with timeout/non-timeout result and ignores other details of individual tests. (In the end this distinction is not important from Atlas point of view because we either get the message back or not.) More technical details about DNS flag day 2019 ---------------------------------------------- In short, the DNS protocol specification does not allow the server to drop (i.e. not respond at all) queries based on options in them, so the tool attempts to test if something in DNS path is dropping queries in violation of DNS protocol or not. DNS client is free to set any flags or add arbitrary options and protocol defines what the other side should do if the flag/option is not understood. Thus, if the test passes you are safe with any version of DNS resolver (without regard to particular configuration). Different resolvers use different set of options by default, and also the set depends on configuration. E.g. latest versions of BIND send DNS cookie option by default and it of course breaks queries to some subset of servers, which will be subject of DNS flag day 2019 (among others). Finally to your question: Is it really needed? ---------------------------------------------- > From a quick look, you're missing ability to set some bits (flags) and > other options in query packet. Majority of tests in linked source code > are using SOA, some other common types in query, which are already > included in options available, some aren't - but they're quite exotic > query types and probably not widely used - so are these really needed? For DNS flag day 2019 in particular we are interested only in queries listed at [2] and tagged with constant EDNS, sorry for not making it clear at the beginning. Of course future DNS flag days will have different requirements ... see below. More generically, as mentioned above, the purpose of test is to answer "will this network work with any standard-compliant resolver" and to answer this we need to test full spectrum. Using only subset of tests would answer sub-questions like: - will this network work with BIND 9.10 - will this network work with BIND 9.11 with cookies disabled but will not answer other sub-questions like: - BIND 9.14 in default configuration (no workarounds for non-compliance) etc. Please note that BIND is just an example, the test matrix is in fact: V vendors * N versions * O options in each implementation, i.e. huge matrix and reducing it to minimal set is not feasible as it changes with each release. Generalization - why are we talking about it at all? ---------------------------------------------------- Having said all that, I now realized this e-mail should have different subject - it is *also* about looking forward *beyond* DNS flag day 2019 itself. Robert Kisteleki made clear during RIPE 77 that we are not going to get anything for DNS flag day 2019 because Atlas planning cycle does not allow to get more features in. That's understood and purpose of this excercise is to find out if there are safe ways to make Atlas more useful for future DNS flag days (and other uses, of course), because in fact we are already too late for (hypotetical) DNS flag day 2020! Problem description by example: 1. Imagine that there will be e.g. a DNS flag day 2020, 2021, 2022, etc. 2. DNS flag day is announced roughly a year before it happens to give operators room for preparation. 3. We test authoritative sides using our own tools like [2] and scripts around it [3]. It basically implements "DNS query carpet bombing" to ~ 23 milion domains. 4. Uncertainity which is left is question of compatibility problems in *client* networks - that's why we are looking at Atlas. / end of introduction / 5. Current state of things forces people who write DNS clients to specify what kind of queries we want to do for DNS flag day 2020 much much earlier than necessary for other purposes so it can get included in Atlas planning cycle. (In fact we would be already late if we wanted to do experiments now and announce it in February 2019, i.e. a year ahead). 6. Naturally if we found out that also a different type of queries is needed (which always happens once you start experimenting) it is either too late to repeat the full cycle, or we have to do experiments years before DNS flag day itself. Such a big delay does not reflect pace of DNS ecosystem development, i.e. is good only for measurement after the fact instead of being usable as precaution/data gathering before the event. In other words we have to hope for the best and let operators to find out what the problem is because there is no way to measure it beforehand (again, in client networks). I hope it illustrates why this limitations and problems steming from them. Proposal -------- Proposal is to allow Atlas user to input wider variety of DNS messages in some form, and do validation on them before sending user-provided DNS message out. This can be done in multiple ways and it up to discussion which way gives reasonable assurance the client query will not cause problem. Assessing impact ---------------- While assessing impact of this proposal we should take into account current state of things. Even the current ability to send out simple A query for user-provided name can trigger wide variety of bugs, including security/denial-of-service bugs in DNS resolvers used by client networks. One example for all is https://doc.powerdns.com/recursor/security-advisories/powerdns-advisory-2017-08.html (not picking on this particular implementation!) An attacker who controls single authoritative server can trigger this bug by sending plain A query from current Atlas to DNS resolver "under attack". Effectivelly all resolvers have had similar bugs in the past, it is certainly not one-off. >From this example I conclude that anyone who can buy own domain (for like ~ 6 USD/year) can mount this attack using current Atlas API, today. In my opinion, an implementation which takes user-provided DNS message and checks it using 3 independent parsers compiled with Valgrind/ASAN (e.g. BIND, Unbound/ldns, Knot DNS, or any other) provides roughly the same level of (in)security as current limited set of options. I hope this clarifies the case. Where do we go from here? [1] https://dnsflagday.net/ [2] https://gitlab.isc.org/isc-projects/DNS-Compliance-Testing/blob/master/genreport.c#L216 [3] https://gitlab.labs.nic.cz/knot/edns-zone-scanner/ Petr ?pa?ek @ CZ.NIC On 19. 12. 18 11:29, Daniel Suchy wrote: > Hello, > > On 12/19/18 10:33 AM, Petr ?pa?ek wrote: >> On 18. 12. 18 14:09, Daniel Suchy wrote: >> I remember from RIPE 77 meeting that there are strong opinions on >> limiting what can be done and that there are reasons for that. Purpose >> of my e-mail is to find out if there is a middle ground. >> >> Does your answer mean "it is not going to happen, go away" >> or is there a room for negotiation? > > In my previous email I tried ask you to more precisely specify, what > tests are really *necesary* (important) for DNS flag-day compability > testing. I'm missing this information from you :-) > > I think if you reduce (and explain) your needs, there's space for > discussion. In general, proposed test is useful in my oppinion - but > you're asking for more than you really need for that purpose, I think. > > With regards, > Daniel From ljsong at biigroup.cn Fri Dec 21 09:22:12 2018 From: ljsong at biigroup.cn (=?UTF-8?B?RGF2ZXkgU29uZyjlrovmnpflgaUp?=) Date: Fri, 21 Dec 2018 16:22:12 +0800 Subject: [atlas] =?utf-8?b?562U5aSNOiAg562U5aSNOiAgRGF0YSBpbiB0aGUgSW50?= =?utf-8?q?ernet_Maps?= In-Reply-To: <00bf01d49783$2b5a38b0$820eaa10$@cn>+567E52F8A641CA73 References: <001501d49696$b4627840$1d2768c0$@cn> <0A986C50-74B9-4880-82DF-F5D9F0EE1229@ripe.net> <00bf01d49783$2b5a38b0$820eaa10$@cn>+567E52F8A641CA73 Message-ID: <009501d49906$46cde860$d469b920$@cn>+569A824C03FCA5D4 Any suggestions ? Or It is not possible to reach my ambitious goal by current tools available. Davey ???: ripe-atlas [mailto:ripe-atlas-bounces at ripe.net] ?? Davey Song(???) ????: 2018?12?19? 18:11 ???: 'Johan Ter Beest' ??: ripe-atlas at ripe.net ??: [atlas] ??: Data in the Internet Maps Hi Johan, Thanks for you information. It works follow you advice to find these built-in measurement. But I need more probes (maybe all of them, or large set) and RTT data for just one day. It is impossible for me to do it manually. Is there any tool for me to select a group of probes or measurements in the rtt-fixed map? Davey ???: Johan Ter Beest [mailto:jterbeest at ripe.net] ????: 2018?12?18? 18:16 ???: "Davey Song(???)" ??: ripe-atlas at ripe.net ??: Re: [atlas] Data in the Internet Maps Hi Davey, On 18 Dec 2018, at 06:58, Davey Song(???) wrote: Hi Atlas people, I?m writing to ask about the Data repo in the intnernet maps. For example I would like to study the DNS root RTT statistics in this page : https://atlas.ripe.net/results/maps/rtt-fixed/ Where and How should I get the access to this data. These maps are based on the built-in measurements that are running on all the probes. You can go to any probe and then look at the built-in tab to see the measurement IDs for these built-in measurements. You can then go to the measurement page for the desired measurement and download the results for the timeframe that you want. For instance: https://atlas.ripe.net/measurements/1009/#!download This is the built-in Ping measurement towards a.root Remember that the results for these measurements are huge so be careful in selecting your timeframes. Let me know if you need any more information. Cheers, Johan ter Beest RIPE Atlas Team Best regards, Davey -------------- next part -------------- An HTML attachment was scrubbed... URL: From jterbeest at ripe.net Fri Dec 21 10:44:21 2018 From: jterbeest at ripe.net (Johan Ter Beest) Date: Fri, 21 Dec 2018 10:44:21 +0100 Subject: [atlas] =?utf-8?b?562U5aSNOiAgRGF0YSBpbiB0aGUgSW50ZXJuZXQgTWFw?= =?utf-8?q?s?= In-Reply-To: <009501d49906$46cde860$d469b920$@cn> References: <001501d49696$b4627840$1d2768c0$@cn> <0A986C50-74B9-4880-82DF-F5D9F0EE1229@ripe.net> <00bf01d49783$2b5a38b0$820eaa10$@cn> <009501d49906$46cde860$d469b920$@cn> Message-ID: <30E41754-121A-4AAB-9FA3-BFE11CB00077@ripe.net> Hi Davey, > On 21 Dec 2018, at 09:22, Davey Song(???) wrote: > > Any suggestions ? Or It is not possible to reach my ambitious goal by current tools available. > > Davey > ? <> > ???: ripe-atlas [mailto:ripe-atlas-bounces at ripe.net] ?? Davey Song(???) > ????: 2018?12?19? 18:11 > ???: 'Johan Ter Beest' > ??: ripe-atlas at ripe.net > ??: [atlas] ??: Data in the Internet Maps > > Hi Johan, > > Thanks for you information. It works follow you advice to find these built-in measurement. But I need more probes (maybe all of them, or large set) and RTT data for just one day. It is impossible for me to do it manually. Is there any tool for me to select a group of probes or measurements in the rtt-fixed map? I think you?re mis-understanding how the built-ins work. These measurements do run on all the probes and therefore have the data for all of the probes. In the download tab, you can then select the day you want and download the results. For instance: https://atlas.ripe.net/api/v2/measurements/1009/results/?start=1545350400&stop=1545436799&format=json This gives you all the results for all the probes for 21 December 2018. You can of course also select a day in the map view itself using the time travel slider. Again, be aware that these results are huge so you should use curl or our Python libraries to get and process these results. You can find all the RIPE NCC tools here: https://github.com/RIPE-NCC The specific ones you want are: https://github.com/RIPE-NCC/ripe-atlas-tools https://github.com/RIPE-NCC/ripe.atlas.sagan https://github.com/RIPE-NCC/ripe-atlas-cousteau Let me know if I answered your question. If not, I will try to help you further. Kind regards, Johan ter Beest > > Davey > > ???: Johan Ter Beest [mailto:jterbeest at ripe.net ] > ????: 2018?12?18? 18:16 > ???: "Davey Song(???)" > ??: ripe-atlas at ripe.net > ??: Re: [atlas] Data in the Internet Maps > > Hi Davey, > > > On 18 Dec 2018, at 06:58, Davey Song(???) > wrote: > > Hi Atlas people, > > > > I?m writing to ask about the Data repo in the intnernet maps. For example I would like to study the DNS root RTT statistics in this page : > https://atlas.ripe.net/results/maps/rtt-fixed/ > > Where and How should I get the access to this data. > > > These maps are based on the built-in measurements that are running on all the probes. You can go to any probe and then look at the built-in tab to see the measurement IDs for these built-in measurements. > > You can then go to the measurement page for the desired measurement and download the results for the timeframe that you want. For instance: > > https://atlas.ripe.net/measurements/1009/#!download > > This is the built-in Ping measurement towards a.root > > Remember that the results for these measurements are huge so be careful in selecting your timeframes. > > Let me know if you need any more information. > > Cheers, > Johan ter Beest > RIPE Atlas Team > > > > > > Best regards, > Davey -------------- next part -------------- An HTML attachment was scrubbed... URL: