From ptyll at nitronet.pl Sat Jan 7 18:12:05 2012 From: ptyll at nitronet.pl (Pawel Tyll) Date: Sat, 7 Jan 2012 18:12:05 +0100 Subject: [members-discuss] RIPE DB NRTM access. Message-ID: <354278486.20120107181205@nitronet.pl> Hello people (and ticket bots :>), I wanted to do some BGP filtering automation today, learned some interesting things in the process. I decided on setting up a local mirror of RIPE IRR and was quite puzzled with reasoning behind this: http://www.ripe.net/data-tools/db/nrtm-mirroring So, I can download the whole database snapshot for free from ftp.ripe.net, wasting bandwidth and other resources, or on the other hand I can pay for doing things the right and efficient way, which in the process is effectively cheaper for all involved parties. So the question is - what gives? :) - why does RIPE insist that I download >200MB of compressed data instead of catching up probably few hundred KB of daily changes? --Pawel. From jorgen at ssc.net Sat Jan 7 20:42:08 2012 From: jorgen at ssc.net (=?ISO-8859-1?Q?J=F8rgen_Hovland?=) Date: Sat, 07 Jan 2012 20:42:08 +0100 Subject: [members-discuss] RIPE DB NRTM access. In-Reply-To: <354278486.20120107181205@nitronet.pl> References: <354278486.20120107181205@nitronet.pl> Message-ID: <4F08A010.1040607@ssc.net> Den 1/7/12 6:12 PM, skrev Pawel Tyll: > Hello people (and ticket bots :>), > > I wanted to do some BGP filtering automation today, learned some > interesting things in the process. I decided on setting up a local > mirror of RIPE IRR and was quite puzzled with reasoning behind this: > > http://www.ripe.net/data-tools/db/nrtm-mirroring > > So, I can download the whole database snapshot for free from > ftp.ripe.net, wasting bandwidth and other resources, or on the > other hand I can pay for doing things the right and efficient way, > which in the process is effectively cheaper for all involved parties. > > So the question is - what gives? :) - why does RIPE insist that I > download>200MB of compressed data instead of catching up probably few > hundred KB of daily changes? > > --Pawel. I'm quite sure it is more expensive to administrate and receive a stream of changes than downloading a static file with no administration:) Bandwidth is cheap and static content scales extremely well. From a.buglak at sumtel.ru Sat Jan 7 20:52:41 2012 From: a.buglak at sumtel.ru (Artem Buglak) Date: Sat, 7 Jan 2012 23:52:41 +0400 Subject: [members-discuss] RIPE DB NRTM access. In-Reply-To: <354278486.20120107181205@nitronet.pl> References: <354278486.20120107181205@nitronet.pl> Message-ID: Hi, Pawel! My guess would be that a full snapshot available via ftp is considered not that NEAR Real Time as the update data stream that you may pull continuously from the server - they're not trying to sell data there but the service of that data being streamed. Although I'd expect that stuff to be available to members without extra pay - at least one host per member. 07.01.2012 21:12 ???????????? "Pawel Tyll" ???????: > > Hello people (and ticket bots :>), > > I wanted to do some BGP filtering automation today, learned some > interesting things in the process. I decided on setting up a local > mirror of RIPE IRR and was quite puzzled with reasoning behind this: > > http://www.ripe.net/data-tools/db/nrtm-mirroring > > So, I can download the whole database snapshot for free from > ftp.ripe.net, wasting bandwidth and other resources, or on the > other hand I can pay for doing things the right and efficient way, > which in the process is effectively cheaper for all involved parties. > > So the question is - what gives? :) - why does RIPE insist that I > download >200MB of compressed data instead of catching up probably few > hundred KB of daily changes? > > --Pawel. > > > > ---- > If you don't want to receive emails from the RIPE NCC members-discuss > mailing list, please log in to your LIR Portal account and go to the general page: > https://lirportal.ripe.net/general/view > > Click on "Edit my LIR details", under "Subscribed Mailing Lists". From here, you can add or remove addresses. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ptyll at nitronet.pl Sat Jan 7 21:14:58 2012 From: ptyll at nitronet.pl (Pawel Tyll) Date: Sat, 7 Jan 2012 21:14:58 +0100 Subject: [members-discuss] RIPE DB NRTM access. In-Reply-To: References: <354278486.20120107181205@nitronet.pl> Message-ID: <1465420718.20120107211458@nitronet.pl> Hi Artem, > My guess would be that a full snapshot available via ftp is > considered not that NEAR Real Time as the update data stream that > you may pull continuously from the server - they're not trying to > sell data there but the service of that data being streamed. I don't need to pull it continuously. It can even be limited to one connection a day, although again, I see no reason for such limitations. I would only like a local mirror that I can hammer however frequently I wish, that would be relatively up-to-date. I can pull the snapshot once a day, restart irrd and be done with it, but...that's just wrong :) > Although I'd expect that stuff to be available to members without > extra pay - at least one host per member. I would expect it to be available to everyone interested that uses BGP. Correct and automated filtering of routing between ASes should be encouraged in every possible way, and abstract pricing that doesn't have any basis in real world isn't encouraging at all. --Pawel. From andrea.cocito at ifom-ieo-campus.it Sat Jan 7 22:07:40 2012 From: andrea.cocito at ifom-ieo-campus.it (Andrea Cocito) Date: Sat, 7 Jan 2012 22:07:40 +0100 Subject: [members-discuss] RIPE DB NRTM access. In-Reply-To: <1465420718.20120107211458@nitronet.pl> References: <354278486.20120107181205@nitronet.pl> <1465420718.20120107211458@nitronet.pl> Message-ID: <5016B8E5-D2E7-4AAF-8D34-574FC7D99220@ifom-ieo-campus.it> On Jan 7, 2012, at 9:14 PM, Pawel Tyll wrote: >> My guess would be that a full snapshot available via ftp is >> considered not that NEAR Real Time as the update data stream that >> you may pull continuously from the server - they're not trying to >> sell data there but the service of that data being streamed. > I don't need to pull it continuously. It can even be limited to one > connection a day, although again, I see no reason for such > limitations. I would only like a local mirror that I can hammer > however frequently I wish, that would be relatively up-to-date. I can > pull the snapshot once a day, restart irrd and be done with it, > but...that's just wrong :) Hi, maybe a simple rsync server would do the job for you and for everyone else with similar needs (described as "I need a periodic update but I don't really need a realtime one"). At our site we mirror a few terabytes of data (genomic repositories) via rsync without even noticing an impact of bandwidth usage (as long as only a small part of those big files changes every day). Anyone at RIPE willing to consider this option ? Ciao, A. From ptyll at nitronet.pl Sat Jan 7 23:18:19 2012 From: ptyll at nitronet.pl (Pawel Tyll) Date: Sat, 7 Jan 2012 23:18:19 +0100 Subject: [members-discuss] RIPE DB NRTM access. In-Reply-To: <5016B8E5-D2E7-4AAF-8D34-574FC7D99220@ifom-ieo-campus.it> References: <354278486.20120107181205@nitronet.pl> <1465420718.20120107211458@nitronet.pl> <5016B8E5-D2E7-4AAF-8D34-574FC7D99220@ifom-ieo-campus.it> Message-ID: <963749669.20120107231819@nitronet.pl> Hi Andrea, > maybe a simple rsync server would do the job for you and for everyone > else with similar needs (described as "I need a periodic update but I > don't really need a realtime one"). It isn't a "realtime" one, it's a simple polling mechanism that hits the whois server every 30 mins (by default), supplies local serial number and requests changes between local sn and current RIPE sn. Nothing realtime about it. Downloading 200MB file isn't a problem. It takes less than a minute. Real trouble is restarting irrd - with 3GB database (which ripe.db.gz unpacks to), it takes about 20 mins on the machine I want to use for this purpose, also creating unnecessary load. > At our site we mirror a few terabytes of data (genomic repositories) > via rsync without even noticing an impact of bandwidth usage (as long > as only a small part of those big files changes every day). Like I said, file size isn't a problem. Unnecessary processing required due to irrd restart is a problem. --Pawel. From andrea.cocito at ifom-ieo-campus.it Sat Jan 7 23:54:57 2012 From: andrea.cocito at ifom-ieo-campus.it (Andrea Cocito) Date: Sat, 7 Jan 2012 23:54:57 +0100 Subject: [members-discuss] RIPE DB NRTM access. In-Reply-To: <963749669.20120107231819@nitronet.pl> References: <354278486.20120107181205@nitronet.pl> <1465420718.20120107211458@nitronet.pl> <5016B8E5-D2E7-4AAF-8D34-574FC7D99220@ifom-ieo-campus.it> <963749669.20120107231819@nitronet.pl> Message-ID: <3BB07846-037C-4FC0-9D27-4AA9AEFD68DD@ifom-ieo-campus.it> Hi, On Jan 7, 2012, at 11:18 PM, Pawel Tyll wrote: > Like I said, file size isn't a problem. Unnecessary processing > required due to irrd restart is a problem. That's a bit different from the initial point... I do not know enough about irrd to comment on this (I just never runned it here), the things we mirror here are MySQL backed and in some way our sysadms convinced MySQL to accept that at some moment some read-only tables get replaced on the fly with new snapshots and that happens magically and atomically. Maybe RIPE might put in some directory one file per day with the "daily updates" and that directory can be rsync'd ? My point was only about the data transfer, I understand that setting up a "push" update requires a significant administrative/management overhead, and it is reasonable to ask for a cost compensation; setting up a public rsync server requires no more effort than setting up a public ftp site (maybe less...): the big dfference is that (as long as data is uncompressed and unencrypted) only the "delta" gets actually transferred. Regards, A. From ptyll at nitronet.pl Sun Jan 8 00:44:17 2012 From: ptyll at nitronet.pl (Pawel Tyll) Date: Sun, 8 Jan 2012 00:44:17 +0100 Subject: [members-discuss] RIPE DB NRTM access. In-Reply-To: <3BB07846-037C-4FC0-9D27-4AA9AEFD68DD@ifom-ieo-campus.it> References: <354278486.20120107181205@nitronet.pl> <1465420718.20120107211458@nitronet.pl> <5016B8E5-D2E7-4AAF-8D34-574FC7D99220@ifom-ieo-campus.it> <963749669.20120107231819@nitronet.pl> <3BB07846-037C-4FC0-9D27-4AA9AEFD68DD@ifom-ieo-campus.it> Message-ID: <34413937.20120108004417@nitronet.pl> > That's a bit different from the initial point... Original point still stands - the fact that it's not a problem anymore to download 200MB over 200KB doesn't make it any less wasteful. > I do not know enough about irrd to comment on this (I just never > runned it here), the things we mirror here are MySQL backed and in > some way our sysadms convinced MySQL to accept that at some moment > some read-only tables get replaced on the fly with new snapshots and > that happens magically and atomically. Operative keyword would be 'magically'. I prefer KISS over magic :) > Maybe RIPE might put in some directory one file per day with the > "daily updates" and that directory can be rsync'd ? I'm not sure why we are still trying to re-invent wheel here. I would really like to hear a sane reason behind non-profit organization expecting a EUR 250 fee for access to, essentially, a whois server with public information that should be mirrored as much as possible, which also is already available for free, but in a form that is only harder to keep updated? I'm trying hard to imagine how running mirror with up to two weeks of serialized updates would be more cost-intensive compared to running FTP server, yet I can't figure this out. I was so perplexed by this that it even made me start this thread on a saturday! ;) > My point was only about the data transfer, I understand that setting > up a "push" update requires a significant administrative/management > overhead, and it is reasonable to ask for a cost compensation; setting > up a public rsync server requires no more effort than setting up a > public ftp site (maybe less...): the big dfference is that (as long as > data is uncompressed and unencrypted) only the "delta" gets actually > transferred. It's already up and running, this point is moot. Another user doesn't require a new set of data - same updates are being sent. It's also running on free and open software. It's also not a push update. From what I learned in short time this saturday, it's just a glorified whois server that allows mirror operation. Besides, others managed to somehow sink their astronomical costs with running such a contraption: http://www.radb.net/resources/databases.php Seriously, maybe I'm missing something here. But what? --Pawel. From yiorgos at tee.gr Mon Jan 9 11:02:53 2012 From: yiorgos at tee.gr (Yiorgos Adamopoulos) Date: Mon, 09 Jan 2012 12:02:53 +0200 Subject: [members-discuss] RIPE DB NRTM access. In-Reply-To: <5016B8E5-D2E7-4AAF-8D34-574FC7D99220@ifom-ieo-campus.it> References: <354278486.20120107181205@nitronet.pl> <1465420718.20120107211458@nitronet.pl> <5016B8E5-D2E7-4AAF-8D34-574FC7D99220@ifom-ieo-campus.it> Message-ID: <4F0ABB4D.2090308@tee.gr> ???? 7/1/2012 11:07 ??, ?/? Andrea Cocito ??????: > maybe a simple rsync server would do the job for you and for everyone > else with similar needs (described as "I need a periodic update but I > don't really need a realtime one"). +1 for rsync on all ftp.ripe.net content From ml at kneip.me Mon Jan 9 11:09:12 2012 From: ml at kneip.me (Armin Kneip) Date: Mon, 09 Jan 2012 11:09:12 +0100 Subject: [members-discuss] RIPE DB NRTM access. In-Reply-To: <4F0ABB4D.2090308@tee.gr> References: <354278486.20120107181205@nitronet.pl> <1465420718.20120107211458@nitronet.pl> <5016B8E5-D2E7-4AAF-8D34-574FC7D99220@ifom-ieo-campus.it> <4F0ABB4D.2090308@tee.gr> Message-ID: <4F0ABCC8.1080000@kneip.me> > +1 for rsync on all ftp.ripe.net content > Dito From denis at ripe.net Tue Jan 10 15:54:57 2012 From: denis at ripe.net (Denis Walker) Date: Tue, 10 Jan 2012 15:54:57 +0100 Subject: [members-discuss] RIPE DB NRTM access. In-Reply-To: <354278486.20120107181205@nitronet.pl> References: <354278486.20120107181205@nitronet.pl> Message-ID: <4F0C5141.4050403@ripe.net> Dear Pawel We have followed this discussion on NRTM issues and would like to thank you for raising the subject. First, let me try to provide some historical context to explain how we got to where we are. In the beginning, NRTM was totally open and free. Over time, it became clear that some users were taking advantage of this service to data mine the RIPE Database for email addresses. Many users whose email addresses were only listed in the RIPE Database were receiving spam. At this point it was decided to register NRTM users. To apply for this service, you had to fill in a form stating who you are, why you wanted the service, what IP address you would use to receive the data stream and provide contact details. The software was adjusted to only accept requests for NRTM data from these registered IP addresses. Some years later our administrative database used for controlling access to the RIPE Database was growing, with many NRTM users listed who no longer made use of the service. In the mean time, the RIPE Data Protection Task Force proposed we filter both the NRTM data stream and the daily split files to remove email addresses. We then moved to the current paid service as a means of covering the administrative overhead and limiting the list of users to only those who actively use the service. The RIPE NCC is open to suggestions for restructuring this service and its administration. Over recent years, it has been discussed at great length at RIPE Meetings with regard to data protection. Perhaps you would like to open a new discussion on the RIPE Database Working Group mailing list [1] to review NRTM and the daily split files services. Technical suggestions, such as the rsync option, can also be covered in the community discussions. If consensus is reached for a better solution, we will be pleased to implement it. [1] https://www.ripe.net/ripe/groups/wg/db Regards, Denis Walker Business Analyst RIPE NCC Database Group On 7/01/12:2 6:12 PM, Pawel Tyll wrote: > Hello people (and ticket bots :>), > > I wanted to do some BGP filtering automation today, learned some > interesting things in the process. I decided on setting up a local > mirror of RIPE IRR and was quite puzzled with reasoning behind this: > > http://www.ripe.net/data-tools/db/nrtm-mirroring > > So, I can download the whole database snapshot for free from > ftp.ripe.net, wasting bandwidth and other resources, or on the > other hand I can pay for doing things the right and efficient way, > which in the process is effectively cheaper for all involved parties. > > So the question is - what gives? :) - why does RIPE insist that I > download >200MB of compressed data instead of catching up probably few > hundred KB of daily changes? > > --Pawel. > > From ptyll at nitronet.pl Tue Jan 10 17:19:42 2012 From: ptyll at nitronet.pl (Pawel Tyll) Date: Tue, 10 Jan 2012 17:19:42 +0100 Subject: [members-discuss] RIPE DB NRTM access. In-Reply-To: <4F0C5141.4050403@ripe.net> References: <354278486.20120107181205@nitronet.pl> <4F0C5141.4050403@ripe.net> Message-ID: <1987308596.20120110171942@nitronet.pl> Hello Denis, > The RIPE NCC is open to suggestions for restructuring this service and > its administration. Over recent years, it has been discussed at great > length at RIPE Meetings with regard to data protection. Perhaps you > would like to open a new discussion on the RIPE Database Working Group > mailing list [1] to review NRTM and the daily split files services. > Technical suggestions, such as the rsync option, can also be covered in > the community discussions. If consensus is reached for a better > solution, we will be pleased to implement it. If spam-protection and long list of allowed hosts are the reasons behind the fee, then surely each interested LIR may ask for addtion of one IP address for free, and to make sure they actually need it and use it, if not used for more than 28 days, request has to be made again after automatic removal. Right? ;) --Pawel. From h.lu at anytimechinese.com Fri Jan 13 10:37:58 2012 From: h.lu at anytimechinese.com (Lu Heng) Date: Fri, 13 Jan 2012 10:37:58 +0100 Subject: [members-discuss] Who elected to the fee task force? Message-ID: Hi Last time there was a opening for 4 members to join the new member fee task force, was the four member already selected? -- -- Kind regards. Lu This transmission is intended solely for the addressee(s) shown above. It may contain information that is privileged, confidential or otherwise protected from disclosure. Any review, dissemination or use of this transmission or its contents by persons other than the intended addressee(s) is strictly prohibited. If you have received this transmission in error, please notify this office immediately and e-mail the original at the sender's address above by replying to this message and including the text of the transmission received. From h.lu at anytimechinese.com Wed Jan 18 16:21:30 2012 From: h.lu at anytimechinese.com (Lu Heng) Date: Wed, 18 Jan 2012 16:21:30 +0100 Subject: [members-discuss] SOPA and Ripe's police order Message-ID: Hi colleagues: As a owner of a company have great relation of fighting censorship in China, I feel strongly about today's bill, and should like to share few of my concerns and thoughts about how the things might going. I personally didn't know we have so many regulations in EU about internet before I raised my ISP business. and I didn't realize that receiving legal latter will be part of the business.(as my lawyer count, on average, we receive about 100-200 legal notice per day, and we need to take down over 100 domains each day for the copy right infringement). Futurenally, I also have a company in China, so that legal costs has been kept in a relative low level. In last year short after the Ripe 63, Ripe was ordered by Dutch police force to take measurement of certain IP ranges.with today's US bill, we are very much concerning what government might take lead to in the future of internet. In China, all ISP have to censer themselves before they allow anything going public, otherwise it will results hash punishment up to a level that entire data center can be disconnected just by single offensive website(which in view of certain interest group). Will that happen one day in EU or US, if this trends continues? People might know that many Chinese use VPN service overseas to get over the firewall built by the government, however, as a company have relation in the business, we also see another type of VPN service being offered in China has increase popularity, it was offered to foreign Chinese in order also them to get back China. The reason for them to "to back China" is this: http://www.iqiyi.com/dianying/20120110/376d083f97061688.html Instead of actual video, You will see a quote "This video is only available for mainland China customers due to copyright restrictions" The reason this happen is because large amount of Chinese company(often legally US based) get sued by copyright groups in US/EU, therefore they impose the censorship to censer the content out of China. Does this censorship has anything different from the government censorship in China? Was that both block of certain content because of the interest of certain group of people? I am personally not interested in politics, the right or wrong of the content does not interest me. but as far as I have been to three Ripe meetings, I do feel a strong sense from the persons built the internet(look at how the basic protocol is written, it was very surprise me that anybody can announcing any range that doesn't require some sign of paper), that the internet should be a open and free place to share what you think what you thought, but not a place that police and government can break in at any minus and tell you what to do. As ISP, we are the ground of the internet, if this trends continues, I was very much worried that one day we have to do what Chinese ISPs are doing, we have to look carefully about every website's content and afraid in one middle night our entire business can be shut down due to something that we don't even know of. I thought it was impossible in western country, until I saw how aggressive the SOPA/Protect IP looks like. I don't know if something similar will happen in EU in a near future, my company lawyer told me that as long as we have data center here in EU, we are bounded by the EU laws. So are you, my colleagues. Here is just my 2 cents thought, I would hear from any colleagues that share the same concern as me and my company do. The raising legal costs and restrictions will not only damage the small business, start ups, it also require us one day become some sort of "internet local government" to police our client, and I wish that day never happen. -- -- Kind regards. Lu This transmission is intended solely for the addressee(s) shown above. It may contain information that is privileged, confidential or otherwise protected from disclosure. Any review, dissemination or use of this transmission or its contents by persons other than the intended addressee(s) is strictly prohibited. If you have received this transmission in error, please notify this office immediately and e-mail the original at the sender's address above by replying to this message and including the text of the transmission received. From nigel at titley.com Wed Jan 18 16:38:31 2012 From: nigel at titley.com (Nigel Titley) Date: Wed, 18 Jan 2012 15:38:31 +0000 Subject: [members-discuss] Who elected to the fee task force? In-Reply-To: References: Message-ID: <4F16E777.3040207@titley.com> On 13/01/2012 09:37, Lu Heng wrote: > Hi > > Last time there was a opening for 4 members to join the new member fee > task force, was the four member already selected? > Yes, we're just finalising the Task Force charter and then will make an announcement of this and its membership. Watch this space Nigel From nigel at titley.com Wed Jan 18 18:23:08 2012 From: nigel at titley.com (Nigel Titley) Date: Wed, 18 Jan 2012 17:23:08 +0000 Subject: [members-discuss] Who elected to the fee task force? In-Reply-To: <4F16E777.3040207@titley.com> References: <4F16E777.3040207@titley.com> Message-ID: <4F16FFFC.6090508@titley.com> On 18/01/2012 15:38, Nigel Titley wrote: > On 13/01/2012 09:37, Lu Heng wrote: >> Hi >> >> Last time there was a opening for 4 members to join the new member fee >> task force, was the four member already selected? >> > Yes, we're just finalising the Task Force charter and then will make an > announcement of this and its membership. Watch this space > Congratulations to The Cork Community Broadband robot on joining the exclusive club of sentient trouble ticket robots. I'll buy you a Guinness next time we meet. Nigel From nigel at titley.com Fri Jan 20 17:10:37 2012 From: nigel at titley.com (Nigel Titley) Date: Fri, 20 Jan 2012 16:10:37 +0000 Subject: [members-discuss] RIPE NCC Charging scheme task force announcement Message-ID: <4F1991FD.3090101@titley.com> Dear colleagues, As announced, the RIPE NCC Executive Board has initiated a Charging Scheme Task Force to look at the principles of the RIPE NCC Charging Scheme. The task force will have the following objectives: - Define the principles of future Charging Schemes - Propose an improved process for the annual Charging Scheme discussions and adoption - Set a pricing structure for several years for legacy address space holders - Advise the Executive Board on a discussion or resolution for the RIPE NCC General Meeting in April 2011 The task force will not discuss or set any actual fees or prices. Participants of the Task Force ------------------------------ To have an efficient task force, the group will be limited in size. The group will consist of participants from three groups, namely: - RIPE NCC Executive Board - Remco van Mook, Dmitry Burkov - RIPE NCC members - Eva Ornberg, Filip Herman, Olga Fomina, Piotr Strzyzewski and Vladislav Potapov - RIPE NCC staff - Jochem de Ruig, Andrea Cima The Executive Board has selected from each membership category (Extra Small, Small, etc.) one member by drawing lots from those who volunteered. The RIPE NCC staff have informed the people involved and will organise the task force. Because there has been a lot of interest in the charging scheme, there will be a separate mailing list (cs-tf at ripe.net) for all those interested and those who want to be informed and discuss this topic. The task force will report back to this mailing list on its progress. You can subscribe to the charging scheme mailing list at: https://www.ripe.net/mailman/listinfo/cs-tf. Regards, Nigel Titley Executive Board Chairman RIPE NCC