You are here: Home > Participate > Join a Discussion > RIPE Forum > Anti-Abuse Working Group > [anti-abuse-wg] [Misc] Research project on blacklists
RIPE Forum v1.4.1

Anti-Abuse Working Group

Threaded
Collapse

[anti-abuse-wg] [Misc] Research project on blacklists

Anushah Hossain

2019-07-17 13:42:21 CET

Hi everyone,

I'm a researcher at UC Berkeley and the International Computer Science
Institute. My colleagues and I are working on evaluating and improving the
accuracy of blacklists. As part of this work, we'd like to hear from you
about the blacklists you currently use, what you perceive as their
strengths and weaknesses, and any thoughts you have on how they might be
improved.

We've prepared an anonymous survey where you can share your views:
https://berkeley.qualtrics.com/jfe/form/SV_200mg5hnQiAOgUl

If you have five to ten minutes free today to fill it out, I would greatly
appreciate your help! Thank you, and please don't hesitate to respond to me
with comments or questions.

(Apologies if you receive this message twice - trying to minimize
cross-posting while still reaching a broad audience)

Best,
Anushah

-- 
Anushah Hossain, PhD Student
Energy and Resources Group, UC Berkeley

Barry Greene

2019-07-17 19:01:16 CET

Not a joke. 

Just a researcher exploring ways to quantify and measure. Always important to have the academic doing the due diligence on our operational assumptions.

> On Jul 17, 2019, at 07:40, ac <ac _at_ main _dot_ me> wrote:
> 
> 
> This is a joke email, right?
> 
> Is it the 1st of April already? :)
> 
> Andre
> 
> On Wed, 17 Jul 2019 13:42:21 +0200
> Anushah Hossain <anushah _at_ icsi.berkeley _dot_ edu> wrote:
> 
>> Hi everyone,
>> 
>> I'm a researcher at UC Berkeley and the International Computer Science
>> Institute. My colleagues and I are working on evaluating and
>> improving the accuracy of blacklists. As part of this work, we'd like
>> to hear from you about the blacklists you currently use, what you
>> perceive as their strengths and weaknesses, and any thoughts you have
>> on how they might be improved.
>> 
>> We've prepared an anonymous survey where you can share your views:
>> https://berkeley.qualtrics.com/jfe/form/SV_200mg5hnQiAOgUl
>> 
>> If you have five to ten minutes free today to fill it out, I would
>> greatly appreciate your help! Thank you, and please don't hesitate to
>> respond to me with comments or questions.
>> 
>> (Apologies if you receive this message twice - trying to minimize
>> cross-posting while still reaching a broad audience)
>> 
>> Best,
>> Anushah
>> 
> 
> 

Serge Droz

2019-07-18 08:03:38 CET

Hi Andreas

I echo Barry's views on the research.

Some valid points, but it's a pity that you tend to void them by mostly
telling others that they are stupid.

I like your idea about studying why certain practices occur. So why not
find a University that is interested in starting a project on this?

Best
Serge


On 18/07/2019 07:20, ac wrote:
> 
> Oh. Lets look more at this then.
> 
> "UC Berkeley" - USA
> "International Computer Science Institute"
> "evaluating and improving the accuracy of blacklists." 
> "including a web link, which is tracked and cross tracked"
> "an anonymous survey"
> 
> Dude, let us be frank: On this list we discuss abuse, in the open and
> directly. People on this list has "skills" and can all be anonymous on
> this list, if they wish to, in fact, many are. (I do not and I am not
> private)
> 
> We are talking about email blacklists? right? as the routing blacklists
> do not bother the evil tech monopolies!
> 
> It is a fact that the spam from the top ten USA tech companies are the
> most challenging abuse on the planet - as this type of abuse, is the
> hardest to combat. - Twitter does not even accept abuse complaints.
> Facebook does not care and Google mixes spam with ham all the time to
> defeat email blacklists
> 
> Why not study the reasons for the percentage increase in the use of
> inspection/tracking/non private/invasive anti abuse technologies in use
> by the largest email and dominant players, Google and Microsoft, of
> ipv6 and the reason why these huge tech players HAVE to push for ipv6
> email servers relay to ensure their future dominance of email relay?
> 
> Instead of "My colleagues and I are working on evaluating and improving
> the accuracy of blacklists"
> 
> As, imnsho, that is absolute USA bullshit. and is not even possible.
> 
> I would go sofar as to state that such research is not intended to
> "improve" anything but to cement the monopolies we fight daily and is
> on the EVIL side of the fight.
> 
> Andre
> 
> 
> 
> On Wed, 17 Jul 2019 10:01:16 -0700
> Barry Greene <barryrgreene _at_ gmail _dot_ com> wrote:
> 
>> Not a joke. 
>>
>> Just a researcher exploring ways to quantify and measure. Always
>> important to have the academic doing the due diligence on our
>> operational assumptions.
>>
>>> On Jul 17, 2019, at 07:40, ac <ac _at_ main _dot_ me> wrote:
>>>
>>>
>>> This is a joke email, right?
>>>
>>> Is it the 1st of April already? :)
>>>
>>> Andre
>>>
>>> On Wed, 17 Jul 2019 13:42:21 +0200
>>> Anushah Hossain <anushah _at_ icsi.berkeley _dot_ edu> wrote:
>>>   
>>>> Hi everyone,
>>>>
>>>> I'm a researcher at UC Berkeley and the International Computer
>>>> Science Institute. My colleagues and I are working on evaluating
>>>> and improving the accuracy of blacklists. As part of this work,
>>>> we'd like to hear from you about the blacklists you currently use,
>>>> what you perceive as their strengths and weaknesses, and any
>>>> thoughts you have on how they might be improved.
>>>>
>>>> We've prepared an anonymous survey where you can share your views:
>>>>
>>>> If you have five to ten minutes free today to fill it out, I would
>>>> greatly appreciate your help! Thank you, and please don't hesitate
>>>> to respond to me with comments or questions.
>>>>
>>>> (Apologies if you receive this message twice - trying to minimize
>>>> cross-posting while still reaching a broad audience)
>>>>
>>>> Best,
>>>> Anushah
>>>>   
>>>
>>>   
> 
> 

-- 
Dr. Serge Droz
Chair, Forum of Incident Response and Security Teams (FIRST)
Phone +41 76 542 44 93 | serge.droz _at_ first _dot_ org | https://www.first.org

John Levine

2019-07-18 08:11:15 CET

In article vVgJ2zmOVgi4zGAm+4Rss3U1GXPg _at_ mail.gmail _dot_ com> you write:
>-=-=-=-=-=-
>
>Hi everyone,
>
>I'm a researcher at UC Berkeley and the International Computer Science
>Institute. My colleagues and I are working on evaluating and improving the
>accuracy of blacklists. 

I looked at the survey and I don't understand what you mean by "blacklists".
I know a lot of different reputation services, many of which might be called
blacklists, but they do a lot of different things.

Can you perhaps give us a few examples of things that are or are not the
blacklists you're interested in?

R's,
John

Fi Shing

2019-07-18 08:17:29 CET

The only organisation that is in a prime position to implement any meaningful blacklist is a RIR like RIPE itself. Anything less than RIR level blacklisting is what is known as "whac a mole"
 
https://en.wikipedia.org/wiki/Whac-A-Mole
 
But, as it comes down to time and money, the likes of which even google and facebook etc are not motivated to part with in terms of accountability, organisations like RIPE, APNIC, ICANN etc. All of them, without exception, refuse to engage in responsible practices. They are happy to take money to issue resources, but taking them away is equated to sacrilege. 
 
In an ideal world, the employees of RIPE etc should be arrested in jailed for aiding and abetting crime.
 
 
 
 
 
 
 
--------- Original Message --------- Subject: Re: [anti-abuse-wg] [Misc] Research project on blacklists
From: "ac" <ac _at_ main _dot_ me>
Date: 7/18/19 3:20 pm
To: anti-abuse-wg _at_ ripe _dot_ net


 Oh. Lets look more at this then.
 
 "UC Berkeley" - USA
 "International Computer Science Institute"
 "evaluating and improving the accuracy of blacklists." 
 "including a web link, which is tracked and cross tracked"
 "an anonymous survey"
 
 Dude, let us be frank: On this list we discuss abuse, in the open and
 directly. People on this list has "skills" and can all be anonymous on
 this list, if they wish to, in fact, many are. (I do not and I am not
 private)
 
 We are talking about email blacklists? right? as the routing blacklists
 do not bother the evil tech monopolies!
 
 It is a fact that the spam from the top ten USA tech companies are the
 most challenging abuse on the planet - as this type of abuse, is the
 hardest to combat. - Twitter does not even accept abuse complaints.
 Facebook does not care and Google mixes spam with ham all the time to
 defeat email blacklists
 
 Why not study the reasons for the percentage increase in the use of
 inspection/tracking/non private/invasive anti abuse technologies in use
 by the largest email and dominant players, Google and Microsoft, of
 ipv6 and the reason why these huge tech players HAVE to push for ipv6
 email servers relay to ensure their future dominance of email relay?
 
 Instead of "My colleagues and I are working on evaluating and improving
 the accuracy of blacklists"
 
 As, imnsho, that is absolute USA bullshit. and is not even possible.
 
 I would go sofar as to state that such research is not intended to
 "improve" anything but to cement the monopolies we fight daily and is
 on the EVIL side of the fight.
 
 Andre
 
 
 
 On Wed, 17 Jul 2019 10:01:16 -0700
 Barry Greene <barryrgreene _at_ gmail _dot_ com> wrote:
 
 > Not a joke. 
 > 
 > Just a researcher exploring ways to quantify and measure. Always
 > important to have the academic doing the due diligence on our
 > operational assumptions.
 > 
 > > On Jul 17, 2019, at 07:40, ac <ac _at_ main _dot_ me> wrote:
 > > 
 > > 
 > > This is a joke email, right?
 > > 
 > > Is it the 1st of April already? :)
 > > 
 > > Andre
 > > 
 > > On Wed, 17 Jul 2019 13:42:21 +0200
 > > Anushah Hossain <anushah _at_ icsi.berkeley _dot_ edu> wrote:
 > > 
 > >> Hi everyone,
 > >> 
 > >> I'm a researcher at UC Berkeley and the International Computer
 > >> Science Institute. My colleagues and I are working on evaluating
 > >> and improving the accuracy of blacklists. As part of this work,
 > >> we'd like to hear from you about the blacklists you currently use,
 > >> what you perceive as their strengths and weaknesses, and any
 > >> thoughts you have on how they might be improved.
 > >> 
 > >> We've prepared an anonymous survey where you can share your views:
 > >> 
 > >> If you have five to ten minutes free today to fill it out, I would
 > >> greatly appreciate your help! Thank you, and please don't hesitate
 > >> to respond to me with comments or questions.
 > >> 
 > >> (Apologies if you receive this message twice - trying to minimize
 > >> cross-posting while still reaching a broad audience)
 > >> 
 > >> Best,
 > >> Anushah
 > >> 
 > > 
 > >
User Image

Sebastian Wiesinger

2019-07-18 09:02:01 CET

* ac <ac _at_ main _dot_ me> [2019-07-18 08:29]:
> It is about: "evaluating and improving the accuracy of blacklists."
> 
> The entire post is arrogant, obnoxious, offensive and inaccurate and is
> an oxymoron.

Tbh the only mails I get this vibe from in this thread are yours.
Could you tune it down a bit? There is no reason to be this
aggressive. It seems this is legitimate research and if you do not
agree with it don't take the survey.

Regards

Sebastian

-- 
GPG Key: 0x58A2D94A93A0B9CE (F4F6 B1A3 866B 26E9 450A  9D82 58A2 D94A 93A0 B9CE)
'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE.
            -- Terry Pratchett, The Fifth Elephant
User Image

Richard Clayton

2019-07-18 14:19:12 CET

In message , ac <ac _at_ main _dot_ me> writes

>Mostly, what makes me very angry is the audacity 

this does seem a reasonable list to ask for assistance on ... but being
around to answer questions promptly would be appropriately polite

surprisingly, I haven't seen the request on any other lists that are (a)
relevant and (b) open -- perhaps they and their project team are not
especially well connected in this space :(  though there is a recent
"anonymous" survey request about router configurations on the NANOG list

>and then the
>"anonymous" 

the Qualtrics platform is available over Tor (unlike some online survey
platforms) so if you declined to answer the questions about which AS and
company you were associated with then there is a substantial amount of
anonymity available to you should you wish to use it...

>and I can already see the "findings" of this research...
>based on random anonymous, hidden and secret inputs....

that is a concern -- this type of questionnaire pretty much never leads
to high quality research directly (since there are significant biases in
who might choose to give replies and there is scope for multiple
responses from a single person, bots filling it in etc)

nevertheless as a starting point for qualitative research (rather than
quantitative) it can be very useful in allowing a researcher to identify
general trends in the answers and -- importantly -- to help the
researcher frame good research questions that are capable of being
investigated in more detail

as John Levine already noted, the questionnaire seems somewhat confused
as to whether it cares about routing issues (bogon lists, the Spamhaus
DROP list etc) or spam filtering (bad domains, phishing feeds, botnet
IPs etc etc)

it also asked if internally generated lists were used, but seemed
curiously uninterested in anything other than if the answer to that was
yes or no -- a missed opportunity I thought.

-- 
richard                                                   Richard Clayton

Those who would give up essential Liberty, to purchase a little temporary 
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

Anushah Hossain

2019-07-18 15:33:39 CET

Apologies for my slow response - I have been traveling and also consulting
with my team members on how best to respond (as you might have gleaned from
my profile linked upthread, my own background is not in networking or
security :)). I hope to share more thorough responses with you once the sun
rises in their timezones.

>surprisingly, I haven't seen the request on any other lists that are (a)
relevant and (b) open -- perhaps they and their project team are not
especially well connected in this space :(

This is true. We were advised to share to RIPE and regional NOG mailing
lists. Are there others you would have recommended?

> as John Levine already noted, the questionnaire seems somewhat confused
as to whether it cares about routing issues (bogon lists, the Spamhaus
DROP list etc) or spam filtering (bad domains, phishing feeds, botnet
IPs etc etc)

Hm, I think we are interested in quite the range of blacklists. Here is a
table of what my colleagues are monitoring:

[image: image.png]

>it also asked if internally generated lists were used, but seemed
curiously uninterested in anything other than if the answer to that was
yes or no -- a missed opportunity I thought.

What would you have recommended probing here?

I do genuinely appreciate your discussion and patience. It is very
interesting and useful for me to see what topics matter to you most and
where we might have misdirected our attention. Just as background, we did
pilot the survey with a smaller set of network operators and felt it had
been straightforward to respond to, given their reactions. But as many of
you have noted, the survey is rather general. I have been conducting
interviews with those working in abuse prevention (even at some of the
companies that have been mentioned upthread) to collect more specific
anecdotes about how dynamic addressing has lowered the accuracy of certain
feeds, for example, or how errors in geo-IP feeds affected them. The
interviews allow for a bit more elucidation, but it has been difficult to
recruit participants. Hence the survey.

All the best,
Anushah



On Thu, Jul 18, 2019 at 2:36 PM Richard Clayton <richard _at_ highwayman _dot_ com>
wrote:

> In message , ac <ac _at_ main _dot_ me> writes
>
> >Mostly, what makes me very angry is the audacity
>
> this does seem a reasonable list to ask for assistance on ... but being
> around to answer questions promptly would be appropriately polite
>
> surprisingly, I haven't seen the request on any other lists that are (a)
> relevant and (b) open -- perhaps they and their project team are not
> especially well connected in this space :(  though there is a recent
> "anonymous" survey request about router configurations on the NANOG list
>
> >and then the
> >"anonymous"
>
> the Qualtrics platform is available over Tor (unlike some online survey
> platforms) so if you declined to answer the questions about which AS and
> company you were associated with then there is a substantial amount of
> anonymity available to you should you wish to use it...
>
> >and I can already see the "findings" of this research...
> >based on random anonymous, hidden and secret inputs....
>
> that is a concern -- this type of questionnaire pretty much never leads
> to high quality research directly (since there are significant biases in
> who might choose to give replies and there is scope for multiple
> responses from a single person, bots filling it in etc)
>
> nevertheless as a starting point for qualitative research (rather than
> quantitative) it can be very useful in allowing a researcher to identify
> general trends in the answers and -- importantly -- to help the
> researcher frame good research questions that are capable of being
> investigated in more detail
>
> as John Levine already noted, the questionnaire seems somewhat confused
> as to whether it cares about routing issues (bogon lists, the Spamhaus
> DROP list etc) or spam filtering (bad domains, phishing feeds, botnet
> IPs etc etc)
>
> it also asked if internally generated lists were used, but seemed
> curiously uninterested in anything other than if the answer to that was
> yes or no -- a missed opportunity I thought.
>
> --
> richard                                                   Richard Clayton
>
> Those who would give up essential Liberty, to purchase a little temporary
> Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755
>


-- 
Anushah Hossain, PhD Student
Energy and Resources Group, UC Berkeley
User Image

Richard Clayton

2019-07-18 19:02:27 CET

In message <CAKcP59JPJT2LsUTrtgAsLeUTDsCrVBWq0_Cuas8LAvNAApQ7UQ _at_ mail _dot_ gma
il.com>, Anushah Hossain <anushah _at_ icsi.berkeley _dot_ edu> writes

>    >surprisingly, I haven't seen the request on any other lists that 
>    are (a)
>    relevant and (b) open -- perhaps they and their project team are 
>    not
>    especially well connected in this space :( 
>
>    This is true. We were advised to share to RIPE and regional NOG 
>    mailing lists. Are there others you would have recommended?

ask the APWG to circulate the request to their members, and you might do
the same with M3AAWG

>    > as John Levine already noted, the questionnaire seems somewhat 
>    confused
>    as to whether it cares about routing issues (bogon lists, the 
>    Spamhaus
>    DROP list etc) or spam filtering (bad domains, phishing feeds, 
>    botnet
>    IPs etc etc)
>
>    Hm, I think we are interested in quite the range of blacklists. 

The issues will vary considerably between different types of list

>    Here is a table of what my colleagues are monitoring:
>
>    image.png
>
>    >it also asked if internally generated lists were used, but seemed
>    curiously uninterested in anything other than if the answer to that 
>    was
>    yes or no -- a missed opportunity I thought.
>
>    What would you have recommended probing here?

you could have asked an open ended question which asked what they did,
how they were built, why they were built in house and how significant
they were.

>    I have been conducting interviews with those 
>    working in abuse prevention (even at some of the companies that 
>    have been mentioned upthread) to collect more specific anecdotes 
>    about how dynamic addressing has lowered the accuracy of certain 
>    feeds, 

we've had DHCP for decades (and everyone knows the issues) ... are you
sure they weren't discussing Carrier Grade NAT ?

>    for example, or how errors in geo-IP feeds affected them.

my own impression of these is that you get what you pay for ... but
unless you are buying proxies I'm sceptical that large scale abuse
filtering systems use this type of info as more than a one indicator
amongst many.

if you buying a proxy you may care a lot more !

    Zachary Weinberg, Shinyoung Cho, Nicolas Christin, Vyas Sekar, and
    Phillipa Gill. How to Catch when Proxies Lie: Verifying the Physical
    Locations of Network Proxies with Active Geolocation. In Proceedings
    of the 2018 ACM Internet Measurement Conference (IMC'18). Boston,
    MA. October 2018.

-- 
richard                                                   Richard Clayton

Those who would give up essential Liberty, to purchase a little temporary 
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755
User Image

Randy Bush

2019-07-18 19:05:36 CET

anushah,

next study: distribution of operator reactions to a simple request
     	    of an academic study questionnaire :)

uncloak: the work icsi does is a couple decade trail of operationally
relevant awesomeness.  so, though i loathe survey quetionnaires, i
answered directly as appended.  i suspect your  question was clear
enough that even i understood it.

randy

---

From: Randy Bush <randy _at_ psg _dot_ com>
Subject: Re: [anti-abuse-wg] [Misc] Research project on blacklists
To: Anushah Hossain <anushah _at_ icsi.berkeley _dot_ edu>
Date: Wed, 17 Jul 2019 08:02:31 -0700

i loathe surveys

i use

    dialups.mail-abuse.org
    dnsbl.sorbs.net
    zen.spamhaus.org

the one i really miss is the whitelist which seems to have died a few
months back, dnswl.org

i specifically whitelist sender addresses of

    *@bigglobe.ne.jp
    *@earthlink.net
    *@google.com
    *@hotmail.com
    *@teleport.com
    *@yahoo.co.jp
    *@yahoo.com

randy

User Image

Sebastian Wiesinger

2019-07-19 15:26:57 CET

* Randy Bush <randy _at_ psg _dot_ com> [2019-07-18 19:06]:
> the one i really miss is the whitelist which seems to have died a few
> months back, dnswl.org

Hi Randy,

dnswl.org is still operating, I'm also involved with them. Is there
anything you need/miss? If so I'm able to relay it to the rest of the
dnswl.org team. Reply off list if you want.

Regards

Sebastian

-- 
GPG Key: 0x58A2D94A93A0B9CE (F4F6 B1A3 866B 26E9 450A  9D82 58A2 D94A 93A0 B9CE)
'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE.
            -- Terry Pratchett, The Fifth Elephant