Bug 243135

Summary:	WebKitGTK based browser detected as bot by botguard
Product:	WebKit	Reporter:	Bartek Sabat <Cloud11665>
Component:	WebKitGTK	Assignee:	Nobody <webkit-unassigned>
Status:	NEW ---
Severity:	Normal	CC:	bfulgham, bugs-noreply, mcatanzaro
Priority:	P2
Version:	WebKit Nightly Build
Hardware:	PC
OS:	Linux
See Also:	https://bugs.webkit.org/show_bug.cgi?id=240041

Description Bartek Sabat 2022-07-23 10:29:04 PDT

botguard (https://www.humansecurity.com/products/botguard-applications) marks WebKitGTK browsers as bots (tested on both Epiphany and a simple WebKitGTK project).

Steps to reproduce:
- go to https://soundcloud.com
- try to log in
after entering your password, you will get prompted with "Our robots think you are a robot. Try reloading the page. If you continue to have this problem, please visit our Help center.". Doing any of the recomended steps doesn't solve the issue.
I'll be creating a ticket on SoundCloud's end, but I doubt it will get resolved, because it'd require them to "reduce" their "security" in order to support a very small subset of browsers.

From what I have found, the issue lies in the way that those 3rd party solutions rely on deprecated features like Navigator.plugins (https://developer.mozilla.org/en-US/docs/Web/API/Navigator/plugins) and others, which can be seen in bot-tests like https://bot.sannysoft.com and https://arh.antoinevastel.com/bots.

PS: I've stumbled upon https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth, which (although for puppeteer) is a great source of information on what these services use to determine whether something is a bot or not.

Comment 1 Michael Catanzaro 2022-07-23 10:34:19 PDT

Thanks for reporting this.

Usually these websites can be fixed by simply adding a user agent quirk. But it's certainly possible you've found the first case that will require something tougher.

Comment 2 Michael Catanzaro 2022-07-23 11:22:13 PDT

Quick summary of discussion on Matrix: we think user agent quirks will not work here. We are not sure specifically what they use to decide to discriminate against us. My pet theory is TLS handshake fingerprinting is most likely, but Bartek proposed a bunch of other possible ways, so who knows.

(In reply to Bartek Sabat from comment #0)
> I'll be creating a ticket on SoundCloud's end, but I doubt it will get
> resolved, because it'd require them to "reduce" their "security" in order to
> support a very small subset of browsers.

You're likely right, but we can hope for better. Make sure they understand that WebKitGTK is part of upstream WebKit, and maybe point them to this bug report so they understand we are discussing.

Currently we do not have a policy for how WebKit should deal with such issues. I will soon propose an antidiscrimination policy to help WebKit address such issues in a more aggressive manner.

Comment 3 Brent Fulgham 2022-09-29 14:19:29 PDT

Some aspect of this might be improved by the Navigator.plugin changes to match spec in Bug 245396.