Forum Challenges

[TazKristi]

Hey, everyone:
We're aware of and still working on the challenges seen on the forum within the last week or so. There is an incredibly aggressive web crawler hitting the forum. Web crawlers are usually pretty benign and don't use many resources - usually. This one is tricky. It's hitting so fast and so frequently that it's slowing down the forum intermittently.

We've taken steps to mitigate the issue, but it is persistent (to put it mildly), so we're continuing to work the problem. One step will be an update that was just released. Sometime in the next few days, the forum will go down for up to 90 minutes to install that update. It should be at night when there are fewer people on the forum.

Thank you for your patience and understanding!

Kristi

[Ankerson]

Thanks for the update. :)

[JSumm]

Thanks Krist!

[riclaw]

Thanks and good luck.

[Cycletroll]

Thanks for the update Kristi!
Good luck!

[RustyIron]

Sounds even more stressful than waiting to go live on Mule Drop Day.

[WilliamMunny]

Awesome, thanks for the update and effort for this forum.

On a side note will you be posting something about the Seconds sale on the 18th on the forum? Just odd to see no official announcements here but seeing them on social medial for a few days.

[Meadowlark]

Dumb question:

What's a web crawler and what's the intention of it?

Thanks.

[RustyIron]

What's a web crawler and what's the intention of it?

Hey, Meadowlark.
A web crawler is an automated web browsing robot that visits every web page on the internet and downloads the information. Google does this. Bing does this. All the search engines do this so they can index the data and provide relevant results to our search requests.

If you scroll to the bottom of this web page, you might see a list of "online users." More than likely, you'll see Google or Bing looking at this very forum. It's what they do. I suspect that there are scores of bad actors running web crawlers that look for vulnerabilities and opportunities to steal information that they shouldn't have.

A well behaved web crawler will "throttle" its activity, limiting the number of web pages it will load in a certain period of time. A rogue web crawler will just try to download the maximum number of web pages in the shortest period of time. It's like a hundred users clicking new web pages every second. The effect can be similar to a Denial of Service attack, or like 9:00 a.m. on Mule Drop Day.

[Meadowlark]

Huh.. that I never knew, thanks rusty.

[kennbr34]

I usually use fail2ban to to handle those