Comments on: How Spiders Fetch & Classify Data

By: Trontastic

Trontastic — Tue, 29 Dec 2009 16:24:51 +0000

Its only going to be a matter of time before Google really starts to put the smack-down on article spinning. With personalized search, LSI, and what they are doing with OCR Google has clearly displayed the ability to stay ahead of the curve.

Okay, leave me alone now. I have to go scrape a few thousand pages before lunch.

By: Raaj

Raaj — Mon, 09 Nov 2009 07:28:40 +0000

Great article. I could never imagaine that Google bots are getting smarter to this extent.

By: Still

Still — Sun, 31 May 2009 19:56:25 +0000

I know I’m all late to the party, but I just wanted to give a quick shoutout –I had gotten a lil complacent in my day-to-day and your blog has made me step back and THINK again.

Especially your EPN experiment! WOW!

**jotting off to spend some quality time at WF…I’ve obviously been missing out…

Thanks Bruh!

By: Mark

Mark — Tue, 26 May 2009 15:35:03 +0000

I think Google is looking at Javascript more in the case when it’s used to trigger circumstances for following pages. So, working out what it needs to pass to get some output and see what the user is seeing.

The problem with an “open-ended” approach to decoding JS/Java is the almost unlimited variations of application. You can do some incredibly advanced stuff with JS and especially Java. If Google started probing large portions of (full app style) code, they would need some cut-off. Can you imagine the amount of resource it would take to fully test inputs/outputs on an unknown app? Not to mention, if this were the case there would be all kinds of ways to trap and trick bots.

I think a full-execution would open up so many more flaws (and costs) than it would reap in benefit.

My guess (and it is just that), is that they look for footprints of “known” code, which will have some kind of predictibility in what input/outputs are going to be and quickly run them, rather than bots that are intelligently trying to decipher how some specific code works / what it is outputting.

Bare in mind pokerbots are very bespoke bits of software.

By: Contempt

Contempt — Tue, 19 May 2009 02:04:59 +0000

Good call Mark. Yeah, there really wasn’t an “ohhh” moment with any of this. But you bring up a good point.

Do you think Google spiders have the ability to probe JS or Java the way that a pokerbot reads off of the poker client? Reading the actual IO and display streams to the screen to decipher what it all is/means?

By: Mark

Mark — Mon, 18 May 2009 11:04:53 +0000

@xentech – robots.txt will request something not to appear in public index, but certainly doesn’t mean it isn’t crawled.

Back to the post. Unless I’m missing something, I was waiting for the big “ohhh” and it never came.

Google have publically annouced that:

1) They can crawl Flash and trying to do so.

2) Can execute Javascript and have been rolling this out.

I didn’t get any insight into Contempt’s thoughts on, for instance – what is the point of a Googlebot joining a Java IRC channel? Think that was just an observed accident?

I was looking through some of David Sontag’s papers on machine learning (http://people.csail.mit.edu/dsontag/) and it would strike me, even using high end stuff like his – you’d have trouble trying to categorize or order any data that came from pluggin around inside unknown Java.

There’s a whole minefield of problems to deal with.

By: xentech

xentech — Tue, 12 May 2009 08:46:35 +0000

I always put my jQuery in /js and disable it in robots.txt, not sure of it’s effectiveness though.

By: Contempt

Contempt — Tue, 12 May 2009 06:28:04 +0000

Hell chea.

By: Chris Monty

Chris Monty — Mon, 11 May 2009 02:13:52 +0000

Hey, nice one. I actually managed to escape from the banking world and get a job as an SEO for a local sports marketing firm. We’ll have to grab lunch soon. I’m off to read your php code post about unique content.