LookSmart asks You to Crawl the Web With Grub
You Can Participate in Crawling the Web! From
Your PC!
by Mike Banks Valentine
Oh, for the simplicity of the good old days of
1995! Where all a web site owner had to do was
put important keywords about her web site in the
metatags of the HTML code of the front page of
her site with a clear description and title to
rank well in the search engines and make it easy
for web searchers to find her.
What made it become so difficult to gain and maintain
a bit of internet visibility? It got crowded on
the web. Crowds made it more difficult to get
noticed - so instead of simply including proper
metatags, web site owners had to begin to make
themselves stand out to get noticed. Many did
that by resorting to trickery, like keyword stuffing
and link farms. Since then, it has become increasingly
difficult to get a web site indexed and ranked
well at the major search engines. Strategies have
evolved that seem to work, then mergers and partnerships
change and so do all of the rules.
It never ends and the confusion grows with each
passing day.
It seemed that paid inclusion and pay-per-click
were the only ways to get your site listed anywhere
besides Google lately, let alone ranked well.
LookSmart has come up with a brilliant idea to
grow their own index that, if it catches on, will
offer huge benefits to both LookSmart and to the
participants in the program. That seems to be
the elusive win-win situation that all but Google
have been looking for. It's called distributed
web crawling and gives web site owners a hand
in growing the Looksmart index as well as the
ability to include their own site in the results!
This win-win situation requires participants to
download the "Grub" distributed crawler
client to their PC and allow it to run when the
computer is otherwise idle. The software to run
a LookSmart Grub web crawler is available at:
http://www.grub.org
It is based on open source software and is made
available to all to distribute and even modify!
Since installing "Grubby" on my own
PC three days ago, I've personally contributed
over 100,000 URL's to the LookSmart database!
It's fascinating to watch and the GUI is both
entertaining and attractive. It can run invisibly
when minimized in the system tray or you can sit
and watch as the URL's fly by during a crawl from
within two available views on your PC desktop.
I love the concept and plan to continue helping
LookSmart grow their database. I've also taken
advantage of the option to have "Grubby"
crawl both my own sites and those of clients by
including a small text file with a digital ID
in the root directory of each server I have access
to.
There are no drawbacks here, just benefits. I
help "Grubby" to crawl the web for LookSmart
and Grubby helps me to get my own sites listed
there by agreeing to crawl them regularly.
No payment involved other than the donated bandwidth
and the PC computing power that would be running
a silly screen saver otherwise. They even provide
a cool screensaver of their own that simulates
the crawling process with colorful orbs that represent
the sites you are crawling to help weave the web!
When you visit the front page at the URL above,
you see the latest statistics on your contribution
to crawling the web.
They've even introduced a measure of competition
into the program by showing top contributors and
ranking them. Sunday the stats at the top of the
page read, "323 clients running - crawling
24,355,058 URLs in the last 24 hours." It
will be interesting and very telling to see if
that number grows significantly or very soon because
that will be some measure of the success of the
program over time.
This process and the software all seem to harken
back to friendlier times on the web where community
mattered more than profit. This could even threaten
the mighty Google if it spreads, since we all
love to participate in things that interest us
and LookSmart has a really interesting little
Grub helping to grow the web from my home office
and yours.
------------------------------------------------------------
Mike Banks Valentine is a Search Engine Optimization
specialist practicing ethical small business SEO
http://SEOptimism.com/
The
Linking Matters Report
A free report to help website owners and publishers
plan effective linking strategies and boost link
popularity - includes guidelines, worksheets and
links to over 50 articles on linking.
Return
to FREE articles index
|
Visit Our Sponsor! WebPosition Gold