Skip to main content

Google Gives Publishers More Control Over How It Crawls Their News




Yesterday, Google threw complaining publishers a bone with its First Click Free program, which lets news sites limit the number of free clicks from Google News for any individual to five a day. News sites have long been accusing Google of profiting off of their news with Google News but today Google is making another concession to publishers.

Google is launching a new crawler that will let publishers keep their content out of Google News and still remain in Google Search. Publishers have always been able to do this by filling out a contact form but now Google is making it easier by automating the technology with a news-crawler.



Yesterday, Google threw complaining publishers a bone with its First Click Free program, which lets news sites limit the number of free clicks from Google News for any individual to five a day. News sites have long been accusing Google of profiting off of their news with Google News but today Google is making another concession to publishers.


Google is launching a new crawler that will let publishers keep their content out of Google News and still remain in Google Search. Publishers have always been able to do this by filling out a contact form but now Google is making it easier by automating the technology with a news-crawler.


Currently, publishers can block Google from including their content in Google’s main index via a Robots Exclusion Protocol (or REP). When Google’s crawler arrives at any site, it checks to see if there’s a robots.txt file to make sure the search engine has permission to crawl the site. This gives publishers the option to block an entire site or certain sections or pages.


Google is applying this technology specifically to News. News sites can now block images from thew news crawlers but not from general web search. Or on the flipside, a news site could choose to index their content on Google News but not Google’s main search index. If a publisher decides to opt out of Google News but still wants to be indexed on Google Search, content will show up in search results, but won’t appear in the block of news results that sometimes shows up in Web Search.


Publishers stand to gain from indexing on Google news, with News sending publishers about 1 billion clicks every month (and 4 billion clicks per month from all of Google). Josh Cohen, Senior Product Manager For Google News writes:


Each of those clicks is an opportunity for publishers, allowing them to show ads, sell subscriptions and introduce readers to the great content they produce every day. While we think this offers a tremendous opportunity for any publisher who wants new readers, publishers are the ones who create the content and they’re in control of it. If they decide they don’t want to be in Google, it’s easy to do.


The new crawler gives publishers like Rupert Murdoch even more ways to block their news from Google. It’s a Christmas wish come true.


Crunch Network: CrunchBase the free database of technology companies, people, and investors












Comments

Post a Comment

Popular posts from this blog

How to find ideas to post new article in your blog

How to find ideas to post new article in your blog    阅读原文»   It is true that sometimes being a blogger may face situations where I would personally like to call it your brain juices got dried up as you have pretty much ran out of topic to blog and you are in crisis as your readers are anxiously waiting for your new posts but you are unable to give in. That’s when you will probably come with excuses like I just posted last week although that post was more directly towards the newbies who stop themselves from making money but it’s still pretty much the same even though you consider yourself not a newbie. The fact is that ideas are everywhere and I mean everywhere if you know where to find it and know how to leverage it. You may be surprised that sometimes these ideas are just right in front of you but you are not observant enough to convert these ideas and turn it into your blog post. Today I will share some tips on where to get these ideas and...
101 comments
Read more

Over A Year After Android Launch, ShopSavvy Finally Comes To The iPhone

ShopSavvy was one of the best early Android applications. It launched in October of last year after winning one of the initial Android Developer Challenge top prizes (when it was still known as GoCart). But despite the success it has seen on Android, one question remained: When would it be available for the iPhone. Today, it finally is. Developed by the guys at Big In Japan , ShopSavvy is an app that allows you to use your device as a portable barcode scanner. You point your phone's camera at any barcode and it will read it, do a product look up, and give you information about the product, as well as where you can find it online or at nearby stores and for how much. Obviously, something like this is a window shopper's dream. ShopSavvy was one of the best early Android applications. It launched in October of last year after winning one of the initial Android Developer Challenge top prizes (when it was still known as GoCart). But despite the success it has seen on Android, o...
8 comments
Read more

Lindsay Lohan Breakup Confirmed by Lohan, Locksmith, Police [Gossip Roundup]

Farewell, last season's Suri Cruise fashions. Goodbye, Amy Winehouse's bathing suit. Adieu, humanoid version of Lauren Conrad. And so long, LiLo and SamRon's fairytale romance. Lindsay Lohan confirmed her split with Samantha Ronson and insisted the decision was part of a very healthy and mature effort to " focus on myself ." Upon hearing this, Ronson changed her locks and discussed a restraining order with police, so confident was she in Lohan's ability to turn productively inward. Lohan promptly had a run-in with the police . Who would have imagined such a messy breakup for this model relationship? Courtney Love's lawyer, on her client's drug-fueled plunge into broke-ness: ""Courtney noticed the money was gone when there wasn't any left." Deadpan gallows humor: the only possible response to having Courtney Love as a client. (Besides asking for a hefty retainer.) [ P6 ] Before Lauren Conrad's contract expired in March, MTV...
39 comments
Read more