Search engines & IDX Part IV: “IDX index fishing”

IDX sites are not naturally prone to be indexed by web search engines like Google. Here’s why:

Typical IDX search engines ‘draw’ pages of listing data dynamically or ‘on the fly.’ When the consumer enters her criteria on the search page, the IDX search engine goes into the database and pulls the data for matching listings. The IDX search engine then formats the listing data for display on the IDX web page. Before the consumer’s search, there was no web page displaying the listing at 123 Elm Street. It was only when the consumer executed her search based on criteria that included 123 Elm Street that a page with information about the listing was drawn.

When Google’s web crawlers encounter a web page for a typical IDX search engine, they index the contents of the IDX search engine’s search page. But the Google web crawler does not execute searches on the IDX search engine, and consequently, the listings on the IDX site are not indexed at Google.

Clever brokers and technologists have found a way around this problem. They have their IDX sites draw a static page for each of the listings in the IDX database, and they may include links to all those pages from a ‘site map’ page on the IDX site. So every day, the IDX site creates a slough of pages, one for every listing in the IDX database. The pages are drawn in such a way as to be easy for Google to index. The display in question may be nothing like what is drawn for the consumer when she executes her search, and in fact, most of those pages will never be seen by any visiting consumer. That’s not important to the site, though – what’s important is web search engine optimization.

The first time I saw this practice, I was impressed by its cleverness. I was shown it by an industry consultant. The consultant had me visit the web site of ABC Realty and look at one of the listings of its agent Nancy Smith at 123 Elm St. (All names and addresses are changed in this example.) The consultant then directed me to Google. He said, “Imagine you saw the ‘for sale’ sign, remembered it was on Elm Street, and that the listing agent was Nancy Smith.” So, I typed “nancy smith elm street” into the Google search engine. The top-matching entry (of the relevancy matches – there were no sponsored links) was for a page on Clever Realty, Inc.’s website. I clicked on the link, and up popped Nancy’s listing, but on Clever Realty’s site, not Nancy’s site or her broker’s. ABC Realty was credited as the listing broker (as required by the IDX rules) but all the contact information on the site was for Clever Realty and its agents.

This is what I call “IDX index fishing.” And it is this practice that makes Google indexing of IDX valuable to the brokers who operate the sites (and the vendors who sell them their technology). To see how it works, try it with one of your own listings: Go to Google; for search criteria, use your name and the street name of one of your listings.

I cannot think of any reason for an IDX site to offer a static page for each IDX listing and a site map for those pages except to fish for web search engine indexing. Consumers do not browse listings by listing number or even street name (usually).

Note that IDX index fishing is not illegal or improper under the IDX rules of most MLSs, as long as the resulting static pages comply with the MLS rules. Note also that IDX sites are not the only sites that do index fishing – Realtor.com and most national aggregators also do it.

Let’s look at another example:

1. I was looking for a cabin/lake home up north in Minnesota. I knew that I was interested in lakes around the town of Moose Lake. I found the name of Majestic Pine Realty, a Moose Lake broker, via a banner ad on a non-real estate site. So I went to Majestic Pine’s site at http://www.majesticpine.com/ and viewed listings.
2. I found a listing I liked – http://www.majesticpine.com/viewListing.aspx?LID=237. I noted that it was on Soper Lake. I don’t know Soper Lake, so I wanted to find out more about the body of water.
3. I googled “Soper Lake mn”.
4. Here’s where things get interesting. The first three matches of my Google search were these:
   1. Soper Lake Fishing in Carlton County, Minnesota Fishing Works
      

      Soper Lake is a lake in Carlton County in the state of Minnesota. The latitude and longitude coordinates for this lake are 46.4272, -92.4916.
      www.fishingworks.com/lakes/minnesota/carlton/holyoke/soper–lake/ – Cached – Similar pages –
      

   2. 1939 Soper Lake Ln, Holyoke, MN, 55749 – MLS ID#145561 – Single …
      

      1939 Soper Lake Ln, Holyoke, MN, 55749 – Single Family Home – MLS ID #145561. 2 Bed, 1 Bath. Real estate listings and homes for sale on REALTOR.com®
      www.realtor.com/realestateandhomes-detail/1939-Soper–Lake-Ln_Holyoke_MN_55749_1108124784 – Cached – Similar pages –
      

      Holyoke, MN, Real Estate Listings and Holyoke Single Family Home …
      

      Photo of 1939 Soper Lake Ln, Holyoke, MN 55749. $149900. 2 Bed, 1 Bath, 984 Sq Ft on 5.67 Acres. Property Type: Single Family Home. Save Listing …
      www.realtor.com/realestateandhomes-search/Holyoke_MN/type-single-family-home – Cached – Similar pages –
      More results from www.realtor.com »
      

   3. 1939 Soper Lake Ln, Holyoke MN 55749, MLS #145561, Weichert.com
      

      Features of this property include 1 level living, tongue and groove pine throughout, open kitchen, vinyl energy windows and attached garage.
      www.weichert.com/24732296/ – Cached – Similar pages –
      

5. I was gratified to find the first link, which told me a little about the lake I was interested in. But I was intrigued by the next two matches – one for Realtor.com, and one for the IDX site of a Weichert office in Minnesota, but both referring to my listing (actually, to Majestic Pine’s listing)…
6. I then went to Google Images and did the same search (looking for pictures of my lake). Here are the first results:

1939 Soper Lake Ln, Holyoke, MN, … 310 x 233 – 25k – jpg www.realtor.com [ More from p.rdcpix.com ]		1939 Soper Lake Ln 320 x 240 – 16k – jpg www.weichert.com [ More from images3.weichert.com ]		1939 Soper Lake Ln 320 x 240 – 24k – jpg www.weichert.com

7. Again I got Realtor.com and the Weichert site. This time I decided to check out the links. I clicked on the Realtor.com link and ended up with a view of the listing with the listing broker’s name and phone number, but no link to back to the listing broker’s page (I guess you have to pay extra for the site the Realtor family owns to link back to the listing broker’s site). In fairness, there was no link to anyone else’s site, either. On the Weichert page, I saw the name of the listing broker (pursuant to IDX rules), but all contact info and links go to Weichert.

8. I searched “soper lake mn” on bing and got similar (though by no means identical) results.

9. I decided to try one further check. What if I remembered just the town and number of bedrooms of my original listing? So I Googled “Holyoke mn 3br” – the listing I’m interested in comes up, displayed on Realtor.com (first matching result) and the Weichert site (third matching result.)

I think we’ve established that some brokers with IDX sites are intentionally using the listings of other brokers in IDX to fish for indexing by Google and other web search engines. In fact, I think that’s the only reasonable explanation for the static pages and site maps on some of these brokers’ sites.

The next question, then, is “So what?” If Realtor.com and third party sites like Trulia and Zillow can fish for web indexing, why can’t broker IDX sites? I guess the answer depends on what you think listing brokers’ rights are and what their expectations are about how other brokers will use their listings in IDX.

-Brian