Search engines indexing IDX sites Part I: What’s the beef?

Just under two months ago, the proverbial stuff hit the fan when a story surfaced that the Metropolitan Indianapolis Board of REALTORS® (MIBOR) told a broker to prevent Google from indexing her IDX web site, citing a common IDX rule:

Participants must protect IDX information from misappropriation by employing reasonable efforts to monitor and prevent “scraping” or other unauthorized accessing, reproduction, or use of the MLS database. (Section 18.2.2 of NAR’s model rules, NAR Handbook on Multiple Listing Policy, 2009 ed.)

MIBOR had confirmed its interpretation by contacting NAR’s technology and policy staff: In their view, what Google does is technologically indistinguishable from ‘scraping.’

What followed was a flurry of blog posts and comments. Some criticized MIBOR’s decision as failing to understand Internet marketing and the importance search engines play in real estate search, accused MIBOR and NAR of seeking to favor Realtor.com and MLS public-facing web sites at the expense of broker-owned sites, and accused NAR and MIBOR of not understanding the technology (good example is Rowles). Most criticized the MLS and NAR, a few took more neutral tones (e.g., Matt Cohen, Mike Wurzer), and hardly anyone dared to defend the associations (but see Victor Lund and an anonymous agent on R.O.B.).

NAR’s multiple listing policy committee met in May in Washington DC and considered a hastily prepared clarification to the rules that would have changed 18.2.2 to read:

Participants must protect IDX information from unauthorized uses. This requirement does not prohibit indexing of IDX sites by search engines.

The policy committee recommended that the NAR board of directors adopt the proposed change. The board of directors accepted a suggestion from a MIBOR representative to postpone the decision until its next meeting in November. This put NAR in for another round of lambasting.

I don’t know whether MIBOR’s approach is the right one or not. I do know that there are good reasons for considering that approach, and I plan to explore them in this and the next several posts here.

In summary, I think we will see that allowing a broker to use an IDX site to ‘fish’ for web indexing results in at least three risks/harms:

1. Listing brokers’ expectations about how their peers will use their listings are not met because this use of IDX data is not consistent with the purposes of IDX as they were previously articulated, and the only remedy currently open to listing brokers is opting out of IDX (which means they can’t have IDX sites of their own);
2. Making an IDX site ‘indexable’ also makes it easier for nefarious data ‘scrapers’ to collect data from it; and
3. As noted elsewhere, redirecting searches related to certain listings or neighborhoods based on SEO practices rather than broker knowledge does a disservice to consumers.

As usual, though, I’ll make you sit through a few posts with some background information and to establish some terminology before we get to the payload…

I want to talk first about what a search engine is, discuss how web search engines like Google index the web, and consider search engine audiences and their objectives. Then I’d like to describe the SEO technique I call “IDX index fishing” and discuss why I think the existing MIBOR/NAR view may be justified. Lastly, I’d like to talk about solutions other than those NAR has considered so far.

I’m breaking this into pretty small pieces, because it’s easier for me to edit that way, and I’ve frequently been reminded that most of y’all do not want to read five-page blog posts. Along the way, I encourage comments and private emails to correct me, argue with me, etc. You know the routine.

-Brian