jeff wrote:
“They are no different than any other client except that they wanted to choose their own domain for linking. And as far as our merchant HawaiianActivites being our “top” merchant according to Project Black Book… let me say that–with no offense to HawaiianActivities.com who is certainly a client we love like all of our clients–they are not our top merchant, and are barely in the top 100.”

You must not have read any bit of the magazine nor subsequent posts on both our PBB blog and on other blogger sites where i provided better/more disclosure on our methodologies.

It is clearly stated that the Top Lists, don’t measure top merchants based on commissions paid out, click traffic, etc… the numbers represent the raw data of the popularity a merchant domain has, based solely on the analysis of number of unique domains and links to the merchant (translated through the third party).

Those sites that run utillize the software of your company that have direct links to their own domain, naturally do not show up.

There is no interest in doing specific dissection of all the merchants that are using your software (now, a competitor may like that information, which can be acquired via t3reports.com, but PBB is looking at the affiliate networks).

If you would like to provide to us who your top merchants are based on clicks, commissions, etc.. we would certainly publish those numbers..since it would offer a good contrast between the “linking popularity” relationships between merchants and domains.

Should I have someone from my staff contact your company about getting this data?

I realize there is a sensitivity that networks have towards not revealing actual data…. much as you have the right to not publish the information, those seeking that information to understanding, make business decisions, etc hunger for the metrics and the analysis.

As far as the CJ issue goes on merchants and merchants IDs, already been posted with straightfoward commentary.

I welcome the constructive critisms and the spotlighting of the flaws. Evolutions and Revolutions are ongoing and evolving, and so is our data, analysis, and journal.

How do you feel about companies like Alexa or Media Metrix that have data on your websites?

Alexa data is skewed and could show low visitors, but in actuality to your data, is high.. Should alexa be damned for publishing that information?

Should businesses who use that information to make decisions be damned as well?

If an analytical company extrapolates or interpolates traffic that comes out to be essentially made up numbers, do you damn those companies as well since they don’t know the real info?

All of this is like a horse race. Bettors use all kinds of qualitative and quantitive means to make an informed guess.

We will always disclose how the information was acquired, what the sample size is (something that other analytical companies don’t disclose), and present the data in its raw form, such that others can take the data and put it into context, which is the transformation of data into information.

The genie is out of the bottle. Businesses want transparency and truth, especially those coming from the offline advertising worlds to move to online.

There is auditing and transparency in offline, and they are looking for the equivalent on the online.

If networks have something to hide, then that’s an internal issue.

Much like our focus and mission, the goal isn’t to knockdown any affiliate program or network, it’s to illuminate what is going on, such that it gives businesses the data and some insight, along with their other data sources, to make their best informed decisions.

Like with all disruptive technologies, there will be those that try to subvert or try to hide from it.

I can personally back up the methodology and the data with any level of scrutiny to show that what you see is what you get.

if you don’t like the numbers, offer up your own and we’ll print it as well.

I won’t say much more about this as I’ve spent too much time on it already. I have no problem with any third party service trying to pull data and printing it; I do have a problem with a publisher defending their faulty interpretation of that data when it is obviously intended as a comparison between providers, and trying to spin responses instead of just saying, “We messed up, we’re sorry, and we’ll add corrections to the pdf and will not make this mistake again.” Do yourself and your readers a favor and fix the mistakes instead of arguing that they aren’t really mistakes.

Brian Littleton wrote:
“I also re-downloaded it today to see what corrections were made after I read your statement, but ”

I didn’t say that i updated the PDF. I posted a blog entry on our website for immediate clarifications to those who come to the site and acquire the PDF.

When I return back home from Ad Tech, I will put notations within the PDF to direct people to a page that explains the methodology (like my first blog post on our site), as well as to add the word “ID”.

On page 11 of my “Stat of Something New” article, i did state “Number of unique merchant IDs found”. The omission of the “ID” on the charts is clear. For those that d/l the PDF, I will send an email to those people to point to a clarification web page once changes have been made in the PDF.

Brian Littleton wrote:
“…the information that was published was wildly innaccurate and though I did use some industry experience to come to that conclusion,”

How is the data inaccurate? It is the raw data. The parsing of the CJ links for the stat on “unique merchant ID” is clearly wrong, given feedback that what i perceived as “unique merchant ID”, were creatives ID, etc.. but the next stat of rollig up all those IDs to get to the number of unique merchants, will be much more interesting.

Those that have inside access to Networks, can easily look at their own “Top List”, which is based on clicks, conversions, and commissions, and may chuckle at our list, seeing how it clearly does not match their list (as Jeff Doak pointed out about hawaiiactivities.com).

But our Top List is based on popularity, the raw data of affiliates who have decided to promote that merchant.

If networks wanted to share their Top Lists, that would be great. It would not replace our stats, it would be another data point for the reader to use.

The popularity index (maybe that’s what i should name the stat instead of Top List), has great validity to the reader.

The “top list” may be “inaccurate” to what the actual “top list” is for a Network, but the data is “accurate” for what it shows. If it weren’t for our new publication, there would never be insights revealed on Networks, and given the great feedback so far, people are hungry for this kind of data.

Brian Littleton wrote:
“On one other note, in looking at your “blue bar” graphs that indicate the top or most “popular” merchant of a given netwok…should I assume that this is to scale.. in other words, should I assume that the Orbitz program on LinkShare has approximately 7 times more links out there than Target? Zappos has twice as many as eBay, and some 10-15 times more than Buy.com?

Is this data accurate?”

The blue and white bars are to the scale of the Max legend values at the top of each graph.

The data is accurate as far as what the webspiders found, given the snapshot of data that was used to produce the charts.

The sample set was over the last 8 months from 1B records. The data has never been presented as the definite number of affiliate domains promoting merchants.

Take the data as a sampling (a modest sized one), that illustrates through all the random spidering, the summation of the linking relationships.

Tomorrow, I install 6 new web spiders that should be able to process 1M+ domains a day (i have 34 other servers that can be added, but i think i would be on data overload if i put those online).

We have 38M subdomain/domains in our tracking database, and the goal is to effectively hit every domain in a month, such that when we produce the next round of stats, that we can say something like:

“The sampling set of data to produce the summary numbers was generated from ##M domains, ###,###,### links from May 1,2006 - May 31, 2006″

What will be very interesting to see, is to be able to spot when new merchants hit the networks, and be able to watch their growth, documenting the “popularity” of the merchant as affiliates promote the merchant.

Jim Kukral wrote:
“Errors happen, but you need to understand that you are pitching yourself and your product as a pretty hefty authority, and when you do that, and your initial release has some questionable ommissions and errors, well, that’s going to play on your brand and people are going to question you.”

I do understand this, and all the constructive criticism is taken seriously. While this kind of information has probably never been seen before due to the holding close to the chest of the cards by Networks, i do understand that some people can look at the data we provide and start to make value judgements such as “XYZ has this number and ABC has that number, so therefore XYZ is better”.

There are false conclusions that people can come up with that are beyond our control, but basing their conclusions on mispresentation or errors (such as the ommission of the “ID” word on the Network stats), is what we can control.

I have a debrief meeting with the team this morning, where we will review this preview issue.

There are many changes that will be made based on feedback, and i have appreciated all comments.

There is no turning back, PBB is a reality, and the publication will refine itself, define itself, expand, evolve, and grow.

With this first public release, we will also work more closely with industry leaders and experts, giving them data, and letting them kick the tires to come up with their interpretations.

The data that I am seeing now about Azoogle leads being used in Google Ad Sense seems to be interesting to those that I have asked about it, and being able to document the numbers, occurrences, etc.. may shed some light on the Adsense area that might offer new insight.

Thanks Brandon