Affiliate Data Feed Delivery via HTTP

I was updating and expanding my affiliate product data feeds and APIs resources at Cumbrowski.com, in particular the resources to the product feeds and store builder feature provided by the PepperJam affiliate network, which was launched about one year ago.

I liked what I saw overall, although some items were, in my opinion a great idea, but then implemented half way only, probably without intending it.

Free? Hurray!

Product data are available to any affiliate without any cost directly via the PepperJam Network web interface. There is also a coupon feed available, in delimited format and RSS format (using some custom tags outside the RSS definition). They also have a tool called “store builder” (product show-case creator), which is similar to the tools available through third party tools providers like PopShops.com, GoldenCan.com, DataFeedFile.com or FeedShare.com.

Data Delivery via HTTP

Providing the feeds directly via HTTP, accessible via any web browser is a smart, efficient and cost effective solution. There is no need to charge any fee for setup or maintenance, because there is really nothing to setup. Via a, what you could call, “special page” are the product data rendered into the web browser in simple delimited text format, without HTML tags or anything like that. This is close to the format how the networks developer gets the same information out of their product catalog database, without the need for the developer to write a fancy, functional and error free user interface to browse the products.

The product information can be pulled and rendered into the browser in real-time.

Large Feeds

Since no FTP account needs to be created and no static product feed files created to be picked up by the affiliate via FTP client software, a whole new set of options become available and possible. No disk space is wasted on files that are not being picked up and no server CPU is spending a single cycle of processing power on it, until the affiliate publisher actually requests it.

True, there are some challenges, if you use HTTP as delivery method, if the amount of product s requested and downloaded is very large. However, 99% of all merchant product feeds do not fall into this category and even the merchants who have such large feeds (e.g. Buy.com, Overstock.com or Walmart.com) learned that it makes sense to break up their huge 1 million+ products big data feeds into smaller chunks, because it is never an easy undertaking to process over 1 million product records all at once, regardless if you download them via FTP or not.

Since HTTP requests can be handled and processed in real-time, the option to provide filters to reduce the amount of data returned, is now suddenly also (or should be) of the interest of the network.

It used to be an interest of the affiliates only (for the most part). I remember the time when I downloaded gigabytes of junk data every day, because there was only the choice between “everything” and “nothing”. Well, nothing was really an option, so you had to deal with all the overhead that you did not need.

Imagine the Possibilities

The real-time factor does allow the setup of virtually an unlimited number of filters. It is sad that affiliate networks who utilize HTTP for product data delivery are not making much use of this. A filter by product category, advertiser and keyword should only be the very least at the beginning.

When I was an affiliate manager in 2003 for a retailer who had the affiliate program with Commission Junction, the first thing I did was the creation of an affiliate intranet where publishers could create an account and pull product information and more. The intranet was initially created to solve the issues of not having contact information of any publisher (a CJ “feature”) , avoiding the $750 setup fee for the product catalog at CJ itself and the up to $250 setup fee for each publisher in our program that did not qualify for the free access to product feeds through CJ.

The intranet required a second registration, which is tough. We had to offer an incentive to do this. That was when we came up with features that were impossible to provide via the traditional network and the way how they did (and to some degree still do) things. For example it was possible to pull the top XXX number of best-selling products for the whole site or specific categories only and also the time-frame of the sales, ranging from “this month” to “all-time”. This enables the discovery of seasonal trends or sudden “hot” items that are new in the product catalog.

Those are only a few examples. Use your imagination and I am sure that the possible opportunities that come out as a consequence of it would be great, if at least some of the ideas will be implemented.

Web Services instead of Delimited Text via HTTP?

You might say that all this is what the new hot feature “web services” is supposed to be doing.

Yes and No. The border between web services and delimited feeds with several filter options provided via HTTP gets blurry, no doubt about that, but the technical details how similar they might appear are still different. The question whether to use a delimited feed or a web service (if available) still remains to be answered on a case-by-case basis. In some cases a combination of both does make sense.

As with web services, providing other data via HTTP beyond product and coupon data makes sense also, including reporting data or general program information about active or possible merchant/advertising partners.

One thing that PepperJam forgot over the nice and easy access to the product data via HTTP is the need for automated access to the data. HTTP is great and the link provided, updated based on my selections even better, but all this is of no use for many affiliates, if they have to be logged-in to the networks web interface in order to use the URL provided to them.

If I log-off and try the feed URL, no records are returned. Not good!

Automated Access without Login

PepperJam were not the first nor last who made this mistake. I remember LinkConnector.com having the same problem. I contacted them a while ago, suggesting to them to remedy this short-coming. I need to check, if they actually did anything or not. I could not send them sample source code as I did with Kolimbo (MyAffiliateProgram), where I could tell that they are using Microsoft Active Server Pages, where I am familiar with.

The affiliate network AvantLink.com did it right from the start. I wrote about it, when they launched, what they called their “AvantLink API Module“. Well, AvantLink had and has the reputation to be very knowledgeable about the subject of feeds and APIs.

You can have a look at their solution to see how they made it possible to provide access to the data in an automated fashion but at the same time doing it in a controlled manner without the option for anonymous access to the content.

If you do not want or like to see what another affiliate network is doing, fine, how about Google?

If you used the Google Calendar, you probably noticed the feature to pull your private events as a feed via an obscure URL provided by Google with the note that you must not give the URL to anybody, because it would give that person access to your feed as well. The protection is the obscure code in the URL that is virtually impossible to guess by anybody.

In the case that you think that the URL did leak out somehow, the option is available to render that URL invalid and generate a new unique and private URL for the same content.

I think you got the idea behind this.

PepperJam Sample Source Codes

Oh, by the way, the coupon feed provided by PepperJam Network does not require to be logged in to pull it. The lack of tags for start and end date of a promotion in the RSS version of the feed limits its possible uses, but I provided an example with source code for how to use it.

I also put the source code of a Visual Basic Script on my website that works around the problem of the need to be logged in to the web interface in order to pull the product feeds. I added a bunch of command line options to make it as flexible as possible for the use by affiliates to automate the pickup of product data from PepperJam.

Documentation? Configuration?

Last but not least. Is it really that hard to provide some notes about the general structure of the delimited feed (column and row separators, first row having column names or not, escaping of content that contains column or row delimiter) and to each column in the file about its use, possible values and format, which columns are required and always must have appropriate data provided by the merchant (and verified by you, the network)? Since the output data are generated on the fly, how about letting the publisher choose the format himself? We did that at our affiliate intranet too. The feed format preferences were simply stored with the affiliates intranet account. Some folks prefer tab-delimited, others pipe and there are even folks who like to deal with the mess that a CSV feed can create, if created improperly. The Linux guys prefer a simple Line-Feed after each record, the Windows guys a carriage-return plus a line-feed and the Mac guys just a carriage-return. Let each have it the way they like it.

I did as in almost every case educated guesses again by looking at actual product feed data from various selected merchants. But why should I do the guessing, if you are in the know and just forgot to write it down and share with your publisher base? “Should” and “Seems Obvious” is not good enough here. Once you automated the download, import, processing and publishing of a feed, having guessed wrong becomes more than a minor inconvenience, especially if it would have been easily avoidable early on.

Cheers

Carsten Cumbrowski
Internet Marketer, Blogger and Entrepreneur

Cumbrowski.com, the resources portal for internet marketers. The content is free, no strings attached.

About Carsten Cumbrowski

Internet Marketer, Entrepreneur and Blogger. To learn more about me and what I am doing, visit my website and check out the “about” section.

Twitter: ccumbrowski
The following two tabs change content below.
Internet Marketer, Entrepreneur and Blogger. To learn more about me and what I am doing, visit my website and check out the "about" section.

4 Responses to Affiliate Data Feed Delivery via HTTP

  1. Kris Jones says:

    Carsten,

    Outstanding post.

    We appreciate your candidness and will be working to resolve some of the issues and expand on many of the perks of our Pepperjam Store Builder and Pepperjam Coupon Feed technology.

    It's truly unbelievable that you'd spend the amount of time necessary to research and draft a post like this…all in the interest of people who read it.

    Kris

  2. My pleasure Kris. Now its the time for your team to go to work :). See you in Vegas. Cheers!

  3. [...] Affiliate Data Feed Delivery via HTTP | ReveNews [...]

  4. [...] Affiliate Data Feed Delivery via HTTP | ReveNews [...]

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>