Superfeedr Documentation

Summary

  1. Publisher
  2. Subscriber
  3. Schema
  4. Miscellaneous

Publisher

The PubSubHubbubb hubs that Superfeedr hosts are compliant with the core spec. In order to enable the hub as a publisher, you need to perform 2 actions : discovery and ping.

Discovery

Discovery is aimed at informing your current (and future) feed subscribers (who poll your feed) that they can get content your Superfeedr hosted hub. It's as easy as adding the following to your feeds:

  • For RSS
    • <?xml version="1.0"?>
<rss>
<channel>
    <title>...</title>
    <description>...</description>
    <link>...</link>
    
    <!-- PubSubHubbub Discovery -->
    <link rel="hub"  href="http://<your-hub-name>.superfeedr.com/" xmlns="http://www.w3.org/2005/Atom" />
    <link rel="self" href="<your-feed-url>" xmlns="http://www.w3.org/2005/Atom" />
    <!-- End Of PubSubHubbub Discovery -->
    ...
</channel>
</rss>
  • For Atom
    • <?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>...</title>
<link href="..." rel="self" type="application/atom+xml"/>

<!-- PubSubHubbub Discovery -->
<link rel="hub" href="http://<your-hub-name>.superfeedr.com/" />
<!-- End Of PubSubHubbub Discovery -->

<updated>...</updated>
<id>...</id>
...
</feed>

Ping (New Content Notification)

The next step is to ping us whenever you update the content of a feed. This will allow us to fetch this specific feed, identify what is new vs what is old in it and finally push the updates to the subscribers to this feed.

  • Send an POST request to http://<your-hub>.superfeedr.com, with the following params :
    • hub.mode : publish
    • hub.url : the url of the feed that was updated
  • If somebody susbcribed to your feed, then, we will fetch it. You should see a request from Superfeedr in your logs.

At Superfeedr, we wanted to make it simple for you to implement the PubSubHubbub protocol, so if you have implemented any type of ping, please get in touch, as we're probably able to receive these too. For example, our XML-RPC endpoint is here.

Subscriber

If you have any question about this documentation or if you still need help with the Superfeedr, please join our Google Group, read our Frequently Asked Questions. We also strongly suggest the use of our dashboard (you must be logged in) to debug the real-time notification.

Color Code

Outbound (requests) traffic have a grey background.
Inbound (responses + notifications) traffic have a green background.

What API to choose?

Superfeedr offers 2 different API : XMPP PubSub and HTTP PubSubHubbub. They have both purposes for which they've been created and, based on your goals using Superfeedr, you might want to select one or another. Here is a small chart that will help you choose the most appropriate in your case. (In this chart, the 'I' represent the endpoint that will receive the notifications).

Api-decision-chart

XMPP PubSub

You can connect your jabber client (or component) to superfeedr by using the JID login-@superfeedr.com and your superfeedr password. You can also specify another JID in your settings.

The XMPP API is based on the XEP-0060 syntax. Additional data has been included within a superfeedr-specific namespace.

Adding a Feed

  • iq[@from] : should match the bare jid you entered in your settings.
  • iq[@id] : random string (enables you to match the superfeedr response with your query).
  • subscribe[@node] : use the url of the feed you want to monitor.
  • subscribe[@jid] : should be equal to the @[from] or, if you have chose an external component jid, should have the same domain. JID resources will be ignored.
  • subscribe[@digest] : This must belong to the Superfeedr namespace (http://superfeedr.com/xmpp-pubsub-ext). If set to true, we will send daily digests to your endpoint.
<iq type="set" from="you@superfeedr.com" to="firehoser.superfeedr.com" id="sub1">
<pubsub xmlns="http://jabber.org/protocol/pubsub" xmlns:superfeedr="http://superfeedr.com/xmpp-pubsub-ext">
    <subscribe node="http://domain.tld/feed1.xml" jid="you@superfeedr.com"/>    
    <subscribe node="http://domain.tld/feed2.xml" jid="you@superfeedr.com"/>
    <subscribe node="http://domain.tld/feed3.xml" jid="you@superfeedr.com" superfeedr:digest="true" />
</pubsub>
</iq>

You can add up to 30 feeds in your subscription query, as long as they have all the same subscribe[@jid]. If you need to subscribe with multiple jids, we suggest that you send multiple subscription queries.

The server acknowledges the subscription(s) and sends the status information for each feed. 

<iq type="result" to="you@superfeedr.com/home" from="firehoser.superfeedr.com" id="sub1">
<pubsub xmlns="http://jabber.org/protocol/pubsub">
    <subscription jid="you@superfeedr.com" subscription="subscribed" node="http://domain.tld/feed1.xml">
        <status xmlns="http://superfeedr.com/xmpp-pubsub-ext">
            <http code="200">9718 bytes fetched in 1.462708s : 2 new entries.</http>
            <next_fetch>2009-05-10T11:19:38-07:00</next_fetch>
        </status>
    </subscription>
    <subscription jid="you@superfeedr.com" subscription="subscribed" node="http://domain.tld/feed2.xml">
        <status xmlns="http://superfeedr.com/xmpp-pubsub-ext">
            <http code="200">9718 bytes fetched in 1.462708s : 2 new entries.</http>
            <next_fetch>2009-05-10T11:19:38-07:00</next_fetch>
        </status>
    </subscription>
    <subscription jid="you@superfeedr.com" subscription="subscribed" node="http://domain.tld/feed3.xml">
        <status xmlns="http://superfeedr.com/xmpp-pubsub-ext">
            <http code="200">9718 bytes fetched in 1.462708s : 2 new entries.</http>
            <next_fetch>2009-05-10T11:19:38-07:00</next_fetch>
        </status>
    </subscription>
</pubsub>
</iq>

In other cases, you will receive an iq with type="error", please check that your subscription query abide by the constraints explained above.

Removing a Feed

  • iq[@from] : should match the bare jid you entered in your settings.
  • iq[@id] : random string (enables you to match the superfeedr response with your query).
  • unsubscribe[@node] : use the url of the feed you want to stop monitoring.
  • unsubscribe[@jid] : use your 'bare' jid .
<iq type="set" from="you@superfeedr.com/home" to="firehoser.superfeedr.com" id="unsub1">
<pubsub xmlns="http://jabber.org/protocol/pubsub">
    <unsubscribe node="http://domain.tld/feed1.xml" jid="you@superfeedr.com"/>
    <unsubscribe node="http://domain.tld/feed2.xml" jid="you@superfeedr.com"/>
    <unsubscribe node="http://domain.tld/feed3.xml" jid="you@superfeedr.com" />
</pubsub>
</iq>

You can remove up to 30 feeds in your unsubscription query.

The server acknowledges the unsubscription.

<iq type="result" from="firehoser.superfeedr.com" to="you@superfeedr.com/home" id="unsub1" />

Listing Feeds

  • iq[@from] : should match the bare jid you entered in your settings.
  • subscriptions[@page] : page number (subscriptions will be sent by batches of 30).
  • subscribe[@jid] : use your 'bare' jid .
<iq type="get" from="you@superfeedr.com/home" to="firehoser.superfeedr.com" id="subman1">
<pubsub xmlns="http://jabber.org/protocol/pubsub" xmlns:superfeedr="http://superfeedr.com/xmpp-pubsub-ext">
    <subscriptions jid="you@superfeedr.com" superfeedr:page="3"/>
</pubsub>
</iq>

The server sends the list of feed to which you're subscribed for the page requested, as well as the status information for each feed.

<iq type="result"  to="you@superfeedr.com/home" id="subman1" from="firehoser.superfeedr.com">
<pubsub xmlns="http://jabber.org/protocol/pubsub"  xmlns:superfeedr="http://superfeedr.com/xmpp-pubsub-ext" >
    <subscriptions superfeedr:page="3">
        <subscription node="http://domain.tld/a/feed/atom.xml" subscription="subscribed" jid="you@superfeedr.com" >
            <status xmlns="http://superfeedr.com/xmpp-pubsub-ext">
                <http code="200">9718 bytes fetched in 1.462708s : 2 new entries.</http>
                <next_fetch>2009-05-10T11:19:38-07:00</next_fetch>
            </status>
        </subscription>
        <subscription node="http://domain2.tld/feed.rss" subscription="subscribed" jid="you@superfeedr.com" >
            <status xmlns="http://superfeedr.com/xmpp-pubsub-ext">
                <http code="200">9718 bytes fetched in 1.462708s : 2 new entries.</http>
                <next_fetch>2009-05-10T11:19:38-07:00</next_fetch>
            </status>
        </subscription>
    </subscriptions>
</pubsub>
</iq>

Notifications

Once you're tracking a feed, your xmpp client must be connected at all times to receive the new entries published on this feed. We won't store any messages if your client is offline.

You can chose in your settings if you want to receive notifications each time we fetch and parse a feed, or just when there are new entries.

Please see the schema information for details on the status, as well as the feed and entry informations. 

<message from="firehoser.superfeedr.com" to="you@superfeedr.com">
<event xmlns="http://jabber.org/protocol/pubsub#event">
    <status feed="http://domain.tld/feed.xml" xmlns="http://superfeedr.com/xmpp-pubsub-ext">
        <http code="200">9718 bytes fetched in 1.462708s : 2 new entries.</http>
        <next_fetch>2009-05-10T11:19:38-07:00</next_fetch>
        <title>Lorem Ipsum</title>
    </status>
    <items node="http://domain.tld/feed.xml">
        <item >
            <entry xmlns="http://www.w3.org/2005/Atom">
                <title>Soliloquy</title>
                <summary>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.</summary>
                <link rel="alternate" type="text/html" href="http://superfeedr.com/entries/12345789"/>
                <id>tag:domain.tld,2009:Soliloquy-32397</id>
                <published>2010-04-05T11:04:21Z</published>
            </entry>
        </item>
        <item>
            <entry xmlns="http://www.w3.org/2005/Atom">
                <title>Finibus Bonorum et Malorum</title>
                <summary>Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt.</summary>
                <link rel="alternate" type="text/html" href="http://superfeedr.com/entries/12345788"/>
                <id>tag:domain.tld,2009:Finibus-32398</id>
                <published>2010-04-06T08:54:02Z</published>
            </entry>
        </item>
    </items>
</event>
</message>

Digest

If you subscribed to feed with the explicit digest=true parameter, then, once a day you should receive a digest notification which will include up to the 10 last entries published in this feed.

These digest notification include a flag in status element

<message from="firehoser.superfeedr.com" to="you@superfeedr.com">
<event xmlns="http://jabber.org/protocol/pubsub#event">
    <status feed="http://domain.tld/feed.xml" xmlns="http://superfeedr.com/xmpp-pubsub-ext" digest="true">
        ...
    <status>
    ...

API Wrappers

XMPP can be scary. For those of you who don't want to mess with that, we have various wrappers (including Ruby, Perl, Python, Java and PHP) for the XMPP API. You can find them on our Github page. If you need one in another language, please contact us.

PubSubHubbub

Superfeedr is a PubSubHubbub hub; our hub is located at http://superfeedr.com/hubbub. We implement the Core Specification. The biggest advantage of using our PubSubHubbub API is that you can re-use your "subscriber" code with any other hub. You can also use any library that supports and implements pubsubhubub.

Implementation

Our hub supports HTTP Basic Authentication, you need to authenticate your HTTP requests. Here are few things to know about Superfeedr's hub :

  • We do not support hub.lease_seconds, this will be ignored
  • You can obviously use https:// and hub.secret for complete security. (Learn more)

Please read the Subscriber Best Practices and do not hesitate to use tools like curl or Hurl for debugging.

Subscriptions

The syntax for subscribe and unsubscribe is identical. You can also use the sync or async mode, based on the capabilities of your app. Please remember that verification will fail if it takes more than 3seconds.

Please refer to the Core Specification for Subscription. Here is a short step by step :

Please note that all superfeedr hubs can send you daily digest notifications. You can use this as an heartbeat to make sure the subscription is still active and that you haven't missed any entry.

  • Send an POST request to http://superfeedr.com/hubbub, with HTTP Basic Authentication and the following params :
    • hub.mode : subscribe or unsubscribe
    • hub.verify : sync or async
    • hub.callback : http://domain.tld/your/callback
    • hub.topic : http//feed.you.want.to/subscribe/to
    • superfeedr.digest : true if you want to receive daily digest notifications.
  • The hub will send a request to your callback url, to verify your intent : your callback must respond with 200 and must echo the hub.challenge provided by the hub.
  • If the hub returns 204, you're subscribed!, If it returns 202, the hub check the subscription later. If it returns anything else, something was wrong, you may want to check the body of hub's the response.

Notifications

One of the benefits of Superfeedr's hub is that you will benefit from our standard schema mapping. In case of failure, our server will retry 3 times to notify your endpoint. If after this they all failed, the notification will be dropped.

Digest

If you've explicitly subscribed with the superfeedr.digest:true option, you should receive daily digest notifications. These notifications may include up to the last 10 entries published.

The feed element will include a sf:digest=true attribute to indicate that the notification is a digest.

Schema

Whatever the original format (RSS, Atom, or any other namespace) is, the notification that we will send to feed subscribers will use standard ATOM. We will match as much as we can into this format. The overall goal here is to make it easy for the subscriber to consume a consistent schema.

Status Schema

XMPP API Only. When you add a feed, list your feeds and when you receive notifications, Superfeedr will send you some status information for the concerned feed.

  • status[@feed] : contains the URL of the feed.
  • http[@code] : last HTTP status code, please see Status Code Definitions.
  • http : the content of that tag is a more explicit log message for your information.
  • next_fetch : the feed will be fetched at most before this time. A good practice is to keep track of that in your client and 'challenge' us if you see that a given feed wasn't fetched in time.
  • title : the feed title.
<status feed="http://domain.tld/feed.xml" xmlns="http://superfeedr.com/xmpp-pubsub-ext">
<http code="200">9718 bytes fetched in 1.462708s : 2 new entries.</http>
<next_fetch>2009-05-10T11:19:38-07:00</next_fetch>
</status>

Entry Schema

Notification entries will have the following form. It is standard ATOM. Here are the fields that you will see in the notification. Please note that an entry might not have all of them.

At the entry level, you may see a stripped attribute, belonging to the Superfeedr namespace. If set to true, it indicates that the content or summary of this entry was bigger than 1MB and that we stripped it.

Here are the components used to build the entries. Please note that they may use specific namespaces.

Link

  • link[@href] : the url related to the parent node
  • link[@rel] (optional) : the type of relation to that parent node (alternate, reply... etc)
  • link[@type] (optional) : MimeType of the link destination (tex/html by default)
  • link[@title] (optional) : the link title. 
<link href="http://domain.tld/entries/12345" rel="alternate" type="text/html" title="The sky is Blue" />
<link href="http://domain.tld/entries/12345/comments.xml" rel="replies" type="application/atom+xml" title="Comments on The sky is Blue" />

Category

  • category[@term] (optional, multiple) : a keyword related to the entry... (tag, category or topic)
<category term="tag" />
<category term="category" />

Point

  • entry[@point] (optional, multiple) : geolocation data. Contains a georss latitude and longitude related to the story. It's either extracted from the story or extrapolated from the content.
<point xmlns="http://www.georss.org/georss">47.597553 -122.15925</point>

Author

  • author(optional, multiple).
    • name : the author's name (or nickname)
    • email (optional) : the author's email address
    • uri (optional) : the author's URI
    • object-type (optional, multiple) : the object type, defined in the ActivityStreams spec
    • link (optional, multiple) : links (see above). They can include links to the author's profile, to the user's avatar...
<author>
<name>John Doe</name>
<email>john@superfeedr.com</email>
<uri>http://twitter.com/superfeedr</uri>
<as:object-type>http://activitystrea.ms/schema/1.0/person</as:object-type>
<link type="image/png" title="John Doe" href="http://domain.tld/john.png" rel="image"/>
</author>

Object

  • objectActivityStreams (optional, multiple).
    • object-type (required, multiple) : the object type, defined in the ActivityStreams spec
    • id (optional) : the unique identifier of the object
    • title (optional) : the title of the object
    • published (optional) : the publication date (iso8601) of the object.
    • updated (optional) : the updated date (iso8601) of the object.
    • content (optional) : the content of the object.
    • author (optional, multiple) : author information (see above).
    • category (optional, multiple) : categories (see above).
    • link (optional, multiple) : links (see above).
<as:object-type>http://gowalla.com/schema/1.0/spot</as:object-type>
<as:object-type>http://activitystrea.ms/schema/1.0/place</as:object-type>
<id>object-id</id>
<title>Title of the Object</title>
<published>2010-04-20T15:00:40+02:00</published>
<updated>2010-04-21T14:00:40+02:00</updated>
<content>hello world</content>
<author>
<name>Second</name>
<uri>http://domain.tld/second</uri>
<email>second@domain.tld</email>
</author>
<link type="text/html" title="" href="http://domain.tld/object/2" rel="alternate"/>
<link type="text/html" title="" href="http://domain.tld/object/2" rel="alternate"/>

Verb

  • verb(optional, multiple) : defined in the ActivityStreams spec.
http://activitystrea.ms/schema/1.0/post

Entries

Entries may include all the above elements. They also contain specific nodes, listed below.

  • entry[@xml:lang] : The language of the entry. It's either extracted or computed from the content (the longer the content, the more relevant).
  • entry[@title] : The title of the entry.
  • entry[@id] : the Unique ID of the entry. If the original entry doesn't have an ID.
  • entry[@published] : The publication date (iso8601) of the entry.
  • entry[@content] : The content of the entry. Check the type attribute to determine the mime-type.
  • entry[@summary] (optional, unique) : The summary of the entry. Check the type attribute to determine the mime-type
<entry xmlns="http://www.w3.org/2005/Atom" xmlns:geo="http://www.georss.org/georss" xmlns:as="http://activitystrea.ms/spec/1.0/" xml:lang="en">
<id>domain.tld:09/05/03-1</id>
<published>2010-04-21T14:00:40+02:00</published>
<title>Entry published on hour ago</title>
<content type="text">Entry published on hour ago when it was shinny outside, but now it's raining</content>
<geo:point>47.597553 -122.15925</geo:point>
<link type="text/html" title="" href="http://domain.tld/entry/1" rel="alternate"/>
<link type="text/html" title="" href="http://domain.tld/entry/1.xml" rel="replies"/>
<link type="image/png" title="A beautiful picture that illustrates the entry" href="http://domain.tld/entry/image_.png" rel="picture"/>
<category term="Things"/>
<category term="Picture"/>
<author>
    <name>First</name>
    <uri>http://domain.tld/first</uri>
    <email>first@domain.tld</email>
    <id>id-first</id>
    <title>First</title>
    <as:object-type>http://activitystrea.ms/schema/1.0/person</as:object-type>
    <as:object-type>http://activitystrea.ms/schema/1.0/dude</as:object-type>
    <link type="image/png" title="A beautiful picture that illustrates the author" href="http://domain.tld/first.png" rel="picture"/>
    <link type="text/html" title="The profile page of the author" href="http://domain.tld/first/profile" rel="alternate"/>
</author>
<author>
    <name>Second</name>
    <uri>http://domain.tld/second</uri>
    <email>second@domain.tld</email>
</author>
<as:verb>http://activitystrea.ms/schema/1.0/post</as:verb>
<as:verb>http://activitystrea.ms/schema/1.0/publish</as:verb>
<as:object>
    <as:object-type>http://gowalla.com/schema/1.0/spot</as:object-type>
    <as:object-type>http://activitystrea.ms/schema/1.0/place</as:object-type>
    <id>object-id</id>
    <title>Title of the Object</title>
    <published>2010-04-20T15:00:40+02:00</published>
    <updated>2010-04-21T14:00:40+02:00</updated>
    <content>hello world</content>
    <author>
        <name>Second</name>
        <uri>http://domain.tld/second</uri>
        <email>second@domain.tld</email>
    </author>
    <link type="text/html" title="" href="http://domain.tld/object/2" rel="alternate"/>
    <link type="text/html" title="" href="http://domain.tld/object/2" rel="alternate"/>
</as:object>
</entry>

Miscellaneous

Track

More than feeds, you want to susbcribe to keywords and receive notifications for each entry published which includes this keyword? Use our track feeds.

Subscribe to any keyword by subscribe to its corresponding track feed like this : http://superfeedr.com/track/<keyword>. The keywords must have more than 4 characters. The case doesn't matter.

IP list

If you use Superfeedr to host your hub, you may want to make sure we can access your content. Similarly, if you're a subscriber, you may want to make sure you don't prevent us from sending you updates. Our IP list is below.

You can also certainly check the UserAgent, however, we have a tendency to change them quite often to communicate with other servers. You can still check that it contains Superfeedr.

© 2009-2010 Superfeedr