dot The future of fast is coming to an event in your city.

Join us at Redis Released

8.5.1 Data to be streamed

back to home

8.5.1 Data to be streamed

As people perform a variety of actions within our social network, those actions are seen at the various functions that defined our API. In particular, we spent most of our time building out the ability to follow/unfollow users, and post/delete messages. If we’d built other pieces of our social network, we’d also find a variety of other events that occur as the result of user behavior. A streaming API is meant to produce a sequence of these events over time as a way of keeping clients or other services updated about a subset of what’s going on across the entire network.

In the process of building a streaming API, a variety of decisions must be made, which can be generally reduced to three major questions:

  • Which events should be exposed?
  • What access restrictions (if any) should exist?
  • What kinds of filtering options should be provided?

For now, I won’t answer the second question about access restrictions. That’s a question that we need to answer when we’re building our social network based on expectations of privacy and system resources. But I’ll answer the other two questions.

Because we focused on posting/deleting messages and following/unfollowing users, we should offer at least some of those events. To keep things simple for now, we’ll only produce message posting and deletion events. But based on the structures that we create and pass around, adding functionality to support follow/unfollow events or events for other actions that we’ve added should be easy.

The types of filtering options that we’ll provide will overlap significantly with the API features and functionality that Twitter provides on the public side of things. In particular, we’ll offer the ability to filter over messages with an equivalent of follow (users), track (keywords), and location filters, in addition to a randomly selected subset of messages, similar to Twitter’s firehose and sample streams.

Now that we know what data we’ll have access to, let’s start looking at how we’ll serve the data.