Streams Example: Simple Data Aggregation

Demonstration of UE Streams utilized to aggregate real time data from multiple sources

This is a quick example of how you can use UE Streams to aggregate data from multiple sources into a single datastore (e.g., database, data lake, etc.).

The general architecture approach looked like this:

You can see the full video here!

Looking for the code?

No problem! Here are the three services I wrote to make it happen. The UI dashboard is not included but we may publish that in the future:

  • Stock Tickers - Query stocks on a cadence and publish to a stream
  • Twitter Trends - Query trending tweets on a cadence and publish to a stream
  • Data Lake Listener - Subscribe to a stream and write all data to mongo as it comes in

Also, here's a generalized version of the stock ticker and twitter trend publish services you can use as a template.

In the future, we will create more streamlined examples by abstracting some of the logic you'll see into new packages. But for now, feel free to explore with UE Streams!