Networking.com

data streaming

By Paul Kirvan

What is data streaming?

Data streaming is the continuous transfer of data from one or more sources at a steady, high speed for processing into specific outputs. Data streaming is not new, but its practical applications are a relatively recent development.

In the early years of the internet, connectivity wasn't always reliable and bandwidth limitations often prevented streaming data to arrive at its destination in an unbroken sequence. Developers created buffers to allow data streams to catch up but the resulting jitter caused such a poor user experience that most consumers preferred to download content rather than stream it.

How data streaming works

The advent of broadband internet, cloud computing and the internet of things (IoT) have made data streaming easier. Today, businesses regularly use data from IoT devices and other streaming sources to make data-driven decisions and facilitate real-time analytics. Many companies have replaced traditional batch processing with streaming data architectures that can accommodate batch processing of high volumes of data.

In batch processing, new data elements are collected in a group and the entire group is processed at some future time. In contrast, a streaming data architecture or stream processor handles data in motion and an extract, load and transform (ELT) batch is treated as an event in a continuous stream of events. Streams of enterprise data are fed into data streaming software, which then routes the streams into storage and processing, and produces outputs, such as reports and analytics.

Examples of data streams

Data streaming use cases include the following:

Data comes in a steady, real-time stream, often with no beginning or end. Data may be acted upon immediately, or later, depending on user requirements. Streams are time stamped because they're often time-sensitive and lose value over time. The streamed data is also often unique and not likely repeatable; it originates from various sources and might have different formats and structures.

For example, various production sensors on a manufacturing production line capture different types of data and aggregate the data. Each sensor's data is then combined with data from the other sensors to provide a detailed view of the production system. A manufacturing resource planning system can use data from the various sensors to further refine how the production systems may be used, when they are scheduled, when maintenance is needed and other important metrics.

Pros and cons of data streaming

Data streaming comes with both advantages and drawbacks. Among the advantages are the following:

The following are some of the drawbacks of data streaming:

Data streaming and big data

To benefit from data streaming at the enterprise level, businesses with streaming architectures require powerful data analytics tools for ingesting and processing information. Popular enterprise tools for working with data streams include the following:

Amazon Kinesis Data Firehose. This real-time big data processing tool can handle hundreds of terabytes of streaming data per hour from data sources such as operating logs, financial transactions and social media feeds.

Apache Flink. This open source distributed data processing platform is used in big data applications, primarily for analysis of data stored in Hadoop clusters. Flink handles both batch and stream processing jobs, with data streaming the default implementation and batch jobs running as special-case versions of streaming applications.

Minimum recommended download speeds for viewing streaming data

To get a reasonable estimate of bandwidth -- also called throughput -- data engineers suggest the use of at least three test apps or sites, such as Fast.com, and that each test be conducted several times to ensure an accurate read.

Various streaming platforms require different download speeds. Some of the more popular consumer services require the following speeds:

Required speeds vary depending on the number of devices connected to the network and the type of media being played. For 4K content and online gaming, higher megabits per second speeds are generally required for the best customer experience.

Find out how streaming analytics can provide insight and value to your organization.

24 May 2023

All Rights Reserved, Copyright 2000 - 2024, TechTarget | Read our Privacy Statement