Want smarter insights in your inbox? Sign up for our weekly newsletters to get only what matters to enterprise AI, data, and security leaders. Subscribe Now
Real-time streaming data can be valuable for numerous applications and purposes across industries. In the case of the New York Stock Exchange (NYSE), streaming data is literally money.
The NYSE is one of the largest financial exchanges in the world and has a lengthy history of being able to share its financial market data.
A hundred years ago it used telegraph based ticker tape to share information. In the modern era it has developed its own low-latency, high-performance technologies deployed on-premises that other organizations can connect with.
Now it’s taking the next step forward, embracing a model based on the open-source Apache Kafka streaming technology that brings NYSE Best Quote and Trades (BQT) data to the AWS cloud.
AI Scaling Hits Its Limits
Power caps, rising token costs, and inference delays are reshaping enterprise AI. Join our exclusive salon to discover how top teams are:
Turning energy into a strategic advantage
Architecting efficient inference for real throughput gains
Unlocking competitive ROI with sustainable AI systems
Secure your spot to stay ahead: https://bit.ly/4mwGngO
To do that, NYSE partnered with streaming data platform vendor Redpanda, which has developed its own implementation of Kafka written in the C++ programming language.
NYSE’s deployment of Redpanda’s C++-based streaming platform achieved 4-5x performance improvements over traditional Kafka competitors, exposing fundamental limitations in how most organizations handle bursty data workloads.
This performance gap becomes critical as enterprises scale AI applications that demand consistent low-latency data access. Kafka-based data streaming also has potential to enable agent-to-agent communications, rivaling other approaches like Google’s A2A and it can also be extended to enable Model Context Protocol (MCP).
“The market thesis is that all of the large foundation models have really indexed the public data sets, and the next frontier is private data sets, and Redpanda really unlocks private data sets for agentic access,” Alex Gallego,founder and CEO of Redpanda told VentureBeat.
What the NYSE is building in the cloud
NYSE built its cloud streaming platform to serve customers who cannot access its data centers directly. The exchange targets fintech companies and retail broker-dealers who need AWS-based access to real-time market data.
“Not every consumer of our market data has the capacity to come to our data center, take the feed and use that feed,” Vinil Bhandari, head of cloud and full stack engineering at NYSE told VentureBeat. “But you know, a small shop in Hong Kong has access to creating their own AWS account, for example, and it’s those audiences that we are trying to cater to.”
NYSE streams its BQT (Best Quotes and Trades) feed, which aggregates real-time data from all seven NYSE excha …