MongoDB announced that Atlas Stream Processing is now in public preview, allowing developers the flexibility and ease of use of the document model alongside the Query API. With Atlas Stream Processing, MongoDB is bringing these same foundational principles to stream processing.
Atlas Stream Processing is redefining the experience of aggregating and enriching streams of high velocity, rapidly changing event data, and unifying how to work with data in motion and at rest, according to the company.
During the private preview, MongoDB saw thousands of development teams request access and the company has gathered useful feedback from hundreds of engaged teams.
In addition to enabling continuous processing of data in Atlas databases through change streams, developers can use Atlas Stream Processing with their Kafka data hosted by valued partners like Confluent, Amazon MSK, Azure Event Hubs, and Redpanda.
The company’s aim with developer data platform capabilities in Atlas has always been to make for a better experience across the key technologies relied on by developers.
New features in public preview include:
- Refined developer experience
- Expanded advanced features and functionality
- Improved operations and security
- Refined developer experience
In private preview, MongoDB established the core of the developer experience that is essential to making Atlas Stream Processing a natural solution for development teams.
And in public preview, the company doubling down on this by making two additional enhancements:
- VS Code integration: The MongoDB VS Code plugin has added support for connecting to Stream Processing instances. For developers already leveraging the plugin, teams can create and manage processors in a familiar development environment. This means less time switching between tools and more time building applications.
- Improved dead letter queue (DLQ) capabilities: DLQ support is a key element for powerful stream processing and in public preview, DLQ capabilities are expanded. DLQ messages will now display themselves when executing pipelines with sp.process() and when running .sample() on running processors, allowing for a more streamlined development experience that does not require setting up a target collection to act as a DLQ.
Atlas Stream Processing already supported many of the key aggregation operators developers are familiar with in the Query API used with data at rest. Now added in public preview are powerful windowing capabilities and the ability to easily merge and emit data to an Atlas database or to a Kafka topic.
Additionally, MongoDB invested heavily in improving other operational and security aspects of Atlas Stream Processing. A few of the highlights include:
- Checkpointing: Atlas Stream Processing now performs checkpoints for saving a state while processing. Stream processors are continuously running processes, so whether due to a data issue or infrastructure failure, they require an intelligent recovery mechanism. Checkpoints make it easy to resume your stream processors from wherever data stopped being collected and processed.
- Terraform provider support: support for the creation of connections and stream processing instances (SPIs) is now available with Terraform. This allows for infrastructure to be authored as code for repeatable deployments.
- Security roles: Atlas Stream Processing has added a project-level role, giving users just enough permission to perform their stream processing tasks. Stream processors can run under the context of a specific role, supporting a least privilege configuration.
- Auditing: Atlas Stream Processing can now audit authentication attempts and actions within your Stream Processing Instance giving you insight into security-related events.
- Kafka consumer group support: Stream processors in now use Kafka consumer groups for offset tracking. This allows users to easily change the position of the processor in the stream for operations and easily monitor for potential processor lag.
For more information about this news, visit www.mongodb.com.