Detect and react intelligently to changes in data with Drasi

Organizations often need to take some action when data stored in a database changes, or, conversely, when it doesn’t change as expected. Why? Because the data is a representation of the current state of the world, and businesses need to adapt when changes occur. Reacting to changes lets them respond faster, streamline processes, improve experiences, and gain a competitive edge. 

Implementing solutions to detect changes across multiple data sources, evaluate those changes to meet specific conditions, and integrate with downstream systems to act on the data can be complex and costly. These solutions usually support critical business processes and, therefore, must be secure, resilient, and meet latency expectations. In our experience, they are often brittle and custom-built, lacking generalization for reuse, which leads to repeated engineering and infrastructure efforts. 

Drasi: A data change processing platform 

Recognizing an opportunity for innovation, we set out to make detecting and reacting intelligently to data changes easy and cost-effective to implement, manage, and maintain. We envisioned a unified platform built from the ground up to achieve this goal, one that enhances and integrates with existing technologies commonly used in similar solutions. Our goal was to create a platform that appeals to developers building event-driven solutions that include the need to detect and react to change, a category of solution we are calling ‘change-driven solutions’. We also hope to appeal to data infrastructure architects and operations teams seeking to simplify and optimize how development teams in large organizations integrate real-time change detection into their data systems. 

The result is Drasi, a data change processing platform, which we are excited to release as an open-source project on GitHub. Although not yet ready for production use, we believe Drasi has matured to the point where its features, reliability, and performance support experimentation across a wide range of change-driven use cases. This initial release, part of our incubation project, marks the beginning of what we hope will be active collaboration with an open-source community of experienced developers, architects, and operations teams. Together, we aim to make Drasi the default choice for building change-driven solutions and establish it as a common component of data infrastructures globally. 

This introductory post will focus on the core concepts of Drasi, and its major components such as Sources, Continuous Queries, and Reactions. As shown in the diagram below, Drasi Sources connect to the change logs and feeds of data sources and pass relevant changes to Continuous Queries. Continuous Queries produce perpetually accurate result sets based on the evaluation of incoming changes. Reactions subscribe to one or more Continuous Queries and trigger meaningful reactions. 

For more information on Drasi’s architecture and technical capabilities, visit Drasi Docs. 

Drasi capabilities include: 

• Full containerization; runs on a Kubernetes cluster. This cluster can scale as needed and can be run on-premises or in the cloud.
• Management through a command line interface (CLI) or application programming interface (API) as well as integration with the Open Telemetry.
• Developer-friendly with the Visual Studio Code IDE and browser-based tools to write and test Continuous Queries (described below).
• Support for data sources such as PostgreSQL and Azure Event Hubs. Customization for new sources and actions is also supported. 

Continuous Queries 

Continuous Queries are at the core of Drasi. As the name implies, these queries run continuously. To understand what is unique about them, it helps to contrast them with instantaneous queries that developers are accustomed to running against databases. 

These instantaneous queries reflect the data at that specific moment, missing any changes made afterward until the query is run again. Running the same query periodically may yield different results due to data changes between runs. To understand what has changed, you need to compare the new results with the previous ones—a common but inefficient technique known as database polling. 

Drasi makes database polling obsolete by providing Continuous Queries that automatically detect and react to data changes as they occur. After the initial startup process (bootstrapping), a Continuous Query queries the source databases once to establish the initial state and then never needs to query them again. This means that as changes happen, the Continuous Query precisely identifies which result elements have been added, updated, or deleted without repeatedly accessing the source databases. It generates a detailed description of these changes—a Query Result Change—that Drasi uses to drive subsequent actions with Reactions. This makes it feasible to use databases directly, eliminating the overhead of copying data to replicas or analytics platforms.

Once started, Continuous Queries run continuously until they are stopped, maintaining an up-to-date query result. This allows you to also request the current query result at any point in time, providing immediate visibility into the data without imposing additional load on the source databases. 

Unlike some stream analytics solutions that require time-windowed aggregation or delays, Continuous Queries update their results immediately without such delays. This ensures full visibility of all changes, allowing you to react to each one as it occurs. 

Currently, Drasi’s Continuous Queries are written in a subset of the Cypher Query Language, augmented with custom functions for Drasi-specific features. We chose Cypher to encourage thinking in terms of entities (nodes) and their relationships (relations), allowing users to express Continuous Queries more directly. We felt its syntax better supports this approach than other options. 

When creating change-driven solutions with Drasi, it helps to define the questions you’re trying to answer and the specific events you want to observe. You can precisely specify what you want to monitor without limiting yourself to data from a single table or database. You then translate this description into a Continuous Query. For example: 

• An online order and delivery system must identify orders ready for pickup and notify their associated driver.
• A smart building maintenance application needs to detect spaces that don’t match the temperature preferences of occupants.
• An employee risk management service must notify managers when their employees are located near a dangerous incident. 

The following Cypher query uses the example of the employee risk management service to describe how a Continuous Query works. 

The query’s MATCH and WHERE clauses identify all employees located in buildings within regions where there are active ’environmental’ incidents with a severity level of ‘critical’ or ‘extreme’. This means any combination of connected nodes with the required property values are included in the query result. 

The RETURN clause outputs the name and email address of the at-risk employee and their manager, along with details about the incident and its region. This defines the schema for results generated by the Continuous Query. 

Notice that you don’t specify the data sources within the query itself; instead, source information is provided as metadata for the Continuous Query. When you create a Continuous Query, you need to provide metadata that tells Drasi which data sources the Continuous Query depends on, how to integrate data from different sources, whether data needs transformation prior to integration, and which index implementation to use for storing the state Drasi maintains for the Continuous Query. 

Continuous Queries currently support the following capabilities: 

• Extensive support for the standard Cypher Query Language, based on the OpenCypher specification.
• Integration of data from multiple sources, including both graph and relational databases such as Microsoft SQL Server, PostgreSQL, and Azure Cosmos DB.
• Aggregation functions such as count, sum, average, min, and max across connected elements.
• Access to historical values to compare data over time and aggregate values within specific time ranges.
• Functions to detect the absence of change, eliminating the need to implement schedulers and timeouts. 

Drasi Sources 

Sources serve as an integration point with existing software systems, establishing connectivity to the sources of change. A Drasi Source continuously monitors the change log or feed generated by a data repository and processes relevant low-level change events that are then forwarded to each Continuous Query subscribed to the Source. Sources also enable Continuous Queries to query data during their initial bootstrap process. 

Drasi Sources are available for the following technologies: 

• PostgreSQL
• Azure Cosmos DB
• SQL Server
• Azure Event Hubs
• Microsoft Dataverse
• Kubernetes

Drasi Reactions 

The final component of Drasi is the Reaction-reusable components that integrate Drasi with downstream systems and act upon observed changes. 

When a Continuous Query receives a source change event that alters its result set, Drasi generates a Query Result Change notification detailing the changes. These notifications are sent to all Reactions subscribed to that Continuous Query. The Reactions determine what action to take based on the result set’s semantics and details in the notification. For example, in the at-risk employee scenario: 

• New employee added to the result set: They have just been identified as at risk. We notify the employee about the incident and inform their manager with the details.
• Employee removed from the set: They are no longer considered at risk. We notify the employee and their manager that the immediate risk has passed.
• Data about an employee modified: They are still at risk, but their situation has changed. We update the employee and the manager with the new information. 

Depending on the organization’s email integration, we might use an Azure Event Grid or Storage Queue Reaction. Alternatively, the organization could write a custom Reaction that integrates directly with their email services. 

Currently, Drasi provides Reactions to connect to: 

• Azure Event Grid
• Azure Storage Queue
• SignalR
• PostgreSQL
• SQL Server
• Microsoft Dataverse
• Debug (developer tool to debug Continuous Queries) 

Fleet management using Drasi 

To demonstrate Drasi’s real-world applicability, we developed a proof of concept with our connected fleet partners to enhance fleet administrators’ operational task management. Administrators need to swiftly respond to urgent vehicle events and process real-time telemetry to anticipate maintenance needs. The existing solution requires multiple integrations with event processing and analytics systems before extracting valuable insights. Drasi was integrated into the Fleet Integration layer as a central hub for system integration. 

In the Drasi-based solution, telemetry events are processed to alert administrators when immediate action is needed and notify them as vehicles approach maintenance milestones. The solution employs two Drasi Sources: one ingests events from an Azure Event Hub “vehicle event” namespace, and the other connects to Dynamics 365 for vehicle maintenance history and asset details. 

We created two Continuous Queries subscribing to these sources for continuous data streams. In the alerting scenario, telemetry events are evaluated against conditions in the first query; if an event contains a matching engine malfunction error code, the result set updates immediately. In the notification scenario, a notification is sent when a vehicle nears a predefined distance since its last maintenance (for example, 10,000 km.) The Continuous Query performs a linear regression on the odometer value, compares it with the maintenance interval for that vehicle model, and updates the result set to predict if service is needed within the next 30 days. 

A single Drasi Reaction triggers Internet of Things (IoT) alerts in Dynamics 365 Field Service for immediate action events and upcoming maintenance notifications. This Reaction monitors updates in the Continuous Queries’ result sets, taking context-aware actions based on incoming events. 

By integrating Drasi, we created a comprehensive and versatile solution that addresses multiple scenarios while allowing for quick updates to adapt to evolving operational and business requirements. 

Learn more and join the community

Drasi is an open-source project licensed under the Apache 2.0 license. Join our growing community to collaborate, innovate, and shape the future of Drasi with fellow developers. For more information and to get involved, visit our website and GitHub repository.  

• Contribute code: Visit our GitHub organization and improve Drasi by contributing to our codebase and documentation or build a Source or Reaction to expand the ecosystem.
• Share feedback: Report bugs and suggest new features to enhance Drasi’s capabilities. Your insights are invaluable in making Drasi more user-friendly, secure, resilient, and cost effective. Join us on our Discord server.
• Spread the word: Advocate for Drasi and help us build a vibrant community of users and contributors.