![\"Drasi](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/10\/Drasi-Logo-1.jpg\"){kind=logo}
<\/figure>\n\n\n\nRecognizing an opportunity for innovation, we set out to make detecting and reacting intelligently to data changes easy and cost-effective to implement, manage, and maintain. We envisioned a unified platform built from the ground up to achieve this goal, one that enhances and integrates with existing technologies commonly used in similar solutions. Our goal was to create a platform that appeals to developers building event-driven solutions that include the need to detect and react to change, a category of solution we are calling \u2018change-driven solutions\u2019. We also hope to appeal to data infrastructure architects and operations teams seeking to simplify and optimize how development teams in large organizations integrate real-time change detection into their data systems. <\/p>\n\n\n\n
The result is Drasi<\/a>, a data change processing platform, which we are excited to release as an open-source project on GitHub<\/a>. Although not yet ready for production use, we believe Drasi has matured to the point where its features, reliability, and performance support experimentation across a wide range of change-driven use cases. This initial release<\/a>, part of our incubation project, marks the beginning of what we hope will be active collaboration with an open-source community of experienced developers, architects, and operations teams. Together, we aim to make Drasi the default choice for building change-driven solutions and establish it as a common component of data infrastructures globally. <\/p>\n\n\n Making it easy and efficient to detect and react to changes in databases<\/p>\n\t\t\t\t\t<\/div>\n\n\t\t\t\t\t\t\t\t\t\t\t This introductory post will focus on the core concepts of Drasi, and its major components such as Sources, Continuous Queries, and Reactions. As shown in the diagram below, Drasi Sources connect to the change logs and feeds of data sources and pass relevant changes to Continuous Queries. Continuous Queries produce perpetually accurate result sets based on the evaluation of incoming changes. Reactions subscribe to one or more Continuous Queries and trigger meaningful reactions. <\/p>\n\n\n\n For more information on Drasi\u2019s architecture and technical capabilities, visit Drasi Docs<\/a>. <\/p>\n\n\n\n Drasi capabilities include: <\/p>\n\n\n\n Continuous Queries are at the core of Drasi. As the name implies, these queries run continuously. To understand what is unique about them, it helps to contrast them with instantaneous queries<\/em>\u202fthat developers are accustomed to running against databases. <\/p>\n\n\n\n These instantaneous queries reflect the data at that specific moment, missing any changes made afterward until the query is run again. Running the same query periodically may yield different results due to data changes between runs. To understand what has changed, you need to compare the new results with the previous ones\u2014a common but inefficient technique known as database polling. <\/p>\n\n\n\n Drasi makes database polling obsolete by providing Continuous Queries that automatically detect and react to data changes as they occur. After the initial startup process (bootstrapping), a Continuous Query queries the source databases once to establish the initial state and then never needs to query them again. This means that as changes happen, the Continuous Query precisely identifies which result elements have been added, updated, or deleted without repeatedly accessing the source databases. It generates a detailed description of these changes\u2014a Query Result Change\u2014that Drasi uses to drive subsequent actions with Reactions. This makes it feasible to use databases directly, eliminating the overhead of copying data to replicas or analytics platforms. <\/p>\n\n\n\n Once started, Continuous Queries run continuously until they are stopped, maintaining an up-to-date query result. This allows you to also request the current query result at any point in time, providing immediate visibility into the data without imposing additional load on the source databases. <\/p>\n\n\n\n Unlike some stream analytics solutions that require time-windowed aggregation or delays, Continuous Queries update their results immediately without such delays. This ensures full visibility of all changes, allowing you to react to each one as it occurs. <\/p>\n\n\n\n Currently, Drasi’s Continuous Queries are written in a subset of the Cypher Query Language, augmented with custom functions for Drasi-specific features. We chose Cypher to encourage thinking in terms of entities (nodes) and their relationships (relations), allowing users to express Continuous Queries more directly. We felt its syntax better supports this approach than other options. <\/p>\n\n\n\n When creating change-driven solutions with Drasi, it helps to define the questions you’re trying to answer and the specific events you want to observe. You can precisely specify what you want to monitor without limiting yourself to data from a single table or database. You then translate this description into a Continuous Query. For example: <\/p>\n\n\n\n The following Cypher query uses the example of the employee risk management service to describe how a Continuous Query works. <\/p>\n\n\n\n The\u202fquery\u2019s MATCH<\/strong>\u202fand\u202fWHERE<\/strong>\u202fclauses identify all\u202femployees\u202flocated in\u202fbuildings\u202fwithin\u202fregions\u202fwhere there are active\u202f\u2019environmental\u2019 incidents with a\u202fseverity\u202flevel of \u2018critical\u2019 or \u2018extreme\u2019. This means any combination of connected nodes with the required property values are included in the query result. <\/p>\n\n\n\n The\u202fRETURN<\/strong>\u202fclause outputs the name and email address of the at-risk employee and their manager, along with details about the incident and its region. This defines the schema for results generated by the Continuous Query. <\/p>\n\n\n\n Notice that you don’t specify the data sources within the query itself; instead, source information is provided as metadata for the Continuous Query. When you create a Continuous Query, you need to provide metadata that tells Drasi which data sources the Continuous Query depends on, how to integrate data from different sources, whether data needs transformation prior to integration, and which index implementation to use for storing the state Drasi maintains for the Continuous Query. <\/p>\n\n\n\n Continuous Queries currently support the following capabilities: <\/p>\n\n\n\n Sources serve as an integration point with existing software systems, establishing connectivity to the sources of change. A Drasi Source continuously monitors the change log or feed generated by a data repository and processes relevant low-level change events that are then forwarded to each\u202fContinuous Query\u202fsubscribed to the Source. Sources also enable Continuous Queries to query data during their initial bootstrap process. <\/p>\n\n\n\n Drasi Sources are available for the following technologies: <\/p>\n\n\n\n The final component of Drasi is the Reaction-reusable components that integrate Drasi with downstream systems and act upon observed changes. <\/p>\n\n\n\n When a Continuous Query receives a source change event that alters its result set, Drasi generates a Query Result Change notification detailing the changes. These notifications are sent to all Reactions subscribed to that Continuous Query. The Reactions determine what action to take based on the result set’s semantics and details in the notification. For example, in the at-risk employee scenario: <\/p>\n\n\n\n Depending on the organization’s email integration, we might use an Azure Event Grid or Storage Queue Reaction. Alternatively, the organization could write a custom Reaction that integrates directly with their email services. <\/p>\n\n\n\n Currently, Drasi provides Reactions to connect to: <\/p>\n\n\n\n To demonstrate Drasi’s real-world applicability, we developed a proof of concept with our connected fleet<\/a> partners to enhance fleet administrators’ operational task management. Administrators need to swiftly respond to urgent vehicle events and process real-time telemetry to anticipate maintenance needs. The existing solution requires multiple integrations with event processing and analytics systems before extracting valuable insights. Drasi was integrated into the Fleet Integration layer as a central hub for system integration. <\/p>\n\n\n\n In the Drasi-based solution, telemetry events are processed to alert administrators when immediate action is needed and notify them as vehicles approach maintenance milestones. The solution employs two Drasi Sources: one ingests events from an Azure Event Hub “vehicle event” namespace, and the other connects to Dynamics 365 for vehicle maintenance history and asset details. <\/p>\n\n\n\n We created two Continuous Queries subscribing to these sources for continuous data streams. In the alerting scenario, telemetry events are evaluated against conditions in the first query; if an event contains a matching engine malfunction error code, the result set updates immediately. In the notification scenario, a notification is sent when a vehicle nears a predefined distance since its last maintenance (for example, 10,000 km.) The Continuous Query performs a linear regression on the odometer value, compares it with the maintenance interval for that vehicle model, and updates the result set to predict if service is needed within the next 30 days. <\/p>\n\n\n\n A single Drasi Reaction triggers Internet of Things (IoT) alerts in Dynamics 365 Field Service for immediate action events and upcoming maintenance notifications. This Reaction monitors updates in the Continuous Queries’ result sets, taking context-aware actions based on incoming events. <\/p>\n\n\n\n By integrating Drasi, we created a comprehensive and versatile solution that addresses multiple scenarios while allowing for quick updates to adapt to evolving operational and business requirements. <\/p>\n\n\n\n Drasi is an open-source project licensed under the Apache 2.0 license. Join our growing community to collaborate, innovate, and shape the future of Drasi with fellow developers. For more information and to get involved, visit our website<\/a> and GitHub repository. <\/p>\n\n\n\n![\"\"](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/WIN23_PRO_Acer_Concept_D_Office_003-1024x560.png\")
\t\t\t\t<\/div>\n\t\t\t\n\t\t\tDrasi<\/h2>\n\n\t\t\t\t\t
![\"Drasi](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/10\/Drasi-Expanded-1.jpg\")
<\/figure>\n\n\n\n\n
Continuous Queries <\/h2>\n\n\n\n
![\"An](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/10\/Drasi2.jpg\")
<\/figure>\n\n\n\n![\"Data](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/10\/Drasi3.jpg\")
<\/figure>\n\n\n\n\n
![\"A](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/10\/Cypher-Query-Example.jpg\")
<\/figure>\n\n\n\n\n
Drasi Sources <\/h2>\n\n\n\n
\n
Drasi Reactions <\/h2>\n\n\n\n
\n
\n
Fleet management using Drasi <\/h2>\n\n\n\n
![\"Diagram](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/10\/Drasi-Connected-Fleet-Example-1.jpg\")
<\/figure>\n\n\n\nLearn more and join the community<\/h2>\n\n\n\n
\n
![\"\"](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/CLO19_Ubisoft_Azure_030-1024x560.png\")
\t\t\t\t<\/div>\n\t\t\t\n\t\t\t