{"id":94670,"date":"2023-07-25T08:00:00","date_gmt":"2023-07-25T15:00:00","guid":{"rendered":""},"modified":"2025-06-30T11:09:32","modified_gmt":"2025-06-30T18:09:32","slug":"connect-fluid-dynamics-machine-learning-and-virtual-reality-with-onnx-runtime","status":"publish","type":"post","link":"https:\/\/opensource.microsoft.com\/blog\/2023\/07\/25\/connect-fluid-dynamics-machine-learning-and-virtual-reality-with-onnx-runtime\/","title":{"rendered":"Connect fluid dynamics, machine learning, and virtual reality with ONNX Runtime"},"content":{"rendered":"\n

This blog is co-authored by Carlos Pe\u00f1a-Monferrer, Co-Founder, SIMZERO, and Carmen D\u00edaz-Mar\u00edn, Co-founder, SIMZERO<\/em><\/p>\n\n\n\n

In this guest blog post, the SIMZERO<\/a> team shares how they used ONNX Runtime<\/a> to unlock the promise of recent developments in science for solving real world problems like preventing hazards and optimizing energy use. <\/p>\n\n\n\n

\n\n\n\n

The advent of machine learning (ML) is transforming every industry, from healthcare to finance, and from transportation to e-commerce. A promising use of machine learning is its combination with computational fluid dynamics (CFD) and other scientific applications. Different techniques, such as data-driven machine learning and physics-informed neural networks (PINNs), are transforming scientific simulations by bringing them into real-world, real-time processes. This development holds great promise for new discoveries, for optimizing the use of energy and materials, or for preventing potential hazards. <\/p>\n\n\n\n

A step further is the generation of virtual worlds or digital twins to form the core of the industrial metaverse. The key part is to be able to interact with the virtual world seamlessly as though it were the real world. In this way, we can test many hypotheses at little to no cost and decide on an optimal solution before implementing it in reality. By thinking outside the box, we can envision creating a virtual multiverse. Within this innovative space, one can propose, evaluate, and decide on multiple hypotheses. Real-world examples of this approach include planning new product configurations, operating a plant, designing heating or cooling systems, or responding to catastrophes. <\/p>\n\n\n

\"A
Figure 1: A virtual city environment with a parameterized inlet wind direction, created using Babylon.js and jsfluids<\/em><\/figcaption><\/figure>\n\n\n\n

To bring CFD to virtual or augmented reality (VR\/AR), there are two main challenges that need to be addressed: the need for significant computational power to perform CFD simulations in real-time or near-real time, and the need for specialized software that can integrate both CFD and VR systems. Next, we will explain how we solved this at SIMZERO<\/a>.<\/p>\n\n\n\n

Accelerating CFD simulations<\/h2>\n\n\n\n

One approach to accelerate CFD simulations is to use mathematical methods such as reduced-order models or ML algorithms to learn and approximate the results of CFD simulations. Another approach is to incorporate physical laws into the network architecture. Although there is still a long way to go in terms of developing physical models for complex processes and learning from them, there are already many applications that can benefit from this progress. In this blog entry we will not focus on the methods themselves but on the deployment of the ML-CFD pre-trained models. To ensure efficient deployment of ML-CFD, we opted to use Open Neural Network Exchange<\/a> (ONNX) models, which allows for seamless interoperability between existing ML-CFD frameworks based on PyTorch, TensorFlow, Keras, and other frameworks. Another important factor in choosing ONNX was the capability for directly running inference in different programming languages thanks to ONNX Runtime<\/a>. This allowed us to use ONNX Runtime to deploy the models to both backend and browser.<\/p>\n\n\n\n

To streamline the process, we have created a Python module `cfdonnx<\/a>` to bring together specific code to convert ML-CFD models to ONNX. The current module supports the U-Net model architecture but we plan to extend this to others in future releases. We have also created some reproducible examples<\/a> with the workflows that cover CFD runs to ONNX using DeepCFD<\/a> as the ML package. The example below shows how to export a PyTorch model to ONNX by passing a PyTorch model. Parameters such as network architecture, input shape, filters, and kernel sizes are read from the PyTorch model in this case, but they can be also passed as arguments for convenience.<\/p>\n\n\n

\npython3 -m cfdonnx --input model.pt --output model.onnx\n<\/pre><\/div>\n\n\n
ML-CFD and VR interface<\/h2>\n\n\n\n
A missing part was the interface between the ML-CFD and VR components. To address this, we developed jsfluids<\/a>, a high-level JavaScript module for real-time CFD inference. This module handles reduced-order models created by ITHACA-FV<\/a> and generic ML models. When using ML, we relied on ONNX Runtime to handle the ML aspects and provide a seamless interface between the virtual world and on-the-fly post-processing. Specifically, ONNX Runtime inference is performed to calculate the new fields that are passed to jsfluids for creating visualization components. The visualization components are compatible with Babylon.js<\/a> which is the real-time 3D engine used. Further analysis can also be performed aside from the visualization components. These tasks were achieved by porting specific parts of VTK<\/a>\/C++ to JavaScript.<\/p>\n\n\n
\"High-level
Figure 2: High-level workflow for jsfluids<\/em><\/figcaption><\/figure>\n\n\n\n
In the end, this approach enables the inference and analysis of fluid dynamics at the edge. By executing models directly on devices, we can reduce infrastructure costs, reduce latency, and improve privacy. In addition, we can bring the models to low consumption devices for connecting them with sensors and real world or to reduce power consumption on deployment.<\/p>\n\n\n\n
The following pseudocode shows how to use an ONNX model for inferring a velocity field, and creating and visualizing GLTF components on Babylon.js:<\/p>\n\n\n
\nconst model = jsfluids.ML;\n\n\/\/ Load the ONNX model from an URL or local data\nort.InferenceSession.create(onnxURL).then((onnx) => {\n\n  \/\/ Load the CFD mesh (unstructured grid) from an URL or local data\n  model.loadMesh(vtuURL).then(() => {\n    \/\/ Create a visualization component\n    \/\/ (e.g. outline surface, plane, streamlines, ...)\n    var gltf = model.setComponent({ type: \"surface\" });\n    var blob = new Blob([gltf]);\n    var url = URL.createObjectURL(blob);\n\n    BABYLON.SceneLoader.ImportMesh(\"\", url, \"\", scene, (meshes) => {\n      \/\/ Manipulate meshes (materials, positions, ...)\n      \/\/ Create the feeds input for your ONNX model\n      \/\/ var feeds = \u2026\n\n      onnx.run(feeds).then((results) => {\n        \/\/ Manipulate results to get the fields if needed\n        \/\/ var U = \u2026\n\n        \/\/ Set operations (if any) to be performed on the new field\n        \/\/ (e.g. vorticity or gradients)\n        model.setOperations({ operations: [\"vorticity\"] });\n\n        \/\/ Assign the field to the model and update\n        model.update({ field: \"U\", data: U });\n\n        \/\/ Render the component and assign the new field colors\n        var render = model.render({\n          field: \"U\",\n          \/\/ range: [0, 0.1]\n        });\n\n        meshes[1].setVerticesData(\n          BABYLON.VertexBuffer.ColorKind,\n          render.colors,\n          true\n        );\n\n        \/\/ Other jsfluids features as probing or integrating a field\n        var point = [0, 0, 0];\n        var probe = model.probe({ field: \"U\", point: point });\n        var integrate = model.integrate({\n          field: \"vorticity\",\n          target: \"grid\"\n        });\n      });\n    }, null, null, \".gltf\");\n<\/pre><\/div>\n\n\n
A playground for CFD, CAD, VR, and ML <\/h2>\n\n\n\n
Getting started with these packages can be complicated, especially as the scope extends to several different domains. For this reason, we have created a live editor playground powered by ChatGPT at SIMZERO Play<\/a> for experimenting, building, and testing. The playground includes other libraries such as babyplots and JSCAD for data visualization and CAD modeling, respectively. JSCAD provides a programmer\u2019s approach to designing 3D models. Although Babylon.js also allows for different geometry operations such as Constructive Solid Geometry (CSG), a user can choose to use JSCAD for convenience or for using models already developed by the community.<\/p>\n\n\n\n
At SIMZERO Play<\/a> you can find different examples and templates for starting building models and worlds while choosing the combination of tools that best suits your needs. For instance, one can just use Babylon.js together with ONNX Runtime for building virtual worlds with artificial intelligence, and add the CFD and CAD features if required. Additional features will be added so users could easily build and use their models and applications outside the playground in both development and production.<\/p>\n\n\n\n
What’s next for ONNX Runtime<\/h2>\n\n\n\n
With the deployment of pre-trained ML-CFD models using ONNX Runtime, we can infer fluid dynamics and seamlessly integrate them with virtual and augmented reality systems using open-source tools. Together with our newly developed JavaScript library jsfluids and the playground, you can easily access this technology to create better worlds.<\/p>\n\n\n\n
Get started today:<\/p>\n\n\n\n