Now on GitHub: The Autonomous Driving Cookbook from Microsoft

At Microsoft, our mission is to empower every person and organization on the planet to achieve more. That’s why I’m happy to present the Autonomous Driving Cookbook which is now available on GitHub. The Autonomous Driving Cookbook is an open source collection of scenarios, tutorials, and demos to help you quickly onboard various aspects of the autonomous driving pipeline. It is an ongoing project developed and maintained by the Deep Learning and Robotics chapter of Microsoft Garage, the team that helped develop the recent expansion of AirSim to include cars for autonomous driving research. By making this cookbook available to everyone, we hope to make the barrier of entry to autonomous driving simulation as low as possible. The cookbook is now available in preview on GitHub as we continue to work hard to add more scenarios and tutorials in the future.
Autonomous driving has transcended far beyond being a crazy moonshot idea over the last half decade or so. It has quickly become one of the biggest technologies today that promises to shape our tomorrow, not very unlike when cars first came into existence. A big driver powering this change is the recent advances in software (Artificial Intelligence), hardware (GPUs, FPGAs etc.) and cloud computing, which have enabled ingest and processing of large amounts of data, making it possible for companies to push for levels 4 and 5 of autonomy. Achieving those levels of autonomy though, require training on hundreds of millions and sometimes hundreds of billions of miles worth of training data to demonstrate reliability, according to a report from RAND. This is where the ability to simulate real driving environments plays a key role and is hence the current focus of the cookbook.

Recipes for everyone

Tutorials in the cookbook are presented as Jupyter notebooks, making it very easy for you to download the instructions and get started without a lot of setup time. To help this further, wherever needed, tutorials come with their own datasets, helper scripts and binaries. While the tutorials leverage popular open source tools (like Keras, TensorFlow, etc.), as well as Microsoft open source and commercial technology (like AirSim, Azure virtual machines, Batch AI, CNTK, etc.), the primary focus is on the content and learning, enabling users to take what they learn here and apply it to their work using tools of their choice.

We aim to make the cookbook a valuable resource for anyone working in autonomous driving, be it research or industry, beginners or experts. For people who are just starting out, we have a tutorial on End-to-End Deep Learning for Autonomous Driving, which serves as a great introduction to the fundamentals of steering models, end-to-end deep learning techniques, and the AirSim simulation platform. For those looking to create scalable models on the cloud, we have a tutorial on Distributed Reinforcement Learning which uses Azure Batch to spin up virtual machines to run training on multiple instances of the simulator simultaneously. We will also be soon releasing a tutorial on Lane Detection using Deep Learning where instead of using popular computer vision-based techniques, which are inherently dependent on camera and image properties, we take data collected from simulations and use techniques like transfer learning to train deep learning models using very little real-world data.

Simulation meets reality

Simulation has become the backbone of the autonomous driving industry, providing a means to collect extensive amounts of data for model training as well as providing a safe testbed to crash-test these models. The closer our simulated scenarios are to reality, the closer we get to making autonomous cars a part of our lives. With that in mind, we are also unveiling a new simulation environment, inspired by the popular Hana Highway in Maui. The famous 64.4 miles long ‘Road to Hana’ is a winding path that goes through lush tropical rainforests, ocean front and grassy mountains with many waterfalls, beaches, tunnels and bridges along the way, all of which serve as the perfect backdrop for training autonomous driving algorithms. Our simulated environment is a condensed 1-mile version of this popular tourist destination and is available to use in the binaries included with the cookbook. We will also make it available on the main AirSim repository soon. This environment has an intermediate difficulty level which fits perfectly between the easy Landscape environment and the more advanced City environment currently available in AirSim. Our upcoming lane detection tutorial also uses this environment to train deep learning models.

Growing together for a better tomorrow

The autonomous driving industry is undergoing an interesting phase. We are living in a reality where Artificial Intelligence has created immense potential for a better tomorrow. This has resulted in a plethora of opportunities that did not exist before. There is a very high demand for skilled talent in this field which is unfortunately not being met. Therefore, it is a great opportunity for those of us already working in the field to come together and create collaborative learning platforms, for example the Eclipse OpenADx initiative, which Microsoft is an active contributor to, together with Bosch and several other stakeholders. A big motivation behind the Autonomous Driving Cookbook for us is to promote this spirit of collaboration and so, we welcome other experts in the area to contribute tutorials to the cookbook to make it a go-to resource for people who wish to enter and grow in this exciting field.
Any feedback or other resources you’d like to see? Let us know in the comments below.

Aditya Sharma

Aditya leads the Deep Learning & Robotics chapter of Microsoft Garage. He is also the lead Program Manager of Project Road Runner, an ADAS/AD initiative out of the Garage that’s behind AirSim autonomous driving and the AD cookbook. For his day job at Microsoft, Aditya works as a PM on the Azure Stack team.

See more articles from this author