In recent years, advancements in computer vision and 3D graphics have pushed the boundaries of what is achievable. One groundbreaking innovation that has gained significant attention is Neural Radiance Fields (NeRFs), due to the decreasing prices of GPUs and the excitement surrounding it. NeRFs combine neural networks and radiance fields to revolutionize how we perceive and interact with digital content.
NeRFs represent a paradigm shift in computer graphics by allowing the creation of realistic 3D models and scenes from limited input data. Through deep learning and sophisticated mathematical models, NeRFs have the potential to transform industries such as entertainment, gaming, architecture, virtual reality and medicine.
Traditional 3D graphics have relied on polygonal mesh models or voxel-based representations, which often fall short in capturing intricate details, realistic lighting and complex reflections. NeRFs address these limitations by providing a more accurate and immersive representation of 3D content.
At its core, a Neural Radiance Field is a learned function that predicts the appearance properties of a 3D point, such as color and opacity. Unlike traditional techniques, NeRFs learn shape and appearance directly from a training dataset, which enables them to capture fine-grained details and reproduce realistic materials and lighting conditions.
The magic of NeRFs lies in their ability to infer the radiance, or the amount of light traveling along a given ray in 3D space. By modeling the radiance field through a neural network, NeRFs can estimate the appearance of objects from different viewpoints and lighting conditions. This not only enables stunning visualizations but also facilitates interactive exploration of the 3D scene, allowing users to navigate and interact with virtual objects in an immersive manner.
NeRFs have vast applications across various industries. In gaming and entertainment, they can create photorealistic virtual worlds that blur the line between reality and simulation. Architects and designers can use NeRFs to visualize and iterate on complex architectural designs, providing clients with realistic virtual walkthroughs. Medical professionals can employ NeRFs for more accurate visualization of anatomical structures, aiding in surgical planning and training.
Join us on this captivating journey as we explore the power of NeRFs and witness the transformation of computer graphics and virtual experiences as we know them.
In conclusion, choosing AWS SageMaker for end-to-end ML-Ops training offers numerous advantages. Its streamlined workflow, scalability, flexibility and integration with Hugging Face make it an attractive choice for data scientists and ML practitioners. By leveraging the power of AWS infrastructure and services, combined with the capabilities of Hugging Face, SageMaker enables efficient and effective ML model development and deployment, setting the stage for successful AI-driven applications.
In this blog post, we will further explore the ML/OPs capability of AWS SageMaker by training a NeRF from a regular video and rendering it into a pixel accurate volumetric representation of the 3D space. We will create a Jupyter Notebook that runs on AWS SageMaker to train and render a complete pixel accurate volumetric representation of a 3D space recorded using a simple handheld device (in this case a Samsung Galaxy S22 Ultra with the default settings).
By the end of this blog, you will have gone through the following:
Part 1: Go through prerequisites for interacting with AWS SageMaker Studio.
Part 2: Import a Jupyter Notebook on AWS SageMaker Studio from a GitHub repository. This can be any regular
.ipynb notebook that you have built on any other cloud provider – for example Google Colab, Azure ML, your local machine, etc. We will be using AWS Accelerated instance types and special containerized workloads that are optimized for deep learning. Additionally, we will address a FAQ section that will discuss the changes we did to the Google Colab NeRF notebook to get it working with all underlying dependencies on SageMaker Studio.
Part 3: Render a full NeRF from a regular video with customizable camera positions, using an AWS SageMaker purpose built Jupyter Notebook that has been tested to generate the results covered in this blog.
So let’s get started.
Once you are done with visiting (1), (2) and (3) above, you can continue to Part 2.
pip wheelsstored and installed from Google Drive. Let’s replace the dependencies for each with the standard ones from their own documentation. You can run a
diffcommand in the terminal to check how the notebooks deviate in their installation steps.
As you can see in the above comparison we are building COLMAP and TinuCUDA from RPM repositories leveraging the latest versions instead of snapshot versions.
Use the terminal in session to debug any installation issues. Sometimes some dependencies might break. For example: The rendering uses NPM repositories that need to be installed. And that requires a specific
gcc version. So you might need to install these libraries on the underlying compute environment.
Once all the dependencies are installed you should be able to go ahead and render your own NeRF.
If you wish to use the data I used for the NeRF generated on the blog, please feel free to reach out by email and I would be happy to share the video.
Environment Configuration 
At this point, you can head over to the notebook itself. It has the necessary documentation per cell to guide you through the rest of the journey!
Technically, yes! Take a look at the following documentation to understand how Jupyter Notebooks are run on Sagemaker Studio. But if you are porting a notebook from other cloud providers, you will have to make some adjustments. Especially for services that the notebook would use in the native provider. In our case for example we had to do away with a Google file until that is used to interactively upload files directly from your local machine to your notebook. But this wasn’t a necessary functionality that we needed to port for the MVP. SageMaker’s file navigation provides upload capability which works faster that some compiled code on the notebook. You could replace the functionality with boto3 sdk for python to interact with S3 and then import the files into the compute environment’s file system, but for the purpose of a demo we didn’t need that.
If it’s a high-resolution video it can take anywhere up to upwards of 1+ hour. You can increase the vCPU count by selecting a higher thread count G4DN instance which will bring the training speed down. But it comes with more costs. So we recommend smaller instances for training experimental workloads and then identify the scaling needed for a production workload. The training uses vCPU and the rendering uses the GPU.
Yes. The video used in this blog is of my own house recorded on a Samsung Galaxy S22 Ultra. I simply walked around my living room taking the video. It’s sampled as High Resolution so the training of this data took about ~=1 hour.
 Project Jupyter
 AWS Deep Learning Containers – Connect your Github account
Learn how to establish a Docker-based Redis cluster on Mac OS for local development. Solve the issue of connecting to the cluster from the host network.
Discover the latest trends, best practices and strategies to safeguard your organization's data while unlocking the full potential of cloud technologies and AI-driven solutions.
High-performance computing (HPC) workloads are demanding and require specialized hardware and software. However, the cloud can provide a cost-effective and scalable solution for HPC.