Camera Mapping a Gi/HDRI Scene

Author : Garry Curtis

February 25 2006 . With Thanks to Henrik Nyquist and Timo for their excellent plugins and to Tijai for first posting a simple Camera Mapping project to the RS list , as well as the matteshadows.r3d material .

A tutorial for Realsoft3D V5

Skill Level Suggested :

It's assumed the reader is an intermediate user , and so very little in the way of RS interface explanations or basic procedures will be offered . I'm afraid this tutorial is elaborate enough already , encompassing several things at once . To make it into a newbie tute would lengthen it by several factors .

As the front page of the Realsoft Wiki says "Keep It Simple Stupid" . Yes , but sorry , some things in 3D just aren't simple and to try and make them that way for newbies would also make the task of writing tutorials 10 times more difficult .

However , I will assume that all levels of RealSoft3D user will be reading this tutorial so I'll do my best to not "speak in tongues" and attempt to make the tute understandable to the novice so that they can mostly follow along and learn something , especially in the area dealing with Gi/HDRI . I've also included examples of desired working methodolgies for good measure .

We will be lighting this tutorial with Gi/HDRI simply for the experience of doing so for those that haven't tried that yet with RS . This very simple example scene will benefit very little from advanced lighting , but in an effort to expand an average RS user's knowledge of various lighting options this tute will spend considerable time away from Camera Mapping and instead will deal with the Realsoft Gi lighting method .

Introduction

Realsoft3D Camera Mapping is a very quick and easy way of combining video with CGI . In this tutorial we will also walk through the creation of an HDRI lit scene , where we have set up a simple Collision detection simulation . The initial high quality HDRI (High Dynamic Range Image) Gi render of only the first frame of that scene will in turn , be Camera Mapped as the the entire Collision Detection simulation is performed , resulting in very very fast renders of that animation , with seamless integration being our goal .

Here's An Early V4.5 Example

    If all this sounds a bit confusing , as exactly what and why and how , we better have a quick look at this example first (posted to the RS list in 2004). The table and blocks in the example animation (linked below) were only rendered once , which is the resulting frame above . A reasonably high render time was used for that one frame only , and that frame was then Camera Mapped to the Simulation scene . An example V 4.5 animation 

The animation shows many balls bouncing off of that table and blocks , but only the balls were rendered for the animation , and 'rendered-in' with much lower non-Gi render settings . The result appears to be a complete Gi animation , but this anim was done in RS V4.5 where a Gi animation would have flickering and noise , and so this work-around not only sped up the process , but also increased the quality . (V5 has flicker-free Gi animation)

Better Get Started

You can build your own RS CD (collision detection) animation project , or just use  this simple example  I have pre-built for this project . If you want to try a little simulation work afterwards ,  check out this link  for a few tips I have to offer about RS and using it's built-in Dynamics system .

The above linked .zip'd project file will run the simulation , and we will start there and begin to build our HDRI lighting . So start RS , load the uncompressed .zip file and have a quick look at it , hit the render button to make sure everything is OK , and that all textures have found their way into their proper folders . (it looks horrible but we'll work on that)

Next we'll need a plugin to import that special deep (24 bits/channel) image into RS . Grab that from Henrik Nyquist's site Henriks Personal Web Site  and add the r3hdr.dll to your plugin folder . Now you'll be able to use HDRI's just as easily as .bmp's ! You'll need a nice bright HDRI and I have this cool link for you ...  an interior of an apartment  (make sure you download the Spherical - Lat/Lon version near page bottom) This nice bright image can be mapped to a sphere as an environment , or as we'll do in this tutorial , just bring it into Realsoft via Timo's cool environment mapping VSL shader 'EnvMap' . Get that  from Timo's materials site  .

That should do it . You have the Realsoft simulation project , the HDRI plugin , the HDRI and the EnvMap shader to map the HDRI to the scene as a surrounding environment . Next we'll open up the scene and get to work on it .

Where Do We Start ?

I would start by doing a quick run through the animation (do an animation preview in OpenGL) just to make sure it runs well and to give you a better idea of what kinds of times we are talking about for just the simulation part . I have the project set for 200 frames , but you can adjust that to another number in the 'Animation Control Window' . Shorten it up in the 'General' Tab , and change the "Custom' time to another number if you like , like 30 or 50 frames .

Leave everything else as it is though . This sim runs well and changing certain settings could throw a wrench into the works . If you have the time , resize the viewport window and set your View Properties to "Continuous Rendering" for the anim preview .

You'll notice the sim is highly inaccurate but we don't care about that . Remember , we'll use only the first frame of the sim and the balls will "appear" to be bouncing off of perfectly still/ well lit/ textured blocks , when in fact they are bouncing off invisible blocks that are misbehaving very badly .

Ready to Add the HDRI ?

Let's get going on improving the scene's rendered appearance . The lack of lights ( except for the camera flashlight) has a tendency to flatten the scene's appearance . The camera's gaudy backdrop image is just a placeholder so delete it and replace it now with the Apartment Lat/Lon HDRI image you downloaded earlier . It won't look very interesting , I'm afraid , but will give us trouble free Gi (some Camera Backdrop textures can be problematic) and allow us to keep moving forward in a straight line for now .

Double click on the correct one of Timo's EnvMap Shaders (there are 3 to choose from) and point it to the Apartment HDRI which is not a probe type but a Lot/Long type HDRI . Leave the HDRI Backdrop and mapping for the scene's camera for now , but keep your eye's open for something cool you'd like to have showing in the background of this animation (maybe some video or another animation ? ) and experiment with that later .

Let's Light the Scene

Gi and HDRI is what we'll use to light the initial frame , and then we'll use something less time consuming but still appropriate for this indoor scene to light the animation (have I said that too many times yet ?) . We only want to render the "blocks and top" so 'drag & drop' the EnvMap shader to the Root , as well as the Gi shader to the Root as well .

We are not going to render the Spheres for our Primary image (frame #1) so simply make them "Raytrace Invisible" for now if you like , but keep in mind we will probably be doing a lot of test renders and scene tweaking and we don't want to accidentally change the Spheres in a way that will not allow them to run properly in the simulation (ie : do a backup now !) .

Time to Hit the Render Button

If you've applied the 2 shaders to the Root of the scene , drag and drop the camera into the view window (just to make sure all is well) and hit the render button to make sure the HDRI is now also your camera backdrop image) . Again , if I were you I'd resize the render either by shrinking the viewport or preferably by using the Render Control and adding around a 320 x 260 "Window" from the list of available formats , and we will start rendering with Gi next .

Incidentally , using the Render Control is something you really must bring into your workflow (in my opinion) . RS AA is another area a user really must spend some time in , simply as a way and means to decrease render times when needed , and to increase quality when required . We can slowly graduate to a High-Quality render once we have tweaked the lighting and found textures we like . So ...

* Change Render Settings From Reasonable Quality to Gi Rendering
* Change Effect/Box from Default Effects to None
* Change Effect/Image to Gi Effects

Hit the Render Button to see your marvelous Gi/HDRI render ! Just kidding , it still looks like crap .

We Waited all that Time For That ?

Now , before you get mad or discouraged , let's just make some quick changes . We have moved into another realm in a sense , and must make some adjustments to our way of thinking , as well as to our render settings

First thing you'll notice about the render is that it's way too bright . That's because it's now lit by the Gi/HDRI as well as the Camera's fIashlight . I would lower the flashlight in the area between 0.0-0.0-0.0 and 0.1-0.1-0.1 (approx) always remembering to drag/drop the camera back into the viewport . Do another render . Still too bright so lower the 'Brightness' control in the EnvMap shader (uncheck the 'Brightness Disable' control box) to about half way , 0.5 0.5 0.5 sounds reasonable , and re-render .

Hmmm ... What can I say ? We see some obvious flaws with the bump mapping on the blocks , and we could sure speed this rendering process up , so let's make some changes ( I could have done this for you but let's just chalk this up to the learning process , shall we ?)

Let's Speed Up the Creation Process !

* Change Render Settings From Gi Rendering to garry Reasonable Quality
* Change Effect/Image from the default Gi Effects to garry Gi Draft
* Change the Bump material for the cube Blocks to 'Wood' (note : change it only via the Material Drop-down list at the top of the select window for each and every cube ! Remapping a new material will destroy the collision detection )
* Remove 'mobile' from the Dist Tab on any Render Settings that has it (to stop those error messages)
* Might as well add more Raycount1 to the default Gi Shader . (maybe just 5 for now)
* We are going to need shadows so think about how we'll add a couple of lights here and there (later) .

Much much faster and perfect for this stage of the game . As we watch the render take place we see that 90% of the scene lighting is coming from the Gi shader after it has finished up in post ! We'll be adding a couple lights soon , but spend some time here tweaking brightness until you're becoming more content with the look .

You can start to see the realism begin to creep into the image now . That's partly because of the 'softer' look . The human eye can easily pick up the slight gradient that is spread around and over everything , as a result of using Realsofts advanced deep maths as well as the HDRI's channel-depth . Quality has gone down but render times have too . We will slowly balance those forever opposing factors out as we slowly progress .

Tweaking the Gi shaders (Pre and Post) would be a good way to 'look and learn' . Some quick tips below for those that might be interested .

Hey , Gi Joe !

Your Gi shader has a ton of controls for dialing in the look of the scene . Some area's that might be worth looking at are :

* Using Gi Draft as a way of first approaching the Gi stage . Take a good look at your Gi Draft settings (in Post Processing) and I would even suggest duplicating it , then stripping it of everything except the Gi Brightness Control & Gi Blur , in an effort to speed up the initial Gi testing . Eventually use the default Gi Draft , then graduate to the Default Gi Effects when you are getting closer to correct lighting .

* The Amount of Raycount1 . The more the merrier ! If you've got the CPU for it , increase this to at least 10 for test renders . Double that for the final render if blotches are showing . This is where the cam-map earns it's living . Crank it up for just one frame , and it will stay good for the 200 or so frames of the animation .

* Keep your eye on the Contrast . This control has a huge effect on the scene ! Make sure you tweak this in your test renders , keeping an eye on how it's lightening or darkening certain areas .

* Gi Blur can be adjusted to suit , and you should do so . Every scene has different requirements . Higher Raycount1 will allow you to turn down the Gi Blur . It's your choice , always . Speed or detailed quality . If you have a scene that does not have high detailed textures , then turn up the Gi Blur . Same if you need speed and want to remove the blotches on smooth coloured/untextured walls . Lot's of options and solutions !

* Gi Blur Specular (Post) is set too high at default , so you will have to adjust this if you have reflecting shaders in your scene . We don't with this scene , but keep it in mind for future scenes with Chrome in them .

* Post Pro Scaling can be 1 to 1 if you have decided to increase AA in your Raytracing settings instead . Work one up , one down , but above all learn to get a feel for what the difference is between the 2 .

* Do strict comparisons . Use XnView (.com) to toggle back and forth between "this render" and "that render" . You can also do that using your browser , by dragging and dropping the files that you've rendered to disk . Learn to label them so that you know how each file is different just by looking at it's name (con,5-rayc1,20-AA0-.bmp for example)

* Learn the AA panel and what each slider does . Gi Bluring can reduce sharpness so learn exactly how and where to increase it and still keep render times down .

* Use soft low lights here and there where the scene requires it . The Gi can only do so much so see if you can help it out in areas where raycounts just aren't 'getting' there .

* Save often under project names that contain clues to where you're at . Nothing worse that hitting the sweet spot only to have XP or RS crash with the last few changes lost forever .

* Above all , learn to throw away any CPU hogs that you don't need . Turn down quality settings for test renders first and foremost . The more renders you do , the more you can learn (if you're paying attention) .

There are many more tweaks and "Tips & Tricks" , obviously . Seems everyone who has worked with Gi has their own set of Gi preferences and rules of thumb . Work with Gi enough until you get a general feel for what each control does .

Ya' Happy Pappy ?

Work with the lighting until you have it fairly set , and then 'adjust-up' your quality settings to begin to dial it in . Switch to Gi Rendering and Gi Effects(in post) to get a better idea of where the finer details are that are going to need your attention .

Last step will be to add some slight shadows with a light or 2 to stop the 'floating' blocks look and add a bit of depth . We must be subtle with this since the glass spheres in the actual sim anim will not be able to interact with any shadows that are "baked" into the camera mapped image , unless we leave the exact same lighting for the next stage as well . Pick a top view and add 4 little point lights or spots on the edges of the checkerboard Top and adjust their Intensity to suit the scene .

Now would probably be a good time to mention that chrome objects are taboo with this method (for now) . Cam-map'd chrome blocks would not reflect the glass spheres , of course , so therefore we choose non-reflecting materials , like 'Wood ' to hide that basic lacking . Our glass sphere's will however reflect the cam-mapped image as if it were actually 'real' !

When you're reasonably satisfied with lighting and a few subtle high-quality shadows , render out a single frame at high quality Gi settings and save it to disk , via the Render Control . This single rendered frame is what we will finally Camera Map to our animation .

Are We There , Yet ?

Well , we're getting there . The next step is the actual Camera Mapping procedure , and as I said it is very quick and easy ... as long as we did all the prep-work thoroughly . However , once we have mapped the high quality image to our scene , we must again begin the test rendering process , in an effort to "Blend" our glass spheres into our (previously) Gi lit scene . Once we're happy with the second stage lighting , we can run the animation , go for a Budweiser then come back and see how we made out .

Save the scene under a different name as a backup (if something goes wrong we can fall back to our previous saved file) then close RS and restart again , fresh . Now A few simple things we need to do to get back from the world of Gi and back into normal lighting modes .

* Change Gi Rendering to Reasonable Quality .
* Change Effect/Image to None .
* Change the Camera's Flashlight back to 1.0 1.0 1.0 (just for now) .
* Render to make sure all is well with our scene before we start to make any changes .

Map That Camera !

Near the bottom of your 'Select' window is a level named "Camera-Map-This" . Time to do so . Change this level from Camera Invisible to visible , and in turn make your Top & Cubes (blocks) levels Camera Invisible and render to make sure you can see this new level of plain , non-textured Top & Blocks . Leave your Camera's Backdrop and it's texture (it's not great but will do for now) .

Select the "Camera-Map-This" level and slap a default mapping on it using the Materials Tab at top , making sure to select the Mapping Type as "Camera" and the Material as the 'matteshadows' VSL material (in your shader collection on the Material tab) , and select "Accept" . Better save this project now under a different name as a new starting point .

Now , set up the 'matteshadow' material by selecting the advanced box and finding the line of vsl code where an image must be added(under the Surface Properties Shader part of the material) . Time to finally bring that high-quality render back into play for this project , so use the file selector and add it .

Note : this material offers no useable AA and any renders at this point will look terrible , but if you click on the "Gradient X & Y" boxs associated with this line of VSL , you will get your AA back where it should be (perfect) .

If you do a render now , it will seem as though you are just observing your HQ rendered image as a backdrop image . Actually , that HQ image is now being magically mapped to your previously untextured surfaces on a plane perfectly parallel to the Camera ... like the Alien in the movie "Predator" !

Camera's Mapped and Ready , Sir !

It's time to light the scene so that these glass sphere's will look good . How is totally up to you . Better run your simulation ahead to a point where the glass spheres are actually on the Top and closer to the camera so that you are able to tweak the scene's lighting as well as the Glass material and the matteshadow material for these particular frames (save out to a new project first) .

You may decide to leave some camera light as a quick test , but I would suggest you at least increase the power coming from the overhead lights that you used to add some shadows in the Gi part of the project keeping in mind that we don't want to over-do it and give away the fact that we're compositing a subtlely lit Gi scene with a scene of another type .

Rewind the anim , save the project under a different name again , and do an animation preview to check caustics and shadows . (do an OpenGL anim preview first just to make sure the collision detection is behaving itself)

Looking at your resulting animation closely you'll notice that shadows and caustics 'appear' to be deposited on the wooden blocks as well as onto the checkerboard top ... but how can this be ? Camera mapping is the answer . Realsoft3D's Camera mapping allows shadows to fall correctly on Cam-map'd objects , like in our scene .

These scene objects are stand-ins for us now and will "hold the Fort" for us while we tweak our collision detection animation . Try to do that using a strict Gi pipeline ! We can rewind and tweak until our hearts content , using low quality render settings and RS will render up a very fast animation result for us of 100 , 200 or 300 frames in no time .

Here's an example animation using this tutorial as a basis . It's obviously not a very good example of Gi or of HDRI rendering , or even of collision detection , but keep in mind that this animation rendered out at 30 seconds per frame because of using a camera mapping solution to get proper shadows /reflection /caustics . Camera Mapping a still photograph to your 3D scene opens up all kinds of possibilities .    Camera Mapped Tutorial Example   

Is That All It Can Do ?

Certainly it can do more than this tutorial shows . For this simple basic scene it may not seem like much but it is a tool you can add to your arsenal . Where this method can really come into it's own is in the combination of a Camera-Mapped video onto Realsoft 'Stand-in' floors and walls and couchs and chairs etc . Our CGI characters can now cast realistic shadows onto your digital picture or video and also reflect that video in chrome and highlights . This is where the blending occurs ... a convergence of a different kind .