Readme

HPCImageRendering

Labversion:1.0.0

Lastupdated:12/21/2018

Contents

Overview

Getting Started

Task 1 - Inspecting the AzureBlobCopy Project

Task 2 – Inspecting the RenderCmd Client Applications

Task 3 – Inspecting the Run.cmd File

Deployment

Task 1 – Deploying the Parametric Sweep Application to On-Premises Nodes

Task 2 – Deploying the Parametric Sweep Application to Windows Azure Nodes

Running the Client

Task 1 – Verifying Cluster State

Task 2 – Running the Client Application

Summary

Overview

TheHPCImageRenderingsampledemonstrateshowtorunaparametricsweepapplicationthatrendersimagesaccordingtoinstructionfiles.TheapplicationdemonstrateshowtodownloadanduploadfilesfromWindowsAzureblobstorage,andhowtocreateaparametricsweepjobfrom.NETcode.

ThissampleusestheAqsisapplicationthatrenders3DimagesaccordingtotheRenderMan®standard.

Note: To use this sample application, download the sample RenderMan data files and extract them to the samples folder.

KeyFeatures

Thissampledemonstratesthefollowing:

  • UploadingaparametricsweepapplicationpackagetoWindowsAzurenodes.
  • UploadinganddownloadingfilestoWindowsAzureblobstorage.
  • CreatingandrunningaparametricsweepjobinaWindowsHPC2008R2SP1cluster.

GettingStarted

Torunthissample,youmustinstalltheHPCPack2008R2ClientUtilitiesRedistributablePackagewithServicePack1andtheHPCPack2008R2SDKwithServicePack1.Inaddition,youneedtohaveadministrativeaccesstoyourHPCcluster’sheadnode.

ToruntheimagerenderingapplicationinWindowsAzurenodes,youmusthaveavalidWindowsAzureaccount,aWindowsAzureworkernodetemplatedefinedinyourheadnode,andseveralWindowsAzureworkernodesintheHPCclusterthatarestartedandonline.FollowtheDeployingAzureWorkerNodesinWindowsHPCServer2008R2SP1Step-by-StepGuideonTechNetforfurtherinformation.

Task1-InspectingtheAzureBlobCopyProject

Inthistask,youwillinspecttheAzureBlobCopyutilityprojecttoseehowtodownloadanduploadfilesfromWindowsAzureblobstorage.TheparametricsweepjobusestheAzureBlobCopyutilitytocopyarenderinginstructionfilefromablobcontainer,andcopytherenderedimagetoadifferentblobcontainer.

  1. OpenMicrosoftVisualStudio2010fromStart|AllPrograms|MicrosoftVisualStudio2010|MicrosoftVisualStudio2010.
  2. OpentheImageRendering.slnsolutionfilelocatedintheImageRendering\Sourcefolder.
  3. IntheSolutionExplorerwindow,expandtheAzureUtilitiesprojectnode,asshowninFigure 1.

Figure1

TheAzureUtilitiesproject

  1. OpentheBlobUtilities.csfileandinspecttheBlobUtilitiesclass.TheclasscontainsseveralmethodsforworkingwithWindowsAzureblobstorage:
  2. InitializeStorage.Themethodvalidatestheexistenceofarequiredblobcontainer,and,ifnecessary,createssuchacontainerandsetsitspermissions.
  3. DownloadFile.Themethoddownloadsafilefromablockblobandsavesittoaspecifiedpath.
  4. UploadFile.Themethoduploadsafiletoablockblob.TheCloudBlob.UploadFilemethoddivideslargecontentintoseveralblocks,andtheblocksareuploadedinparallel,accordingtotheminimumnumberofI/Othreadsinthethreadpool.Ifyouwanttochangethenumberofparalleluploads,youcansettheCloudBlobContainer.ParallelOperationThreadCountproperty,asshowninthefollowingexample:

C#

publicvoidUploadFile(

stringcontainerName,stringpath,stringfileName)

{

varblobStorage=InitializeStorage(containerName);

if(blobStorage!=null)

{

CloudBlobContainercontainer=

blobStorage.GetContainerReference(containerName);

blobStorage.ParallelOperationThreadCount=8;

CloudBlobblob=container.GetBlockBlobReference(fileName);

blob.UploadFile(Path.Combine(path,fileName));

}

}

  1. DeleteFile.Themethoddeletesafilefromablockblob.
  1. IntheSolutionExplorerwindow,locatetheAzureBlobCopyproject,andinspectthecontentsoftheProgram.csfile.
    AzureBlobCopyisaconsoleapplicationthatsupportscopyingfilestoandfromWindowsAzureblobstorage.Forexample,thefollowingcommandlinedownloadsafilenamedframe-1.zipfromtheinputblobstoragetoalocalfolder:

CMD

AzureBlobCopy.exe-ActionDownload-BlobContainerinput-LocalDirc:\frames\-FileNameframe-1.zip

  1. OpentheAzureBlobCopyproject’ssettingsandinspectthefollowingtabs:
  2. Build.Theaqsisfoldercontainstheaqsisrenderingapplicationthatrunsoneachcomputenode;thisistheonlyfolderthatis copiedtothecomputenodes(on-premisesandWindowsAzure).Beforerunningtheaqsisapplication,theAzureBlobCopyutilityiscalledinordertodownloadtherenderinginstructionfilefromtheblob.Theoutputpathoftheprojectisthereforesettotheaqsis\binfolder.
  3. BuildEvents.ThePost-buildeventrunsthehpcpackcommandtopacktheaqsisapplicationandtheAzureBlobCopyutilitytoacompressedzipfilefordeploymenttoWindowsAzurecomputenodes.
  4. OpentheApp.configfilefromtheAzureBlobCopyproject.TheappSettingssectioncontainstwokeys,StorageAccountNameandStorageKey.
  5. ChangethestorageaccountnameandthestorageprimarykeytomatchyourWindowsAzurestorageaccountsettings.
  6. BuildtheAzureBlobCopyproject.

Task2–InspectingtheRenderCmdClientApplications

Inthistask,youwillexploretheRenderCmdclientapplicationthatuploadstheinstructionfilestotheWindowsAzureblobstorage,submitstheparametricsweepjob,anddownloadstherenderedimagesfromtheoutputblob.

  1. IntheSolutionExplorerwindow,expandtheRenderCmdprojectnode.

Figure2

TheRenderCmdproject

  1. IntheRenderCmdproject,opentheRenderCommand.csfile,andinspectthecontentsoftheRenderCommandclass.
  2. UploadFiles.Thismethoditeratesthefilesintheinputfolderanduploadsalltherenderinstructionfiles(.ribfiles)toablobaftercompressingeachfilewiththerar.exeutility.
  3. DownloadFiles.Thismethoddownloadsthecontentsoftherenderedimages’outputblobtoalocalfolder.
  4. Run.ThismethodcallstheUploadFilesmethod,submitstheparametricsweepjob,andthencallstheDownloadFilesmethodtodownloadtherendered imagescreatedbytheparametricsweepapplication.
  5. IntheRenderCmdproject,opentheHPC.csfileandinspecttheCreateJobmethod.
  6. Themethodfirstconnectstothejobschedulerandcreatesanewjob,asshowninthefollowingcode snippet:

C#

Schedulerscheduler=newScheduler();

scheduler.Connect(headnode);

//Definejobsettings

ISchedulerJobjob=scheduler.CreateJob();

job.Name="AqsisonAzure";

job.MinimumNumberOfCores=1;

job.MaximumNumberOfCores=1;

job.UnitType=JobUnitType.Core;

//Lettheschedulercalculatetherequiredresourcesforthejob

job.AutoCalculateMax=true;

job.NodeGroups.Add(targetNodes);

  1. Aftercreatingthejobsettings,themethodaddsaparametricsweeptasktothenewjobandsubmitsittothescheduler:

C#

//Createaparametricsweeptask

ISchedulerTasktask=job.CreateTask();

task.Type=TaskType.ParametricSweep;

task.StartValue=0;

task.EndValue=endValue;

task.IncrementValue=1;

//Runtheaqsiscommandtorendertheimages

//The(*)wildcardisusedasaplaceholderforthecurrentindexvalue

task.CommandLine=@"%CCP_PACKAGE_ROOT%\Aqsis\bin\run.cmdframe-*";

task.WorkDirectory="%CCP_PACKAGE_ROOT%";

Console.WriteLine("Runningjob");

job.AddTask(task);

scheduler.SubmitJob(job,username:null,password:null);

Note:TheCCP_PACKAGE_ROOTenvironmentvariableisdefinedinWindowsAzurenodesandpointstothefolderinwhichHPCpackagedapplicationsaredeployed.

  1. Aftersubmittingthejob,theclientapplicationwaitsuntilthejobcompletes,periodicallycheckingthejob’sstatus.
  1. IntheRenderCmdproject,opentheapp.configfileandinspecttheappSettingssection.ThissectioncontainsfoursettingsthatyouneedtochangetoaccordwithyourHPCclusterdefinitionsandWindowsAzurestorageaccountsettings:
  2. HeadNodeName.Thenameofyourheadnodemachine.
  3. NodeGroup.Thegroupofcomputenodesthatwillruntheparametricsweepapplication.
  4. StorageAccountName.Thenameoftheblob storageinwhichtheinputandoutputfileswillbestored.
  5. StorageKey.TheprimarystoragekeyofthestoragespecifiedinStorageAccountName.
  6. BuildtheRenderCmdproject.

Task3–InspectingtheRun.cmdFile

Inthistask,youwillinspectthecontentsoftherun.cmdfiletoseewhichcommandsrunwhentheparametricsweepjobexecutesineachcomputenode.

  1. IntheSolutionExplorerwindow,right-clickthesolutionandselectOpenFolderinWindowsExplorer.
  2. IntheWindowsExplorerwindownavigatetotheaqsis\binfolder,locatetherun.cmdfile,andviewthefile’scontents:

RUN.CMD

REM Use the input parameter as a frame index.

set frame=%1

REM Setup the executable, input, and output folders.

set root=%CCP_PACKAGE_ROOT%\Aqsis

set inputdir=%CCP_WORKDIR%\%CCP_JOBID%\%CCP_TASKID%\input

set outputdir=%CCP_WORKDIR%\%CCP_JOBID%\%CCP_TASKID%\output

if not exist %inputdir% mkdir %inputdir%

if not exist %outputdir% mkdir %outputdir%

REM Pull input data from blob storage.

%root%\bin\AzureBlobCopy.exe -Action Download -BlobContainer input -LocalDir %inputdir% -FileName %frame%.zip

REM Unzip the input file, run the executable, and create output data.

%root%\bin\rar.exe e -y %inputdir%\%frame%.zip %outputdir%

cd %outputdir%

%root%\bin\aqsis.exe -shaders:"%root%\displacement:%root%\shaders\imager:%root%\shaders\light:%root%\shaders\surface:%root%\shaders\volume" -displays="%root%\bin" %outputdir%\%frame%.rib

REM Upload the output files to blob storage.

%root%\bin\AzureBlobCopy.exe -Action Upload -BlobContainer output -LocalDir %outputdir% -FileName %frame%.tif

  1. Therun.cmdfileexecutesthefollowingcommands:
  2. RunstheAzureBlobCopyutilitytodownloadthecurrentrenderinginstructionfilefromtheinputblobanddecompressit.
  3. Runstheaqsis3Drenderingapplication.

Note:Thisapplicationusestherootenvironmentvariable,whichissetaccordingtotheCCP_PACKAGE_ROOTenvironmentvariablementionedinTask2inthissection.

  1. RunstheAzureBlobCopyutilitytouploadthegenerated.tiffimagetotheoutputblob.

Deployment

Inordertorunthissample,youwillneedtodeploytheparametricsweepapplicationtoyourcomputenodes(bothon-premisesandinWindowsAzure).

Task1–DeployingtheParametricSweepApplicationtoOn-PremisesNodes

Inthistask,youwilldeploytheaqsisparametricsweepapplicationtoyourHPCcluster’son-premisesnodes.Instead of manually copying the application to each on-premises node, you can use the clusrun command to run the xcopy command in all the nodes.

Note:IfyouaregoingtotestthissampleonWindowsAzurenodesonly,youcanskipthistask.

  1. Inyourheadnodemachine,createasharenamedapps.Verifythatalltheon-premisesnodescanaccessthisshare.
  1. In the SolutionExplorer window, right-click the solution and select OpenFolderinWindowsExplorer.
  1. Copy the aqsis folder and its sub-folders to the shared folder you created in the head node.
  2. OpenthecommandpromptwindowfromStart|AllPrograms|MicrosoftVisualStudio2010|VisualStudioTools|VisualStudioCommandPrompt(2010).
  3. Runthefollowingcommandlinetoallowtheparametricsweepapplication to use the same command line paths whether it runs in on-premises nodes or Windows Azure nodes:

CMD

cluscfgsetenvs"CCP_PACKAGE_ROOT=c:\apps"

Note:SincetheCCP_PACKAGE_ROOTenvironmentvariableusedintheparametricsweepjobisdefinedonlyinWindowsAzurecomputenodes,andwewantthecommandlinetoalsorunonon-premisesnodes,weneedtosetitmanuallyforeachoftheon-premisesnodes.

  1. Run the following command line to instruct all the nodes to copy the application from the shared folder:

CMD

clusrun /nodegroup:ComputeNodesxcopy \\MyHeadNode\apps\aqsis\*.* %CCP_PACKAGE_ROOT%\aqsis\ /EY

Task2–DeployingtheParametricSweepApplicationtoWindowsAzureNodes

Inthistask,youwilldeploytheaqsisparametricsweepapplicationtoyourWindowsAzurenodes.

  1. OpenthecommandpromptwindowfromStart|AllPrograms|MicrosoftVisualStudio2010|VisualStudioTools|VisualStudioCommandPrompt(2010).
  2. NavigatetotheImageRenderingsamplefolder,andrunthefollowingcommandtouploadthedeploymentpackagetotheWindowsAzurepackagestorage:

CMD

hpcpackuploadSource\aqsis.zip/nodetemplate:"Azure nodetemplate"/relativePath:aqsis

Note:Change the value of the nodetemplate parameter to the name of your Windows Azure node template.

  1. IfyoualreadyhaveWindowsAzurenodesstartedintheHPCcluster,youneedtocopythenewpackagetothem.TosynctheWindowsAzurenodeswiththenewpackagesstoredintheapplicationpackagesblob,runthefollowingcommand:

CMD

clusrun/nodegroup:WindowsAzureNodeshpcsync

Running the Client

Task1–VerifyingClusterState

Inthistask,youwillverifythatthenodesinyourclusterareonline.

  1. OpentheHPC2008R2ClusterManagerapplicationfromStart|AllPrograms|MicrosoftHPCPack2008R2|HPCClusterManager.
  2. IntheClusterManagerapplication,entertheNodeManagementsectionandverifythattheWindowsAzurenodesintheclusterareonline,asshowninFigure 3

Figure3

VerifyingtheStateoftheWindowsAzureNodes

Task2–RunningtheClientApplication

Inthistask,youwillruntheclientapplicationthatcreatestheparametricsweepjob.

  1. OpenthecommandpromptwindowfromStart|AllPrograms|MicrosoftVisualStudio2010|VisualStudioTools|VisualStudioCommandPrompt(2010).
  1. NavigatetotheImageRenderingsamplefolder,andfromtheretoSource\RenderCmd\bin\debug,andthenrunthefollowingcommandtostarttheparametricsweepjob:

CMD

RenderCmd..\..\..\..\SampleInput

  1. Whileyouwaitfortheapplicationtofinishrendering,youcaninspectthestatusoftheparametricsweepjobusingtheHPC2008R2ClusterManagerutility,asshowninFigure 4

Figure4

StatusoftheParametricSweepRenderingJob

  1. WaitacoupleofminutesfortheapplicationtocompleteandthennavigatetotheImageRendering\SampleInput\outputfoldertoinspecttherendered.tiffimages.

Summary

AfterrunningtheImageRenderingsample,youshouldhavelearnedthefollowing:

  • HowtoworkwithWindowsAzureblobstorage.
  • HowtopackageanapplicationforWindowsAzure.
  • HowtodeployanapplicationtoWindowsAzurenodesandtoon-premisesnodes.
  • Howtosubmitaparametricsweepjobfroma.NETclientapplication.
  • Howtocheckarunningjob’sstatus.