Getting ACUSIM's AcuSolve to run on Windows HPC Server 2008 in a distributed mode.
I've recently worked on getting ACUSIM's AcuSolve to work successfully on Windows HPC Server 2008 and I thought I would share this with the community.
Some points to note though, ACUSIM recommends that only AcuSolve 1.7f works on Windows HPC Server 2008 in a distributed mode, and since this is the version that I got my hands on, I can only ensure that this version works, and it did.
Now that we have gotten that out of the way, I'm going to elaborate a little on the environment set up that I used.
1) I have a cluster of 3 compute nodes running Windows HPC Server 2008. Excluding the head node.
2) ActivePerl for 64-bit has been installed on the nodes, individually. If you plan on running a single instance of Perl installation, please do get in touch, I’ll try and see if I can get it to work. (This will come into play later when I outline the steps to get things to work.)
3) Things to take note, the batch scripts provided by ACUSIM has typos in them. Note: I performed the installation to a D: and the scripts were seemingly hardcoded to a C: install, and even then, I’m not too sure if they are right because they point to a C:\Acusim\classic, but my installation went into a D:\Acusim only. I have since fed this back to ACUSIM. :)
Steps to get it to work:
1) First, we need to fix the paths, from this point onwards, I am assuming that you have performed a first-cut installation to your head node and shared the Acusim installation directory out already. Now, you need to go to your Acusim installation “bin” directory and locate this perl script file called acuFixPath.pl. Edit line 190 to read “ $outfile= "$home\@TMP_$file" ; ” Reason is this, the script was coded for *nix based systems and the original syntax didn’t work. Also this syntax will assume that you will specify a $home for ACUSIM home, which we will.
2) Second, you need to run acuFixPath.bat with the following options, in my case it would be “acuFixPath.bat –files _all –home \\hpchead\Apps2\Acusim -machine WIN64 -version V1.7f –v 1” This would update all the paths to now point to the head node shared directory as the home installation for ACUSIM. This would fix all the path typos.
3) Now that the paths are fixed, the next step would be to get the compute nodes to accept UNC paths in its command processor. Now, log into each of the compute nodes individually, and execute “ \\hpchead\Apps2\Acusim\WIN64\V1.7f\bin\acuFixUnc.bat”. Unfortunately, this has to be done individually on each of the nodes. Once the script has executed, the compute nodes are ready to accept UNC paths on its command processor.
4) Now we need to customize the acuSub.bat to point to the right paths. In my case, I referenced acuEnv.bat to get certain parameters into the acuSub.bat file. Once you open up your acuSub.bat file you need to paste the following lines above the line that reads, “perl %ACUSIM_HOME%\%ACUSIM_MACHINE%\%ACUSIM_VERSION%\bin\acuSub.pl %opt% ”
rem -----------------------------------------------------------------------
rem Basic parameters
rem -----------------------------------------------------------------------
set ACUSIM_HOME=\\hpchead\Apps2\Acusim
set ACUSIM_VERSION=V1.7f
set ACUSIM_MACHINE=WIN64
set ACUSIM_ROOT=%ACUSIM_HOME%\%ACUSIM_MACHINE%\%ACUSIM_VERSION%
set ACUSIM_SYSTEM_CNF=%ACUSIM_ROOT%\script\Acusim.cnf
set ACUSIM_CNF_FILES=.\Acusim.cnf:~\Acusim.cnf:%ACUSIM_SYSTEM_CNF%
set PATH=%ACUSIM_ROOT%\base\lib\site-packages;%PATH%
set PATH=%ACUSIM_ROOT%\base\DLLs;%PATH%
set PATH=%ACUSIM_ROOT%\base\bin;%PATH%
set PATH=%ACUSIM_ROOT%\bin;%PATH%
set PATH=%ACUSIM_ROOT%\ODB_6.9\lib;%PATH%
set Visual3_CP=%ACUSIM_ROOT&\script
set PERL5OPT=-I%ACUSIM_ROOT%\script %PERL5OPT%
set PYTHONPATH=%ACUSIM_ROOT%\base\bin
set PYTHONPATH=%ACUSIM_ROOT%\script;%PYTHONPATH%
set P_SCHEMA=%ACUSIM_ROOT%\script\schema
set PV_INTERFACE_PATH=%ACUSIM_ROOT%\pv\packages
set ACUSIM_PARAVIEW_LIB=%ACUSIM_ROOT%\pv\bin\vtkAcuSolveReader.dll
This is to set up the ACUSIM environment when it runs in the compute nodes. Please do let me know if you want to know more and why.
5) We also need to edit the acuSub.pl file. At line 217 of the acuSub.pl file you will see, “$perlPath = "C:\\Perl\\bin" if $perlPath eq "_auto";” You need to change it to read “$perlPath = "C:\\Perl64\\bin" if $perlPath eq "_auto";” (This is because I’m using ActivePerl’s 64-bit installation, I guess if you are using the 32-bit one, you are fine.)
6) Because we are using the Windows HPC Server MPI layer, we need to submit it through the batch scheduler. But before you do that, you need to edit your system Acusim.cnf. You need to change the 2 lines, to read as follows:
message_passing_type= msmpi
view_message_passing_type= msmpi
You need to use msmpi. Original setting was using HPMPI.
7) So, once you are done, you can submit your job and start having fun! I used the following parameters. “D:\Jerald_Share\AS_door>acusub -np 12 -sched ccs -pdir \\hpchead\Jerald_Share\AS_Door -pb door”