LUXSimUsingHTCondor

Submit HTCondor Jobs

• In order to use the computing resources at Wisconsin, one can submit HTCondor jobs to run simulation.

  ssh lzlogin01.hep.wisc.edu

  # Make a valid voms-proxy
  voms-proxy-init --voms=lz --valid=142:00

  # Run farmout script to submit jobs
  python /cvmfs/lz.opensciencegrid.org/sw/bin/farmoutGeneric.py \
       --WorkFlow="test” \
       --nJobs=2 \
       --Executable=“/data/tapas/lzsim/submit/RunLUXSim.sh” \
       --TransferInputFile="/data/tapas/lzsim/submit/test.mac” \
       --UserEnv="MyOutputName=test_ “ \
       -o "test” 
       --Arguments="test.mac”

• Above input macro and Executable script can be used an example
• RunLUXSim.sh uses HEAD version of LUXSim

• Example RunLUXSim.sh looks like this :

 
#!/bin/bash

wdir=$(pwd)
ver=HEAD

# Source the environment needed to run LUXSim
export ROOTSYS=/cvmfs/lz.opensciencegrid.org/ROOT/v5.34.32/slc6_gcc44_x86_64/root
export PATH=${ROOTSYS}/bin:/sbin:/usr/sbin:/usr/bin:/bin:/usr/local/bin
export LD_LIBRARY_PATH=${ROOTSYS}/lib:/lib64:/usr/lib64:/lib:/usr/lib
source /cvmfs/lz.opensciencegrid.org/geant4/etc/geant4env.sh

# Temporary directory that gets created on the execute node
tmpDir=$(mktemp -d --tmpdir="./")
echo "Linking CVMFS LUXSim area to temp dir. $tmpDir"
cd $tmpDir
ln -s /cvmfs/lz.opensciencegrid.org/LUXSim/$ver/* .

echo "MyOutputName=$MyOutputDir/$MyOutputName$MyRandomNumber"
cp ../*.mac .

# Run the LUXSim
./LUXSimExecutable $@

uname=$(whoami)
for i in *.bin
do
    if [ -s "${i}" ]; then
     # Copy .bin files to storage /hdfs/lz/user/
     echo Trying... lcg-cp "${i}" -bvvD srmv2 srm://cmssrm2.hep.wisc.edu:8443/srm/v2/server?SFN=/hdfs/lz/user/${uname}/${MyOutputDir}/"${i}"
     lcg-cp "${i}" -bvvD srmv2 srm://cmssrm2.hep.wisc.edu:8443/srm/v2/server?SFN=/hdfs/lz/user/${uname}/${MyOutputDir}/"${i}"
      # Convert .bin file to .root file
     /cvmfs/lz.opensciencegrid.org/LUXSim/$ver/tools/LUXRootReader "${i}"
   else
     echo "Something went wrong... No bin files made. Look at the log"
   fi
done

# Copy root files to storage
for n in *.root
do
    if [ -s "${n}" ]; then
     echo Trying... lcg-cp "${n}" -bvvD srmv2 srm://cmssrm2.hep.wisc.edu:8443/srm/v2/server?SFN=/hdfs/lz/user/${uname}/${MyOutputDir}/"${n}"
     lcg-cp "${n}" -bvvD srmv2 srm://cmssrm2.hep.wisc.edu:8443/srm/v2/server?SFN=/hdfs/lz/user/${uname}/${MyOutputDir}/"${n}"
    else
     echo "Something went wrong... No Root files made. Look at the log"
    fi
done

• Following is an example test.mac :

/run/verbose      0
/control/verbose  0
/tracking/verbose 0
/grdm/verbose     0

/run/initialize

# Output folder
/LUXSim/io/outputDir .
/LUXSim/io/updateFrequency 1
/LUXSim/physicsList/useOpticalProcesses true


/LUXSim/detector/select LZDetector
/LUXSim/detector/gridWires on

/LUXSim/detector/printEFields

/LUXSim/detector/update

/LUXSim/materials/LXeTeflonRefl 0.95
/LUXSim/materials/GXeTeflonRefl 0.75
/LUXSim/materials/LXeAbsorption 100 m
/LUXSim/materials/GXeAbsorption 500 m
/LUXSim/materials/LXeSteelRefl 0.2
/LUXSim/materials/GXeSteelRefl 0.2
/LUXSim/materials/AlUnoxidizedQuartzRefl 0.9 
/LUXSim/materials/GXeRayleigh 500 m
/LUXSim/materials/LXeRayleigh 0.3 m

/gps/particle  e-
/gps/energy   5 keV
/gps/position 0 0 75 cm
/gps/ang/type iso

# Output .bin files

/control/getEnv MyRandomNumber
/control/getEnv MyOutputName
/LUXSim/io/outputName {MyOutputName}
/LUXSim/randomSeed {MyRandomNumber}
/LUXSim/physicsList/driftElecAttenuation 1.5 m
/LUXSim/beamOn 10

exit