LUXSimUsingHTCondor

Submit HTCondor Jobs

In order to use the computing resources at Wisconsin, one can submit HTCondor jobs to run simulation

ssh <username>@lzlogin01.hep.wisc.edu

# Make a valid voms-proxy
voms-proxy-init --voms=lz --valid=142:00

# Run farmout script to submit jobs
# To get help about options 
 # python  /cvmfs/lz.opensciencegrid.org/sw/bin/farmoutGeneric.py -h 

 python /cvmfs/lz.opensciencegrid.org/sw/bin/farmoutGeneric.py \
        --WorkFlow="test.mac_test1" \
        --Executable="/afs/hep.wisc.edu/user/cvuosalo/work/lz/farmoutluxsim.sh" \
	-e lz \
        --nJobs=10 \
        --TransferInputFile="/afs/hep.wisc.edu/user/cvuosalo/work/lz/test.mac" \
	-f '*.bin *.root' \
        --UserEnv="UserName=`whoami` " \
        -o "test.mac" \
        --Arguments="test.mac"


#The above python scipt generates job file using your input macro. \
#  It uses the shell script to run LUXSim and copy your output to LZ storage area. 

1. !/bin/bash

wdir=$(pwd) ver=release-4.4.2

1. Source the environment needed to run LUXSim

export ROOTSYS=/cvmfs/lz.opensciencegrid.org/ROOT/v5.34.32/slc6_gcc44_x86_64/root export PATH=${ROOTSYS}/bin:/sbin:/usr/sbin:/usr/bin:/bin:/usr/local/bin export LD_LIBRARY_PATH=${ROOTSYS}/lib:/lib64:/usr/lib64:/lib:/usr/lib source /cvmfs/lz.opensciencegrid.org/geant4/etc/geant4env.sh geant4.9.5.p02

1. Temporary directory that gets created on the execute node

tmpDir=$(mktemp -d --tmpdir="./") echo "Linking CVMFS LUXSim area to temp dir. $tmpDir" cd $tmpDir

1. ln -s /cvmfs/lz.opensciencegrid.org/LUXSim/release-4.3.2/* .

ln -s /cvmfs/lz.opensciencegrid.org/LUXSim/$ver/* .

1. echo "MyOutputName=$MyOutputDir/$MyOutputName$MyRandomNumber"
2. export MyOutputName=$MyOutputDir/$MyOutputName$MyRandomNumber

cp ../*.mac .

1. Run the LUXSim

./LUXSimExecutable $@

for i in *.bin do

   if [ -s "${i}" ]; then

     # Convert .bin file to .root file
    echo /cvmfs/lz.opensciencegrid.org/LUXSim/$ver/tools/LUXRootReader "${i}"
    /cvmfs/lz.opensciencegrid.org/LUXSim/$ver/tools/LUXRootReader "${i}"
   ln -s "${i}" ..

  else
    echo "Something went wrong... No bin files made. Look at the log"
    exit 1
  fi

done ln -s *.root ..

Example test.mac

/run/verbose      0
/control/verbose  0
/tracking/verbose 0
/grdm/verbose     0

/run/initialize

# Output folder
/LUXSim/io/outputDir .
/LUXSim/io/updateFrequency 1
/LUXSim/physicsList/useOpticalProcesses true


/LUXSim/detector/select LZDetector
/LUXSim/detector/gridWires on

/LUXSim/detector/printEFields

/LUXSim/detector/update

/LUXSim/materials/LXeTeflonRefl 0.95
/LUXSim/materials/GXeTeflonRefl 0.75
/LUXSim/materials/LXeAbsorption 100 m
/LUXSim/materials/GXeAbsorption 500 m
/LUXSim/materials/LXeSteelRefl 0.2
/LUXSim/materials/GXeSteelRefl 0.2
/LUXSim/materials/AlUnoxidizedQuartzRefl 0.9 
/LUXSim/materials/GXeRayleigh 500 m
/LUXSim/materials/LXeRayleigh 0.3 m

/gps/particle  e-
/gps/energy   5 keV
/gps/position 0 0 75 cm
/gps/ang/type iso

# Output .bin files

/control/getEnv MyRandomNumber
/control/getEnv MyOutputName
/LUXSim/io/outputName {MyOutputName}
/LUXSim/randomSeed {MyRandomNumber}
/LUXSim/physicsList/driftElecAttenuation 1.5 m
/LUXSim/beamOn 10

exit

Files in the storage area

• Some events are available in the Wisconsin LZ storage area and can be accessed using the following command :

  xrdfs g22n02.hep.wisc.edu:31094 ls /lz/user/tapas/LZSim/pmt-Co_gdecay_1_pro_sci_ioni_drift/
  xrdfs g22n02.hep.wisc.edu:31094 ls /lz/user/tapas/LZSim/pmt-Co_gdecay_1_pro_sci_ioni_drift_gammma100KeV

• copy the files using xrdcp

  xrdcp -d 1 -v \
   root://cmsxrootd.hep.wisc.edu//lz/user/tapas/LZSim/pmt-Co_gdecay_1_pro_sci_ioni_drift_gammma100KeV/filename.root .