2,295
edits
LUXSimUsingHTCondor: Difference between revisions
Jump to navigation
Jump to search
→In order to use the computing resources at Wisconsin, one can submit HTCondor jobs to run simulation
TapasSarangi (talk | contribs) |
|||
| (28 intermediate revisions by 2 users not shown) | |||
|
== Submit HTCondor Jobs ==
<nowiki>
ssh <username>@lzlogin01.hep.wisc.edu
# To get help about options
# python /cvmfs/lz.opensciencegrid.org/sw/bin/runWiscJobs.py -h
python /cvmfs/lz.opensciencegrid.org/sw/bin/runWiscJobs.py \
--WorkFlow="test.mac_test1" \
--Executable="/afs/hep.wisc.edu/user/cvuosalo/work/lz/RunLUXSim.sh" \
--Experiment=lz \
--nJobs=10 \
--TransferInputFile="/afs/hep.wisc.edu/user/cvuosalo/work/lz/test.mac" \
--HDFSProdDir=outfiles \
--UserEnv="UserName=`whoami` " \
--OutputDir="test.mac" \
--Arguments="test.mac"
# If you want your output copied to HDFS, have your executable copy the output
# files to the directory you specify with --HDFSProdDir.
</nowiki>
==== Executable example: RunLUXSim.sh ====
<nowiki>
#!/bin/bash
wdir=$(pwd)
ver=release-4.4.2
# Source the environment needed to run LUXSim
export ROOTSYS=/cvmfs/lz.opensciencegrid.org/ROOT/v5.34.32/slc6_gcc44_x86_64/root
export PATH=${ROOTSYS}/bin:/sbin:/usr/sbin:/usr/bin:/bin:/usr/local/bin
export LD_LIBRARY_PATH=${ROOTSYS}/lib:/lib64:/usr/lib64:/lib:/usr/lib
source /cvmfs/lz.opensciencegrid.org/geant4/etc/geant4env.sh geant4.9.5.p02
# Temporary directory that gets created on the execute node
tmpDir=$(mktemp -d --tmpdir="./")
echo "Linking CVMFS LUXSim area to temp dir. $tmpDir"
cd $tmpDir
# ln -s /cvmfs/lz.opensciencegrid.org/LUXSim/release-4.3.2/* .
ln -s /cvmfs/lz.opensciencegrid.org/LUXSim/$ver/* .
# echo "MyOutputName=$MyOutputDir/$MyOutputName$MyRandomNumber"
# export MyOutputName=$MyOutputDir/$MyOutputName$MyRandomNumber
cp ../*.mac .
# Run the LUXSim
./LUXSimExecutable $@
mkdir -p ../outfiles
for i in *.bin
do
if [ -s "${i}" ]; then
# Convert .bin file to .root file
echo /cvmfs/lz.opensciencegrid.org/LUXSim/$ver/tools/LUXRootReader "${i}"
/cvmfs/lz.opensciencegrid.org/LUXSim/$ver/tools/LUXRootReader "${i}"
(cd ../outfiles ; ln -s ../$tmpDir/${i} .)
else
echo "Something went wrong... No bin files made. Look at the log"
exit 1
fi
done
cd ../outfiles
ln -s ../$tmpDir/*.root .
</nowiki>
==== Example test.mac ====
<nowiki>
/run/initialize
/LUXSim/io/alwaysRecordPrimary true
/LUXSim/io/updateFrequency 100
/LUXSim/io/outputDir .
/LUXSim/io/outputName EnergyDeposition_
/LUXSim/detector/select LZDetector
/LUXSim/detector/gridWires off
/LUXSim/detector/topGrid on
/LUXSim/detector/update
/LUXSim/detector/recordLevel LiquidXenonTarget 3
/LUXSim/detector/recordLevelOptPhot LiquidXenonTarget 3
/LUXSim/detector/recordLevelThermElec LiquidXenonTarget 3
/gps/particle e-
/gps/energy 30 keV
/gps/position 0 0 75 cm
#this is approximate center. See ElectronCollection for whole volume
/gps/ang/type iso
/LUXSim/physicsList/useOpticalProcesses true
/LUXSim/beamOn 5
exit
</nowiki>
==== Files in the storage area ====
* Some events are available in the Wisconsin LZ storage area and can be accessed using the following command :
xrdfs g22n02.hep.wisc.edu:31094 ls /lz/user/tapas/LZSim/pmt-Co_gdecay_1_pro_sci_ioni_drift/
xrdfs g22n02.hep.wisc.edu:31094 ls /lz/user/tapas/LZSim/pmt-Co_gdecay_1_pro_sci_ioni_drift_gammma100KeV
* copy the files using xrdcp
xrdcp -d 1 -v \
root://cmsxrootd.hep.wisc.edu//lz/user/tapas/LZSim/pmt-Co_gdecay_1_pro_sci_ioni_drift_gammma100KeV/filename.root .
| |||