¶ Source Scan
Required for the FINAL COLD stage of full QC. See the pixel module QC page on LBNL. The source scan is done with the source box, a separate setup from the thermal control cold boxes used for the majority of QC.
¶ Quick Start Guide
To start using the source scan setup, it is assumed you have remote access to the SpikePig computer, and have the following repositories installed and compiled: YARR (devel branch), PixelModuleQC, Software Interlock, and for the latter two labRemote (interlockUpdate branch). It is also assumed you have access to a Grafana dashboard to monitor box conditions, either the primary Sourcebox_6238 dashboard or a similar one. For detailed instructions on setting up these requirements, see the Software Environment section below. It is also assumed you will use the default sourcebox configuration files included in the PixelModuleQC and Software Interlock repositories. These will generally be somewhat up to date, but if you need to alter them see the Configs and Connectivity section below.
The quick start guide is divided into sections that you can skip if not needed. The first covers physically installing a module in the box along with vacuum and dry air. The second covers running monitoring and the software interlock in the background. The third is about actually operating the box: changing temperature and powering the module safely, as well as safely shutting everything down. The final section covers running the necessary scans with YARR.
¶ Module Setup
The first step is to install the module. Remove the baseplate from the carrier case, being careful to only touch the case when handling the module. Turn on vacuum to the box by fully opening the valve labeled "VACUUM"; you should hear a hissing sound from the vacuum chuck. Gently place the module on the chuck so that it completely covers it, with no gaps between the module and gasket. If done correctly you should hear the vacuum sound stop and feel the module be suctioned down slightly, and the vacuum regulator dial should display between 50-60 kPa.
Plug the module data and power pigtails into their respective adaptor cards as shown. BE CAREFUL with the power pigtail -- if it is misaligned such that the wrong pins are contacted, powering the low voltage or high voltage could break the module. A good check that it is properly aligned is that once monitoring is started, the module NTC reading should be a reasonable, physical temperature. If it is failing to read or giving absurd values, it is likely an indication that the power pigtail is misaligned.
Next, turn on dry air flow to the box by fully opening the nearby valve labeled "DRY AIR". Ensure that air is flowing into the box; the flow meter should show between 30-60 SCFH. After this you should be able to close and lock the box. When monitoring begins you should see the box humidity and dewpoint dropping as dry air enters; the dewpoint should stabilize at approximately -40˚C.
With the module connected, lid closed, and dry air flowing, the hardware interlock traffic light should soon shift from red to yellow. You can then press the reset button on it to make it green. From this point forward if the HW interlock triggers it means something is actually wrong.
¶ Monitoring and Software Interlock
Next begin monitoring of all relevant box conditions using PixelModuleQC. It is helpful to run this in a separate tmux session:
tmux new -s monitor
cd ~/pixelmoduleqc
source qcenv/bin/activate
python ./scripts/coolman.py -c conf/sourcebox_config.json monitor
This should begin running with no terminal output. You can confirm it is functioning by looking at the Grafana dashboard, where all plots should begin displaying data. To safely leave the tmux session running, use CTRL+B -> D; do not use the exit command as it will close the session.
The Software Interlock should be set running in a similar way:
tmux new -s swint
cd ~/pixelmoduleqc
source qcenv/bin/activate
python interlock.py -c sourcebox-cfg.json
It should produce only the following lines of output:
INFO:root:Using software interlock config file: software-interlock/sourcebox-cfg.json
INFO:root:Checking communication with all devices
INFO:root:Monitoring quantities
Again safely exit with CTRL+B -> D.
¶ Box Operation
The coolman program can control temperature automatically by operating the peltiers, but we avoid automatic control of the module low voltage and high voltage. To control all three, paste these commands into your .bash_profile to operate each power supply individually:
alias hv='~/pixelmoduleqc/build/bin/powersupply -e ~/pixelmoduleqc/conf/sourcebox_config.json -n DT5472N'
alias lv='~/pixelmoduleqc/build/bin/powersupply -e ~/pixelmoduleqc/conf/sourcebox_config.json -n HMP4040 -c 1'
alias peltier='~/pixelmoduleqc/build/bin/powersupply -e ~/pixelmoduleqc/conf/sourcebox_config.json -n NGP804 -c 1'
and run source ~/.bash_profile. Each power supply has the same set of commands, so with these aliases you can simply run hv <command>, lv <command> or peltier <command> as needed. For full command information use --help as the command. The most important ones are:
get-voltage: read the set output voltage (current)
meas-voltage: measure the actual output voltage (current)
set-voltage -- <V>: set the output voltage to V in Volts (current to I in Amps)
power-on: output power at the set voltage or current
power-off: stop outputting power; in the case of high voltage this safely ramps down automatically
The low voltage power supply should be set to output 5.88A (max 3V) -- this should not be changed, as 5.88A is what is required to power the module. The high voltage power supply should be set to bias at -100V -- just power this on or off as needed and it will automatically ramp safely, but be warned your terminal will be occupied for the ~minute it takes to ramp. The peltier power supply should be controlled automatically with coolman.py, but to use it manually set it to ~7.2V (max ~11A) -- this is approximately what is needed to keep a powered module at -15˚C, but can be changed by small amounts to adjust the temperature (more voltage = lower temperature).
To properly cool and bias the module, make sure the dewpoint is low (< -30˚C) then run (with qcenv activated):
hv power-on
lv power-on
python ~/pixelmoduleqc/scripts/coolman.py -c ~/pixelmoduleqc/conf/sourcebox_config operate <temperature>
where the temperature is in degrees Celsius.
After a few minutes the module should settle within about 1˚C of -15˚C, and the coolman controller will keep it there. The operate command also takes a time (in seconds) as an optional second command-line argument, in which case it will keep the module at the given temperature for the given time, then turn it off and warm it back up.
If at any point the peltier and low voltage power supplies turn off on their own, the software and/or hardware interlock has likely been triggered, due to either high dewpoint, high or low module temperature, low air flow, or the box opening. Let the box safely shut down and return to stability around room temperature before operating it again, and check what caused the interlock to trigger. In the case of the hardware interlock let it return to standby (yellow) and reset it with the button on the board.
To safely warm up the box again manually, make sure to first run peltier power-off and then lv power-off, so the module does not cool unexpectedly. Once the module is safely warming to room temperature with both of the former depowered, you can run hv power-off.
¶ Running Scans
A full explanation of tuning, masking, and running scans in YARR is best left to the YARR Docks. Assuming you have the connectivity configs for whichever module is installed in the box, and that it is properly masked if it has core column issues, you should start with an eye diagram scan. You can then follow the example tuning procedure here, or whatever is most up to date.
After tuning, you should run a digital scan with the -m 1 option to reset the mask, followed by an analog scan and then a noise scan. The noise scan time can be edited in the appropriate scan config. Finally, to actually operate the source box, expose the module to the source (by opening the primary shutter) and run a selftrigger_source scan. The time taken can be edited in the relevant config; 5 minutes is sufficient to test the system, while 30 minutes or 1 hour should be used for a proper source scan. For a chip with core columns disabled, note that there are two separate places this must be done: the EnCoreCol parameters and the HitOrMask parameters; they should take the same values. AdditionallyAfter the scan is finished close the primary shutter before running anything else.
¶ Software Environment
The source scan box is controlled and monitored through SpikePig, a CentOS 7 machine. It can be accessed remotely via ssh <username>@spikepig.dhcp.lbl.gov. Paste the following two commands into your .bash_profile, or run them before operating the box:
source /opt/rh/devtoolset-9/enable
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
The first enables GCC version 9 for building YARR; the second is necessary for the python virtual environments to run PixelModuleqQC and the software interlock.
The box can be run by any user on SpikePig provided they have the following repositories installed:
¶ YARR
To communicate with the module you should clone the devel branch of YARR into your home directory:
git clone --branch devel https://gitlab.cern.ch/Yarr/Yarr.git ~/Yarr
and compile the minimum build:
cd Yarr
mkdir build
cd build
cmake3 ../
make -j4
make install
For further information about YARR or troubleshooting see the YARR Docks
¶ PixelModuleQC
To control and monitor the box hardware and sensors you should clone the PixelModuleQC repository into your home directory:
git clone https://gitlab.cern.ch/berkeleylab/labremote-apps/pixelmoduleqc.git ~/pixelmoduleqc
and run the setup script:
cd pixelmoduleqc
source setup.sh
This will clone a branch of labRemote and compile it, then create and activate a Python virtual environment, qcenv, which should be used when running PixelModuleQC. To exit the venv type deactivate, and to reactivate it later use source qcenv/bin/activate. See the documentation for further information about PixelModuleQC or labRemote.
¶ Software Interlock
To run the software interlock you should clone software-interlock into your home directory:
git clone https://gitlab.cern.ch/berkeleylab/labremote-apps/software-interlock.git ~/software-interlock
The Software Interlock also uses labRemote. It should be run in the same python virtual environment as PixelModuleQC, so use source ~/pixelmoduleqc/qcenv/bin/activate before running the interlock.
Make sure to add the ~/pixelmoduleqc/scripts directory to your Python paths so software interlock can import the HeartBeat module by running this command in your software interlock tmux session:
export PYTHONPATH=$PYTHONPATH:/home/user/pixelmoduleqc/scripts
Where you replace "user" with your account name.
For more information about the software interlock see the documentation.
¶ Grafana Monitoring
The most convenient way to monitor the source box is Grafana, a web app that allows for real-time plotting from a remote SQL database.
This is an example snapshot of the current source box Grafana dashboard. You should be able to log in using your LDAP credentials (lbl email up to the @ sign, and password). Some Grafana tutorials are available, but it may be easier to just play around with the source box monitoring dashboard (Sourcebox_6238) to get a feel for it. The dashboard is already set up to pull from pixel_module_qc in InfluxDB; currently source box monitoring is all written to the sourcebox_test table. The name of this table, along with all sensor names and other database information, is generally controlled in the pixelmoduleqc config in the section defining sensors.
¶ Configs and Connectivity
There are two main JSON config files, one used by PixelModuleQC to define hardware, sensors, monitoring, and control and another used by the Software Interlock to set alarm and shutoff boundaries.
¶ PixelModuleQC Config
The following is a list of sections of the JSON config file, together with whatever information you may need to change or understand while running the box.
-
setupname: There should be no need to change this from "sourcebox" unless you want to mark monitoring data in InfluxDB with a different "setup" tag. -
datastreams: This governs output streams, there should be no need to change it. -
datasinks: This contains InfluxDB port and credential info, there should be no need to change it. -
devices: This lists connected equipment like power supplies and meters.nameis your name for the device, to be used elsewhere in the config.hw-typeandhw-modeldefine the labRemote device type.communicationcan mostly be left alone, with two exceptions:interlock-filecannot be the last list item in this section, because of JSON idiosyncrasies.portcan in theory change whenever the computer or devices reboot or when devices are unplugged. To check the port of a device, runls /devwith the device unplugged, then again after connecting it to the computer -- the new item in the/devfolder corresponds to that device. Generally the prefix is consistent (power supplies start with "ttyACM", e.g.) but the following number may change.
-
channels: This lists controllable power output channels on any devices.nameis your name for the channel, to be used elsewhere in the config.devicemust be an item in the above list of devices.channelis the output channel on the equipment. Numbering starts from 1.- The other options should be self-explanatory.
-
monitorsensors: Confusingly named, this is the list of sensors and other connected intermediates like the FT232 or arduinos.nameis your name for the sensor, to be used elsewhere in the config.typeis defined inpixelmoduleqc/scripts/pixmodqc/config.pyquantitygoverns the sensor name tag in InfluxDBprescaleis the prescale factor applied to a measurementfieldslists the measurements taken by a sensor. The key governs the measurement name in InfluxDB and the value is the corresponding method name for whatever object type this sensor is.- Most other options here are specific to certain sensor types and should be mostly self-explanatory. For more information about any of these values reading through
pixelmoduleqc/scripts/pixmodqc/config.pyand other scripts referenced there should help.
-
monitoring: This governs monitoring groups and measurement interval.groups: It is recommended to keep measurements through the FT232 together because they are all quite fast. Different measurement speeds is the most important thing to consider when grouping sensors.allsensors: Just list all the sensorsmeasurementName: This tags the entire monitoring stream in InfluxDB. Be careful about writing to existing tables or streams.
-
watchdog: This controls the watchdog program. It will likely soon be deprecated in favor of the Software Interlock. -
control: This governs everything about the controller function of PixelModuleQC. [I will return to this when I have it running] -
cycles: This defines temperature cycles for the cold box. Irrelevant for the source box.
¶ Software Interlock Config
The following is a list of sections of the JSON config file, together with whatever information you may need to change or understand while running the software interlock.
-
setup-details: This tells the interlock where to look for ongoing monitoring.lab-remote-cfg-fileshould point to the config used by PixelModuleQC, described above. The interlock will get most of its information from that JSON.interlock-filemust match the file listed in the PixelModuleQC config for each of the "devices".setup-monitoringmust match the setup name in the PixelModuleQC config.
-
quantities: This lists each quantity that the interlock can trigger on.nameis your name for the quantity.enabledis the flag that tells the script whether or not to monitor this quantity (0 = NO, 1 = YES).typeis defined insoftware-interlock/interlock.py.sensorsmust match the corresponding sensor names in the PixelModuleQC config.fieldsmust match the corresponding measurement names in InfluxDB (and the PixelModuleQC config).n-of-retry-influxDB-queryis the max number of attempts allowed for the script to fail reading sensor data for this quantity before triggering interlock, default is 3.measure-timeis the time interval the interlock requests from InfluxDB, so it governs how long without a measurement will trigger the "no data" shutdown from the interlock.upper-cutoffis the measurement threshhold above which the interlock will trigger and shut all devices downupper-warningis the measurement threshhold above which the interlock will send a warning message both in the terminal output and to the email listed below.upper-enabledgoverns whether both upper limit points are active- The corresponding "lower" keys control the same things, but for dropping below certain values.
-
notification-settings: Please please please set this to your email so I don't get alerts every time you test out the system. -
shutdown-settings: This governs the safety ramp-down of the high voltage power supply, but I believe it is mostly irrelevant because the labRemotepower-offcommand used here already safely ramps down any HV supplies.
¶ Source Box Hardware
¶ Radioactive Source
An Am-241 source mounted in the top of the box is used to irradiate the module for source scans. It has a nominal activity of approximately 3.7 GBq, primarily alpha and gamma production. The effective activity for a 200 µm silicon module mounted 10 cm away from the source inside the box is approximatly 0.8 MBq. The activity outside the box is negligible. The source is shielded by two independent shutters, both of which must be open to expose it to the inside of the box. The primary shutter is manually opened and closed:
The secondary shutter is automatically closed when the box door is opened, and vice versa:
NOTE: there is currently a mechanical issue with the primary shutter sticking, will update when it is fixed.
¶ Plumbing
¶ [REVISIT ONCE TUBING IS FIXED]
The box uses house vacuum and dry air, both piped through (sealed) holes in the back. The dry air flow can be controlled manually using the valve labeled "DRY AIR". The air flow can be visually monitored with the flow meter next to the computer. The air is also put through a disposable filter before entering the box, to catch some oil that has been building up in the tubing and flow meter. Vacuum can be controlled manually by the valve labeled "VACUUM" and can be visually monitored with the attached vacuum regulator, but is also digitally monitored by PixelModuleQC.
The module stackup inside the box sits on a cold plate, hooked up to a VWR 1197P recirculating chiller. Unfortunately this model is not currently implemented in labRemote, so the chiller is just manually controlled for now, using the screen on top of the bath. For the expected source scan module temperature of -15˚C it is not necessary that the chiller go below +15˚C, so it is set to that by default. The coolant was topped up as of March 2024; if the chiller seems to have issues staying at +15˚C check the bath for ice buildup or similar coolant problems.
¶ Major Equipment
The following is a list of hardware used in the setup, along with their purpose and details should they need to be replaced. The names in quotes at the end of each entry are its hardware type and hardware model in labRemote, which are used in the configs.
- High Voltage Power Supply: DT5472N power supply from CAEN. Single channel, 500V/1mA, USB-B powered from computer. Provides bias voltage for the sensor. "PS", "DT5472NPs".
- Low Voltage Power Supply: HMP4040 power supply from Rohde & Schwarz. Quad channel, 32V/10A, Wall-powered, USB-B connection to computer. Provides power to the module, and separately provides input voltage for the analog vacuum pressure sensor. "PS", "RS_HMP4040".
- Peltier Power Supply: NGP804 power supply from Rohde and Schwarz. Quad channel, 32V/20A, Wall-powered, USB-B connection to computer. Provides power to both peltiers through a single channel. "PS", "RS_NGP804"
- Multimeter: DMM6500 digital multimeter from Keithley. Single channel front, plus 10 two-pole channels through scanner card in back. Not currently used, replaced with simple ADCs because it's slow and I hate it, but available. "Meter", "DMM6500".
- Chiller: 1197P refrigerant circulator from VWR. RS232 connection to computer, but not currently implemented in labRemote, so it is manually controoled for now.
¶ Minor Equipment and Sensors
The following is a list of sensors, devices, and smaller equipment used in the setup, along with their purpose and details should they need to be replaced. If a sensor is implemented in labRemote it will be noted.
- Canary Board: Lab environmental condition monitoring board, custom (?). Data uploaded to InfluxDB using python script `/home/lgrossman/canary-env/readSerial_new_influx.py. Make sure measurement name and tag are set to "6238-environment" and "6238", respectively, to write to the correct place in InfluxDB. Run in a separate tmux session using any python virtual environment.
- FT232: FT232H FTDI chip on Adafruit breakout board, USB-C to I2C connection. Used to read out several I2C devices, including ADCs and HYT271 temp/RH sensors. Serial number: "pixel" (Don't ask). Make sure switch labeled "I2C Mode" is set to "on".
labRemote.devcom.FT232H - I2C Multiplexer: PCA9548A mux on Qwiic breakout board, 8-channel I2C multiplexer. Default I2C address
0x70, configurable from0x70-0x77.labRemote.devcom.PCA9548ACom - Analog to Digital Converters: ADS1015 ADC on Qwiic breakout board, 12-bit, 4-channel. Default I2C address
0x40. Operating voltage 3.3V.labRemote.devcom.ADS1015 - Digital Temp Sensor: HYT271 digital temperature/humidity sensor, board mount with I2C connection. Default I2C address
0x28. Extremely delicate.labRemote.devcom.HYT271 - Vacuum Pressure Sensor: MPX5100 analog vacuum sensor, read out through ADC. Operating voltage 5V, powered by HMP4040 PS. Measurement parameters taken from datasheet.
- NTCs: Both on-module and chuck NTC resistors currently have the same parameters: 10kΩ at 25˚C, BNTC=3435. Read out through ADCs.
- Peltiers: Thermoelectric cooling elements used in module stackup. 1487-387004685-ND on bottom, 926-1029-ND on top. Currently no room for Thermoelectric Interface Material (TIM), if this changes use squares of P122035-ND at each Peltier boundary.
- Relays: Panasonic AR1029 relays used to break Peltier and Low Voltage power supply if hardware interlock triggers.
- Air Flow Meter: King flow meter, monitored visually, range approx. 2 to 30 l/min.
- Vacuum Regulator: SMC vacuum pressure regulator, monitored visually, rated for -1.3 to -100 kPa.