Monitoring of the OPERA detector RPCs Prof. Riccardo ... - Infn

Monitoring of the OPERA detector RPCs Prof. Riccardo ... - Infn

Universitá degli Studi di PadovaFacoltá di Scienze MM·FF·NNDipartimento di Fisica G. GalileiCorso di Laurea Triennale in FisicaMonitoring of the OPERA detector RPCsRelatori:Prof. Riccardo BrugneraDott. Alessandro BertolinLaureando:Andrea Pigato, 578134Padova, Settembre 2010

ContentsIndex 11 The OPERA experiment 22 RPCs 42.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2 Operating principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.3 OPERA RPCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.3.1 Technical details . . . . . . . . . . . . . . . . . . . . . . . . . 52.3.2 Quality control . . . . . . . . . . . . . . . . . . . . . . . . . . 52.3.3 The slow-control system . . . . . . . . . . . . . . . . . . . . . 63 RPCs monitoring 73.1 Available data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2 The OperaMon software . . . . . . . . . . . . . . . . . . . . . . . . . 73.2.1 Automatic tools . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2.2 On-demand tools . . . . . . . . . . . . . . . . . . . . . . . . . 94 Data analysis 104.1 High voltage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.2 Current . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.3 Pressure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.4 Temperature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 Further development 206 Final considerations 21A OperaMon technology 22A.1 The choice of Ruby . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22iii

A.2 The (giant) SC database . . . . . . . . . . . . . . . . . . . . . . . . . 23A.3 The full technology stack . . . . . . . . . . . . . . . . . . . . . . . . . 23Bibliography 24Thanks 27iv

AbstractThe aim of this work is to explain why RPCs need continuousmonitoring and to present a near real-time set of tools(both automatic and on-demand) we developed to monitor theOPERA detector ones.

Chapter 1The OPERA experiment. / Physics Letters B 691 (2010) 138–145OPERA 1 is a long baseline experiment (successfully! see figure 1.1 and [1]) designedto provide direct evidence of neutrino oscillation ν µ → ν τ through the appearanceof ν τ in a pure ν µ beam produced at CERN’s SPS 2 (Switzerland, 730 km far) [2].Figure 1.1: View of the first τ − candidate event (Sat Aug 22 21:27:33 2009 UTC), trasverse toneutrino direction.1 Oscillation Project with Emulsion-tRacking Apparatus2 Super Proton Synchroton2

The detector (figure 1.2) is located in the underground Gran Sasso Laboratory(LNGS in Italy) at a depth of 1400 m (2.7 km water equivalent), and consists oftwo twin supermodules made of a massive target and a muon spectrometer each.The basic constituent of the target is the “brick”, a modular structure made oflead plates and nuclear emulsion layers sandwiched together to provide, respectively,the large mass and the high precision tracking needed for τ detection. In each supermodule,78336 bricks (more than 600 ton) are arranged in 24 walls instrumentedwith electronic tracker panels to allow event location and drive the (automatic)collection and scanning of the right brick(s).More prominent for the purpose of this work is a description of the muon spectrometers:each supermodule’s target is followed, downstream, by a dipolar magnetmade of two iron walls interleaved by 6 (front, between, behind) high precision drifttubes trackers planes. Copper coils sorrounding the top and bottom return pathsprovide a 1.55 T magnetic flux density in the tracking region. Each wall is made of12 5 cm-thick iron slabs separated by 2 cm air gaps in which 7 rows of 3 bakeliteRPCs 3 are embedded. 2 additional RPCs planes are placed upstream the magnets(XPCs) and used to resolve ambiguities in high tracks density events.The main task of the OPERA RPC system is to (coarse) reconstruct tracks insidethe magnets, especially for stopping muons, and to provide start/stop triggers forthe higher precision trackers electronic [3]. The whole OPERA RPC system hosts1008 RPCs, corresponding to ∼ 3200 m 2 of instrumented surface.Figure 1.2: Wideshot of the OPERA detector.spectrometers are well visible.Target Trackers (TT) and RPCs intrumented3 Resistive Plate Chambers3

Chapter 2RPCs2.1 IntroductionResistive Plate Chambers (RPCs) are gas particle detectors largely used in manyexperiments (LHC, Babar, Argo. . . ), often chosen because of their good space-timeresolution and mechanical simplicity.A single chamber (figure 2.1) is made of two plane parallel electrodes made ofhigh-resistivity bakelite, with a gas mixture in between. High voltage is applied on athin graphite layer placed on the external side of each electrode [4]. The outermostplastic layer provide the needed insulation. The 2 mm gap uniformity betweenthe electrodes is provided by a lattice of plastic spacers. The inner surface of eachelectrode is coated with a few microns thick polymerized linseed oil layer.Figure 2.1: RPC cross section.4

2.2 Operating principleRPCs’ sensitive element is the 2 mm thick gas layer [5]. Ionizing particles crossingthe gas give rise to a discharge by freeing electrons. Components in the gas mixturequench the discharge by avoiding auto photoionization and capturing outer electrons.Typical discharge size and time assure good space-time resolution. Signal readoutis performed by means of pickup electrodes outside the chamber.2.3 OPERA RPCs2.3.1 Technical detailsThe OPERA RPC system hosts 1008 rectangular detectors, arranged in planes madeof 7 rows of 3 chambers each. The size of a single RPC is 2.9 × 1.1 m 2 , thus theoverall sensitive surface is around 3200 m 2 (figure 2.2). The thickness of every RPCis between 6 and 7 mm.Electrodes volume resistivity ρ has been measured and found ∼ 10 11 - 10 12 Ω · cm[6]. An opposite potential of ∼ 2800 V is applied on the graphite layer of both sides,creating a uniform, steady electric field [4].The inside of each chamber is filled with a pre-mixed mixture of Ar, C 2 H 2 F 4 , iso-C 4 H 10 and SF 6 in the proportion of 75.4/20/4/0.6%, and flushed indipendentlyat a rate of 5 refills per day [7]. The copper and stainless pipes used in the gasdistribution system prevent excessive humidification.The RPCs are operated in streamer mode, which provide large amplitude easyto-discriminatesignals. The double coordinate read-out is performed by means of∼ 3 cm pitch crossed strips placed on both side of each plane.2.3.2 Quality controlBecause of their position in the experimental setup (the vast majority is embeddedin the iron walls of the spectrometer, thus inaccessible for the whole experimentlifetime), a strict quality control program, performed in a dedicated facility at theGran Sasso external laboratory, preceded RPCs installation [4].Long-term tests were performed on real size both prototype and pre-productionRPCs. At the low counting rates measured on the experiment site (∼ 20 Hz/m 2 ),detectors aging can only derive from intrinsic defects causing very high local dischargerates [6].To prevent premature aging, mechanical (gas leakage, curvature, gluing) andelectrical (resistivity, ohmic and operating current) properties have been verified on5

Figure 2.2: The assembly of an OPERA RPCs plane. It is possible to distinguish a partial planeof RPCs detector (5 black rows at the top), and the copper pickup strips (at the bottom).every single chamber. To follow, the response to cosmic rays has been tested interms of efficiency, counting rate and noise map [8]. Stringent requirements lead toa 30% RPC rejection rate.2.3.3 The slow-control systemRPCs are delicate detectors: parameters like temperature, gas pressure and humidity,voltage and working current can influence bakelite properties, performances andaging.The OPERA slow-control (SC) system has been designed to real-time control everyhardware component, and consists of data structures stored in a enterprise-classrelational database and a set of tasks to manipulate the data [9]. Data acquisition isperformed by a pool of hardware connected clients on a time basis. Other tasks monitorselected parameters and trigger alarms when out of ranges. Until now, someparameters have been saved almost exclusively for offline data correlation duringdata analysis.With this thesis we present a set of tools, both automatic and on-demand, developedto exploit a larger fraction of the SC database for a faster and easier routinemonitoring of the OPERA RPCs.6

Chapter 3RPCs monitoring3.1 Available dataData stored in the SC database include:• voltage data, from 4 SY127 CAEN [10] power supplies;• current data, from positive channels active distributors (imeters)• gas pressure data from a centralized sensor;• temperature data from termistors placed on each RPCs plane.3.2 The OperaMon softwareWe developed a set of tools, both automatic and on-demand, to manipulate theavailable data. During the development, we proposed a new set of statistical tablesto properly store data manipulation results, as well as the update of some existingtables. At the beginning of August 2010 these changes have been applied to the SCdatabase.The software has been successfully deployed to the operaweb server( at the Gran Sasso data center. For further details on thetechnology stack, please see Appendix A.3.2.1 Automatic toolsThe automatic part of OperaMon consists of two Ruby scripts run on a time basis.Monitor runs every 12 hours, and is integrated with the OpRealIO system, whichprovides timestamped apparatus statuses for offline correlation (figure 3.1). Monitorbasically checks if any of the high voltage channels underwent undervoltage issues7

during the last OPERA data taking round and highlight which RPCs supermodules,planes and rows were affected.Figure 3.1: A sample output of the Monitor script. The presented format is human readable:OpRealIO needs the Unix timestamp as first value and an error code instead of “undervoltage”.Latest three columns represent the affected supermodule, plane and row.Stats runs every 3 hours and provides statistical data on a set of new tablescreated on purpose. Data include current mean, median and standard deviation,pressure mean, temperature mean and overall high voltage channels downtime. Exceptfor the centralized pressure sensor, the other statistical data are given persupermodule. The high voltage downtime table provides additional RPCs planeand row information. Stats timespan has been chosen after various tests that consideredthe acceptable delay in the monitoring interface, the level of details neededin the graphs and the database performances.The automatic, time based computation of the statistical data and the storage onnew tables has become necessary because of two main reasons. First, tables hostingvoltage, current, pressure and temperature data become very large (tens of millionsrecords). Querying data from these tables is a very heavy and time consumingoperation, so it was a better choice to automatically analyze small bunches of data(every 3 hours) and store results on another table for faster reading. Second, becauseof their fast increasing size RAW data tables need a few weeks long “dump &clean” cycle. Statistical data would have been limited to few weeks in the past andinaccessible at the beginning of each cycle.The script has been also run on SC database backups to pre-fill the statisticaltables with data since June 2010.8

3.2.2 On-demand toolsA user interface has also been developed to provide easy access to the database mostrelevant data. It consists of a small, Ruby on Rails based web application (figure3.2) specifically designed to ease the everyday, routine monitoring of the OPERARPCs. Available features include:• high voltage channels real-time time series;• high voltage channels overall downtime;• current snapshot histograms, both real-time and past;• current, temperature and pressure statistical data;• basic data correlations.Other minor features have been added on request of the Padua OPERA team.Figure 3.2: A screenshot of the RPCs monitoring web interface provided by OperaMon. The graphsat the top are the supermodules’ currents snapshots.9

Chapter 4Data analysisIn this chapter we will explain in detail the data on which we built the OperaMonsoftware. We will also be able to present some early results.4.1 High voltageThe electric potential between the electrodes is an important parameter for RPCs,because they rely on the uniform, steady and intense electric field to produce a fastdischarge in the gas layer [5].In the OPERA RPC system high voltage is provided by 4 SY127 CAEN powersupplies with 40 channels each, divided in 32 positive, 4 negative and 4 unused.Every channel, included unused ones, writes its own status on the SC database at arate of about 1 record per minute.OperaMon is cable-connection aware, so it extracts only connected channels voltagevalues to check if they are below a preset threshold (currently 1500 V for Monitorand 2600 V for Stats, well below the optimal value). When this happens, the downtimeis written both on the SC database and in the OpRealIO file format, togetherwith the affected supermodules, planes and rows. On the web interface, per channelvoltage time series are available (figure 4.1; the undervoltage peaks were caused byan unscheduled shutdown, see section 4.2). There is also a graph showing the highvoltage downtime on each supermodule, followed by a list of the affected channels(figure 4.2).OperaMon has already collected more than 30 days of downtime on about 150high voltage channels.10

Figure 4.1: Time series for both a good (left) and bad (right) hv channel from the end of July tothe end of August 2010. These graphs are printed by OperaMon itself.11

Figure 4.2: Overall hv downtime for supermodule 1, from the beginning of June to the end ofAugust 2010. This graph is printed by OperaMon itself.4.2 CurrentAn out of range current flowing in the gas between the electrodes can lead to sensorpremature aging, performance issues and even damages, thus this parameter mustbe constantly monitored [6].Positive high voltage channels are connected to imeter active distributors, whichhandle 8 inputs and 24 outputs each. Negative high voltage channels are connectedto passive distributors, which handle 2 inputs and 48 outputs each. Both thesekind of distributors are direcly connected to some RPCs rows on the spectrometers.Every imeter output channel, included unused ones, writes its own status on the SCdatabase at a rate of about 2 records per minute.Being cable-connection aware, OperaMon extracts only connected channels currentvalues to provide a near real-time current distribution histogram, together withstatistical data computed every 3 hours.In figure 4.3 we provide the supermodule 2 data since the beginning of June 2010.There is a linear currents increase during the first 18 days of June, which is beinginvestigated by the Padua OPERA team. After that, the currents mean is almostconstant, but a little noisy. Note also the slow increase of the standard deviation.Narrow peaks are well recognizable on the 29th July around 5 pm, caused by the12

sudden recharge of the whole RPC system after an unscheduled shutdown of thehigh voltage channels due to an error.Per channel current time series are also available (figure 4.4), limited to fewhours because of the computational effort. Good imeter channels have low, constantcurrents. Bad channels show higher and much more noised ones.13

Figure 4.3: Current statistical data for the second supermodule, from the beginning of June to theend of August 2010. These graphs are printed by OperaMon itself.14

Figure 4.4: About 2 days time series for both a good (left) and bad (right) imeter channel; thefirst is connected to sm: 2, plane: 24, row: 3, while the latter to sm: 1, plane: 13, row: 7. Thesegraphs are printed by OperaMon itself.15

4.3 PressureGas pressure changes modify gas density inside the RPCs and therefore trigger voltageadjustements [11] (figure 4.5). Pressure value is read from a centralized sensor,and written together with the voltage data. This parameter is almost constant, andOperaMon is trying to highlight possible seasonal fluctuations, computing statisticaldata every 3 hours.Data since the beginning of June are provided in figure 4.6. Note that the undergroundlaboratory pressure is artificially kept under the atmospheric value forsecurity reasons.Figure 4.5: Correlation between average voltage and pressure. This graph is printed by OperaMonitself.16

Figure 4.6: Pressure statistical data from the beginning of June to the end of August 2010. Thisgraph is printed by OperaMon itself.17

4.4 TemperatureHigh temperatures affect RPCs operations modifying bakelite electrical propertiessuch as resistivity ρ and dielectric constant ɛ, worsening the detector performanceespecially in streamer mode ([12] and [13]).On every RPCs plane 10 termistors (3 at the top, 4 in the middle, 3 at thebottom) write temperature data in the SC database at a rate of about 2 records perminute each.OperaMon analyzes top and bottom temperature values every 3 hours to providestatistical data about the expected temperature gradient and to highlight out ofrange values (the working temperature is about 17 ◦ C) and seasonal fluctuations.In figure 4.7 we provide both supermodules’ data since the beginning of June.As expected, temperatures are higher at the top and lower at the bottom. Theabnormal fluctuation of the supermodule 1 temperatures at the beginning of Augustwas caused by the failure of the magnet power supply.Thanks to OperaMon, abnormal temperatures as high as 50 ◦ C have been localizedon a single RPCs plane of the first supermodule, and reported to the OPERA teamfor further investigations.18

Figure 4.7: Temperature statistical data for both supermodules, from the beginning of June to theend of August 2010. Upper lines are top temperatures, while lower lines are bottom ones. Thesegraphs are printed by OperaMon itself.19

Chapter 5Further developmentOperaMon is still a work in progress. Even if the automatic part of the software hasbeen through several weeks of testing and the basic features of the web interface havebeen implemented, we plan to continue the development during the next months.We have already selected new functionalities, e.g. make the graphs embeddable inthe OPERA logbook and generate downloadable ROOT-uple files for each visualizedset of data; along with these, we have also thought about some other, more excitingones and we are collecting feedback from the users.We will be rolling out new features as soon as they will become available; also,technical documentation will be produced for the OPERA team.20

Chapter 6Final considerationsThe routine monitoring of the RPCs statuses is very important for the experimentsusing such delicate detectors. This task can be very much improved by means ofdedicated software tools.Even with its still quite basic interface, the web based part of OperaMon canalready ease the effort of the assigned personnel: it retrieves and makes available(safely and worldwide) relevant data from the SC database without requiring anySQL 1 knowledge nor other software than a web browser. The software is not final,and will be improved and extended over the next months.The automatic part of OperaMon is already in place, and meet the requiredspecifications to integrate with the OpRealIO system for offline data correlation.Also, data from the statistical tables will be relevant for RPCs aging studies andcould potentially lead to future publications.Noticeably, the development of OperaMon pinpointed some subtle issues in theSC database overall design: these issues have been fixed when possible, for thebenefit of the whole OPERA slow-control software ecosystem.1 Structured Query Language, used to manage data in a relational database21

Appendix AOperaMon technologyIn this chapter we would like to summarize some technical details about the softwareand the technology stack.A.1 The choice of RubyThe OperaMon software is written in Ruby [14]. The OPERA experiment slowcontrolsoftware ecosystem is dominated by C and C++ applications, so it wasquite a challenge to suggest and use this rather “exotic” language: nobody elsein the OPERA team had never used it before, and it required some extra workby OPERA sysadmins to make interpreter and needed libraries available on thedestination server.In the end we think it has proven to be an effective choice. Being interpreted,the software coincides with its source code, and even inexperienced personnel cansafely inspect a Ruby script because of its natural syntax (figure A.1). We couldtake advantage of a wide variety of open source libraries available for Ruby, some ofwhich written in C for heavy duty tasks (e.g. database connectors).The web based interface is based on Ruby on Rails [15], a modern and elegantopen source web framework. Being based on the MVC 1 pattern and favouringconvention over configuration, it allows faster development and easier maintenance.1 Model, View, Controller22

Figure A.1: A sample of Ruby code: this is the beginning of the Rails controller for the temperaturedata. Note the elegance and the natural syntax of the time functions used.A.2 The (giant) SC databaseThe SC database is powered by the enterprise-class PostgreSQL engine running ona dedicated computer.OperaMon needs to query some very big tables (up to more than 10 millions rows),so we had to deal with many database related performance issues, bottlenecks beingboth intrinsic and Ruby dependent. As a solution, we proposed a set of new tablesto store values computed every few hours on small bunches of data; we also createda full set of hand-crafted SQL queries to transfer most of the computational effortfrom the Ruby side to the database engine.During the development we noticed some subtle issues in the database overalldesign. These issues have been fixed when possible and worked around when not,and included time correlation of data from different tables and cables connectionsmapping.A.3 The full technology stackWe thought a full list of all the technologies we used to develop and deploy theOperaMon software could be of some interest.Ruby and Ruby on Rails, plus the needed libraries (or Gems), are at the coreof the whole application. For the web interface, we used a pretty standard mix ofHTML, CSS and Javascript, including some libraries like jquery and jqplot. Wehad to deal with two different database engines, PostgreSQL (already in place forthe slow-control system) and SQLite, a smaller and lighter one based on plain files.OperaMon source code is hosted on a remote GIT repository, for version control23

and backup purposes. The deploy is automatized by Capistrano, and the target( is a Red Hat based linux server with the Apache webserverand the Phusion-Passenger module.Every single piece of technology we used is open source software.24

Bibliography[1] Opera Collab.; Agafonova N. et al., Observation of a first ν τ candidate event inthe OPERA experiment in the CNGS beam, Phys. Lett. B 691, 138 (2010).[2] OPERA Collab., An appearance experiment to search for ν µ → ν τ oscillationsin the CNGS beam - Experiment Proposal, CERN/SPSC 2000-028.[3] Bertolin A. et al., The RPC system of the OPERA experiment, Nucl. Instrum.Meth. A 602, 631 (2009).[4] Bergnoli A. et al., Tests of OPERA RPC Detectors, IEEE Trans. on Nucl. Sc.52, 2963 (2005).[5] Cardarelli R. et al., Progress in resistive plate counters, Nucl. Instrum. Meth.A 263, 20 (1988).[6] Bergnoli A. et al., Tests on OPERA RPCs, Nucl. Phys. B Proceed. Suppl. 158,93 (2006).[7] Mengucci A. et al., Gas mixture studies for streamer operation of Resistive PlateChambers at low rate, Nucl. Instrum. Meth. A 583, 264 (2007).[8] Bergnoli A. et al., The quality control tests for the RPCs of the OPERA experiment,Nucl. Instrum. Meth. A 533, 203 (2004).[9] Garfagnini A. et al., The OPERA muon spectrometers, Nucl. Instrum. Meth.A 572, 177 (2007).[10] CAEN S.p.A. (, Viareggio (LU), Italy.[11] De Vincenzi M. et al., Study of the performance of standard RPC chambers asa function of bakelite temperature, Nucl. Instrum. Meth. A 508, 94 (2003).[12] Aielli G. et al., RPC operation at high temperature, Nucl. Instrum. Meth. A508, 44 (2003).25

[13] Bruni F, Hull G and Mari SM, Effect of temperature on bakelite properties usedfor standard RPC chambers, Nucl. Phys. B Proceed. Suppl. 158, 182 (2006).[14] Matsumoto Yukihiro et al., .[15] Heinemeier Hansson David et al., .26

ThanksThis work would not have been possible without the help of many people I wouldlike to thank here.First of all my two relators, prof. Brugnera and Sc.D. Bertolin. They not only gaveme the opportunity to work on something useful and needed, but also managedto stand and second (especially the latter, I apologize) my continuous struggle toavoid every kind of workaround during OperaMon development. They have beenabsolutely necessary, with their advices and explanations, in keeping the project ontrack.Sc.D. Garfagnini (Padua OPERA team), a very valuable aid for the SC system (evenat weekends!) and an immediate OperaMon enthusiast.Ing. Meschini, (OPERA Computing Support), who got everything up and runningon the operaweb server for the OperaMon deploy, and Diego Giorgini, (IT studentin Padua), for his not-just-technical help with the software.My family, because they are my first supporters.Claudia, for letting me think I’m smart enough.27

More magazines by this user
Similar magazines