Data Editing

Data Editing 

Flagging 

Compression 

The first challenges: 

*Data are BIG 

*Data has significant RFI 

We need to FLAG & COMPRESS. 

Louise Ker, LOFAR-UK Data School, 30/08/11

The LOFAR Flagging Tools 

-Our subband is 12Gb in size, with 1s integrations and 64 channels – need to compress in time 

and frequency. (Use msoverview in the tutorial to check this for yourself) 

-It's also noisy, and we need an automated way to remove RFI (no AIPS!) 

Images from Offringa et al. (2010) 

Automated Flagging and Compression done in NDPPP (New Default Pre-Processing Pipeline). 


Example Usage 

NDPPP is usually run by the Radio Observatory automatically. 

One line: 

ker@lce072> NDPPP NDPPP.parset 

NDPPP 

Flagging 

Compress 

AOFlagger 

MADFlagger Time Frequency 

-Use the parset file to specify what you want to do. 

Result: A compressed & flagged file of a much more managable size 

12Gb------>113Mb 


Example Usage 

msin = /data/scratch/tutorials/cyga/L24921_SB005_uv.MS 

msin.startchan = 1 

msin.nchan = 62 

msin.datacolumn = DATA 

msout = "L24921_SB005_uv.MS" 

msout.datacolumn = DATA 

steps = [preflag,flag1,count,avg1,flag2,avg2,count] 

preflag.type=preflagger 

preflag.corrtype=auto 

flag1.type=madflagger 

flag1.threshold=4 

flag1.freqwindow=31 

flag1.timewindow=5 

flag1.correlations=[0,3] 

avg1.type = squash 

avg1.freqstep = 64 

avg1.timestep = 1 

flag2.type=madflagger 

flag2.threshold=3 

flag2.timewindow=51 

avg2.type = squash 

avg2.timestep = 5 

Flag the first & last channels 

Steps required 

flags the autocorrelations 

Flag with MADFlagger 

flags the XX and YY polarizations 

Compress the data 

(Compresses to one channel) 

2 nd round of Flagging 

compresses 5 time-slots i.e. 15 s 


Data Inspection 

CASABROWSER 

CASAPLOTMS 

Variety of DIY Python scripts too for data inspection (see Neil's tutorial later this week) 

-Casa should only be run on compressed datasets. 

-Can manually flag any missed RFI (in practice this is rarely necessary) 

-Most importantly, ALWAYS check the data at this point. 


Try it Yourself 

Summary: 

-need to COMPRESS to have manageable dataset 

-need to FLAG to remove RFI in an AUTOMATED way. 

-need to INSPECT – LOFAR is still in commissioning! 

First Interactive Session: 

-Logging into the cluster 

-Run msoverview to get details of the Measurement Set. 

-Running NDPPP to compress & flag our Cygnus A subband. 

-Inspecting the data 

(P99-104 in the LOFAR Imaging Cookbook) 

Please ask questions! 


Advanced Things to Try 

Some Notes 

Demix 

Adv Models 

Global Calibration 

Beam Correction 

New Imager 

To test soon? 

Flag 

NDPPP 

Calibrate 

BBS 

Image 

Casapy 

Model 

Self-calibrate 

Steps in red will get you started... 

Steps in green will take us up to more or less the current commissioning stage. 

All well documented in Cookbook and can be easily worked through 


Demixing 

Amplitude 

Time 

Required for essentially all LBA, & some HBA 

datasets. 

-The 'A-Team' CassA, CygA etc dominate at 

low frequencies & need to be removed. 

-demixing allows subtraction of these sources 

prior to calibration 

Ripples from 

CasA etc. 

Phase 

Raw data, one baseline 3C196 

Time 

Figures courtesy 

George Heald 

Same baseline, demixed, calibrated, 

3C196 subtracted 


Demixing 

-How to? 

Follow Chapter 7 in the Cookbook on a raw subband. 

> /home/diepen/scripts/do_demixing.py 

BUT... 

- does not work when A-Team source within 30 degrees of target. 

-VERY compute intensive, only one demixing should ever be run at a time per 

node. 

-takes a long time … 

-Make sure you have enough room (demixing requires ~100Gb space), and 

remove all the intermediate products afterwards. 


Beam Correction 

-BBS has an option to calibrate taking into account the LOFAR beamshape. 

-Just need to add a couple of lines to parset file: 

Step.solve.Model.Beam.Enable = T 

Step.correct.Model.Sources = [CygA] 

Step.correct.Model.Beam.Enable = T 

need to specify direction 

-*Big* caveat. Self-calibration cannot be done for wide-field imaging, as the 

Casa Imager cannot apply the LOFAR beam correction. 

Work on an imager which can is in progress, and should be ready for testing 

soon.. 


Making Models 

-Some models in wide use such as the A-Team, & some 3C sources etc are available in 

/globaldata/COOKBOOK/Models. 

-casapy2bbs.py converts a casa .model to a bbs source catalogue (as you did in the tutorial to self-calibrate). 

-Can also use pyBDSM, which takes a fits image & produces a bbs source file from detected sources 

(Chapter 9 in Cookbook). 

-Always worth asking other commissioners for existing models. 

>use LofIm 

>use LUS 

>pybdsm 

BDSM [1]: inp process_image 

BDSM [2]: filename = ’CygA.fits’ 

BDSM [3]: go 

BDSM [4]: write_gaul(bbs_patches='source') 

--> Wrote BBS sky model ’CygA.pybdsm.sky_in’ 

VLSS Image 

PyBDSM 


Global Calibration 

We are also able to calibrate multiple subbands simultaneously. 

Again, just a few lines needed in the parset file.. 

For example, 10 subbands of our Cygnus A observation: 

>Strategy.UseSolver = T 

>Step..Solve.CalibrationGroups = [10] 

No more than 5-10 subbands (1-2MHz bandwidth) should be used due to 

increasing problems with ionosphere & station clock drift. 

Global calibration is particularly helpful for fainter fields, where S/N from just one 

subband is low. 


Troubleshooting 

The software sometimes fails …. 

[FAIL] error: setupsourcedb or remote setupsourcedb-part process(es) failed 

Things to check.... 

-Are your parset files in the correct format? 

-INSPECT your data. 

-Look at the end of the .log files produced by the software – more explicit error 

messages can usually be found there. Still not sure? Post log output on LOFAR Users 

Forum. 

-Note sometimes software fails due to changes on the cluster, new bugs in bbs etc, 

again Forum a good place to look for help.. 


Typical Reduction 

LBA 

LBA or HBA? 

HBA (typically done by RO) 

Demix 

High res clean 

component model 

BBS 

YES 

Bright, central 

source? 

Long baselines may need to be 

flagged with coarser VLSS models 

Global calibration useful for all, 

but essential for faint fields 

No 

NDPPP 

Model from 

pyBDSM+VLSS 

BBS 

Check & flag solutions, 

corrected data 

Check & flag solutions, 

corrected data 

Self-Cal: casapy image, 

bbs, casapy2bbs 

Image in Casapy 


Commissioning 

Sign up to sources of help: 

-LOFAR Wiki – there's a record of commissioners' work under 'Commissioning/busy Wednesdays. A lot 

include the bbs parset & casapy parameters used – good reference! 

-LOFAR Users Forum – up to date record of any system downtime, bugs etc. 

-Join a Busyweek/Busy Wednesday (can do this remotely). 

-Getting started: 

-Check wiki for up to date list of data available 

-HBA bright central 3C sources (there are lots of these!) are good 'starters'. 

A 3C HBA Source: 

-Same basic procedure as Cygnus A. 

-Usually the radio observatory flags and compresses HBA datasets, and transfers 4 subbands per 

observation to the compute nodes for commissioners to get started (so no initial NDPPP). 

-You then need to 1)Inspect the data, do any additional flagging necessary. 2) Make a good sky model (e.g. 

pyBDSM), 3) Calibrate in BBS, 4) Check & flag solutions & corrected data, 5) Image in Casapy, 6) Selfcalibrate 

as required. 

-A good point to start is the uv-plane-cal.parset given in the cookbook for BBS. Have a browse through the 

wiki for other parsets. 


That's All! 

Find out more... 

LOFAR User's Forum 

LOFAR Busy Days & Busyweeks (can join remotely) 

Plenty of Data & exciting science on the way! 

Courtesy van Weeren 

Remember things change quickly – hopefully this 

talk will be out of date soon! 

Bootes Field at 50MHz, Ker.

Data Editing

Create successful ePaper yourself

Delete template?

Save as template?