Versioning Infrastructure Michael Mirman - Perforce

Versioning Infrastructure
Michael Mirman
Perforce Administrator
1

Introduction
2

Architecture Overview
§ ~750 users worldwide
§ Several million archive files
§ ~50 Perforce-related hosts
§ Almost all possible triggers;
several daemons
§ Mirroring our own bug
database into Perforce
§ Interfaces: P4, P4V,
P4Perl, P4Java, Emacs,
P4Eclipse, P4DB, P4EXP
3

Describing Servers
4

Using markup language
brokerhost=p4brokerprod
p4broker
Primary
$brokerhost
[/local/perforce/bin/p4broker -d -c /local/perforce/config/broker.conf]
5

Using markup language (cont.)
§
Defining primary and secondary servers
master=p4host-00
standby=p4host-01
Primary
$master
[$bin/p4d -d -p 1666 -In Master -r /local/p4/db -J /local/p4/jdir/journal]
Secondary
$standby
[$bin/p4d -d -p 1666 -In Standby -r /local/p4/db -J off]
Master
(primary)
p4broker
Standby
(secondary)
6

Using configurables
§
Configurables complement start-up commands
-> p4 configure set P4LOG=/local/perforce/logs/p4d.log
-> p4 configure set Standby#serviceUser=service
-> p4 configure set ‘Standby#startup.1=pull -i 2 -J /local/perforce/jdir/journal’
-> p4 configure set ‘Standby#startup.2=pull -i 1 –u’
-> p4 configure set FwdReplica#db.replication=readonly
-> p4 configure set FwdReplica#lbr.replication=readonly
-> p4 configure set FwdReplica#rpl.forward.all=1
-> p4 configure set FwdReplica#serviceUser=service
-> p4 configure set ‘FwdReplica#startup.1=pull -i 2 -J /local/perforce/jdir/journal’
-> p4 configure set ‘FwdReplica#startup.2=pull -i 1 –u’
7

Maintenance Commands (start up scripts)
§
§
Keep commands as similar as possible.
They can read the configuration file(s)
and make the right decisions.
Start-up commands – based on the
configuration file and local host name
decide what processes (services) must
be started
§
/etc/init.d/p4d.current -> /local/perforce/
support/init.scripts/p4d.prod
8

Maintenance Commands (crontabs)
§ Crontabs on different hosts look practically
identical, making repurposing hosts easy
# Daily maintenance
00 03 * * * /local/perforce/cronjobs/p0X.cron
# Rolling journal
35,05 * * * * /local/perforce/cronjobs/roll_journal -conf /local/perforce/CONFIG/test_config.xml
9

Deployment of maintenance scripts (version 1.0)
§
“Push” using change-commit trigger:
push1 change-commit //depot/p4admin/... "updateRefTree -p %serverport% -c
p4serv.p4adm.p00,p4serv.p4adm.t00“
§ Comma-separated list defines clients to be updated
§
§
Problem: some clients are on different hosts
Attempt to solve it:
push2 change-commit //depot/p4admin/... "updateRefTree -p %serverport% -c
p4serv.p4adm.p00,p4serv.p4adm.t00:ssh:p4test00“
§
Problem: Triggers line is too long and cumbersome
10

Deployment of maintenance scripts (version 1.1)
§
Leverage spec depot, file spec, and client spec:
push3 change-commit //depot/p4admin/... "updateRefTree -p %serverport% -c p4serv.p4adm.p…“
§
§
§
Spec depot contains all client specs
§ Trigger gets all effective clients
p4 files -e //spec/client/p4serv.p4adm.p…
p4 print //spec/client/p4serv.p4adm.p…
Client specs contain Host
§ Trigger runs ssh to the given host to update client
Problems:
§
§
Sometimes trigger runs for a long time
If target host is down, we have no mechanism to update the target area
11

Deployment scalability issue
§
2009/12/31: 5 targets
• 2010/12/31: 13 targets
• 2011/12/31: 20 targets
• 2012/12/31: 49 targets
12

Deployment of maintenance scripts (version 2.0)
§
Pull, don’t push
set1 change-commit //depot/p4admin/... “set_mwsync -p %serverport% -c p4serv.p4adm.p…“
13

Deployment of maintenance scripts (cont.)
Set counter in the trigger
p4 set counter mwsync.host_name.client_name .me_stamp
14

Deployment of maintenance scripts (cont.)
§
§
Run cron job on every host that needs to
pull
Cron retrieves the counters, corresponding
to the local host name and syncs client(s)
*/5 * * * * /local/perforce/cronjobs/watch2sync -­‐p perforce:1666
• Monitoring verifies that no counters with “old” Omestamps exist on the master
00 6,12,18 * * * /local/perforce/scripts/lag_in_sync
15

Deployment of crontabs
§
§
§
§
Use change-commit trigger to install crontab
commit.cron change-commit //depot/crontabs/p4serv/... "install.crontab -p4port
%serverport% -changelist %changelist%"
Crontab files in the depot are //depot/crontabs/userid/host
For every crontab file the trigger retrieves from the change, the
trigger deploys by piping the content to this command:
ssh user@host crontab -
Additional insurance (what if ssh fails?) in every crontab
p4 -p perforce:1666 print -q //depot/crontabs/p4serv/$HOST > /tmp/crontab.$$; [ -s /
tmp/crontab.$$ ] && crontab /tmp/crontab.$$; rm /tmp/crontab.$$
16

Conclusion
17

Summary
§
§
§
§
Use meta language to describe infrastructure to allow change it
easily
Keep infrastructure description files, supporting scripts, and crontabs
in Perforce (possibly on different servers)
Use triggers to deploy changes in the infrastructure
If your servers often change roles:
§
§
Keep the details about the servers in configuration files, read by different scripts.
Keep crontabs and boot-up scripts identical between servers.
18

Q&A
Michael Mirman: Michael.Mirman@mathworks.com
19