Author: Jim Ray Subject: final report on IGS station clocks Date: 28 December 2011 Since the start of the former IGS/BIPM time transfer pilot project in the late 1990s, I have informally monitored the performance of station and satellite clocks in the IGS products and notified station operators of problems and other issues. With this final summary report, that personal effort now ends. Clock Products website ---------------------- Station operators, esp those with external atomic clocks, should get into the habit of periodically checking the plots posted by Ken Senior at: Final clocks = https://goby.nrl.navy.mil/IGStime/igst_plots_monthly.php Rapid clocks = https://goby.nrl.navy.mil/IGStime/igrt_plots_monthly.php Both are organized into monthly bins. The Rapids are updated daily (by 17:00 UTC) while the Finals are updated weekly (usually Thursdays). Ken recently added new plots of the form [YYMMM]_ssss_data_vs_timescale.ps where "ssss" is the station name, YY is the year, and MMM is the month. These plots are now available for all stations, including those without external atomic clocks, and show the time domain clock variations (as viewed from the GPS data). These are obviously useful to monitor the performance of external clocks, but they can sometimes also be diagnositic even for receivers using internal clocks. Plots of the form [YYMMM]_ssss_states_and_sigmas.ps give more detailed information for those clocks used in the IGS timescale algorithm, but are probably too technical for most users. Note that the new Hadamard deviation plots are insensitive to quadratic variations (normal for clock comparisons) whereas the previous Allan deviations are insensitive only to linear differences. Note also that Ken no longer removes the day-boundary jumps in computing Hadamard deviations, while they were removed in the old Allan deviations. This leads to visible stability differences at about 1 d intervals and longer, but the stability of the IGS timescales remains near 1e-15 at 1 d; it is improved over longer intervals thanks mostly to a better steering algorithm. Another valuable source of monitor information are plots of the clock discontinuities between successive days, which are only sensible for H-maser stations. Those are posted at: https://goby.nrl.navy.mil/IGStime/daybdy/ For background information, please see the review at: http://acc.igs.org/clocks/clock-pilot-proj_metro03.pdf and references therein. While there are ~72 IGS stations equipped with H-masers according to site logs, only about 20-25 of these contribute significantly to the IGS Rapid and Final ensemble timescales, and less than 15 dominate. Many of the best-performing clocks are at timing labs, which is understandable given their natural concern. Reasons for the shortfall in availability include: * lack of RINEX data or slow delivery * use by less than 2 ACs, the minimum required for a combined IGS clock product * observed stabilities much worse than H-maser performance The 2nd factor is aggravated when multiple IGS stations are located near one other, each with competing utilities. This dilutes AC usage and really should be avoided by consolidating such multiple occupations into a single super-station. (Data from ancillary stations is very useful and highly recommended but it should be reserved for local studies rather than distributed externally.) Sources of clock instability ---------------------------- There are many factors that can degrade the stability of H-maser signals before they reach the GNSS receiver, most often downstream from the maser itself. Here are some common sources of problems: * frequency multipliers -- Even when they operate normally, they can add diurnal variations due to temperature sensitivity. * frequency distribution systems -- These often cause problems but are needed at facilities like VLBI sites in order to make multiple copies of the H-maser reference signal. * poor environmental control -- Diurnal variations are common due to exposure of the H-maser (least likely), frequency signal distribution system, any multipliers, antenna cables, any signal splitters, or GPS receiver to temperature variations if the thermal sensitivity of any component is significant. In addition to these degradations in the timing systems, other GNSS general problems are also relevant: * poor cable connections -- Single cable runs are always preferable, of minimal length and number of connectors. Connectors should not be reset or disturbed unnecessarily, and should not be opened when there is any chance of invasion by water. Cables should not be bent or unnecessarily exposed to outside conditions. * poor RF environment near antenna -- Opportunities for multipath reflections should be minimized, esp from surfaces close to the antenna. Any reflections from behind and near the antenna (unless blocked by Eccosorb, for instance) will leak into the observations and introduce biases into the clock estimates as well as position and other parameters. These can vary as the electrical properties change due to rain and ice, for example. Consequently, the best antenna mounts have the least reflecting material near the antenna. This fact is seen in the repeability of time domain clocks and position time series among different stations. * faulty antenna or receiver -- Good antennas tend to operate for long periods without attention, apart from the clearing of snow or other accumulates or lightning strikes. Receiver failures are more common and often go unnoticed for long periods. Strong indicators of problems include unexplained loss of SNR, increased phase cycle slips (which should not exceed about 1 per 1000 obs), or changes in observed clock behavior (even for non-atomic clocks). Current status of H-maser stations ---------------------------------- Final comments on individual H-maser stations are included here. Cesium and rubidium stations are listed further below, but without comments. "High weight" refers to weight in the IGS timescale ensembles and is typically about 5%. "(lab)" marks to stations at timing labs. Due to length, the rest of this report is only available in the complete version, available at: http://acc.igs.org/clocks/final-report.txt algo -- still recovering from power failure on 2011-11-30 but normally gets high weight; day-boundary clock jumps are very high (800 to 900 ps RMS) & show very strong seasonal modulation due to mounting of antenna ~1 wavelength above pillar top amc2 -- very good stability & normally gets high weight; day jumps are (lab) good to moderate (140 to 220 ps) due to tripod antenna mount; this is a backbone station of the IGS clock products & is used for timescale steering due to its close linkage to UTC badg -- new IGS station colocated with VLBI; no clock results yet brew -- good stability & normally gets moderate weight in Finals; cycle slips about 1.5 to 2 per 1000 obs; day jumps are good (~160 ps) but have biased mean of -28 ps (Ashtech UZ-12 receiver) brft -- H-maser not used since 2011-02-07 due to problems when used with Leica GRX1200PRO receiver brus -- currently uses Cs due to H-maser failure ~2010-11-05 (lab) cedu -- observed stability is much poorer than H-maser presumably due to problems in freq distribution system cont -- little used due to competition with conz; intended for time transfer (lab) ops but antenna is mounted in very hostile environment conz -- ref frame station but competes with cont for AC usage; often good stability & high weight in Rapids but recurring data quality issues have sometimes limited usefulness; suffered from 50 ns resets till Leica GRX1200PRO was replaced with GRX1200+GNSS but cycle slips are high (~7 per 1000 obs); day jumps are fair (170-200 ps) but have a biased mean of about -20 ps cro1 -- good stability & normally gets moderate weight in Finals but AC usage is spotty in Rapids; data yields are low & cycle slips are high (~6 per 1000 obs) probably due to old receiver (Ashtech UZ-12); day jumps are good (~145 ps) but have biased mean of -12 to -20 ps gode -- performance degraded severely during June-July 2011 & it appears to run from internal clock now (despite log); data quality very poor (old Ashtech UZ-12 receiver) with highly excessive phase cycle slips; historic day jumps are high (225 to 245 ps) & variable with small or no bias godz -- no data for >1 month but usable clock results have always been spotty gol2 -- not normally used in favor of colocated gold (a JPS EGGDT receiver but using internal crystal), probably mostly because old receiver (Ashtech UZ-12) guao -- limited usage in Finals & not at all in Rapids so cannot be well evaluated, but phase cycle slips usually >1 per 100 obs (old Ashtech UZ-12 receiver); very unfortunate situation because of colocation with VLBI station URUMQI hers -- AC usage has been spotty in Finals & very rare in Rapids, which is highly unfortunate given the very high quality of this station; since start in late 2010, day jumps excellent at 124-133 ps with perhaps a small bias of -8 ps hob2 -- AC usage is good but the clock stability is quite poor with maser drift & frequent small jumps, possibly due to problems with Leica GRX1200GGPRO receiver using external clock (as seen at several other stations); so weight is low to zero; day jumps have always been quite high (537 to 581 ps) with a modest bias of -20 ps hrao -- stability is sometimes fair but can degrade quickly; data quality is fair given old receiver (Ashtech UZ-12); day jumps are moderate at ~200 ps with no bias ieng -- clock stability is only fair for a timing lab but its weight & AC (lab) usage are usually high, esp in the Rapids where good clocks are often more limited; data quality is OK (Ashtech Z-XII3T receiver) but day jumps are among lowest in IGS at 107-120 ps, which is somewhat puzzling given antenna mount over metal parapet on roof; on the other hand, bias of day jumps is large, about -35 ps irkj -- results are spotty due to competition with irkm & irkt even though this is a ref frame station & data quality from JPS LEGACY receiver seems excellent; historic day jumps are very high (about 700 ps) possibly related to unique monument involving concrete pillar encased in metal tube covered by a metal lid under the antenna irkm -- no recent clock results are available due to competition with irkj & irkt irkt -- original Irkutsk station but still equipped with ancient Rogue SNR-8000 receiver so data yields are very low & quality is poor; historic day jumps poor at about 300 ps with large bias of -49 to -68 ps; this station should be decommissioned & ACs should not use it any longer kgni -- no clock results & no data reported since 2011-07-26 (lab) khaj -- GPS clock has regular resets which make stability assessment difficult; only used in Finals with moderate to zero weights; data quality seems mostly OK; day jumps increased hugely in late 2009 but rare log updates & lack of response from station personnel do not allow correlation with any external event kokb -- clock stability & weight are both moderate; phase cycle slips >1 per 1000 obs, probably due to old Ashtech UZ-12 receiver; historic day jumps quite poor at about 323 ps, with significant bias of about -25 ps, but level of jumps has varied over time; last major increase in jumps occurred in summer 2008, probably associated with installation of powered signal splitter on 2008-09-24 kokv -- no clock results for this relatively new station that competes with colocated kokb, which shares its antenna; data quality seems considerably better than kokb (JPS EGGDT receiver); IGS ops should consolidate on this receiver & the kokb Ashtech receiver decommissioned kour -- clock stability is poor with >1 ns subdaily variations that repeat from day to day & with large offsets between days; cycle slips are >1 per 1000 obs (JPS LEGACY receiver); despite obvious day jumps in monthly time domain plots, computed day jumps are excellent at <150 ps & modest bias; apparent conflict might be caused by auto- editting out of most bad jumps so that only best days remain ksmv -- no clock results, probably due to competition with other Kashima stations, long RINEX latency, very low data yields, & excessive cycle slips mad2 -- no clock results since early 2009, probably due to competition from other, better stations in the Madrid area mat1 -- only very infrequent AC usage due to competition with colocated mate but data yields & cycle slips also need improvement mate -- clock stability & weight are both usually good except for 50 ns resets typical with Leica GRX1200PRO receivers; cycle slips are too high at 4 to 5 per 1000 obs; historic day jumps have been poor at 253 to 290 ps (possible modest bias) but the jumps have declined somewhat with recent generation of receivers mdvj -- the intrinsic clock stability is one of the best in the IGS network (lab) but the AC usage in the Rapids is infrequent unfortunately, probably due to ~3 hr RINEX latency; the data quality also seems to be excellent (TPS NETG3 receiver); however, the historic day jumps were the worst in the IGS for a long period, which was enventually traced to a faulty component in the freq distribution system, though the most recent jumps are still not great mdvo -- no results for this station, which is evidently no longer active (lab) medi -- clock stability suffers from slow freq drifts, probably due to the maser itself, so it gets only modest weight; it is almost never in the Rapids, probably because the RINEX latency is usually >1 hr; data yields are modest & cycle slips somewhat high (~3 per 1000 obs); day jumps are very high (650-700 ps), probably due to using a concrete pillar mount, with a large bias of about -65 ps mets -- regular clock results are not available since summer 2010, when highly unstable, non-maser performance was reported; AC usage has since dwindled, perhaps in part due to favoring the colocated metz station that has markedly better data quality but also a RINEX latency >3 hr; the historic mets day jumps were very bad, at 734 to 931 ps & with large biases nist -- clock stability & IGS weights are both excellent; data quality is (lab) usually good (NOV OEM4-G2) but the cycle slips can vary greatly from one day to the next; day jumps are moderate (~245 ps) but there is a very large bias of -73 to -115 ps. nlib -- no data has been available from this VLBI colocation station since fall 2009 due to antenna/radome installation issues nnor -- this station is unique in having an external crystal oscillator slaved to an H-maser; while much better than an isolated crystal clock, the hi-freq stability not adequate to be used in the IGS timescale ensemble & there is a pronounced diurnal variation with an amp of 1.0 to 1.5 ns not1 -- AC usage has been very limited for >2 years in both Rapids & Finals so it is not possible to make any current assessment; data quality could be better (cycle slips >1 per 1000 obs) & historic day jumps were poor at 295 to 320 ps, with some bias of -15 to -41 ps novm -- clock stability is normally quite good & weight is moderately high (lab) in Finals but not used in Rapids, probably due to >3 hr RINEX latency (at CDDIS); data quality (JPS LEGACY) seems generally good; day jumps are poor at 315 ps with modest bias of 15 ps npld -- this station discontinued operations ~2008.0 but has not been (lab) formally decommissioned nrc1 -- clock stability is normally good & weight is fairly high in both (lab) Rapids & Finals; data quality seems good despite age of receiver (Ashtech Z-XII3T); day jumps are very bad (730 to 928 ps) with extreme seasonal modulation (as with most Canadian stations), presumably related to winter snow/ice effects even though antenna is not mounted over a concrete pillar like other NRCan stations nrl1 -- basically no clock results available for this station due to heavy (lab) competition with numerous other IGS stations nearby nya1 -- non-maser performance since early 2010, probably due to problems with freq multiplier; same situation as with nyal but current data look like nya1 receiver is unlocked from external clock; highly unfortunate situation, esp very long unusable state; both stations seem to have excellent data with very high yields; historic day jumps were very high at 502 to 629 ps, mostly due to an extremely bad period in 2004; day jumps have moderate bias of about -30 ps nyal -- non-maser performance since early 2010, probably due to problems with freq multiplier; same situation as with nya1 though receiver reaction is different; highly unfortunate situation, esp very long unusable state; both stations seem to have excellent data with very high yields; historic day jumps were poor at ~390 ps onsa -- long history of very reliable, high-quality performance; clock stability is normally very good with very high weight in Rapids & Finals; this is a backbone station in the IGS clocks; data quality (JPS E_GGD) is very good; day jumps have always been among very best in IGS network at 95 to 110 ps (no significant bias), probably because of the unique use of an Eccosorb matrix surrounding the antenna (though radome is uncalibrated) opmt -- excellent clock stability & high weight in Finals but rarely used (lab) in Rapids because RINEX latency is ~2.4 d(!); data quality is good despite old Ashtech Z-XII3T receiver, but daily RINEX files always lack data for initial 00:00 epoch; day jumps are 300 ps with bias of 29 ps park -- sparse clock results show +-5 ns scatter & are unusable, probably due to major problems in local freq distribution system; data quality is very erratic from day to day, maybe caused by the input clock instabilities pie1 -- clock stability & weights are normally quite good in Rapids & Finals; data quality is OK despite old Ashtech UZ-12 receiver; historic day jumps about 304 to 323 ps (with small bias of about -10 ps) but improved greatly after new antenna & cable installed on 2007-01-23 prds -- clock stability fairly poor, with excessive subdaily variations for normal H-maser, possibly due to inadequate environmenal isolation; Rapid & Final weights are therefore only ~2% even though data quality is excellent for this very old AOA BENCHMARK ACT receiver; day jumps have always been very high, ~500 ps, as is normal for NRCan stations ptbb -- clock stability is normally very good & weights are high in Rapid (lab) & Final products; cycle slips too high at 8 to 10 per 1000 obs, probably due to old Ashtech Z-XII3T receiver; Rapid AC usage is surprising given a daily RINEX latency of ~200 min (at CDDIS), which is excessive & should be reduced; day jumps are ~best in IGS at 93 to 98 ps, probably due to antenna mount on mast, but bias is 23 to 43 ps roap -- no clock results available probably due to competition with (lab) colocated sfer sfer -- pronounced subdaily variations, presumably due to inadequate (lab) environmental isolation of some timing component; data quality is usually OK but day-to-day variations are significant; day jumps are moderately large at 334 to 412 ps with bias of 9 to 22 ps shao -- for unknown reasons, H-maser was disconnected on 2009-12-31 according to log even though maser-like performance has been seen at times since then till early 2011; data quality is OK; historic day jumps about 200 ps but volume of clock data is limited spt0 -- clock stability is normally good & weights high in Rapids & Finals; (lab) but most recent data quality is degraded, probably due to receiver according to station operators; this is the only antenna besides onsa that uses Eccosorb (since 2007-06-08) to block non-direct signals but evidently less successfully; as with ptbb, Rapid AC usage is surprising given a daily RINEX latency of ~200 min (at CDDIS), which is excessive & should be reduced; historic day jumps are 379 to 389 ps, which did not change when Eccosorb was installed but did improve markedly in late 2009 for unknown reasons stjo -- no maser-like clock results were observed from mid 2009 till very recently, for unknown reasons; maser-like performance returned in mid-December 2011 so it is too soon to make any assessment; data quality is very good considering old AOA SNR-8000 ACT receiver; historic day jumps were 314 ps but there are no strong seasonal variations, unlike most other NRCan stations svtl -- the intrinsic stability of the clock at this VLBI colocation site is among the very best in the IGS but large resets occur too often & results are only available in the Rapids since early 2011; data quality seems very good (TPS NETG3); day jumps are 224 ps; this station should get more AC usage! tid1 -- usable clock results are only available in the Rapids from 2004- 2005, probably due to competition with colocated tid2 & tidb; data quality seems OK (TRIMBLE NETR8); historic day jumps were very poor at 532 ps (with large bias of -99 ps) but they greatly increased in early 2005 for unknown reasons (also for tidb); IGS ops should consolidate on just a single station at this site tid2 -- clock results have only been spotty from this station & no data (old Ashtech UZ-12 receiver) have been available since July 2010 even though it has not been formally decommissioned tidb -- clock stability is usually OK with Rapid & Final weights of ~2% or less; data quality is fine despite old Ashtech UZ-12 receiver; historic day jumps have been very poor at 493 to 522 ps (with large bias of -49 to -52 ps); day jumps have varied greatly over time & last increased greatly in early 2005 for unknown reasons (also for tid1); IGS ops should consolidate on just a single super-station at this site tskb -- clock stability is rather poor & not often usable in timescales; cycle slips >1 per 1000 obs & variable; day jumps not computed for this station due to oversight twtf -- clock stability usually very good with high weights in Rapid & (lab) Final timescales; cycle slips ~4 per 1000 obs, probably due to old Ashtech Z-XII3T receiver; day jumps fairly good in Rapids (141 ps) but much greater in Finals (249 ps) due to outliers in 2007-2009; day jump bias about -23 ps in both usn3 -- clock stability normally very good with high weights in Rapid & (lab) Final timescales; data quality surprisingly good for this old Ashtech Z-XII3T receiver; day jumps quite good at about 132 ps, probably due to antenna mount aloft, but some outliers occur each winter, possibly due to snow accumulation in chokerings; this is a backbone station for clock products & used for timescale steering due to its close linkage to UTC usno -- clock stability also normally quite good for this station, which (lab) gets moderately high weights in Rapids & Finals, but not so much as usn3; data quality is excellent despite old Ashtech Z-XII3T receiver; historic day jumps have always been high, at ~400 ps, probably due to mounting on roof metal parapet but got much worse when Ashtech antenna was replaced with a different model & higher elevation cutoff (10 deg vs previous 5 deg) on 2009-09-19; this station is a backup to usn3 for timescale steering due to its close linkage to UTC usud -- clock stability is fine at high freq but poorer over days; weight in timescales is moderate with usage much greater in Finals than Rapids; data quality is fine despite old Ashtech UZ-12 receiver; day jumps ~170 ps wab2 -- clock stability is very good over subdaily intervals but poorer (lab) over days; weights are very high in Rapids & Finals, among very highest in network; data quality is fine despite old Ashtech UZ-12 receiver; day jumps are excellent at 109 ps in Rapids but greater in Finals (207 ps) due to isolated outliers from unknown causes; day jump biases are 21 to 48 ps; this station is a key contributor to the IGS timescales wdc3 -- data temporarily suspended since 2010-06-09 while waiting for (lab) calibration of new antennas; prior to that, day jumps were excellent at 122 ps, probably because of antenna mount on mast wdc4 -- no clock results available for this backup to wdc3 (lab) wes2 -- clock stability is OK over subdaily intervals but much poorer over longer periods so timescale weights vary between 0 & ~4%; 50 ns clock resets occur regularly as with other Leica GRX1200PRO receivers; cycle slips usually >1 per 1000 obs despite modern receiver but vary with time; day jumps have always been high, at ~300 ps but levels have been lower for past ~2 years wsrt -- subdaily clock stability is very good & so timescale weights are normally very high, but occasional breaks up to ~1 ns seen; cycle slips are extremely low for this old AOA SNR-12 ACT receiver; day jumps rather consistently high at ~300 ps with modest bias of -13 ps; this is a backbone station for IGS clock products because of its high reliability & AC usage, as wellas good stability wtzr -- despite numerous problems over the years, including environmental (lab) influences on receiver, current subdaily stability is excellent, apart from occasional 50 ns resets despite change of receiver from Leica GRX1200PRO to GRX1200+GNSS on 2011-04-14; so timescale weights are usually about 3 to 5%; data quality seems to be very good; day jumps have consistently been around 173 to 190 ps, probably affected by back-reflections from nearby metal roof parapet wtzs -- no clock results are available for this station, probably because (lab) of competition with the various other colocated stations; IGS ops should consolidate on a single station at Wettzell wtzz -- only very sparse clock results are available, probably because of (lab) competition with the various other colocated stations; IGS ops should consolidate on a single station at Wettzell yebe -- clock stability is poor with repeating subdaily variations & large jumps between days, which has been attributed to construction of an adjacent, taller bldg in 2008; so the timescale weights are only ~1%; data quality is usually OK; day jumps were ~250 ps before the construction in 2008 but are now ~522 ps yell -- subdaily clock stability is very good & timescale weights normally among highest in IGS; data quality from this old AOA SNR-12 ACT is excellent; like most NRCan stations, day jumps are very high, ~700 ps, with strong seasonal modulations; this is a backbone station for IGS clock products because of its high reliability & AC usage, as wellas very good subdaily stability despite large day jumps zeck -- H-maser installed 2009-05-13 & has mostly performed well since then, although moderately large subdaily variations sometimes occur; timescale weights vary as a result though usually fairly high; data quality is very good; day jumps not computed for this station due to oversight List of Cesium stations (according to log files) ----------------------- ade1/ade2 aira bhr1/bhr2 brus bue1/bue2 ccj2/ccjm daej dlft eil1/eil2 gmas gmsd gope harb hrm1/hrm2 kiru mas1 mizu ohi3 osn1/osn2 pert pre1/pre2 qui1/qui2 sant stk2 suwn sydn syog tah1/tah2 tsk2 vill wel1/wel2 wtza yakt List of Rubidium stations (according to log files) ------------------------- ban2 bogi chur coco dubo iqal jplm lck2/lcki lpgs mar6 mcil ous2 pbr2/pbri pots qaq1 scor tash thu2/thu3 tnml tro1 vis0 whit wuhn yar2