N310 Sync With White Rabbit

Hi all,

I'm trying to use the White Rabbit switch to synchronize the acquisition across multiple N310s receivers.  I'm trying to use the instructions on this Ettus article:

https://kb.ettus.com/Using_Ethernet-Based_Synchronization_on_the_USRP%E2%84%A2_N3xx_Devices

I'm using the USRP sink block in gnuradio and set the "clock source" to "internal" and the "timing source" to "sfp0" as it indicates in the article

However I get the following error

[INFO] [UHD] linux; GNU C++ version 7.3.0; Boost_106501; UHD_3.14.0.HEAD-0-g6875d061                          

[INFO] [MPMD] Initializing 1 device(s) in parallel with args: mgmt_addr=192.168.3.2,                           type=n3xx,product=n310,serial=316A5C2,claimed=False,addr=192.168.3.2

[INFO] [MPM.main] Launching USRP/MPM, version: 3.14.0.0-g6875d061

[INFO] [MPM.main] Spawning RPC process...

[INFO] [MPM.PeriphManager] Device serial number: 316A5C2

[INFO] [MPM.PeriphManager] Initialized 2 daughterboard(s).

[INFO] [MPM.PeriphManager] init() called with device args `clock_source=internal,time_source=internal'.

[INFO] [MPM.RPCServer] RPC server ready!

[INFO] [MPM.RPCServer] Spawning watchdog task...

[INFO] [0/Replay_0] Initializing block control (NOC ID: 0x4E91A00000000004)

[INFO] [MPM.PeriphManager] init() called with device args `clock_source=internal,product=n310,mgmt_addr=192.168.3.2,time_source=internal'.

[INFO] [0/Radio_0] Initializing block control (NOC ID: 0x12AD100000011312)

[INFO] [0/Radio_1] Initializing block control (NOC ID: 0x12AD100000011312)

[INFO] [0/DDC_0] Initializing block control (NOC ID: 0xDDC0000000000000)

[INFO] [0/DDC_1] Initializing block control (NOC ID: 0xDDC0000000000000)

[INFO] [0/DUC_0] Initializing block control (NOC ID: 0xD0C0000000000002)

[INFO] [0/DUC_1] Initializing block control (NOC ID: 0xD0C0000000000002)

[INFO] [0/FIFO_0] Initializing block control (NOC ID: 0xF1F0000000000000)

[INFO] [0/FIFO_1] Initializing block control (NOC ID: 0xF1F0000000000000)

[INFO] [0/FIFO_2] Initializing block control (NOC ID: 0xF1F0000000000000)

[INFO] [0/FIFO_3] Initializing block control (NOC ID: 0xF1F0000000000000)

[ERROR] [RPC] TDC Failed to reset.

Traceback (most recent call last):

  File "/home/cmatson/data_lts/muddi_char/test_multi_radio.py", line 206, in <module>

    main()

  File "/home/cmatson/data_lts/muddi_char/test_multi_radio.py", line 194, in main

    tb = top_block_cls()

  File "/home/cmatson/data_lts/muddi_char/test_multi_radio.py", line 78, in __init__

    self.uhd_usrp_source_0.set_time_source('sfp0', 0)

  File "/usr/local/lib/python2.7/dist-packages/gnuradio/uhd/uhd_swig.py", line 3067, in set_time_source

    return _uhd_swig.usrp_source_sptr_set_time_source(self, source, mboard)

RuntimeError: RuntimeError: Error during RPC call to `set_time_source'. Error message: TDC Failed to reset.

[INFO] [MPM.Magnesium-0] Re-initializing daughter board. This may take some time.

[ERROR] [MPM.Sync-0] TDC Failed to Reset! Check your clocks! Status: 0x0

[ERROR] [MPM.RPCServer] Uncaught exception in method set_time_source :TDC Failed to reset.

 Traceback (most recent call last):

  File "/usr/lib/python3.5/site-packages/usrp_mpm/rpc_server.py", line 182, in new_claimed_function

    return function(*args)

  File "/usr/lib/python3.5/site-packages/usrp_mpm/periph_manager/n3xx.py", line 626, in set_time_source

    self.set_sync_source(source)

  File "/usr/lib/python3.5/site-packages/usrp_mpm/periph_manager/n3xx.py", line 723, in set_sync_source

    skip_rfic=args.get('skip_rfic', None)

  File "/usr/lib/python3.5/site-packages/usrp_mpm/dboard_manager/magnesium.py", line 385, in update_ref_clock_freq

    self._reinit(self.master_clock_rate)

  File "/usr/lib/python3.5/site-packages/usrp_mpm/dboard_manager/magnesium.py", line 344, in _reinit

    self.init(args)

  File "/usr/lib/python3.5/site-packages/usrp_mpm/dboard_manager/magnesium.py", line 293, in init

    args, self._init_args, fast_reinit)

  File "/usr/lib/python3.5/site-packages/usrp_mpm/dboard_manager/mg_init.py", line 615, in init

    args

  File "/usr/lib/python3.5/site-packages/usrp_mpm/dboard_manager/mg_init.py", line 555, in _full_init

    args)

  File "/usr/lib/python3.5/site-packages/usrp_mpm/dboard_manager/mg_init.py", line 221, in _sync_db_clock

    target_offset=trace_delay_offset)

  File "/usr/lib/python3.5/site-packages/usrp_mpm/cores/tdc_sync.py", line 201, in run

    self.configure(force=True)

  File "/usr/lib/python3.5/site-packages/usrp_mpm/cores/tdc_sync.py", line 254, in configure

    raise RuntimeError("TDC Failed to reset.")

RuntimeError: TDC Failed to reset.

 

I can't seem to find anything about "TDC" online.  Once I get this error, it doesn't seem to recover even if I change the parameters back, and the only way I have found to fix it is to reset the device.

I believe the white rabbit switch is configured properly because I am able to synchronize between two daughterboards within a single N310 by setting the "sync" parameter to "unknown PPS" which then calls the set_time_next_pps.  If I don't set the "clock/timing_source" parameters in the usrp block and just have the "sync" parameter set to "unknown PPS" on two N310s, the individual pairs daughterboards are sync'd but not between the two radios and there's a seemingly random delay (of up to several ms) between the two radios.

Thanks,

Cameron