1St Meeting of the Detector Advisory Committee

Clock and cControl Signal and Messagingfast signal sSpecification

M.Postranecky, M.Warren and D.Wilson 161.Dec.2009

- 1 -

1 Introduction 1

2 Overview of 2D pixel detector DAQ and control 1

3 Timing interfaces 1

3.1 XFEL timing interface 2

3.2 Non XFEL timing interface 3

3.3 Standalone timing interface 3

4 FEE fast signal interface 3

4.1 Line 1: clock 4

4.2 Line 2: control commands 4

4.3 Line 3: VETO command 4

4.4 Line 4: FEE status 5

1 Introduction 1

2 Overview of 2D pixel detector DAQ and control 1

3 Clock and Control Interfaces 2

3.1 Inputs from the Timing System 2

3.1.1 XFEL timing 3

3.1.2 Internal standalone timing 3

3.1.3 LCLS timing … to be finished 3

3.1.4 Spring8 timing … to be finished 4

3.1.5 Other light source timing … to be finished 4

3.2 Inputs and outputs to FEE (Fast Signals) 4

3.2.1 Line 1: CLOCK 4

3.2.2 Line 2: START and RESET Commands 5

3.2.3 Line 3: VETO / NO-VETO Commands 5

3.2.4 Line 4: STATUS Commands 6

4 Network Messages 6

5 Error Handling 6

6 Timing Considerations 7

6.1 Magnetic Coupler Skew 7

7 Veto Definition 8

7.1 Improved definition of 2D veto 8

7.2 Input signals 9

7.3 Veto unit (VU) 9

7.4 Output signals 10

7.5 FEE requirements 10

7.6 Issues 10

8 Work to be done 10

1  Introduction

This document is aimed at updating the definition of fast signals and network messages used by the Clock and Control system of XFEL 2D pixel detectors. Error handling and configuration issues are also addressed.

2  Overview of 2D pixel detector DAQ and control

Three 2D pixel detector collaborations (AGIPD, DSSC and LPD) are building detectors for use at XFEL. All detectors will use common backend DAQ and control systems and consist of three sub-systems:

§  Front End Electronics (FEE) of the detectors. The FEEs areis detector specific andcustom is being implemented by the each collaborationations separately.

§  Train Builder (TB) data readout. The TB is used to build data into frame ordered bunch train specific contiguous blocks which are then sent on to a computing farm layer for processing.

§  Clock and Control (C&C). The C&C interfaces to the XFEL timing system, generates and receives fast signals used to synchronize detector FEEs, and distributes configuration information using the fast signals or network messages.

3  Clock and Control Timing iInterfaces

This section describes thC&Ce timing interfaces of the clock and control system (C&C)interfaces. Timing interfaces (clocks and triggers) are required to allow detector operations at XFEL and other, non XFEL, light sources like LCLS. Additionally standalone running using internally generated timing signals is required.

A schematic showing the proposed fast signal connectivity is show in Figure 1., which is connected to both the Timing/Receiving (TR) computer and the Front-end electronics (FEE) of the 2D pixel detectors at XFEL. Additional external inputs for timing are also defined to be used to interface to other systems (i.e. LCLS, Spring-8, etc.)

RJ45 connectors are used for both the inputs and outputs. Socket A is the default input from the XFEL Timing Receiver (TR) board. Socket B is used for the external non XFEL timing sources like LCLS. The pin assignments of A and B are defined so that the sockets can be combined into a single socket if there are physical space limitations. Timing input cable lengths are expected to be short, e.g. the XFEL TR resides in the same crate as the C&C.

Socket C contains the outputs to the FEE along with the status input. Socket A and B can be combined into a single socket if physical space is limited.

Figure 1 Timing and FEE connection schematic

The different timing interfaces required are described below.

3.1  XFEL timing interface

Figure 1 C&C Socket Definitions

The pins of the external inputs (Socket A and B) are defined so that the sockets can be combined into a single socket if there are physical space limitations. The Status line is the only input from the FEE to C&C.

Cable lengths between the C&C and FEE modules are ≤5m. TR and C&C are in the same crate and cable lengths are short.

Inputs from the Timing System

This section defines input signals coming from the different external timing systems foreseen.

3.1.1  XFEL timing

The XFEL TR board will provide front panel and backplane timing signal outputs.

Four 5-pair har-link (http://www.harting-connectivity-networks.de/en/produkte/produktauswahl/interface-steckverbinder/metrische-interface-steckverbinder-im-20-mm-kontaktraster-har-link) front panel connectors provide access to the encoded trigger message data stream, the 4.51 MHz bunch clock and to programmable outputs. The later can provide trigger pulse (train START, STOP…), delayed trigger pulses (START + 15ms…), gates, bursts and patterns. The backplane PCIe will allow interrupt driven C&C crate CPU synchronization to the TR. Note that no technical or user documentation exists for the TR board which is currently being prototyped.

Table 1 following table defines the signals sent by the XFEL Timing Receiver (TR) boardTR signals required by the C&C..

Table 1 XFEL TR timing signals required

Table 2 Input from Timing Receiver board.

The encoded trigger message stream contains START train, STOP train, and other trigger messages.

The C&C requirements for trigger messages should be made known to the XFEL timing group, as should input signal characteristics (LVDS, CML…).

3.2  Non XFEL timing interface

Table 2 defines non XFEL timing signals required by the C&C.

Table 2 Non XFEL timing signals required (not finalized)

The information in this section can only be finalized when timing interface information is know from LCLS, Spring8, FLASH, etc. Open issues:

§  Where do the signals come from?

§  Must additional in crate hardware be foreseen to access timing information

§  Are unique bunch numbers used and, if so, how can they be accessed.

In view of these uncertainties the following variations are possible:

Bunch clock: LVTTL or NIM, single ended, LEMO 00 permissible range 750 kHz - 6 MHz. Slower clocks below 750 kHz acceptable but will not meet jitter etc. specs. Faster clocks above 6 MHz acceptable but will not permit synchronous veto.

Trigger input: LVTTL or NIM, single ended, LEMO 00

3.3  Standalone timing interface

The inputs from the TR computer to C&C are clock at either 4.5Mhz or 5Mhz and a start of train trigger. The external inputs are a clock between 750kHz and 6Mhz and an external trigger.

Clock Inputs:

a) Bunch Clock from T.R. board in crate:

5 MHz or 4.5 MHz, Differential LVDS on RJ45 input socket A pins 1+ and 2-

b) External clock inputs:

i) Differential LVDS, on RJ45 input socket B pins 7+ and 8-

ii) LVTTL or NIM, single ended, LEMO 00 permissible range 750 kHz - 6 MHz

(Slower clocks below 750 kHz acceptable but will not meet jitter etc. specs)

(Faster clocks above 6 MHz acceptable but will not permit synchronous veto)

TRAIN-START (Trigger) Inputs:

a) TRAIN-START pulse from T.R. board in crate:

The Train Start is sent ~15msec before the first bunch on RJ45 input socket A pins 4+ and 5-

This line is also used for Train Stop and Additional Messages

b) External input:

i) Differential LVDS, on RJ45 input socket B pins 3+ and 6-

ii) LVTTL or NIM, single ended, LEMO 00

Inputs a) for run-mode with XFEL detectors and T.R. installed

Inputs b) for stand-alone/test mode or other unknown detectors

3.4  Internal standalone timing

An internally generated 4.51 MHz clock (4.5 or 5MHz) isis provided for standalone running. XXX. What signal is used to trigger the system and what are its timing and electrical characteristics?

FEE fast signal interface

This section describes the C&C FEE fast signal interface. Fast signals are clocks and pulses transferred over cables (≤ 5 m).

The FEE interface is used to distribute synchronization and bunch train configuration information train ID, bunch pattern ID…) to the FEEs. Currently only one quantity is sent from the FEE to the C&C.

3.4.1  A schematic showing the proposed connection functionality is show in Figure 1 and fast signal definitions in Table 3.LCLS timing … to be finished

The external clock input discussed here is provided by the light source timing system XFEL, LCLS, Spring8 where the C&C system operating

> b) External clock inputs :

> i) Differential LVDS,

> on RJ45 input socket B pins 7+ and 8-

> ii)LVTTL or NIM, single ended, LEMO 00

> permissible range 750 kHz - 6 MHz

> ( slower clocks below 750 kHz acceptable

> but will not meet jitter etc. specs )

> ( faster clocks above 6 MHz acceptable

> but will not permit synchronous veto )

>

3.4.2  Spring8 timing … to be finished

3.4.3  Other light source timing … to be finished

3.5  Inputs and outputs to FEE (Fast Signals)

The following table defines all fast signals sent between C&C and FEE.

Table 3 C&C FEE (FPGA-FPGA) fast signal definitions

3.6  Line 1: clock

Table 1 Fast Signals

3.7  Line 1: CLOCK

Line 1 distributes a ~99 MHz synchronization clock derived from the glitch less input bunch clock. On RJ45 socket C pins 1+ and 2-

Line 1: CLOCK is an integer multiple of the input Bunch Clock. If input Bunch Clock is 5.0 MHz, this output clock will be 100.00 MHz. If input Bunch Clock is 4.5 MHz, this output clock will be 99.00 MHz. It is PLL-ed to the Bunch Clock. It is un-interrupted during a run. It is in fixed phase relationship to the Bunch Clock.

3.8  Line 2: control commands

Line 2 distributes control commands to the FEEs by encoding information on a 100 MHz carrier clock.

Table 4 Line 2 control commands

3.9  Line 2: START and RESET Commands

On RJ45 socket C pins 4+ and 5-

Line 2: A) START Command consists of the following data

a) Start = 1100 ( 4 bits )

b) Train-ID = 32 bits, counts-up by '1' for each train (i.e. no GPS-derived timing info)

c) Bunch-Pattern-ID = 8 bits

d) Checksum of b) and c) = 8 bits

------

Total = 52 bits of data

NOTE: Simple Checksum = ex-OR of five 8-bit words of b) and c)

B) STOP = 1010 (4 bits)

After all ~3000 bunches have been sent.

C) RESET = 1001 (4 bits)

Used to reset the FEE micro-controller via the FEE FPGA

D) Reserved = 1111 (4 bits)

NOTE: using four bit (1100/1010/1001/1111) commands prevent single-bit errors.

More that 4 opcodes can be defined later by increasing the number of bits.

The bit error rate (BER) should ideally be none and certainly not more than a few/day. If high BER rates are seen during running this means that there is a fault, which must be repaired, or the timing is wrong and must be corrected.

3.10  Line 3: VETO command

3.11  Line 3: VETO / NO-VETO Commands

On RJ45 socket C pins 7+ and 8-

Line 3 distributes a VETO command to the FEEs. Two protocols are defined for distributing the command: a 100 MHz clock with the bunch number to be vetoed encoded on it, or a 4.51 MHz clock with yes/no (high/low) encoding. The later provides backwards compatibility with the original veto signal definition. Which protocol is distributed is defined during run configuration.