4.2 Hardware Verification of the Digital Design
109
data should be reduced. This is done by utilizing one of the extra bits available when
60 bits are packed into 64-bit words before the buffers. When a buffer is almost full,
the module will set this bit to one and wait until the next packet. When the truncated
packet arrives at the memory-writer module, this bit will be detected in the stream
and the first 64-bit word in the packet will be updated with another bit that indicates
to the later systems that only the first 64-bit word (the header) is valid and that the
payload size as indicated in the header is invalid. The program on the Linux system
that forwards packets to the computer would then see this bit as being set and only
send the header instead of the full packet.
As the incoming serial data arrives at a high speed, there is a possibility that the
data will be sampled at the changeover point, generating invalid data. To prevent
this, the system uses the DDR registers available in the input pads of the FPGA to
provide the value at both the rising and falling edge of the system clock. An auto
adjustment function in the control program on the computer selects which input to
use for the de-serializer based on the ability of the de-serializer to synchronize to
the incoming data stream. In the packet-based mode, the serial links are operating
independently so there is no need to have the incoming data in phase across the links,
but in the direct readout mode, this is necessary. To compensate for phase differences
of plus/minus a clock cycle, an extra register can be selectively included to the input
path of each serial link. The FPGA that was used does not have run-time-configurable
delay compensation in the input/output cells and implementing delays with fabric
cells was not found to be reliable. Oversampling was also not an option as the device
is not able to operate at much higher speeds than the 320 MHz that is the maximum
speed of the serial link.
4.2.1.3 Data Server
The data server is a program running on the embedded Linux system of the microprocessor. It is in charge of handling the data transmission and TCP/IP connection
control with the remote analysis program on the computer. As the data from the
SAMPA is packet based, with a header and payload, it can be transmitted verbatim
to the controlling computer with Transmission Control Protocol (TCP), minimizing overhead. The data throughput of the DAQ system is primarily limited by the
throughput of the underlying embedded Linux system and the Gigabit Ethernet. The
throughput using TCP was tested with the iPerf3 tool [25] to be 670 Mbps between
the Linux system and a computer, which is lower than expected for a 1 Gbps connection. User Datagram Protocol (UDP) is generally faster than TCP, as it does not
automatically re-transmit lost packets, but testing with UDP showed only a slight
increase in speed to 690 Mbps with a 1% packet loss, indicating that the issue lies in
the microprocessor to Ethernet interface. The extra work of manually implementing
packet re-transmission to work with UDP did not seem worth it for the small speed
gain. In contrast, the transfer speed from the computer to the DAQ was measured to
be 950 Mbps with 43% packet loss for UDP and 716 Mbps for TCP.
109
data should be reduced. This is done by utilizing one of the extra bits available when
60 bits are packed into 64-bit words before the buffers. When a buffer is almost full,
the module will set this bit to one and wait until the next packet. When the truncated
packet arrives at the memory-writer module, this bit will be detected in the stream
and the first 64-bit word in the packet will be updated with another bit that indicates
to the later systems that only the first 64-bit word (the header) is valid and that the
payload size as indicated in the header is invalid. The program on the Linux system
that forwards packets to the computer would then see this bit as being set and only
send the header instead of the full packet.
As the incoming serial data arrives at a high speed, there is a possibility that the
data will be sampled at the changeover point, generating invalid data. To prevent
this, the system uses the DDR registers available in the input pads of the FPGA to
provide the value at both the rising and falling edge of the system clock. An auto
adjustment function in the control program on the computer selects which input to
use for the de-serializer based on the ability of the de-serializer to synchronize to
the incoming data stream. In the packet-based mode, the serial links are operating
independently so there is no need to have the incoming data in phase across the links,
but in the direct readout mode, this is necessary. To compensate for phase differences
of plus/minus a clock cycle, an extra register can be selectively included to the input
path of each serial link. The FPGA that was used does not have run-time-configurable
delay compensation in the input/output cells and implementing delays with fabric
cells was not found to be reliable. Oversampling was also not an option as the device
is not able to operate at much higher speeds than the 320 MHz that is the maximum
speed of the serial link.
4.2.1.3 Data Server
The data server is a program running on the embedded Linux system of the microprocessor. It is in charge of handling the data transmission and TCP/IP connection
control with the remote analysis program on the computer. As the data from the
SAMPA is packet based, with a header and payload, it can be transmitted verbatim
to the controlling computer with Transmission Control Protocol (TCP), minimizing overhead. The data throughput of the DAQ system is primarily limited by the
throughput of the underlying embedded Linux system and the Gigabit Ethernet. The
throughput using TCP was tested with the iPerf3 tool [25] to be 670 Mbps between
the Linux system and a computer, which is lower than expected for a 1 Gbps connection. User Datagram Protocol (UDP) is generally faster than TCP, as it does not
automatically re-transmit lost packets, but testing with UDP showed only a slight
increase in speed to 690 Mbps with a 1% packet loss, indicating that the issue lies in
the microprocessor to Ethernet interface. The extra work of manually implementing
packet re-transmission to work with UDP did not seem worth it for the small speed
gain. In contrast, the transfer speed from the computer to the DAQ was measured to
be 950 Mbps with 43% packet loss for UDP and 716 Mbps for TCP.