Write latency on PCIe from PC (host) to FPGA

Question

Hello,
In my application, the target is to write 64 Bytes from the PC to the FPGA with the minimum latency.
I have implemented the “Virtex-7 FPGA Gen3 Integrated Block for PCI Express(3.0)” IP, that seems the best candidate for that and I am using the AXI STREAM “m_axi_cq” interface to write data to internal FPGA memory.
The configuration is this one:
PCIe GEN 3 / 8 lanes.
AXI at 250 MHz with data bus = 256 bits.
On the µP = XEON E5 V2, the right core is bound to the FPGA.
I am using, specific Intel instruction to transfer directly bloc of 256 bits (in my example I am using 2 times this instruction in C code). I do not want to call a DMA, because I am afraid to lose more time to call the DMA than to make a direct memory write.
In the FPGA, I put an ILA core to monitor the access (see waveforms).
- First bus is the output of the Xilinx core (“m_axi_cq” bus)
- Second bus is the “memory write signal” of 8 memories of 32 bits data width => 16 x writes of 32 bits = 64 Bytes
Question:
- Even if I use 2 Intel instruction to transfer 2x256 bits, I see 4 transfers of 128 bits => is it normal ?
- It seems that we cannot have a TLP bigger than 256 bits (without DMA access) => do you confirm?
- Between 2 write access, I have 23 clock cycles = 23*4ns = 92 ns, and I do not arrive to decrease this score => did I reached the minimum possible ?
Many thanks for your attention

Answer 1

 
Be careful with watching the latency. 
It's HIGHLY variable in our cases.
We've got our PCIE block (endpoint on the FPGA) issuing block reads to the cpu host and/or an NVIDIA GPU.
We've measured on various systems, average latencies in the 180-240 clocks range.  (Clock = pcie user clock, 250 MHz).  That's not's great, but for us tolerable. 
The problem was the distribution in the latencies.  The WORST case latencies can be terrible.  > 1600 clocks in some cases.    Plotting a few series of results, we see a bimodal distbution of latencies.  I.e. a bunch hovering around 150, and a bunch hovering around 300.  The average being the above results.  Usually around the point where the latencies are tending to move from one "average" to the other, we get the outlier extra long latency.
Probably something to do with cache flushes/fills happening over on the CPU.
In any event, because of this high variability of latencies, our first designs broke real time.  We had to re-architect things to handle.
Bandwidth's not a problem for PCIE. Predicable latency, however is troublesome...
Regards,
Mark