Problem with a DDI Device driver for Solaris 8 and 9 on a Sun-Blade-1500

Question

The problem has to do with dma allocated memory and the little-endian format.
When I copy blocks bigger than 256 bytes to the dma memory, using a uiomove() or bcopy(), the data after the first 32 bytes is swapped around over 8 bytes.
The bcopy from a driver (stack) buffer into the dma virtual memory has the same affect.
So it has to do with the kernel dma allocated memory and the endianess.
When I set the byte order to big-endian inside the ddi_device_acc_attr the data seems to be alright, also with bigger transfer sizes.
My device however needs the little-endian format.
I have the same the same device driver running without any errors on a Sun Blade-150 Solaris 8 for a long time now.
Can somebody please advise me?
Data example,
test data: 0x00,0x01,0x02,?.
64, 128 or 256 bytes blocks:
000: 07 06 05 04 03 02 01 00 0F 0E 0D 0C 0B 0A 09 08
016: 17 16 15 14 13 12 11 10 1F 1E 1D 1C 1B 1A 19 18
032: 27 26 25 24 23 22 21 20 2F 2E 2D 2C 2B 2A 29 28
048: 37 36 35 34 33 32 31 30 3F 3E 3D 3C 3B 3A 39 38
064: 47 46 45 44 43 42 41 40 4F 4E 4D 4C 4B 4A 49 48
080: 57 56 55 54 53 52 51 50 5F 5E 5D 5C 5B 5A 59 58
096: 67 66 65 64 63 62 61 60 6F 6E 6D 6C 6B 6A 69 68
112: 77 76 75 74 73 72 71 70 7F 7E 7D 7C 7B 7A 79 78
512 or 1024 bytes blocks:
000: 07 06 05 04 03 02 01 00 0F 0E 0D 0C 0B 0A 09 08
016: 17 16 15 14 13 12 11 10 1F 1E 1D 1C 1B 1A 19 18
032: 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F <----- ?
048: 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F <----- ?
064: 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F <----- ?
080: 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F <----- ?
096: 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F <----- ?
112: 70 71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F <----- ?
Code example, how I allocate the DMA (kernel) memory:
DDI_STRUCTURE_LE_ACC = Little Endian format
DDI_STRUCTURE_BE_ACC = Big Endian format
static struct ddi_device_acc_attr sse_dma_attr =
DDI_DEVICE_ATTR_V0, /* The version number of this structure */
DDI_STRUCTURE_LE_ACC, /* see above */
DDI_STRICTORDER_ACC, /* How CPU will reference data, default */
The dma definitions for allocating the DMA memory
static ddi_dma_attr_t dmaattr= {
DMA_ATTR_V0, /* version */
0, /* starting address for DVMA */
0xffffffff, /* end address for DVMA */
0xffffffff, /* max transfer count in one cookie */
0x1, /* address restrictive alignment, 1 = byte alignment */
0x7, /* burst sizes */
1, /* min number of byes */
0x00ffffff, /* max number of bytes device can transmit/receive */
0xffffffff, /* upper bound of the DMA engine's address */
1,
512,
0, /* DDI_DMA_FORCE_PHYSICAL doesn't work */
ddi_dma_alloc_handle()
if((ret=ddi_dma_alloc_handle(xsp->dip,
&dmaattr,
DDI_DMA_SLEEP,
NULL,
dmahandle_out)) != DDI_SUCCESS)
cmn_err(CE_CONT, "ucr_dma_alloc_memory, "
"ERROR ddi_dma_alloc_handle status = %d\n",
ret);
return(ret);
if((ret=ddi_dma_mem_alloc(*dmahandle_out,
(uint_t) size,
dma_acc_attr,
DDI_DMA_CONSISTENT,
DDI_DMA_SLEEP,
NULL,
(caddr_t *)&raw_kern_addr,
&real_len,
dma_acc_handle_out)) != DDI_SUCCESS)
cmn_err(CE_CONT, "ucr_ucr_dma_alloc_memory, "
"ERROR ddi_dma_mem_alloc status = %d\n",
ret);
ddi_dma_free_handle(dmahandle_out);
return(ret);
This will also init the PCI - IOMMU for
address translation.
if((ret=ddi_dma_addr_bind_handle(*dmahandle_out,
NULL,
(caddr_t)raw_kern_addr,
real_len,
DDI_DMA_RDWR | DDI_DMA_CONSISTENT,
DDI_DMA_SLEEP,
NULL,
&dma_cookie,
&count)) != DDI_SUCCESS)
cmn_err(CE_CONT, "ucr_ucr_dma_alloc_memory, "
"ERROR ddi_dma_addr_bind_handle status = %d\n",
ret);
ddi_dma_mem_free(dma_acc_handle_out);
ddi_dma_free_handle(dmahandle_out);
return(ret);
if((ret=ddi_dma_sync(*dmahandle_out,
0,
real_len,
DDI_DMA_SYNC_FORDEV)) != DDI_SUCCESS)
cmn_err(CE_CONT, "ucr_ucr_dma_alloc_memory, "
"ERROR ddi_dma_sync status = %d\n", ret);
ddi_dma_unbind_handle(*dmahandle_out);
ddi_dma_mem_free(dma_acc_handle_out);
ddi_dma_free_handle(dmahandle_out);
return(ret);

Answer 1

I am having the similar problem identified in this thread.
OS: Sun Solaris 10 01/06
Hardware: Sun Blade 2500
PCI Memory card.
Problem: When driver moves more then 40 bytes via PIO from a PCI device memory card, the long words are swapped starting at 40 byte transfer size and greater.
device memory description:
static ddi_device_acc_attr_t sdram_access_attr =
DDI_DEVICE_ATTR_V0, /* Boilerplate value */
DDI_STRUCTURE_LE_ACC,
DDI_STRICTORDER_ACC /* Don't reorder accesses */
The user application does a read call, which goes to xxread. xxread calls physio( xxstrategy, bp, dev, B_READ, xxminphys, uio );
In xxstrategy the transfer is done using
ddi_mem_rep_get8(ucb->ucb_sdram_accHndl,
(uint8_t*) kaddr,
(uint8_t*) raddr,
(size_t) bp->b_resid),
(uint_t) DDI_DEV_AUTOINCR );
PCI Device memory is loaded with incrementing address (8bit), 0,1,2... This is confirmed using a PCI Bus Analyzer.
A read of 36 bytes or less returns:
03 02 01 00 07 06 05 04 ...
A read of 40 bytes returns:
07 06 05 04 03 02 01 00 ...
If I peek 1 byte at a time using ddi_get8(ucb->ucb_sdram_accHndl, ...) I get
00 01 02 03 04 05 06 07
Why are the long words swapped starting at 40 byte transfer size and greater?