How to update a large (over 4 million item) List(Of Byte) quickly by altering indexes contained in a Dictionary(Of Integer, Byte) where the Dictionaries keys are the indexes in the List(Of Byte) that need to be changed to the values for those indexes?

Question

   I'm having some difficulty with transferring images from a UDP Client to a UDP Server. The issue is receiving the bytes necessary to update an original image sent from the Client to the Server and updating the Servers List(Of Byte) with the
new bytes replacing bytes in that list. This is a simplex connection where the Sever receives and the Client sends to utilize the least amount of bandwidth for a "Remote Desktop" style application where the Server side needs image updates of whatever
occurs on the Client desktop.
   So far I can tranfer images with no issue. The images can be be any image type (.Bmp, .Gif, .JPeg, .Png, etc). I was working with sending .JPeg's as they appear to be the smallest size image when a Bitmap is saved to a memory stream as type
.JPeg. And then I am using GZip to compress that byte array again so it is much smaller. However on a loopback on my NIC the speed for sending a full size screen capture is not very fast as the Server updates fairly slowly unless the Clients screen capture
Bitmap is reduced in size to about 1/3'd of the original size. Then about 12000 bytes or less are sent for each update.
   Due to .JPeg compression I suppose there is no way to get the difference in bytes between two .JPegs and only send those when something occurs on the desktop that alters the desktop screen capture image. Therefore I went to using .Bmp's as each
.Bmp contains the same number of bytes in its array regardless of the image alterations on the desktop. So I suppose the difference in bytes from a second screen capture and an inital screen capture are what is different in the second image from the initial
image.
   What I have done so far is save an initial Bitmap of a screen capture using a memory stream and saving as type .Bmp which takes less than 93 milliseconds for 4196406 bytes. Compressing that takes less than 118 milliseconds to 197325 bytes for
the current windows on the desktop. When that is done PictureBox1 is updated from nothing to the captured image as the PictureBox's background image with image layout zoom and the PictureBox sized at 1/2 my screens width and 1/2 my screens height.
   Then I save a new Bitmap the same way which now contains different image information as the PictureBox is now displaying an image so its back color is no longer displayed (solid color Aqua) and the cursor has moved to a different location. The
second Bitmap is also 4196406 in bytes and compressed it was 315473 bytes in size.
   I also just found code from this link Converting a Bitmap to a Byte Array (and Byte Array to Bitmap) which gets a byte array
directly from a Bitmap and the size of that is 3148800 for whatever is full screen captured on my laptop. So I should be able to work with smaller byte arrays at some point.
   The issue I'm having is that once the Client sends an image of the desktop to the Server I only want to update the server with any differences occuring on the Clients desktop. So what I have done is compare the first screen captures bytes (stored
in a List(Of Byte)) to the second screen captures bytes (stored in a List(Of Byte)) by using a For/Next for 0 to 4196405 where if a byte in the first screen captures List is not equal to a byte in the second screen captures List I add the index and byte of
the second screen captures list to a Dictionary(Of Integer, Byte). The Dictionary then only contains the indexes and bytes that are different between the first screen capture and second screen capture. This takes about 125 milliseconds which I think is pretty
fast for 4196406 byte comparison using a For/Next and adding all the different bytes and indexes for each byte to a Dictionary.
    The difference in Bytes between the inital screen capture and the second screen capture is 242587 as an example which changes of course. For that amount of bytes the Dictionary contains 242587 integers as indexes and 242587 bytes as different
bytes totaling 485174 bytes for both arrays (keys, values).  Compressed the indexes go from 242587 to 43489 bytes and the values go from 242587 to 34982 bytes. Which means I will have to send 78, 481 bytes from the Client to the Server to update the display
on the server. Quite smaller than the original 4196406 bytes of the second Bitmap saved to type .Bmp or the compressed size of that array which was 315473 bytes. Plus a few bytes I add as overhead so the server knows when an image array ends and how many packets
were sent for the array so it can discard complete arrays if necessary since UDP is lossfull although probably not so much in current networks like it may originally have been when the internet started.
    In reality the data from the Client to the Server will mostly be the cursor as it moves and updating the Server image with only a few hundred bytes I would imagine at a time. Or when the cursor selects a Button for example and the Buttons
color changes causing those differences in the original screen capture.
   But the problem is if I send the Dictionaries Indexes and Bytes to the Server then I need to update the original Bitmap List(Of Byte) on the server by removing the Bytes in the received informations Index locations array from the Servers Bitmap
List(Of Byte) and replacing those Bytes with the Bytes in the received informations Byte array. This takes so long using a For/Next for however many indexes are in the received informations Index array to update the Bitmap List(Of Byte) on the server using
"Bmp1Bytes.RemoveAt(Index As Integer)" followed by "Bmp1Bytes.Insert(Index As Integer, Item As Byte)" in the For/Next.
    I've tried various For/Next statements including using a new List(Of Byte) with If statements so If the the integer for the For/Next ='s the Key in a Dictionary(Of Integer, Byte) using a Counter to provide the Dictionaries Key value then
the Dictionaries byte value will be added to the List(Of Byte) and the counter will increas by one Else the List(Of Byte) adds the original "Bmp1Bytes" byte at that index to the new List(Of Byte). This takes forever also.
   I also tried the same For/Next adding to a new Dictionary(Of Integer, Byte) but that takes forever too.
   I think I could use RemoveRange and AddRange to speed things up. But I don't know how to retrieve a contiguous range of indexes in the received indexes that need to be updated in the servers "Bmp1Bytes" List(Of Byte) from the received
array of indexes and bytes which are in a Dictionary(Of Integer, Byte).  But I believe this would even be slower than some realistic method for replacing all Bytes in a List(Of Byte) when I have the indexes that need to be replaced and the bytes to replace
them with.
   Even if I just used AddRange on a new List(Of Byte) to add ranges of bytes from the original "Bmp1Bytes" and the changes from the Dictionary(Of Integer, Byte) I think this would be rather slow. Although I don't know how to do that
by getting contiguous ranges of indexes from the Dictionaries keys.
   So I was wondering if there is some method perhaps using Linq or IEnumerable which I've been unable to figure anything out which could do this.
   I do have some copy and pasted code which I don't understand how it works that I am using which I would guess could be altered for doing something like this but I can't find information that provides how the code works.  Or even if I did
maybe I can't understand it. Like the code below which is extremely fast.
   Dim strArray() As String = Array.ConvertAll(Of Integer, String)(BmpComparisonDict.Keys.ToArray, Function(x) x.ToString())
La vida loca

Answer 1

Monkeyboy,
That was quite a bit to read, but still a bit unclear. Could you put a specific list of goals/questions, asked in the smallest possible form?
It seems like either you're making a program that monitors activity on your computer, or you're writing some kind of remote pc app.
When you do get your bytes from using lockbits, keep in mind all the files header info would be lost. I think retaining the header info is worth the extra bytes.
The other, thing: I'm not sure if you're taking 32bpp screen shots, but also keep in mind that the "whole desktop" is the final destination for blended graphics, if that makes sense. What I mean is that there is no need to capture an "alpha"
channel for a desktop screenshot, as alpha would always be 255, this could save you 1 byte per pixel captured... Theres nothing "behind" the desktop, therefore no alpha, and every window shown above the desktop is already blended. I suggest using
24Bpp for a full screen capture.
Your X,Y information for the mouse could be stored as UINT16, this would save you a measly 2 bytes per location update/save.
When you update your byte arrays, maybe you can turn the array into a stream and write to whatever index, however many bytes, that should prevent a "Shift" of bytes, and instead overwrite any bytes that "get in the way".
ex
Dim example As String = "This is an example."
Dim insertString As String = "was"
Dim insertBytes As Byte() = System.Text.Encoding.ASCII.GetBytes(insertString)
Dim bytes As Byte() = System.Text.Encoding.ASCII.GetBytes(example)
Dim modifiedBytes As Byte() = {}
Using ms As New System.IO.MemoryStream(bytes)
ms.Position = 5
ms.Write(insertBytes, 0, 3)
modifiedBytes = ms.ToArray
End Using
Dim newString As String = System.Text.Encoding.ASCII.GetString(modifiedBytes)
'Notice how below there isn't the word "is" anymore, and that there isn't a
'space.
'This demonstrates that you overwrite existing data, versus shifting everything to
'the right.
'Returns: This wasan example.
MsgBox(newString)
“If you want something you've never had, you need to do something you've never done.”
Don't forget to mark
helpful posts and answers
! Answer an interesting question? Write a
new article
about it! My Articles
*This post does not reflect the opinion of Microsoft, or its employees.
Well it's too much to read. I was really tired when I wrote it. Even the below is too much to read but perhaps gets the point across of what I would like to do which I think
Joel Engineer may have answered but I'm not sure. As I'm still too tired to understand that yet and research what he said in order to figure it out yet.
But maybe the code below can provide the concept of the operation with the comments in it. But seeing as how I'm still tired it may be confused.
Option Strict On
Imports System.Windows.Forms
Imports System.IO
Imports System.IO.Compression
Imports System.Drawing.Imaging
Imports System.Runtime.InteropServices
Public Class Form1
Dim Bmp1Bytes As New List(Of Byte)
Dim Bmp1BytesCompressed As New List(Of Byte)
Dim Bmp2Bytes As New List(Of Byte)
Dim BmpComparisonDict As New Dictionary(Of Integer, Byte)
Dim BmpDifferenceIndexesCompressed As New List(Of Byte)
Dim BmpDifferenceBytesCompressed As New List(Of Byte)
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
SomeSub()
End Sub
Private Sub SomeSub()
' Pretend this code is in UDP Client app. A screen capture is performed of the desktop. Takes about 90 milliseconds.
Bmp1Bytes.Clear()
Using BMP1 As New Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height)
Using g1 As Graphics = Graphics.FromImage(BMP1)
g1.CopyFromScreen(0, 0, 0, 0, BMP1.Size)
Cursor.Draw(g1, New Rectangle(Cursor.Position.X, Cursor.Position.Y, Cursor.Size.Width, Cursor.Size.Height))
Using MS As New MemoryStream
BMP1.Save(MS, System.Drawing.Imaging.ImageFormat.Bmp)
Bmp1Bytes.AddRange(MS.ToArray)
End Using
End Using
End Using
Bmp1BytesCompressed.AddRange(Compress(Bmp1Bytes.ToArray))
' UDP Client app sends Bmp1BytesCompressed.ToArray to UDP Server which is the entire image of the desktop that the UDP
' Client is on. This image takes awhile to send since compressed it is about 177000 bytes from over 4000000 bytes.
' I will be using different code just to get the bytes from the actual Bitmap in the future. That is not important for now.
' Pretend the UDP Server has received the bytes, decompressed the array received into a List(Of Byte) and is displaying
' the image of the UDP Clients desktop in a PictureBox.
' Now the image on the UDP Clients desktop changes due to the mouse cursor moving as an example. Therefore a new Bitmap
' is created from a screen capture. This takes about 90 milliseconds.
Bmp2Bytes.Clear()
Using BMP2 As New Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height)
Using g1 As Graphics = Graphics.FromImage(BMP2)
g1.CopyFromScreen(0, 0, 0, 0, BMP2.Size)
Cursor.Draw(g1, New Rectangle(Cursor.Position.X, Cursor.Position.Y, Cursor.Size.Width, Cursor.Size.Height))
Using MS As New MemoryStream
BMP2.Save(MS, System.Drawing.Imaging.ImageFormat.Bmp)
Bmp2Bytes.AddRange(MS.ToArray)
End Using
End Using
End Using
' Now I have the original images bytes in Bmp1Bytes and the new images bytes in Bmp2Bytes on the UDP Client. But I don't
' want to send all of the bytes in Bmp2Bytes to the UDP Server. Just the indexes of and the bytes that are different in
' Bmp2Bytes from Bmp1Bytes.
' This takes less than 100 milliseconds for what I've tested so far where over 500000 bytes in Bmp2Bytes are different
' than the bytes in Bmp1Bytes. Usually that amount would be much less. But during testing I was displaying the image
' from Bmp1 bytes in a PictureBox so a large amount of data would change between the first screen shot, the PictureBox
' then displaying an image on the same PC and then the second screen shot.
BmpComparisonDict.Clear()
For i = 0 To Bmp1Bytes.Count - 1
If Bmp1Bytes(i) <> Bmp2Bytes(i) Then
BmpComparisonDict.Add(i, Bmp2Bytes(i))
End If
Next
' So now I have all the difference bytes and their indexes from Bmp2Bytes in the BmpComparisonDict. So I compress
' the indexes into on List and the Bytes into another List.
BmpDifferenceIndexesCompressed.Clear()
BmpDifferenceBytesCompressed.Clear()
BmpDifferenceIndexesCompressed.AddRange(Compress(BmpComparisonDict.Keys.SelectMany(Function(d) BitConverter.GetBytes(d)).ToArray()))
BmpDifferenceBytesCompressed.AddRange(Compress(BmpComparisonDict.Values.ToArray))
' Now pretend the UDP Client has sent both those arrays to the UDP Server which has added both decompressed arrays
' to a Dictionary(Of Integer, Byte). And the server has the original image decompressed bytes received in a List
' called Bmp1Bytes also.
' This is where I am stuck. The UDP Server has the Dictionary. That part was fast. However there is no
' fast method I have found for creating a new List(Of Byte) where bytes in the originally received List(Of Byte) that
' do not have to be altered are placed into a new List(Of Byte) except for the indexes listed in the
' Dictionary(Of Integer, Byte) that need to be placed into the appropriate index locations of the new List(Of Byte).
' The below example for doing so is exceptionally slow. Pretend UpdateDictionary has all of the decompressed indexes
' and bytes received by the UDP Server for the update contained within it.
Dim UpdateDictionary As New Dictionary(Of Integer, Byte)
Dim UpdatedBytes As New List(Of Byte)
Dim Counter As Integer = 0
For i = 0 To Bmp1Bytes.Count - 1
If i = UpdateDictionary.Keys(Counter) Then ' Provides the index contained in the Keys for the Dictionary
UpdatedBytes.Add(UpdateDictionary.Values(Counter))
Counter += 1
If Counter > UpdateDictionary.Count - 1 Then Counter = 0
Else
UpdatedBytes.Add(Bmp1Bytes(i))
End If
Next
' So what I'm trying to do is find an extremely fast method for performing something similar to what the
' above operation performs.
End Sub
Private Function Compress(BytesToCompress() As Byte) As List(Of Byte)
Dim BytesCompressed As New List(Of Byte)
Using compressedStream = New MemoryStream()
Using zipStream = New GZipStream(compressedStream, CompressionMode.Compress)
zipStream.Write(BytesToCompress, 0, BytesToCompress.Count)
zipStream.Close()
BytesCompressed.AddRange(compressedStream.ToArray)
End Using
End Using
Return BytesCompressed
End Function
Private Function Decompress(BytesToDecompress() As Byte) As List(Of Byte)
Dim BytesDecompressed As New List(Of Byte)
Using DecompressedStream = New MemoryStream()
Using zipStream = New GZipStream(DecompressedStream, CompressionMode.Decompress)
zipStream.Write(BytesToDecompress, 0, BytesToDecompress.Count)
zipStream.Close()
BytesDecompressed.AddRange(DecompressedStream.ToArray)
End Using
End Using
Return BytesDecompressed
End Function
End Class
La vida loca