Network Stream Max Buffer Size

Hello,
I recall an AE on here once mentioning that network streams can exhibit problems if you set a buffer size greater than 9MB, but I haven't been able to find any concrete explanation of this. The reason behind me asking is that, I'm currently running into some memory problems in my application in which I initialize my buffer to 30MB.
Basically, my application starts out running fine, but after 30-45 minutes, all of a sudden, my network stream buffer starts building up until it reaches a max value, even though I am still reading data out of it at seemingly the same rate. It's as if routine memory allocation/deallocation elsewhere in my application causes issues with the network stream.
Anyone have any insight to this?
thanks in advance,

The size of the buffer you set for your network streams will just determine how much memory is set aside within your application. If you make your stream too large, LabVIEW will throw an error such as 'Memory is Full' telling you there is not enough space to create that large of a buffer.
The important benchmarking data to look at concerning network streams is throughput and latency. The following KB has data in section 6 regarding how quickly data can be passed through a network stream:
http://www.ni.com/white-paper/12267/en/#toc4
Since you are able to function properly for the first 30-45 minutes, it does not seem that you are violating network capabilities with your setup. However, there seems to be something that is building up/slowing down after that period of time.
How did you determine that you need a 30 MB buffer size? If you are originally only storing between 0-8000 items on the buffer, it may make sense to make a buffer smaller than 30000000 to free up resources for other parts of your application.
Here is some more information about how to profile performance and memory within LabVIEW: http://zone.ni.com/reference/en-XX/help/371361J-01/lvdialog/profile/
Applications Engineer
National Instruments

Similar Messages

Max buffer size for HDS streams

How does OSMF (v2.0) compute the max buffer length for HDS streams (pre-recorded)?
I have mediaPlayer.bufferTime = 5 seconds, but I see that even on very good b/w connections, the buffer length does not exceed 7.5 secs. Is there a way to increase the maximum buffer length value, so that clients on higher b/w can accumulate a larger buffer?
By comparison, for RTMP, the buffer length seems to go up to 60sec which is consistent with what the docs say here:
http://livedocs.adobe.com/flashmediaserver/3.0/hpdocs/help.html?content=00000175.html
Thx,
- abey

No. Did not find a way to set the maxBuffer value, but I was able to workaround the problem by setting the mediaPlayer.bufferTime dynamically (i.e different values for various stages of playback, like loading, seeking, buffering, normal playback etc.)..So setting bufferTime to 60 secs during normal playback allows the buffer to grow to a decent size.
- Abey

Doing Buffered Event count by using Count Buffered Edges.vi, what is the max buffer size allowed?

I'm currently using Count Buffered Edges.vi to do Buffered Event count with the following settings,
Source : Internal timebase, 100Khz, 10usec for each count
gate : use the function generator to send in a 50Hz signal(for testing purpose only). Period of 0.02sec
the max internal buffer size that i can allocate is only about 100~300. Whenever i change both the internal buffer size and counts to read to a higher value, this vi don't seem to function well. I need to have a buffer size of at least 2000.
1. is it possible to have a buffer size of 2000? what is the problem causing the wrong counter value?
2. also note that the size of max internal buffer varies w
ith the frequency of signal sent to the gate, why is this so? eg: buffer size get smaller as frequency decrease.
3. i'll get funny response and counter value when both the internal buffer size and counts to read are not set to the same. Why is this so? is it a must to set both value the same?
thks and best regards
lyn

Hi,
I have tried the same example, and used a 100Hz signal on the gate. I increased the buffer size to 2000 and I did not get any errors. The buffer size does not get smaller when increasing the frequency of the gate signal; simply, the number of counts gets smaller when the gate frequency becomes larger. The buffer size must be able to contain the number of counts you want to read, otherwise, the VI might not function correctly.
Regards,
RamziH.

Max buffer size in sql promt

hi all
what is the max buffer in sql window..
if it is 1000000 then can we able to increase the size
thanks

If you are talkin about SQL*Plus editor. you can look for 'Screen Buffer' at the Menubar (Options-> Environment).
Screen Buffer
This area has two text boxes: Buffer Width and Buffer Length.
     not applicable
* Use the Buffer Width text box to set the number of characters available to display on one line. The Buffer Width value must be at least as big as the LINESIZE value. Buffer Width has a default value of 100, a minimum value of 80, and a maximum value of 32,767 characters.
In the Buffer Length text box, you set the number of lines that SQL*Plus displays on the screen. The default value of the Buffer Length parameter is 1000 lines. You can specify from 100 to 2000 lines on one screen.
     Notes: When you change the Screen Buffer option, SQL*Plus displays a dialog to alert you that if you shorten the size of your screen buffer, some data may not be displayed on your screen Click OK to proceed.
     If you use SET MARKUP to send output to an HTML table, the number of lines specified in the Buffer Length variable specifies the number of HTML table rows Each HTML table row may contain more than one text line.

DBMS_LOB.WRITEAPPEND Max buffer size exceeded

Hello,
I'm following this guide to create an index using Oracle Text:
http://download.oracle.com/docs/cd/B19306_01/text.102/b14218/cdatadic.htm#i1006810
So I wrote something like this:
CREATE OR REPLACE PROCEDURE CREATE_INDEX(rid IN ROWID, tlob IN OUT NOCOPY CLOB)
IS
BEGIN
     DBMS_LOB.CREATETEMPORARY(tlob, TRUE);
     FOR c1 IN (SELECT ID_DOCUMENT FROM DOCUMENT WHERE rowid = rid)
     LOOP
          DBMS_LOB.WRITEAPPEND(tlob, LENGTH('<DOCUMENT>'), '<DOCUMENT>');
          DBMS_LOB.WRITEAPPEND(tlob, LENGTH('<DOCUMENT_TITLE>'), '<DOCUMENT_TITLE>');
          DBMS_LOB.WRITEAPPEND(tlob, LENGTH(NVL(c1.TITLE, ' ')), NVL(c1.TITLE, ' '));
          DBMS_LOB.WRITEAPPEND(tlob, LENGTH('</DOCUMENT_TITLE>'), '</DOCUMENT_TITLE>');
          DBMS_LOB.WRITEAPPEND(tlob, LENGTH('</DOCUMENT>'), '</DOCUMENT>');
          FOR c2 IN (SELECT TITRE,TEXTE FROM PAGE WHERE ID_DOCUMENT = c1.ID_DOCUMENT)
          LOOP
               DBMS_LOB.WRITEAPPEND(tlob, LENGTH('<PAGE>'), '<PAGE>');
               DBMS_LOB.WRITEAPPEND(tlob, LENGTH('<PAGE_TEXT>'), '<PAGE_TEXT>');
               DBMS_LOB.WRITEAPPEND(tlob, LENGTH(NVL(c2.TEXTE, ' ')), NVL(c2.TEXTE, ' '));
               DBMS_LOB.WRITEAPPEND(tlob, LENGTH('</PAGE_TEXT>'), '</PAGE_TEXT>');
               DBMS_LOB.WRITEAPPEND(tlob, LENGTH('</PAGE>'), '</PAGE>');
          END LOOP;
     END LOOP;
END;
Issue is that some page text are bigger than 32767 bytes ! So I've got an INVALID_ARGVAL...
I can't figure out how can I increase this buffer size and how to manage this issue ??
Can you please help me :)
Thank you,
Ben
Edited by: user10900283 on 9 févr. 2009 00:05

Hi ben,
I'm afraid, that doesn't help much, since you have obviously rewritten your procedure based on the advise given here.
Coluld you please post your new procedure, as formatted SQL*Plus, embedded in {noformat}{noformat} tags, like this:SQL> CREATE OR REPLACE PROCEDURE create_index(rid IN ROWID,
2 IS
3 BEGIN
4 dbms_lob.createtemporary(tlob, TRUE);
5
6 FOR c1 IN (SELECT id_document
7 FROM document
8 WHERE ROWID = rid)
9 LOOP
10 dbms_lob.writeappend(tlob, LENGTH('<DOCUMENT>'), '<DOCUMENT>');
11 dbms_lob.writeappend(tlob, LENGTH('<DOCUMENT_TITLE>')
12 ,'<DOCUMENT_TITLE>');
13 dbms_lob.writeappend(tlob, LENGTH(nvl(c1.title, ' '))
14 ,nvl(c1.title, ' '));
15 dbms_lob.writeappend(tlob
16 ,LENGTH('</DOCUMENT_TITLE>')
17 ,'</DOCUMENT_TITLE>');
18 dbms_lob.writeappend(tlob, LENGTH('</DOCUMENT>'), '</DOCUMENT>');
19
20 FOR c2 IN (SELECT titre, texte
21 FROM page
22 WHERE id_document = c1.id_document)
23 LOOP
24 dbms_lob.writeappend(tlob, LENGTH('<PAGE>'), '<PAGE>');
25 dbms_lob.writeappend(tlob, LENGTH('<PAGE_TEXT>'), '<PAGE_TEXT>');
26 dbms_lob.writeappend(tlob
27 ,LENGTH(nvl(c2.texte, ' '))
28 ,nvl(c2.texte, ' '));
29 dbms_lob.writeappend(tlob, LENGTH('</PAGE_TEXT>'), '</PAGE_TEXT>')
30 dbms_lob.writeappend(tlob, LENGTH('</PAGE>'), '</PAGE>');
31 END LOOP;
32 END LOOP;
33 END;
34 /
Advarsel: Procedure er oprettet med kompileringsfejl.
SQL>
SQL> DECLARE
2 rid ROWID;
3 tlob CLOB;
4 BEGIN
5 rid := 'AAAy1wAAbAAANwsABZ';
6 tlob := NULL;
7 create_index(rid => rid, tlob => tlob);
8 dbms_output.put_line('TLOB = ' || tlob); -- Not sure, you can do this?
9 END;
10 /
create_index(rid => rid, tlob => tlob);
FEJL i linie 7:
ORA-06550: line 7, column 4:
PLS-00905: object BRUGER.CREATE_INDEX is invalid
ORA-06550: line 7, column 4:
PL/SQL: Statement ignored
SQL>

How to reduce max buffer/cache size?

Hi,
every time I copy a file which is bigger or similiar in size to my total RAM (4gb) I notice very low responsibility from firefox (which is totally unresponsive, can't switch tabs or scroll for 30-60s). Of course my free memory is very low (something like 50-100mb) and I notice some swap usage. AFAIK linux caches everthing that is being copied, but in case of such big files it seems unnecessary.
Is there a way to reduce max buffer size?
I know that buffering is good in general, but I get a feeling that firefox is giving up ram and he has to read everything again from disk which slows him down. I always have many tabs open, so often it has around 30% of memory.
I searched many times on how to reduce buffer sizes, but I've always found only articles with "buffering is always good and never an issue" attitude.
I would be very happy to hear any suggestrions,
cheers,
kajman

This seems a popular problem, going back years. The default Linux setup is bad for responsiveness, it seems.
Here's the summary of what I do:
Firstly, install a BFS-patched kernel, for a better kernel scheduler, and also so that the ionice and schedtool commands will work. Bonus points for switching to BFQ while you're at it - or stick with CFQ, which also supports ionice.
In /etc/fstab, use commit=60 rather than default of 5 seconds, and also noatime, e.g.:
UUID=73d55f23-fb9d-4a36-bb25-blahblah / ext4 defaults,noatime,nobarrier,commit=60 1 1
In /etc/sysctl.conf
# From http://rudd-o.com/en/linux-and-free-software/tales-from-responsivenessland-why-linux-feels-slow-and-how-to-fix-that
vm.swappiness=0
# https://lwn.net/Articles/572921/
vm.dirty_background_bytes=16777216
vm.dirty_bytes=50331648
In ~/.bashrc - see post, e.g.:
alias verynice="ionice -c3 nice -n 15"
In /etc/security/limits.d/ - see post. Read CK's excellent blog article, for info.
In your cp command, add the word verynice to the start, to stop the large batch copy from having the same priority as your UI.
Compile sqlite without fsync, to make e.g. firefox smoother.
Potentially use threadirqs to prioritize the interrupt-handling.
Edit: Updated vm.swappiness from 0 to 10, from CK's blog.
Edit2: Also see patch and e.g. nr_requests in thread.
Edit3: Using nice instead of schedtool - not sure whether schedtool can hog the CPU.
Edit4: Added threadirqs.
Edit5: Tweaked sysctl.conf settings.
Edit6: Added nobarrier option to mount, and sqlite's fsync.
Edit7: Removed swap comment - I do use a swapfile, these days, mainly because firefox needs so much virtual RAM to compile.
Last edited by brebs (2014-03-10 09:51:34)

Sychronize AO/AI buffered data graph and measure data more than buffer size

I am trying to measure the response time (around 1ms) of the pressure drop indicated by AI channel 0 when the AO channel 0 gives a negetive single pulse to the unit under test (valve). DAQ board: Keithley KPCI-3108, LabView Version: 6.1, OS system: Win2000 professional.
My problem is I am getting different timed graph between the AI and AO channels every time I run my program, except the first time I can get real time graph. I tried to decrease the buffer size less than the max buffer size of the DAQ board (2048 samples), but it still does unreal time graph from AI channel, seems it was still reading from old data in the buffer when AO writes the new buffer data, that is my guessing. In my p
rogram, the AO and AI part is seperated, AO Write buffer is in a while loop while AI read is not. Would that be a problem? Or it's something else?
Also, I am trying to measure data much larger than the board buffer size limit. Is it possible to make the measurement by modifying the program?
I really appreciate any of your help. Thank you very much!
Best,
Jenna

Jenna,
You can modify the X-axis of a chart/graph in LabVIEW to display real-time. I have included a link below to an example program that illustrates how to do this.
If you are doing a finite, buffered acquisition make sure that you are always reading everything from the buffer for each run. In other words, If you set a buffer size of 5000, then make sure you are reading 5000 scans (set number of scans to read to 5000). This will assure you are reading new data every time you run you program. You could always put the AI Read VI within a loop and read a smaller number from the buffer until the buffer is empty (monitor the scan backlog output of the AI Read VI to see how many scans are left in the buffer).
You can set a buffer size larger than the FIFO
buffer of the hardware. The buffer size you set in LabVIEW is actually a software buffer size within your computer's memory. The data is acquired with the hardware, stored temporarily within the on-board FIFO, transferred to the software buffer, and then read in LabVIEW.
Are you trying to create a TTL square wave with the analog output of the DAQ device? If so, the DAQ device has counters that can generate a highly accurate digital pulse as well. Just a suggestion. LabVIEW has a variety of shipping examples that are geared toward using counters (find examples>>DAQ>>counters). I hope this helps.
Real-Time Chart Example
http://venus.ni.com/stage/we/niepd_web_display.DISPLAY_EPD4?p_guid=B45EACE3E95556A4E034080020E74861&p_node=DZ52038&p_submitted=N&p_rank=&p_answer=&p_source=Internal
Regards,
Todd D.
National Instruments
Applications Engineer

Network Stream Error -314340 due to buffer size on the writer endpoint

Hello everyone,
I just wanted to share a somewhat odd experience we had with the network stream VIs. We found this problem in LV2014 but aren't aware if it is new or not. I searched for a while on the network stream endpoint creation error -314340 and couldn't come up with any useful links to our problem. The good news is that we have fixed our problem but I wanted to explain it a little more in case anyone else has a similar problem.
The specific network stream error -314340 should seemingly occur if you are attempting to connect to a network stream endpoint that is already connected to another endpoint or in which the URL points to a different endpoint than the one trying to connect.
We ran into this issue on attempting to connect to a remote PXI chassis (PXIe-8135) running LabVIEW real-time from an HMI machine, both of which have three NICs and access different networks. We have a class that wraps the network stream VIs and we have deployed this class across four machines (Windows and RT) to establish over 30 network streams between these machines. The class can distinguish between messaging streams that handle clusters of control and status information and also data streams that contain a cluster with a timestamp and 24 I16s. It was on the data network streams that we ran into the issue.
The symptoms of the problem were that we if would attempt to use the HMI computer with a reader endpoint specifying the URL of the writer endpoint on the real-time PXI, the reader endpoint would return with an error of -314340, indicating the writer endpoint was pointing to a third location. Leaving the URL blank on the writer endpoint blank and running in real-time interactive or startup VI made no difference. However, the writer endpoint would return without error and eventually catch a remote endpoint destroyed. To make things more interesting, if you specified the URL on the writer endpoint instead of the reader endpoint, the connection would be made as expected.
Ultimately through experimenting with it, we found that the buffer size of the create writer endpoint for the data stream was causing the problem and that we had fat fingered the constants for this buffer size. Also, pre-allocating or allocating the buffer on the fly made no difference. We imagine that it may be due to the fact we are using a complex data type with a cluster with an array inside of it and it can be difficult to allocate a buffer for this data type. We guess that the issue may be that by the reader endpoint establishing the connection to a writer with a large buffer size specified, the writer endpoint ultimately times out somewhere in the handshaking routine that is hidden below the surface.
I just wanted to post this so others would have a reference if they run into a similar situation and again for reference we found this in LV2014 but are not aware if it is a problem in earlier versions.
Thanks,
Curtiss

Hi Curtiss!
Thank you for your post! Would it be possible for you to add some steps that others can use to reproduce/resolve the issue?
Regards,
Kelly B.
Applications Engineering
National Instruments

Best buffer size for BufferedInput/Output Streams?

I'm developing a client-server application where one server serves an undetermined number of clients. The server spins off threads to service each client individually. The clients send data (serialized objects) to each other via BufferedInput/OutputStreams wrapped in ObjectInput/OutputStreams. The streams are created like this:
toServer = new ObjectOutputStream(new BufferedOutputStream(theSocket.getOutputStream()));
fromServer = new ObjectInputStream(new BufferedInputStream(theSocket.getInputStream()));I've never had more than a handful (5-6) users on the system at one time, and it works fine. However, I haven't the resources to test the system with a large number of clients, and I'm wondering if the default buffer size is enough to handle a potentially large amount of traffic to a client. Should I increase the buffer size just in case, or should I not worry? Does the buffer size even matter when using an ObjectInput/OutputStream?
I'd appreciate any good advice.

There is an underlying buffer size of 1024 in ObjectIn/OutputStream anyway so the buffer should be at least this large. Generally 4k is adequate unless you are doing very high data volumes in which case 16k, 32k, 63k may help. If you are on Windows you should also up the socket send/receive buffers which by default are 8k: change this to at least 28k. The precise value depends on the network link & in theory should be >= the bandwidth-delay product.

Different max photo sizes in emails, beams, photo stream?

I've noticed differences in file sizes when emailing photos from the iPhoto app, when inserting in an email, and when emailing directly from the photo. I've found that iOS 6 is able to send the largest file when inserting it into an email. If I select email from iPhoto, the full size option is a fraction of that. If I email a photo directly from the photo itself, the full size seems to be somewhere in-between. I sent a panoramic photo three times, once using each method. Each time I selected to send the FULL file sizes, and each time they were different:
iPhoto: 1918K
insert in email: 12004K
email from photo itself: 3529K
Is there a reason iOS 6 does this?
Why are the full photo file sizes not universal?
Are the max file sizes being uploaded to photo stream?
What is the best way to get the largest photo jpg off of the iPhone aside from syncing?

John,
You have several options for re-sizing photos, though (continuing from what V.K. has already stated). If you want to "customize" the size of each photo, you'll need to do so before you attach them to an email. They can be re-sized in iPhoto, or they can be opened in Preview and exported as whatever size you like.
If your images are in iPhoto, select one, then choose File>Export. Use the export dialogue to select the size and compression of the resulting file, save it to someplace like your Desktop, then attach it to an email. The choices, here, are the same as they would be within Mail, but will be applied on an image-by-image basis.
Scott

How do you increase the Streaming buffer size in iTunes 10

When I try to listen to the radio in ITunes it plays for 30 seconds then says rebuffering stream.
I just upgraded to an iMac and my old dinasaur G4 does not have this problem.
I've read things saying to increase the Streaming buffer size but I don't see how to do this anywhere.
J

I don't think that is possible with newer iTunes versions.

"Streaming Buffer Size" pop-up menu does not exist under Advanced when I use iTunes - Preferences - Advanced. How can I increase buffer size?

How can I change buffer size in iTunes 10? "Streaming Buffer Size" pop-up menu does not exist under Preferences--Advanced.

Hi all, iTunes streaming victims.
I struggled long with the streaming buffer problem of iTunes and indeed could solve it with increasing the buffer size, which features has been removed in iTunes 10. And this without updating the instructions by Apple!
Here is my solutions:
http://www.rogueamoeba.com/airfoil/
perfect for all wireless audio streaming not only for itunes, but any source on the Mac.
solved the problem completely for my 3 MacBooks, 2 airports, iphones etc.
it is not for free but a very good value for money.
cheers, Wil

Will increasing the buffer size make flash streaming video stop stalling?

I have surmised all this on my own so it could be just a fantasy, but:
I am often the victim of slow internet connections. This means that when I try and watch streaming video it constantly pauses. It's my understanding that increasing the buffer size of flash would help prevent this but I can't figure out how to do that.
Thanks

You can pause the video just long enough for the player to load enough content. Most Flash player show a loading bar if you hover the player. For short pieces you can load the entire content before starting the play.

Increase stream buffer size

Hello everyone,
my internet radio streams often stop or have to buffer although I have a bandwidth > 14.000 kBit/s. I read on forums that I should increase the stream buffer through the iTunes preferences. But I learned that the newer versions of iTunes don't have these preferences. Is there an alternative way? Maybe via terminal?
Thank you in advance for your help!
Greetings
Danny

Hi all, iTunes streaming victims.
I struggled long with the streaming buffer problem of iTunes and indeed could solve it with increasing the buffer size, which features has been removed in iTunes 10. And this without updating the instructions by Apple!
Here is my solutions:
http://www.rogueamoeba.com/airfoil/
perfect for all wireless audio streaming not only for itunes, but any source on the Mac.
solved the problem completely for my 3 MacBooks, 2 airports, iphones etc.
it is not for free but a very good value for money.
cheers, Wil

Effect of Changing Stream Buffer Size

What is the downside (if any) of changing the Stream Buffer Size from Medium to Large when playing music files?

Streaming Buffer Size
Change how much audio or video is downloaded before it starts to play. Choosing a larger buffer can help video play more smoothly. If you have a fast Internet connection, a smaller buffer can help the movie start playing more quickly.

Network Stream Max Buffer Size

Similar Messages

Maybe you are looking for