Multiple streams and sockets

Currently I have client/server software that sends messages to each other. These messages are delimited strings.
Now I have a need to make one of the delimited fields binary data.
I am using a BufferedReader and BufferedWriter to read the strings. Can I easy just write to the BufferedWriter, and then switch to a DataOutputStream and write the bytes, and convert back to the Writer? This is assuming that I create both with the socket's output stream. How should I handle the writing of strings and binary data in the same message?

Are you sure?
Let's say the message was like so:
"I am sending you 1kb, connect to me at 11.111.12.11 on port 5984"
(or in code):
send("<message><packet>1</packet><measure>kb</measure><ip>11.111.12.11</ip><port>5984</port></message>");On the other end of the 'command' line, it opens up a Data socket to that port/ip, and reads 1kb, and terms.
You could include descriptors in the message, too...
..or not.

Similar Messages

Multible I/O Streams on Socket

tachwohl
How can a Server send multible I/O Stream to a single User at the same time?
Ok, I have the following problem. This days, I'm coding a Client/Server Application. The Client/Server Connection is "should" be static during runtime.
The Server and the application should be possible sends and recieves Objects all the time.
The Client have a Socket, whitch connects to the Server. After connecting, the Server creates a InputStream and OutputStream and the Client has to buid a InputStream and a OutputStream right?
After this, when my Client want to send or recieve something, I have to use the I/O Streams and socket created before, right?
The Problem now, wenn I Stream one Object at the time through the I/O Stream, all works fine.
But when my Server starts beeing Multithread, sending more than one object to the client (per thread started I/O Stream), I can't read the Input on the pipes end.
What's the solution? Always creating a new I/O Stream? Working with SocketChannel (Non Blocking I/O)?
plz need help
thx

I see, I have to do the OutputStreams one after one, right?
Isn't there any way to create multible "outputpipes" over one socket at the same time?

Greetings: I have multiple iPads and iPhones. I want all to be able to be able to stream to our Apple TV. All of the docs I see say you must have the same Apple ID for all the devices, but we each have our own Apple ID. Is this just a doc short coming?

Greetings: I have multiple iPads and iPhones. I want all to be able to be able to stream to our Apple TV. All of the docs I see say you must have the same Apple ID for all the devices, but we each have our own Apple ID. Is this just a doc short coming?

You can each have your own ID for your own iTunes accounts, but in order for a device to stream via AirPlay to the same Apple TV, everything must use the same homesharing ID. This is not the same as your iTunes account ID (although it can be for one of the devices)

How can you use multiple stream types with one socket?

Hi,
I'm working on a large program that currently uses Object Input/Output streams for all of the messaging in a client/server application. This works fine except when files need to be transferred. In many cases using a Object stream to send a file can be 30 times slower than using a Buffered input/output stream. I've found I can combined the two. Here are some code snippets to give you a basic idea of what's happening...
BufferedInputStream bis = new BufferedInputStream( serverSocket.getInputStream( ) );
ObjectInputStream ois = new ObjectInputStream( serverSocket.getInputStream( ) );
//this code runs on a thread on the server
while( true ){
switch( whichKindOfStreamUsedNext ){
case OBJECT_STREAM:
Object object = ois.readObject( );
doSomethingWithObject( object );
break;
case BUFFERED_STREAM:
readFromBuffer( bis );
break;
Obviously there is a lot missing here. Basically the variable whichKindOfStreamUsedNext is changed in the methods doSomethingWithObject( ) and readFromBuffer depending on what the current state of the server is and what is passed to these methods from the client.
Here is the problem. If readFromBuffer( ) does a very small task and the client sends an object through an object stream everything is okay. I've switched whichKindOfStreamUsedNext = OBJECT_STREAM before that point and by the time the client sends the object the server is waiting on Object object = ois.readObject( );. However if the method readFromBuffer( ) does a very time intensive task and it takes a while to return and meanwhile the client sends an object then the server never gets that object. Does anyone have an easy solution to this problem. (Changing the whole program to just using BufferedStreams is not a solution).
Thanks.

Thanks a lot for the response.
I guess I didn't realize I could do that.
I changed how I am doing the program anyways. Sending flags to switch streams was a little messy. but now I have a new problem. I've discovered that mixing object streams with buffered streams also leads to significant speed increases. I do that in this manner...
int ONE_MEG = 1024*1024;
ObjectInputStream ois = new ObjectInputStream( new BufferedInputStream( socket.getInputStream( ), ONE_MEG ) );
and I do the same thing for the ObjectOutputStream. It works very well when I just set up the client's output stream and the servers input stream in this manner. Upload times are increased from 60 seconds to 2-5 seconds.
Unfortunately when I try to do the same thing with the servers output stream and the clients input stream to speed up downloads I get deadlock! As soon as the socket connection is opened and I try to set up the streams in this manner I get deadlock. Does anyone have any idea why this occurs and what I can do to fix it?

Any way to combine multiple incoming streams and send it back to clients?

I am trying to make a server that receives HUNDREDS of incoming audio streams and then combines them into one stream and send it back to all clients- like audio conferencing. Is it possible to combine those streams together into a single one and send it to the receiving ends? I read on some topics in the forum and there really doesnt seem to be any answers.
Also, if its not possible to combine those streams then is the next "best" alternative be having the server send all the audio streams from everyone to everyone? Its going to take a lot of resources doing that i think... any ideas? Thanks!

hi bgl. I read that thread and its a litle bit hard to follow. it seems that you managed to grab Bytes of the incoming audio stream and continuing grabing them in a loop. I guess you use the player to play the Bytes? I also dont really see how i can mix all the Bytes together from lets say 10 users. Can you show me an example of it? Like making an incoming stream into Bytes and playing it. I assume it would have the same effect as playing the stream using a player? thanks

Can I have multiple stream types in one object?

For my final project in my data commucnications class I'm writing a client/server socket application that will allow multiple clients to play TicTacToe simultaneously against the game on the server. The teacher is a C/C++ jock, and knows very little Java. We're free to choose any language that will do sockets, and I love Java and don't love C++.
I've built the GUI, and got it to the point that I can reliably connect multiple clients on different machines in the school lab to the Server object, which accepts the new sockets and creates a new thread of ServerGame to do the playing in response to the client's moves. A mousePressed() detects clicks in the grid, and modifies a string to contain the status of the game. I've used a BufferedReader and PrintWriter combination on both sides, to send the GameString back and forth, starting with "---------", then the client makes a move and it changes to "X--------" and the PrintWriter sends it over and the ServerGame makes a move to "X---O----" and send it back, etc, etc. You get the idea. I have a ways to go with the implementation of strategy stuff for the ServerGame's moves, but I like it so far. But now I realize it would be really cool to add to it.
What I want to do, since there can be multiple players, is have a way that it can be like a TicTacToe club. At any one time, it would be nice to be able to know who else is playing and what their won/loss record is. I plan a Player object, with String name, int wins, losses, ties, gets and sets. With Textfields and a Sign In button I should be able to send the Player name to the Server, which can look up the name in a Vector of Player objects , find the one that matches, and sends the Player object for that name over to the Client, and also provide it to the ServerGame. Each player's won/loss record will be updated as the play continues, and the record will be "stored" in the Server's vector when he "logs off". Updates shouldn't be too hard to manage.
So, with that as the description, here's the question -- most streams don't handle Objects. I see that ObjectInputStream and ObjectOutputStream do. So -- am I headed for trouble in using multiple stream objects in an application? I suppose I could just use the object streams and let them send out a serialized String. In other words, I want to add this to my program, but I don't want to lose too much time finding out for myself if it works one way or the other. I already have working code for the String. Without giving too much away, can anyone give me some general guidance here?

Anyway, here's the present roadblock that's eating into the time I have left. I've spent many happy hours looking for what I'm missing here, and I'm stumped, real-time.
I found it was no problem to just send everything over and back with ObjectInputStream and ObjectOutputStream. From the client I send a String with the state of the game, and can break it down and code for decisions in the server as to a response and send the new String back to the client. I now have a Player class with Strings name and password, ints wins, losses, ties. I have a sign-in in the client GUI and old "members" of the club are welcomed and matched with their Player object stored in a Vector, and new members are welcomed and added to the Vector. My idea is to make the Vector static so the clients can have access to the Vector through their individual threads of the Game in the server. That way I should be able to make it so that any one player can have in his client window a TextArea listing who's playing right now, with their won-loss record, and have it updated, real-time.
The problem is that in my test-program for the concept, I can get a Player object to go back and forth, I can make changes in it and send it back and have it display properly at either end after a change. What I'm aiming at in the end is the ability to pass a copy of the Vector object from the server to the client, for updating the status of everyone else playing, and when there's a win or loss in an individual client, it should be able to tell its own server thread and through that update the Vector for all to access. Sounds OK to me, but what's happening is that the Vector that goes into the pipe at the server is not the same as the Vector that comes out the pipe into the client. I've tried all the tricks I can think of using console System.out.println()'s, and it's really weird.
I build a dummy Vector in the constructor with 4 Players. I can send a message to the server that removes elementAt(0), and tell it to list the contents of the Vector there, and sure enough the first element is gone, and the console shows a printout of the contents of all the remaining Player objects and their members. But when I writeObject() back to the client, the whole Vector arrives at the client end. Even after I've removed all the Player elements one by one, I receive the full original Vector in the client. I put it into a local Vector cast from the readObject() method. I believe that should live only as long as the method. I even tried putting clear() at the end of the method, so there wouldn't be anything lingering the next time I call the method that gets the Vector from the server.
What seems the biggest clue is that now I've set up another method and a button for it, that gets the elementAt(0) from the server Vector, changes the name and sends it back. Again, after the regular call to get the Vector sent over, it shows in the server console that one element has been removed. And one by one the element sent over from (0) is the one that was bumped down to fill the space from removeElementAt(). But in the client, the console shows 4 objects in the Vector, and one by one, starting at (0), the Player whose name was changed fills in right up to the top of the Vector.
So something is persisting, and I just can't find it.
The code is hundres of lines, so I hesitate to send it in here. I just hope this somewhat lengthy description tips off someone who might know where to look.
Thanks a lot
F

Using Threads And Sockets

Hello,
I want to create a program that can send 2 or more files through the socket to the same client at the same time. Is it possible? I am trying to use threads to do this but I am not sure what to do in those threads. Should I send the socket to the thread class and create BufferedReader and DataOutputStream again in the thread?
Also should I create threads in the client-side?
Thanks
Edited by: bcaputcu on May 18, 2010 2:19 AM

bcaputcu wrote:
Hello,
I want to create a program that can send 2 or more files through the socket to the same client at the same time. Is it possible?No. At least not in the way you're thinking.
I am trying to use threads to do this but I am not sure what to do in those threads. Should I send the socket to the thread class and create BufferedReader and DataOutputStream again in the thread?No, because you can't do that. The socket won't create multiple streams for your threads.
Also should I create threads in the client-side?No.
You need to send the files one at a time. While you could basically send the data interleaved, it would still only involve only one thread, one socket and one set of streams. And it would be a silly thing to do.

Stream based socket , server implementation

Please help me with this simple server and socket problem.
SERVER_
package com.supratim;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;
public class SocketServer {
     private static ServerSocket server;
     public static void main(String[] args) throws IOException {
          server = new ServerSocket(9999);
          new SocketServer().go();
     private void go() throws IOException {
     while(true) {
          Socket socket = server.accept();
          DataInputStream dataInputStream = new DataInputStream(socket.getInputStream());
          DataOutputStream dataOutputStream = new DataOutputStream(socket.getOutputStream());
          byte[] buff=new byte[dataInputStream.available()];
          int i;
          while((i=dataInputStream.read())>0) {
               dataInputStream.read(buff, 0, buff.length);
          String inputMessage="INPUT MESSAGE OBTAINED : "+new String(buff);
          byte[]outputBuffer = inputMessage.getBytes();
          dataOutputStream.write(outputBuffer);
          dataOutputStream.flush();
          socket.close();
CLIENT+
package com.supratim;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.Socket;
import java.net.UnknownHostException;
public class SocketClient {
     public static void main(String[] args) throws UnknownHostException, IOException {
          new SocketClient().go();
     private void go() throws UnknownHostException, IOException {
          Socket socket = new Socket("localhost",9999);
          InputStream dataInputStream = socket.getInputStream();
          OutputStream dataOutputStream = socket.getOutputStream();
          String inputMessage="ABCDEFGHIJKLMNOPQRSTWXYZ";
          byte[]outputBuffer = inputMessage.getBytes();
          dataOutputStream.write(outputBuffer);
          dataOutputStream.flush();
          byte[] buff=new byte[dataInputStream.available()];
          int i;
          while((i=dataInputStream.read())>0) {
               dataInputStream.read(buff, 0, buff.length);
          System.out.println("RESPONSE : "+new String(buff));
          socket.close();
}When I try to connect to the SERVER from CLIENT, nothing happens. As soon as I terminate the CLIENT, the following stack trace comes...
       Exception in thread "main" java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at java.net.SocketInputStream.read(SocketInputStream.java:182)
        at java.io.FilterInputStream.read(FilterInputStream.java:66)
        at com.supratim.SocketServer.go(SocketServer.java:26)
        at com.supratim.SocketServer.main(SocketServer.java:14)Please tell me, am I missing something??
I dont want to use PrintWriter,BufferedReader,BufferedWriter...

jverd wrote:
supratim wrote:
jverd wrote:
supratim wrote:
tjacobs01 wrote:
          while((i=dataInputStream.read())>0) {
               dataInputStream.read(buff, 0, buff.length);
          }This is your problem. InputStream.read is going to block progressif it is so...how am I suppose to check the end of the stream....what is the way out??I'm not really sure what problem you're having or what you're asking. However, if your problem is that you want your server to be able to accept new connections while it's servicing an earlier conneciton--that is, service multiple requests at once--then, clearly, you need to use multiple threads. At the very least, one for accepting connections and one for servicing the requests that come on those connections.The problem is when the client is connecting to send a message as a byte stream...the message does not reach there..eventually the connection resets... why is that?? which part of the SocketServer is being blocked...and what is its way out??
Let's say it is used in a single threaded environment..only one client is connecting at a time...Does the client flush()? Does it close() the Socket's OutputStream?
If you don't flush(), data that you have sent may not be received.
If you don't close(), the server won't know the client is done sending, so he'll never get the -1.
(Or, alternative to calling OutputStream.close(), you could call Socket.shutdownOutput(), I think.)
Edited by: jverd on Jul 12, 2011 10:49 AMplease check the code...flushing...closing are all there...

SIGPIPE and socket accept() calls

Hi!
I am developing a client/server application that utilises TCP stream sockets for communication between multiple clients and a single server. The main server thread creates a socket, binds it, listens and waits to accept incoming socket connections. When a connection is established it farms of a thread to handle the client's request. The server is being developed on SunOS 5.8 (sparc).
My problem is that the server receives a SIGPIPE during a blocking call to accept() shortly after a one (and only one) client initiates a connection.
I can't find any suporting documentation as to why SIGPIPE would be raised during an accept() call.
I mean, accept() accepts the socket and returns a file descriptor to the socket. If the client teared down the connection (which the client ISN'T doing) after connecting then I would expect the server to have a valid socket descriptor and EPIPE or SIGPIPEs to occur whenever the server calls socket operations like send() or recv().
Any ideas why the SIGPIPE might be happening in accept()?

Get a new iPhone. it's very difficult to get all the water out of the phone and corrosion will occur inside. Your phone will likely not be reliable even if it recovers.

How do we split our iCloud accounts but keep one iTunes account so we can share purchased content for our multiple iPhones and iPads?

How do we split our iCloud accounts but keep one iTunes account so we can share purchased content for our multiple iPhones and iPads?

You can migrate a copy of the data to a new account, then delete the other person's data from each account. To do this, on the phone that will be changing accounts, if you have any photos in photo stream that you want to keep on the phone, save these to your camera roll by opening the photo stream album in the thumbnail view, tapping Edit, then tap all the photos you want to save, tap Share and tap Save to Camera Roll. If you have any synced notes that you want to keep on the phone, email these to yourself so you can create new notes in the new account.
Once this is done, go to Settings>iCloud, scroll to the bottom and tap Delete Account. (This will only delete the account from this phone, not from iCloud. The phone that will be keeping the account will not be effected by this.) When prompted about what to do with the iCloud data, be sure to select Keep On My iPhone. Next, set up a new iCloud account using a different Apple ID (if you don't have one, tap Get a Free Apple ID at the bottom). Then turn iCloud data syncing for contacts, etc. back to On, and when prompted about merging with iCloud, choose Merge. This will upload the data to the new account. You will create a new icloud email address with you turn Mail to On.
Finally, to un-merge the data you will then have to go to icloud.com on your computer and sign into each iCloud account separately and manually delete the data you don't want (such as deleting your wife's data from your account, and vice versa).

How to set up multiple streams on one PC

Hi there,
can anyone guide me in setting up multiple streams using the FLME and different sources?
thanks,

Hi,
Did you know that you can have several instances of Flash Media Live Encoder running on your computer?
Open FMLE and choose the camera you want to use and set the parameters etc.
double click the FMLE application again and other instance will open. You can then set up a different camera on this instance, this way you can have two cameras streaming at the same time.
the only issue here is that you need a computer powerful enough to handle the multple streams and the bandwidth of multiple streams from the computer to FMS for example.
Is this what you meant?

PhotoStream: Multiple Devices and Performance Issues

Had a bad night last night with PhotoStream. I have an iPhone 4S (AT&T), my wife has an iPhone 4 (VZW), we have an iPad 1 (Wi-Fi), Apple TV 2 (Ethernet), and a 2011 Mac Mini (Wi-Fi)... all in the same house that has a Time Capsule and is connected to AT&T DSL service, 3.0Mbps service (average speed test results are 2.65Mbps down, 315kbps up). Oh yeah, there's a micro-cell in there too but anytime there's trouble that gets shut off first so its irrelevant for this discussion.
So yesterday was Halloween and we both took alot of photos away from the house with the iPhones. Got home, started using the internet, noticed the performance was HORRIBLE. Speed test showed over 1,000 ms ping latency (I still can't believe I actually waited for it to finish the ping test) and nothing was getting through for the download / upload test. I have never (believe it or not) with AT&T DSL seen it get this bad. Its been bad before, don't get me wrong, but this was surprising even to me.
How did I come to the conclusion that PhotoStream was my issue you may ask? Well, its sort of a long story, but bear with me... The other day, we started having trouble with the Apple TV 2, which for the most part is pretty solid. My wife was streaming Netflix movies and it was stopping every 30 seconds. It was looking at this issue that had me first raise suspicions about PhotoStream. Just days before this problem started, I had enabled PhotoStream on the Apple TV 2. SO I did the first most logical thing anyone could do, I took a step back and disabled it. Problem solved. Thought nothing of it, yeah I was a little discouraged that I couldn't use it on the Apple TV 2, but not a big deal, my favorite part of PhotoStream is using it to get the photos into iPhoto without having to plug the iPhones in so I said to myself "I'll just keep putting the photos into Events and Albums until they work out the kinks on the AppleTV 2" thinking perhaps the Apple TV 2 needs to not update PhotoStream while videos are being played. They can fix that at some point, right? Sure they can.
Now, back to this other issue. My bandwidth last night was completely crapped out, and I was determined to prestige on Black Ops one last time so this had to change STAT! I mean, come on! I was at Level 47 and I was ready to GO! My internet connection just wasn't having it. I powered off the router, rebooted the modem, and powered on the rotuer again. Everything came up beautifully. I jumped into some Nuketown 24/7, 4 green bars next to my name and I was happy. After we lost Bravo, which on Nuketown takes about a minute and a half if both teams are fighting over it, I'm lagging like crazy and I have that little red bar that no one likes to see next to my name. Sure enough, speedtest results are all whacked out again.
So I'm sitting there thinking to myself, what could this be? My wife was watching cable TV, my son was asleep... essentially the Xbox 360 was the only thing trying to use the internet... I thought... until I remembered our iPhones and the photos we had just taken and all the devices that would be simultaneously trying to update PhotoStream. Let's list them again, shall we?
iPhone 4S - Upload to Stream and Download from Stream
iPhone 4 - Upload to Stream and Download from Stream
iPad 1 - Download from Stream
Mac Mini - Download from Stream
So how can I prove this crazy theory of mine that the great and wonderful PhotoStream was choking my internet connection? Simple.
iPhone 4S - PhotoStream OFF
iPhone 4 - PhotoStream OFF
iPad 1 - PhotoStream OFF
Mac Mini - PhotoStream OFF
Speed test results immediately after doing this?
2.73Mbps down / .31Mbps up with 26ms ping time (used the Speedtest.NET app on the 4S).
Sadly, this means that for now, I cannot use PhotoStream, however I did manage to Prestige on Black Ops which made me happy, and the iPhone 4S can still be plugged into the Mac Mini to download my wonderful looking photos.
That's right, I didn't throw away the charger cables just yet and if you want to start talking about iTunes Wireless Synching with multiple devices in the same house, I can gladly take up some more of your time. Thats right, its OFF as well!

I just wanted to report that this is still an open and particularly nefarious bug. I've got a fairly idealized Apple household with 2 iPhone 4's, a 'new' iPad, a 13" Macbook Pro, Airport Express and our most recent addition this weekend... an Apple TV.
While I had no particular interest in enabling Photo Stream, I turned it on while messing with the screen saver settings and didn't think anything about it. As I began to play with my new toy I was immediately disappointed with all the long spinners and timeouts in Netflix. Later that day I noticed my computer and iPad were having similar connection issues. My first thought was 'oh, Comcast must be acting up, that's why my new Apple TV isn't working so well'.
Fast forward 3 days to today and our network issues were still occuring and I decided I needed to figure out what was going on. After mucking around in network settings for a couple hours I half-jokingly decided to disconnect the AppleTV on the whim that the issues started at the same time I hooked it up. Sure enough my network issues disappeared. I even went so far as to run continuous pings to various IP's outside my home network and watch the timeouts come & go as i plugged/unplugged my ATV. Now knowing the ATV was the problem I set out to figure out what in particular was causing it. As soon as I signed out of Photo Stream everything was working perfect again. Netflix on the ATV is quick & responsive, no more spinners or timeouts. My other devices aren't plagued with random disconnects, dns failures or slow downloads.
Long story short, something is bugged in AppleTV's Photo Stream support. Mine didn't appear to be downloading any new photos, it had stopped with 255 photos downloaded. Not sure if that was all I had in iCloud or maybe AppleTV can only pull down 255 max? Maybe 255 is the magic number where it hits the bug that causes this behavior? Either way, it appeared to be only affecting the outgoing internet connection, when I had a continous ping going to another machine on my internal LAN it didn't appear to get affected.
Hope this helps someone at Apple track down the bug. While someone technical can probably troubleshoot these issues I could see someone like my mom never putting the 2 + 2 together that her internet problems started around the time she enabled Photo Stream on her AppleTV and instead blaming it on 'the internet acting up again' and of course calling me months later to look into it for her.

Improving multiple streams, Apple TV streaming?

Anyone have a time capsule and experience a lot of stuttering with streaming files to appletv? I replaced my time capsule with my dlink router and this works much better, one stutter compared to 20 with the time capsule in a given time (this is with a wired connection). I like my time capsule too, does anyone know of a fix or how to improve QOL (multiple streams etc)?
Thanks
Mo

Since the Apple TV is not equipped with Gigabit-Ethernet ports you can expect the best streaming performance if you have both the Time Capsule (TC) and Apple TV running in the "802.11n only (5 GHz)" radio mode. Next best would be to have them connected via 100 Mb Ethernet. Anything less than these would challenge anything above SD video.

Mixing byte & character stream over socket

Hi,
I would like to transmit a file name first to my server ( string )
then I transmit the contents of an arbitrary file ( byte )
I tried to chain two different filters to the InputStream instance returned by the socket, but it does not seem to work.
What is the best way to do that ?
Do I have to read my initial file name string by a byte stream and rebuild it ?
Tks

I Use this code
InputStream bin = socket.getInputStream();
OutputStream bout = socket.getOutputStream();
PrintWriter in = new PrintWriter( bout, true);
BufferedReader out = new BufferedReader(
new InputStreamReader(bin) );
on server and on client just change the socket reference,
The programs works fine just the first and second time and after
the second time, the PrintWriter and BufferedReader works
as an InputStream and an OutputStream, Do you know how to indicate
to the Print Writer and OutputStream that are reading text ?
TNX, Alex

Using OWB mappings with Oracle CDC/Streams and LCRs

Hi,
Has anyone worked with Oracle Streams and OWB? We're looking to leverage Streams to update our data warehouse using Streams to apply changes from the transactional/source DB. At some point we seem to remember hearing that OWB could leverage Streams, perhaps even using the Logical Change Records (LCRs) from Streams as input to mappings?
Any thoughts much appreciated.
Thanks,
Jim Carter

Hi Jim,
We've built a fairly complex solution based on streams. We wanted to break up the various components into separate entities so that any network failure or individual component failure wouldn't cause issues for the other components. So, here goes:
1) The OLTP source database is streaming LCR's to our Datawarehouse where we keep an operational copy of production, updated daily from those streams. This allows for various operational reports to be run/rerun in a given day with the end-of-yesterday picture without impacting the performance on the source system.
2) Our apply process on the datamart side actually updates TWO copies of data. It does a default apply to our operational copy of production, and each of those tables have triggers that put a second copy of the data into daily partitioned tables. So, yesterday's partitions has only the data that was actually changed yesterday. After the default apply, we walk the Oracle dependency tree to fill in all of the supporting information so that yesterday's partition includes all the data needed to run our ETL queries for that day.
Example: Suppose yesterday an address for a customer was updated. Streams only knows about the change to the address record, so the automated process would only put that address record into the daily partition. The dependency walk fills in the associated customer, date of birth, etc. data into that partition so that the partition holds all of the related data to that address record for updates without having to query against the complete tables. By the same token, a change to some other customer info will backfill in the adress record for this customer too.
Now, our ETL queries run against views created against these partitoned tables so that they are only looking at the data for that day (the view s_address joins from our control tables to the partitiond address table so that we are only seeing one day's address records). This means that the ETL is running agains the minimal subset of data required to update dimensions and create facts. It also means that, for example, if there is a problem with the ETL we can suspend running ETL while we fix a problem, and the streaming process will just go on filling partitions until we are ready to re-launch ETL and catch up - one day at a time. We also back up the data mart after each load so that, if we discover an error in ETL logic and need to rebuild we can restore the datamart to a given day and then reprocess the daily partitions in order very simply.
We have added control fields in those partitioned tables that show which record was inserted/updated/or deleted in production, and which was added by the dependency walk so, if neccessary, our ETL can determine which data elements were the ones that changed. As we do daily updates to the data mart as our finest grain, this process may update a given record in a given partition multiple times so that the status of this record at the end of the day in that daily partition shows the final version of that record for the day. So, for example, if you add an address record an then update it on the same day the partition for that day will show the final updated version of the record, and the control field will show this to be a new inserted record for the day.
This satisfies our business requirements. Yours may be different.
We have a set of control tables which manage what partition is being loaded from streams, and which have been loaded via ETL to the datamart. The only limitation is that, of course, the ETL load can only go as far as the last partition completely loaded and closed from streams. And we manage the sizing of this staging system by pruning partitions.
Now, this process IS complex, and requires a fair chunk of storage, but it provides us with the local daily static copy of the OLTP system for running operational reports against without impacting production, and a guaranteed minimal subset of the OLTP system for speedy ETL runs.
As for referencing LCRs themselves, we did not go that route due to the dependency issues (one single LTR will almost never include all of the dependant data from which to update a dimension record or build a fact record, so we would have had to constantly link each one with the full data set to get all of that other info).
Anyway - just thought our approach might give you some ideas as you work out your own approach.
Cheers,
Mike

Multiple streams and sockets

Similar Messages

Maybe you are looking for