Design pattern / data loading solution

Hello all!
I have been working on a few projects which involve loading data, sometimes remotely, sometimes local, sometimes JSON, sometimes XML. The problem I am having is that due to the speed of development and the changing minds of various clients I am finding my designs are too rigid and I would like them to be more flexable. I have been trying to think of a reusable solution to data loading, and would like some advice as I imagine many of you out there have had the same problem.
What I would like to do is to create a generic LoadingOperation abstract class, which has member variables of type Parser and Loader, which have parse() and loadData() methods respectivly. The Parser and Loader classed are interfaces and classes that implement these could be XMLParser and JSONParser, LocalLoader and RemoteLoader etc. With something like this i would like to have a new class which extends LoadingOperation for each thing to be loaded, weather thats a local XML file, or remote JSON, or whatever.
The problem is is that specific Parser implementation cannot return custom data types without breaking the polymorphic behavior of the LoadingOperation class. I have been messing around with generics and declaring subclasses of LoadingOperation like
class SpecificLoader extends LoadingOperation<CustomDataType>
and doing similar things with Parser classes, but this seems a bit weird.
Does anyone have any suggestions on what im doing wrong / could be doing better. I want to be able to react quickly to changing specifications (ignoring the fact that they shouldnt be changing that much!) and have a logical seperation of code etc...
thanks for any help!
psi have also asked this question here [http://stackoverflow.com/questions/4329087/design-pattern-data-loading-solution]

rackham wrote:
Hello all!
I have been working on a few projects which involve loading data, sometimes remotely, sometimes local, sometimes JSON, sometimes XML. The problem I am having is that due to the speed of development and the changing minds of various clients I am finding my designs are too rigid and I would like them to be more flexable. I have been trying to think of a reusable solution to data loading, and would like some advice as I imagine many of you out there have had the same problem.
What I would like to do is to create a generic LoadingOperation abstract class, which has member variables of type Parser and Loader, which have parse() and loadData() methods respectivly. The Parser and Loader classed are interfaces and classes that implement these could be XMLParser and JSONParser, LocalLoader and RemoteLoader etc. With something like this i would like to have a new class which extends LoadingOperation for each thing to be loaded, weather thats a local XML file, or remote JSON, or whatever.
The problem is is that specific Parser implementation cannot return custom data types without breaking the polymorphic behavior of the LoadingOperation class. I have been messing around with generics and declaring subclasses of LoadingOperation like
class SpecificLoader extends LoadingOperation<CustomDataType>
and doing similar things with Parser classes, but this seems a bit weird.
Does anyone have any suggestions on what im doing wrong / could be doing better. I want to be able to react quickly to changing specifications (ignoring the fact that they shouldnt be changing that much!) and have a logical seperation of code etc...That depends on the specifics.
The fact that it seems like processes are similar doesn't mean that they are in fact the same. My code editor and Word both seem to be basically the same but I am rather sure that generalizing between the two would be a big mistake.
And I speak from experience (parsing customer data and attempting to generalize the process.)
The problem with attempting to generalize is if you generalize functionality that is not in fact the same. And then you end up with conditional logic all over the place to deal with differences dependent on the users. Rather than saving time that actually costs time because the code becomes more fragile.
Doesn't mean it isn't possible but just rather that you should insure that it is in fact common behavior before implementing anything.

Similar Messages

  • Proxy design pattern

    Hey,
    I have jar that run in different enviroments (client,server,partners).
    In my jar i have some method that in case they called the action that executed is depend on the implementor.
    For example let say that i have method "updateXXX(long id,String name){
    Notify.notify(...)
    if this called in the client than it need to lookup to this service and send the class name, method name and the method parameters to the service.
    if it run in the server than it need to run XXX with the same parameters (this infrastrcture need to server some method with different parameters).
    Is the proxy design patterns is the solution for this?
    I thought to demand from each imlementor that want to use those kind of methods to implement a plugin with a given class name and in notify to load this class with class.forName, what are you say about this soultion?
    any other ideas will be more than welcome.

    The proxy pattern could help here, but isn't the simplest solution. You just want to define an interface that has those updateXXX methods declared on it, and have several classes implement those methods, such as one which executes some code in the same process, or one that does so over RMI, that sort of thing. Write your calling code against that interface, and configure at runtime which class they'll be calling
    interface Updater {
        void updateXXX(Object params);
    class LocalUpdater implements Updater {
         public void updateXXX(Object params) {
           // code to do work here
    class RemoteUpdater implements Updater {
    public void updateXXX(Object params) {
           // code to do remote work here
    class Client {
       Updater myUpdater;
       Client(Updater updater) { // constructor takes an instance of something which implements Updater
           myUpdater = updater;
        void doSomething() {
           myUpdater.updateXXX("params"); // see how this class is decoupled from your Updater implementations
    }How you inject your concrete dependencies is up to you.

  • What is a design pattern?

    Howdy,
    I hear a lot of talk of deasign patterns in ABAP...
    My question is what on earth is a design pattern and why/how would it be useful?
    Any ideas anyone

    Hi Steve.
    The good point to start with design patterns is to read book "Design Patterns - Elements of Reusable Object-Oriented Software" by Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides. It gives strong, consistent understanding of patterns basis with real life examples.
    Short introduction extract:
    "Christopher Alexander says, "Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice". Even though Alexander was talking about patterns in buildings and towns, what he says is true about object-oriented design patterns. Our solutions are expressed in terms of objects and interfaces instead of walls and doors, but at the core of both kinds of patterns is a solution to a problem in a context.
    In general, a pattern has four essential elements:
    The pattern name is a handle we can use to describe a design problem, its solutions, and consequences in a word or two. Naming a pattern immediately increases our design vocabulary. It lets us design at a higher level of abstraction. Having a vocabulary for patterns lets us talk about them with our colleagues, in our documentation, and even to ourselves. It makes it easier to think about designs and to communicate them and their trade-offs to others. Finding good names has been one of the hardest parts of developing our catalog.
    The problem describes when to apply the pattern. It explains the problem and its context. It might describe specific design problems such as how to represent algorithms as objects. It might describe class or object structures that are symptomatic of an inflexible design. Sometimes the problem will include a list of conditions that must be met before it makes sense to apply the pattern.
    The solution describes the elements that make up the design, their relationships, responsibilities, and collaborations. The solution doesn't describe a particular concrete design or implementation, because a pattern is like a template that can be applied in many different situations. Instead, the pattern provides an abstract description of a design problem and how a general arrangement of elements (classes and objects in our case) solves it.
    The consequences are the results and trade-offs of applying the pattern. Though consequences are often unvoiced when we describe design decisions, they are critical for evaluating design alternatives and for understanding the costs and benefits of applying the pattern. The consequences for software often concern space and time trade-offs. They may address language and implementation issues as well. Since reuse is often a factor in object-oriented design, the consequences of a pattern include its impact on a system's flexibility, extensibility, or portability. Listing these consequences explicitly helps you understand and evaluate them."
    Regards,
    Maxim.

  • Choice of design pattern for data acquisition system

    Hello all
    I have a trouble about selecting the suitable design pattern / architecture for a data acquisition system. 
    Here is the details of the desired system:
    There is data acquisition hardware and I need to use it by observing parameters on User interface. 
    the data acquisiton period, channel list to scan should be chosen on User interface. Besides, there are many user interface interactions. e.g. if user selects a channel to add scanlist, then I need to enable and make visible some other parts on user interface. 
    When user completes the channel selection, then he will press the button to start data acquisition. Then I also need to show the scanned values on a graph in real time and log them in txt file.
    I know that I cannot use producer consumer pattern here. because the data acquisition loop should wait for parameters to scan channels. and it works in a given period by user. so the user interface loop performs higher rate then consumer loop (data acquisition loop). it means queue will be bigger bigger. if I use notifier it will loss some data come from user interface. 
    is there any idea about that ? is there any suitable design pattern for this case ? 
    Thanks in advance
    best regards 
    Veli BAYAR
    Embedded Systems Software and Hardware Engineer 
    "You live in a graphical world. Why not program in one?"
    Solved!
    Go to Solution.

    johnsold wrote:
    Veli,
    I recommend the Producer/Consumer model with some modifications.
    You might need three loops.  I cannot tell for sure from your brief description.
    The User Interface loop responds to the user inputs for configuration and start/stop of acquisition.  The parameters and commands are passed to the Data Acquisition loop via a queue. In this loop is a state machine which has Idle, Configuration, Acquisition, and Shutdown states (and perhaps others). The data is sent to the Processing loop via a different queue. The Processing loop performs any data processing, displays the data to the user, and saves it to file. A notifier can be used to send the Stop or shutdown command from the User Interface loop to the other loops.  If the amount of processing is minimal and the file write times are not too long, the Processing loop functions might be able to occur in the Timeout case of the UI loop Event structure.  This simplifies things somewhat but is not as flexible when changes need to be made.
    I am not sure that a Design Pattern for this exact setup exists but it is basically a combination of the Producer/Consumer (Events) and Producer/Consumer (Data) Design Patterns.
    Lynn
    Check out this thread: http://forums.ni.com/t5/LabVIEW/Multiple-poll-case-structures-to-event-help/td-p/2551309
    There are discussions there about a 3-loop architecture that may help you.
    Jeff
    Jeffrey Zola

  • How to design data load process chain?

    Hello,
    I am designing data load process chains for the first time and would like to get some general information on best practicies in that area.
    My situation is as follows:
    I have 3 source systems (R3 and two for which I use flat files).
    How do you suggest, should I define one big chain for all my loading process (I have about 20 InfoSources) or define a few shorter e.g.
    1. Master data R3
    2. Master data flat file system 1
    3. Master data flat file system 2
    4. Transaction data R3
    5. Transaction data file sys 1
    ... and execute one after another succesful end?
    Could you also suggest me any links or manuals on that topic?
    Thank you
    Andrzej

    Andrzej,
    My advise is to make separate chains for master & transaction data (always load in this order!) and afterwards make a 'master chain' where you insert these 2 chains one after the other (so: Start process -> Master data chain -> Transaction data chain).
    Regarding the separate chains; paralellize as much as possible (if functionally allowed). Normally, the number of parallel ('vertical') chains equals the nr of CPU's available (check with basis-person).
    Hope this provides you with enough info to start off with!
    Regards,
    Marco

  • Design patterns for Dynamic Class Loading

    Hi,
    I have to develop a program for uni that dynamically loads classes based on names in a text file. All the classes subclass an abstract class with a constructor that takes one argument. This means I can't use the Class.forName(className).newInstance() method of class loading. I have researched some design patterns I could use and have come up with the following 3:
    Factory pattern; "Robocode" pattern (not it's real name, but Robocode uses it); and, the "one I made up myself" pattern.
    The robocode pattern instantiates a class using the default no-argument constructor then immediately sets all properties that shoud have been provided in the constructor:
    Object o = Class.forName(myClass).newInstance();
    o.setProperty(property);Personally I think this is ugly and a cheap fix instead of doing it properly.
    My own pattern finds the constructor that takes the arguments I need then calls it to create the object:
    Class c = Class.forName(myClass);
    Constructor cons = c.getConstructor(new Class[]{Class.forName("java.lang.String")});
    Object o = cons.newInstance(new Object[]{"hello"});What's the best to use? Are there any other patterns I should consider?

    My own pattern finds the constructor that takes the
    arguments I need then calls it to create the object:
    Class c = Class.forName(myClass);
    Constructor cons = c.getConstructor(new
    Class[]{Class.forName("java.lang.String")});
    Object o = cons.newInstance(new Object[]{"hello"});
    I have followed this basic 'pattern' several times though I would use
    Constructor cons = c.getConstructor(new Class[]{String.class});
    It works great.

  • Design patterns  for data access

    . What design patterns can be used for data access to ensure that only a minimum amount of data is maintained in session

    . What design patterns can be used for data access to
    ensure that only a minimum amount of data is
    maintained in sessionThat's not a design pattern, that's business logic.
    Your application has to determine what "minimum" means.
    Martin Fowler has lots of patterns for data access in his "Patterns of Enterprise Application Architecture". Check them out.

  • Data Lista Handler Design Pattern and paging queries

    I'm using Toplink in an MVC (Regular Beans, JSP and Struts) application. I'm not using ADF. At least not until i make up my mind on the features-portability tradeoff.
    Anyway, my application will have to use queries to show the data to the user. This can easily be accomplished by putting the search parameters on my jsp and using ReadAllQuery to compose my query.
    My problem is that the result can return a large amount of records and show all this records at once on the jsp might take a time the user won't be willing to wait. That's when paging the results will come handy.
    In another application of mine i use paging manipulating the rownum of the query, but in that case, i had control of the whole sql statement. As this is the first time i'm working with Toplink, i'm feeling that can't be done.
    There's the Data List Handler design pattern too, but that one also manipulates the sql statement.
    I'm wondering now whether paging can be accomplished using Toplink..? Will i have to use ADF...? This kind of stuff.
    Thanks a lot to all.
    - Eduardo

    Eduardo,
    What you need to do is use Cursors in TopLink.Scrollable cursors enable you to scroll through a result set from the database without reading the whole result set in a single database read. The ScrollableCursor class implements the Java ListIterator interface to allow for direct and relative access within the stream. Scrollable cursors also enable you to scroll forward and backward through the stream.
    Here is an example:
    ReadAllQuery query = new ReadAllQuery();
    query.setReferenceClass(Employee.class);
    query.useScrollableCursor();
    ScrollableCursor cursor = (ScrollableCursor) session.executeQuery(query);
    while (cursor.hasNext()) {
    System.out.println(cursor.next().toString());
    cursor.close();
    I hope this answers your question.
    Deepak

  • Resources for designing redundancy and load balancing among data centers

    Hello all,
    I'm looking for resources for designing redundancy and load balancing between two physically separate data centers. I'm looking for some "best practice" links, tips, or recommendations. Any suggestions are appreciated!
    Thanks.

    I think that we can do per packet load balancing by using CEF.
    Please go to the following URL:
    http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/122cgcr/fswtch_c/swprt1/xcfcefc.htm#xtocid5
    Also, you may need local director or distributed director. What resource/application is availalbe in the data centre? (e.g. http server, ftp server, TN3270 server, and so on)

  • ETL design pattern with Data Integrator

    Hi all
    I have searched a lot on the SAP website and also on the Google to find a reliable document about ETL design pattern which applies to BO Data Integrator, but it was not successful. Most of the links I found was for  SSIS and Informatica,....
    I would be grateful if you could guide me to a good link about ETL design pattern.
    Thanks.

    I would start with these two:
    https://wiki.sdn.sap.com:443/wiki/display/BOBJ/ETLProjectGuidelines
    https://wiki.sdn.sap.com:443/wiki/pages/viewpage.action?pageId=49414406

  • Data Transfer Objects Design Pattern

    Hi All,
    Could anyone tell me more about this DTO design pattern, as such when one should prefer it.
    Second is this pattern transaction safe or one would have to implement the same manually. Any updates would prove benefecial to me.

    Hi
    I do not understand what you mean.
    A dto is just an object which you populate in one layer and use in another layer.
    you do not need any jndi look for DTO , instead you lookup for ejbs , datasources ,....
    for example you lookup for some CMP , search and find some CMPs , populate DTO with those CMP and send the DTO to front layer.
    if it is not the answer , can you explain more about your requirements(S)

  • Design Pattern / Best Practice Question

    Hi,
    I have been using Flex for a while now, but there is a
    scenario which I still have not found a solution I'm entirely happy
    with. I'm wondering if anyone else out there might have suggestions
    on a design pattern or best practice.
    Suppose I have a view which depends on model data which
    resides in some back end systems. That model data may or may not
    have been loaded (e.g. via a web service or remote object call) at
    the time the view is displayed.
    I don't know if the user will ever visit this part of the
    application so I would prefer to defer retrieval of the data until
    the user actually navigates to this view. Or I want to retrieve the
    data each time the view is displayed because the data is dynamic
    and could change between one presentation of the view and the next.
    Because the data comes from several systems, I cannot simply
    make one service call and display the view when it completes and
    all the data is available. I need to call several services which
    could complete in any order but I only want to display my view
    after I know all of them have completed and all of the model data
    is available. Otherwise, I can present the user an incomplete view
    (e.g. some combo boxes are empty until the corresponding service
    call to get the data completes).
    The solution I like best so far is to dispatch a single event
    (I am using Cairngorm) handled by a single command which acts as
    the caller and responder for all of the services. This command then
    remembers which responses it has received and dispatches another
    event to navigate to the view once all the results have returned.
    If the services being called are used in different
    combinations on different screens, this results in proliferation of
    events and commands. An event and command for each service and
    additional events and commands to bundle the services and the
    handling of their responses in the right combinations for each of
    the views.
    Another approach is to have some helper class listen for all
    of the model changes and only display the view when the model
    enters some state that is acceptable. It is sometimes difficult to
    determine just by looking at the model whether it is in the right
    state (e.g. how can I tell that a collection is the new collection
    that should just have been requested versus an old one lingering
    from a previous call). The logic required can get kind of
    convoluted and brittle.
    Basically, all of the solutions I've come up with so far seem
    less than ideal and a little hackish. I keep thinking there is some
    elegant solution out there that I am just missing ... but so far,
    no luck finding it. Thoughts?
    Thanks.
    Bill

    i think a service class is right - to coordinate your calls.
    i would have 1 event per call (so you could listen to individual
    responses if you wanted to).
    then i would use a flag. if you want to check for staleness,
    you would probably want two objects to map your service flag to
    lastRequested and lastCompleted. when you check, check if it's
    completed, and if it's not stale and that your lastRequested is
    less than lastCompleted (meaning that you're not currently waiting,
    i.e. you've returned since making a request). then make the request
    and update your lastRequested.
    here's a snippet of what i mean.
    ./paul
    public static const SVC1_LOADED:int = 1;
    public static const SVC2_LOADED:int = 2;
    public static const SVC3_LOADED:int = 4;
    public static const SVCALL_LOADED:int = 7;
    private var completedFlag:int = 0;
    then each call would have it's own callback.
    private function onSvc1Complete( evt:Event):void {
    completedFlag |= SVC1_LOADED;
    lastCompleted[ SVC1_LOADED ] = getTimer();
    dispatchEvent( new Event("svc1complete") );
    checkDone();
    private function checkDone():void{
    if( completedFlag == SVCALL_LOADED )
    dispatchEvent(new Event( "allLoaded" ));

  • JDBC Design Patterns

    Hi All,
    I am new to patterns and have started understanding them using the Head First Series.
    I would like to know what all design patterns are there in JDBC?
    Can i say that JDBC uses a Facade design patterns as it hides the database specific details and provides us a interface which helps us to connect to a database. What other design patterns exist as part of the JDBC?
    Request you to clarify my doubts.
    Many thanks in advance

    I would really appreciate a good discussion on it
    rather than any spoon feeding.
    A typical JDBC code will appear as follows:
    try
    /* Load the jdbc-odbc driver
    Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");
    // Open a connection to data source
    con
    =DriverManager.getConnection("jdbc:odbc:DBName","","")
    // Get a statement from the connection
    Statement stmt = conn.createStatement() ;
    // Execute the query
    ResultSet rs = stmt.executeQuery( "select * from
    table_name" ) ;
    1. Is it ok to look into each statement of the above
    code snippet in terms of Design Patterns.it's ok, yes. not necessarily appropriate or worthwhile, though
    2. How can one dissect each line of the code in terms
    of various design patterns?you can't. design patterns exist at a higher level of abstraction than code. 'in terms of' is also disconcertingly vague.
    3. Is it correct to conclude that java.sql package
    overall uses Abstract Factory Pattern.
    As it uses lot of interfaces and implemetation is
    provided by the Vendors. yep. that sounds fair enough. not very useful, though
    4. Consider a particular package like
    java.sql.Statement,java.sql.Blob,java.sql.Connection
    etc. is is correct to conclude that
    its an example of a Factory Method Pattern as any
    client would instantiate. Here the client would be
    Statement object,
    Creator is Connection Interface and createStatement()
    is the factory method.possibly. there's little value in trying to define everything as a "design pattern", though. rather than thinking "oh, they must be using PATTERN X here", think "if I were writing this code, I think PATTERN X would be appropriate". or, more usefully, "is there a pattern that solves this problem?". there isn't, necessarily
    5. I am not able to understand if a statement like
    Class.forName() should be viewed only in terms of a
    programming instruction or
    any design patternit's a line of code, nothing more. stop trying to make everything into a pattern. until you realise where patterns are and aren't applicable, you'll never understand them
    6. Can "stmt.executeQuery()" can be viewed as A
    Strategy Pattern? nope. what makes you think that?
    I would really welcome a good discussion on the above
    questions.does the discussion have to involve design patterns? you know, of course, that design patterns aren't magic beans, right? I know you've just discovered patterns, and are all excited by them, but seriously, they're only ideas, not Infallible Solutions To All Software Problems &#8482;. the most common mistake people make using design patterns is to see them everywhere, and try to bend every problem to fit a particular pattern
    for the record, nobody really views JDBC in terms of patterns. there's little value in viewing existing technologies in those terms, since the most you can "gain" is to have guessed what some other developer did, before.

  • Is there a design pattern for splitting up files into smaller files?

    I am developing a project where I have to load very large files (upto 50 MB). Currently I am loading these files completely into (consecutive) memory. This has the advantage that I can very easily change bytes at certain locations, because I do not know the
    structure of all bytes.
    However, my intention is to also change the structure, e.g. removing/adding 'chunks'. Now I have the idea to remove the 'known' parts out of it, store them in classes with a data chunk only containing those parts and make a sort of reference list to those chunks.
    E.g.:
    Original file:
    Header
    ChunkA 1
    ChunkA 2
    Intermediate
    ChunkB 1
    Footer
    The result will be:
    ChunkA 1 and ChunkA 2 instance. ChunkB 1 instance
    'File' instance and a reference with base offsets + reference to all chunks.
    At the end I have to 'recreate' or write the original file (with changes) back.
    Is this in general a good idea or is there some design pattern helping me in this?

    50MB is not much in the modern era of 6GB+ machines. If you want to optimize memory then consider using a
    memory mapped file.
    But you mentioned making data structure changes. This is generally dangerous as you have to be concerned about things like disaster recovery. What happens if you are in the middle of saving the modified structure when the program dies? You just corrupted
    your file. A better solution is to stream the existing file using BinaryReader to read in the existing file in parts based upon the structure.  Write out the data to a new, temporary file using BinaryWriter. This may be the original data or your modifications
    depending upon need.  Once you've generated the new file replace the old file (with optional backup). So even if something catastrophic happens during saving you don't lose the original file.
    Michael Taylor
    http://blogs.msmvps.com/p3net

  • Data load from variable file names

    I have multiple files that I want to load into a cube, each starting with the same 5 characters but ending differently. EG GM1010104 GM1010204 What's the best option for MAXL script to automate this data load ? Can you use a wildcard name in the script to pick up anything starting with GM101**** ?

    No - you need to specify the file name as it appears properly (I've never tried it but I am pretty sure it wouldn't work). One solution to this problem though is to have a shell script (or DOS commands) auto-generate an ESSCMD/MaxL script based on the files that exist in a directory.Most scripting environments should allow you to loop through a list of files that match some pattern - you can then create a script with the results and execute it.Another option is to build a MaxL script that accepts a parameter (file name) and have a shell script call it as it loops through the file list.Hope that helps.Regards,Jade----------------------------------Jade ColeSenior BI ConsultantClarity [email protected]

Maybe you are looking for