Local data storage with XML vs SQLLite

Short version:
I have a mobile app that gathers and stores a large amount of data. The current XML solution (all in one file) is having performance issues as the volume of data increases beyond a certain point.
Can I be confident that using SQLLite and a local database will be a better option and improve performance?
Long version:
I have built a mobile app that is used by my client to collect data.
When we started the project the amount of data was relatively small. Given that, and given the past experiences of my team, we chose to use an XML file to store the collected data. When the app starts it loads the whole xml file into a set of data objects in memory for use by the app. The data is not loaded again for as long as the app stays running. We then run a routine whenever the app wants to save to the HD that converts all of the data back into XML.
The data is split into 'projects' and each project naturally has its own xml node. To improve performance when saving, each instance of the Project class stores its loaded node string. Then, when the save routine is called, if the classes data has not been changed it simply returns the original string instead of going through the whole process of re-compiling it.
Of course this does not change the fact that whenever a save is performed, the whole xml structure must be saved back to the file on the HD.
The app has now been used in anger for quite some time and we are starting to get reports of performance issues. The main culprit is during loading which only happens once but can apparently now take an exeptionally long time and causes problems during startup to the point where users think it has crashed.
I am now tasked with trying to improve the data storage for the app. My original reaction was to try breaking down the xml into multiple files which can then we loaded and saved when needed instead of handling everything all at once. I am worried though about the implications of the app trying to handle in excess of 2000 xml files in some cases. It may not be an issue at all but it just stands out to me as something to be concerned about.
So the best other option that I am aware of is to use SQLLite to save all the data into a local database. I have very little personal experience with sql and databases though so while a lot of the documentation and blog posts I see seem to suggest that this is the way to go for mobile apps with a large amount of data, I cannot be quite certain. The big issue is that I cannot afford to sell the idea of such a radical change (and more importantly the time it would take to implement it) to my client without being quite sure that this will be a valid solution to the problem.
So my question to you, Flex community, is would you continue to use XML or is SQLLite the better way to go for large amounts of data? Do you think that I would see an improvement by using SQLLite? Additionally, do you have any tips or experience with a similar situation that might be helpful to me?
Thank you
Jamie

I've been very happy using the internal SQLite database with Flex.  You should definitely get much better performance from it.  Just being able to load your data asynchronously if nothing else you should get a great benefit.  The sheer amount of data in your XML file that is being passed around is probably quite a hit (especially on mobile).  Breaking up the XML shouldn't be too crazy to do (as you're already doing it somewhat, I'd imagine, when you're accessing the information from within your app).  Give switching to SQLite a try, but I'd say that's a great place to start.

Similar Messages

  • Dreamweaver CS4 Cannot get Spry.Data.NestedXMLDataSet with XML to work

    Please help!
    I've created a page in Dreamweaver CS4 with a spry dataset. This works great. I've downloaded Spry.Data.NestedXMLDataSet and installed it but I keep getting an error that nested datasets are not supported.
    Thoughts solutions.
    Thanks,
    M

    I have found the reason that I could not make it work, I did not have the correct JavaScript file installed. 
    Having made the correction, here is a code that does work but most likely does not have the formatting that you want. I will leave that up to you.
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
    <html xmlns="http://www.w3.org/1999/xhtml" xmlns:spry="http://ns.adobe.com/spry">
    <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <title>Untitled Document</title>
    <script src="SpryAssets/xpath.js" type="text/javascript"></script>
    <script src="SpryAssets/SpryData.js" type="text/javascript"></script>
    <script src="SpryAssets/SpryNestedXMLDataSet.js" type="text/javascript"></script>
    <link href="SpryAssets/SpryMasterDetail.css" rel="stylesheet" type="text/css" />
    <script type="text/javascript">
    <!--
    var ds1 = new Spry.Data.XMLDataSet("scoring2.xml", "wcartist/mainartist");
    ds1.setColumnType("picture", "html");
    ds1.setColumnType("biotext", "html");
    ds1.setColumnType("audioplaylist", "html");
    var ds2 = new Spry.Data.NestedXMLDataSet(ds1, "videos/videochild");
    ds2.setColumnType("videothumb", "html");
    ds2.setColumnType("videourl", "html");
    //-->
    </script>
    </head>
    <body>
      <div class="MasterDetail">
        <div spry:region="ds1" class="MasterContainer">
          <div class="MasterColumn" spry:repeat="ds1" spry:choose="">
            <div spry:when="{ds_CurrentRowNumber} == {ds_RowNumber}" spry:setrow="ds1" spry:select="select" spry:hover="hover" spry:selected="">{artistnamebtn}</div>
            <div spry:default="" spry:setrow="ds1" spry:select="select" spry:hover="hover">{artistnamebtn}</div>
          </div>
        </div>
        <div class="DetailContainer">
          <div spry:detailregion="ds1">
            <div class="DetailColumn">{genre}</div>
            <div class="DetailColumn">{picture}</div>
            <div class="DetailColumn">{artistnameblock}</div>
            <div class="DetailColumn">{biotext}</div>
            <div class="DetailColumn">{audiotracklist}</div>
            <div class="DetailColumn">{audioplaylist}</div>
          </div>
          <div spry:region="ds2">
              <p> </p>
              <p>videos</p>
              <div spry:repeat="ds2">     
              <p>{videoname}</p>
              <p>{videothumb}</p>
              <p>{videourl}</p>
              <p> </p>
            </div>
          </div>
        </div>
        <br style="clear:both" />
      </div>
    </body>
    </html>

  • Date Filtering with XML data

    I'm displaying events in a spry :repeat region with my source
    being XML.
    The date is set in the XML like:
    <event>
    <name>EventName</name>
    <desc>Christmas Eve</desc>
    <stopdate>12/25/2007</stopdate>
    </event>
    How would I hide this event after its given <stopdate>
    value?

    <script>
    function getCurrentDate(){
    objToday = new Date();
    year = objToday.getFullYear();
    month = objToday.getMonth() + 1;
    if ( month < 10 )
    month = "0" + month;
    day = objToday.getDate();
    if ( day < 10 )
    day = "0" + day;
    strToday = day + "/" + month + "/" + year;
    return strToday;
    </script>
    And maby use
    <tr spry repeat>
    <td spry:if="{stopdate} >
    getCurrentDate()">{stopdate}</td>
    </tr>
    Something like that would do.
    If not maby set the row to date EXAMPLE:
    var ds1 = new
    Spry.Data.XMLDataSet("myxml.xml","races/standings");
    // Example of setting the type of one column:
    ds1.setColumnType("start", "date");
    I think that is what u want.
    It checks if the {stopdate} greater than the current date..
    if it true, it will show the date. else it wont... ofcourse u can
    change the operator of the spry if ;)

  • Data template with xml over http

    Hi!
    I have a data template what should get data from xml file.
    <dataTemplate name="ccbXML" description="For xml output">
    <properties>
    <property name="debug_mode" value="on"/>
    <property name="include_parameters" value="true"/>
    <property name="include_null_Element" value="true"/>
    <property name="scalable_mode" value="on"/>
    <property name="db_fetch_size" value="1000"/>
    </properties>
    <dataQuery>
    <xml name="DATA_XML">
    <url method="GET" realm="" username="" password="">http://my.site.sise/00000000070000000001.xml</url>
    </xml>
    </dataQuery>
    </dataTemplate>
    The result is:
    <CCBXML>
         <LIST_data_xml></LIST_data_xml>
    </CCBXML>
    When I use:
    <url method="GET" realm="" username="" password="">file:///D:\files\result.xml</url>
    then I get a parsal result from the XML (it is not structuated and it does not show all the xml content, only random tag's).
    In the end it is not possible for me to use the file protocol because the xml is generated dynamically in a http page. Is it even possible to use the http protocol in data template? Or is anything else wrong with my data template that I'm not getting the right result (even with file:///)?
    I work BI Publisher Enterprise Release 10.1.3.4
    Evelyn

    Could u please send the working example for http binding on SOA Suite 11 g.
    Thanks and Regards,
    Anil verma

  • Error message when using Spry Data set with XML

    Hi,
    I have what i see as a big problem, might just be me being
    dumm, but here we go.
    When I try to use the feature of spry XML Data Set and insert
    spry table I get error message when trying it in IE 7.
    There are even a problem when I use fireworks automated slide
    show function, where you can choose to use the Spry/XML gallery
    feature.
    The error message I get everytime (incl the auto gallery in
    fireworks) is:
    The tag: "html" doesn't have an attribute: "xmlns:spry" in
    currently active versions.[XHTML 1.0 transitional]
    The tag: "div" doesn't have an attribute: "spry:region" in
    currently active versions.[XHTML 1.0 transitional]
    The tag: "th" doesn't have an attribute: "spry:sort" in
    currently active versions.[XHTML 1.0 transitional]
    and so on.
    What´s wrong?!

    Hello barvid,
    Ok, let's take each problem step by step:
    1. The first error you see in the browser: " Exception while
    loading ..." normally appears when you try to load files that are
    not permitted by the browser security model. This means the XML
    file you try to load is located on a different server than your
    HTML which is not allowed. The browser is not permitting to load
    any files that are not in the same domain as the current page
    because of the complex security risks. You'll have to either get
    the XML file from its original location and save it in your website
    or use a proxy on your web server that should request that file
    from the external server and send it to the browser as an XML from
    the same domain as the HTML origin.
    1.2 The "Exception while loading ..." error you see is a
    generic error and my previous advise is based on my experience. In
    case this is not your scenario you'll have to open the SpryData.js
    and around line 112 you'll see the following code:
    Spry.Debug.reportError("Exception caught while loading " +
    url + ": " + e);
    Please change this line with the following line of code:
    Spry.Debug.reportError("Exception caught while loading " +
    url + ": " + (e.message?e.message:e));
    to obtain a more verbose error message that may clarify the
    reasons of your errors.
    2. The problems you describe are not browser errors but they
    are w3c validation errors. At this point the XHTML standard allows
    every application that define custom namespace attributes to also
    link a DTD to the page so the page to continue to be valid.
    Unfortunately at this point this feature is not correctly supported
    by the FF and IE so we wrote an articles about how you should
    link
    the Spry DTD to your page so the validator understand the
    custom attributes Spry use. You'll find inside the full description
    of the problems and all the technical details you'll need to know
    to solve this issue.
    Regards,
    Cristian

  • Local Data Storage in Objects

    Hello,
    I have been trying to program more in Objects.  i.e. Data and Functions are bundled together in a Class or in a project library.  I have used FGV a lot in the past.  One of the things I like about a FGV is the ability to locally store data in an uninitialized shift regester.  Only the cases in the FGV can access the locally stored data.
    Is there an equivilant storage method using Project Libraries or Classes?  Is it even a good idea to have locally stored data?
    Thanks

    ASTDan wrote:
    Is there a way to store that object data within the object?
    That's what that Private Class Data control is for.
    There are only two ways to tell somebody thanks: Kudos and Marked Solutions
    Unofficial Forum Rules and Guidelines

  • Can I create member log-in and data storage with my Muse account?

    I'm looking into the Muse product for a possible new client who will need to collect member data and post it onto the site. I want to know if this function can be created in Muse and with my Muse account.

    Hi,
    Please refer to the following link Re: Password/login
    Regards,
    Aish

  • SSL/TLS security certificate data match with XML Payload in SAP PI

    Hi,
    We are working on a solution where we would want to use SSL/TLS or WS Security with client server mutual authentication using client server certificates.
    But, once the sender is authenticated using the certificates, can the XML payload be matched for the correctness with the certificate information? Is this available to PI integration engine at any time? Like Sender A autheticated as A using certificates, must be stopped if his XML payload is saying that he is sender B (which is most unlikely if we trust the senders but did not want to leave a loophole).
    Any ideas here?
    Thanks and Regards,
    Vijay

    Hi Wolfgang,
    Cross-posting is discouraged and against the forum rules, because it is misused and makes a mess of the search due to distributed discussions and answers.
    I will move it to the PI forum and add a watch on it as it is security forum related.
    Unfortunately, the forum software does not have the option to "mirror" threads.
    Cheers,
    Julius
    Edited by: Julius Bussche on Sep 14, 2009 9:50 PM

  • How to import repeated data cells with XML?

    Hello!
    I have some little troubles with the XML import functionality.
    Let's consider a billing table:
    Item
    Price
    Tax percentage
    VAT
    Total price
    Item description
    $80
    20%
    $20
    $100
    Total
    $80
    $20
    $100

    Yes, I just missused Ctrl+S, and I posted my question too early. Then I lost the update of my question
    I'll have to re-type the whole stuff..
    Well, here is a shot version of the question; this is what I get when importing XML:
    Item
    Price
    Tax percentage
    VAT
    Total price
    Item description
    $80
    20%
    $20
    $100
    Item description
    $80
    20%
    $20
    $100
    Item description
    $80
    20%
    $20
    $100
    Total
    $80
    $20
    $100
    The XML structure repeats the cells. But, in InDesign, new lines are not generated accordingly: extra cells are added to the last cell of the line...
    Thanks

  • TREX - Configuring Distributed Slave with Decentralized Data Storage

    I am creating a distributed TREX environment with decentralized data storage with 3 hosts.  The environment is running TREX 7.10 Rev 14 on Windows 2003 x64.  These are the hosts:
    Server 01p: 1st Master NameServer, Master Index Server, Master Queue Server
    Server 02p: 2nd Master NameServer, Slave Index Server
    Server 03p: Slave NameServer, Slave Index Server (GOAL; Not there yet)
    The first and second hosts are properly set up, with the first host creating the index and replicating the snapshot to the slave index server for searching.  The third host is added to the landscape.  When I attempt to change the role of the third host to be a slave for the Master IS and run a check on the landscape, I receive the following errors:
    check...
    wsaphptd03p: file error on 'wsaphptd03p:e:\usr\sap\HPT\TRX00\_test_file_wsaphptd02p_: The system cannot find the file specified'
    wsaphptd02p: file error on 'wsaphptd02p:e:\usr\sap\HPT\TRX00\_test_file_wsaphptd03p_: The system cannot find the file specified'
    slaves: select 'Use Central Storage' for shared slaves on central storage or change base path to non shared location
    The installs were all performed in the same with, with storage on the "E:" drive using a local install on the stand-alone installation as described in the TREX71InstallMultipleHosts and TREX71INstallSingleHosts guides provided.
    Does anybody know what I should try to do to resolve this issue to add the third host to my TREX distributed landscape?  There really weren't any documents that gave more information besides the install documents.
    Thanks for any help.

    A ticket was opened with SAP customer support.  The response to that ticket is below:
    Many thanks for the connection. We found out, that the error message is wrong. It can be ignored, if you press 'Shift' and button 'Deploy' (TREXAdmin tool -> Landscape Configuration).  We will fix this error in the next Revision (Revision 25) for TREX 7.1.

  • AIR desktop app (Windows 7) - Big data set, XML or SQLlite?

    I am developing an AIR desktop application for a Windows 7 touch screen kiosk that needs to access and allow users to search a set of data, which could initially contain anywhere between 17,000 and 100,000 records. Each record could have perhaps 15 to 20 fields, and all the data will be stored locally on the computer, and updated with new data by memory stick, perhaps once per month. The kiosk will not be connected to the internet.
    Can anyone offer any advice as to the viability / limitations of using XML, SQLLite, or some other means of data storage, to access this information from an AIR application?

    You can do a general search for XML vs SQLite and find that a good deal of the answers say to go with SQLite for this sort of data set and one of that size. SQLite will probably be faster from an access standpoint but has a lot more overhead on setting up and getting working. XML in AS3, in my opinion of working with it for many years, is definitely easy to work with, especially compared to how other languages implement methods of reading/writing to XML. Have you done any work with either before? Below is a link to an Adobe article on working with SQLite in an AIR application.
    Adobe Flash Platform * Working with local SQL databases in AIR

  • Fastest way to extract data out of xml with following constraints.

    10.2 on linux
    xml files are being dropped off into a queue. in the queue the documents must be stored as clobs so that control can be given back to the client as soon as possible.
    once in the queue we would like to extract all the data from the xml and place it in relational staging tables. the data is then moved from these tables into production.
    the only thing that can change is what happens between the queue and the staging tables. currently i am just using extract statements to pull the data out of the clob.
    the files are around 20mb and currently take over 20 minutes to process which is way too long.
    i looked at DBMS_XMLSTORE but we cannot alter the xml format.
    i looked at Oracle text but if i understand it correctly, we would have to rebuild the entire index after every new queue item.
    i have very little experience with xml so i want to make sure i know all my options.
    from what i can tell my only option is to take the clob and let xml db parse it into o-r tables. ...but even that seems like a horrible waste.
    is there anything else i can do? any pointers?
    thanks for any help!
    by the way...this forum has been of great help. my only problem is that i don't seem to ask the right questions at the right time.

    Chris
    Most people seem to find that allow XML DB to persist the XML using the object based storage and nested tables and then using insert as select operations is the most effective way to do what you want. There are a number of threads on how to best do this..
    The question to ask is do you really need the relational staging tables. If you read through the forum you'll see that once the XML has been persisted as objects, and the XML objects have been stored using a nested table storage models you can easily create relational views to represent the staging tables.
    This process will work very well if there are no updates to the staging tables. Effectively you will process the XML once, when you insert into the Schema based tables, and then use the relational views as the source for the migration from staging to production.
    If you haven't already done so, reading the following posts will help you with this
    XMLType column based on XML Schema: several questions
    http://forums.oracle.com/forums/thread.jspa?threadID=347820&tstart=0
    problem with sql/xml
    XML Query Performance on Nested Tables
    Basically you'll need an XML Schema that describes your XML and you'll need to set up nested table storage for each of the collections in your XML Schema in order to the required performance when using the views.
    The easiest way will be to use the default table that is creted when registering the XML Schema and the annotation xdb:storeVarrayAsTable="true" and then ensure that you sequence each collection correctly.

  • Problem with data template in XML publisher

    When I create data template in xml publisher and save it, I get next message:
    dbase:
    Failed to save data.
    Error occurred when creating xml data.
    undefined
    NaN
    NaN
    dbase is a database server
    JDBC connection name:asu_fox
    JDBC connection URL: jdbc:oracle:thin:@dbase:1521:ora03
    Text in data template:
    <?xml version="1.0" encoding="UTF-8"?>
    <dataTemplate name="data_report_2" description="data template for reports 2 and 10"
    dataSourceRef="asu_fox" version="1.0">
    <dataQuery>
         <sqlStatement name="Q1">
              <![CDATA[select kol_pl,kol_sv,trud_pl,trud_sv,nom_per_p,kod_polu_p,p_v,kod_tmc,liter,kol_skl,nom_per,shifr_dse,shifr_izd,kod_polu,kod_post,trud,kol_m,tip_rin,kod_op,kod_op0,order1,n_val,shifr_usel_val,sum_trud_val,sum_trud_val_sv from reporter.t_plan_report_2]]>
         </sqlStatement>
    </dataQuery>
    <dataStructure>
         <group name="group_all" source="Q1">
              <element name="kol_pl_l" value="kol_pl"/>
              <element name="kol_sv_l" value="kol_sv"/>
              <element name="trud_pl_l" value="trud_pl"/>
              <element name="trud_sv_l" value="trud_sv"/>
              <element name="nom_per_p_l" value="nom_per_p"/>
              <element name="kod_polu_p_l" value="kod_polu_p"/>
              <element name="p_v_l" value="p_v"/>
              <element name="kod_tmc_l" value="kod_tmc"/>
              <element name="liter_l" value="liter"/>
              <element name="nom_per_l" value="nom_per"/>
              <element name="shifr_dse_l" value="shifr_dse"/>
              <element name="shifr_izd_l" value="shifr_izd"/>
              <element name="kod_polu_l" value="kod_polu"/>
              <element name="kod_post_l" value="kod_post"/>
              <element name="trud_l" value="trud"/>
              <element name="kol_m_l" value="kol_m"/>
              <element name="tip_rin_l" value="tip_rin"/>
              <element name="kod_op_l" value="kod_op"/>
              <element name="kod_op0_l" value="kod_op0"/>
              <element name="order1_l" value="order1"/>
              <element name="n_val_l" value="n_val"/>
              <element name="shifr_usel_val_l" value="shifr_usel_val"/>
              <element name="sum_trud_val_l" value="sum_trud_val"/>
              <element name="sum_trud_val_sv_l" value="sum_trud_val_sv"/>
         </group>
    </dataStructure>
    </dataTemplate>
    I get this error with different data templates.
    Help me, please.
    Vladimir

    Hi Vladimir
    Can you test the data template posted here. http://static7.userland.com/oracle/gems/leslieStuddard/EMPLOYEES.xml
    It needs to connect to the sample scott schema, hopefully you have it installed?
    Let me know if you get the same error.
    thanks
    Tim

  • How to Create a Table in Oracle with XML data type.

    Dear ALL,
    What are the requirements for creating a table with xml datatype in Oracle: The steps would help very much to know the scripting of the table and how to query and either insert/update and remove data from that table.
    Any help, direction, advise would be highly appreciated.
    Thanks.

    Reffer to this Note.243554.1.
    In a nut shell you will need to run catqm.sql

  • Append value with Data Storage Vis

    Dear,
    I would like to use the Data Storage VIs to collect my data but I can't achieve my target.
    I'm acquiring 1000 samples at 1kHz (N samples on demand) and I make the mean value of the samples. At the end of this proces I have a scalar value and the initial time at wich I have acquired the data. With this two I build a waveform and then I use the Write Data Storage Vi (TDMS) with "append" write mode to save my data.
    When I read the Data Storage all the value have lost the time information. They start from 1.00.00,000 01/01/1904 and are equal spaced in time.
    How can I keep the time information?
    Thanks

    Sorry totaly misread what you were doing.
    You need to create ensure that waverform that is saved has the correct values for X0 and dx when you save it. Use the build waveform function to acheive this.
    edit:
    the default value for X0 is timestamp 0 (1904), use get datetimestamp at the very start of the cycle to get the correct value.
    James
    Message Edited by James W on 04-21-2010 01:06 PM

Maybe you are looking for

  • When I am in safari, I try to bookmark a web page and it is not doing it, it worked before I updated to the ios6 version

    When I am in safari, I try to bookmark a web page and it is not doing it, it worked before I updated to the ios6 version, does any one know why ? It pulls up the screen to save it but then it is not showing up on the bookmark drop down menu, it conti

  • Internet not working while on skype call

    Greetings everyone I'm having this..specific problem for the past few days during Skype calls. Whenever I'm in a call with my friends, my intertnet completely stops working. I'm unable to browse it at all, as well as send and receive messages on Skyp

  • What is the purpose of Clearing and Revenue account in Asset Disposal : F-92?

    Hi All, Please clarify my doubt ; In Asset Disposal - Sale to Customer : F-92 with customer Suppose an asset with historical cost $1000 and accumulated depreciation of $100 is being sold to a customer at $1100, the posting entries will be as follows

  • My iPhone doesn't charge

    Good day All Since 3 weeks ago, I found something new with my iphone. First, I had to plug, unplug and plug the lightning cable into the phone to make it charging. Somedays after, I tryed to plug it a thousand time but nothing happens. When battery w

  • Keyboard/Mouse battery icon????

    My TouchSmart 600-1055 uses HP's wireless keyboard and mouse.  The computer runs Windows7.  There is no mouse or keyboard icon in the system tray, so there is no way to see the state of the battery charge on the mouse or keyboard.  Windows7 is suppos