Logging large .ctl to citadel database is slow

Hi, All
I'm using LV 8.5 with DSC.
I'm logging my own.ctl types shared variables to citadel database. This own.ctl includes 52 piece doubles, strings and boolean datatypes. I have 200 piece this tape of shared variables and I record all that shared variable information to citadel database. All operations as writing, reading, achieving, and deleting database are very slow. They also consume lot CPU time. Is there any capacity limits on citadel database ? Which is best structure to record this kind of data to database? I have tested (see attacment) structure but it seems to be very slow because I have to delete and create database trace references after 1st database is full.
BR,
Attachments:
code_structure.jpg ‏51 KB

When you say "...to Log Data only when necessary... " I assume you are using Set Tag Attribute.vi to establish this behavior.
National Instruments recently added a feature (as a hot-fix) to the DSC Engine to be able to ignore timestamps coming from servers to prevent to log values with "back-in-time" timestamps. Citadel is really critical taking values back in time (logs a ) and therefore retrieval of Citadel data with such back-in-time traces could act wired.
You can find more info from:
Why Do I See a Lot of NaN (Not-a-Number) In My Citadel Database When I Use the Set Tag Attribute.vi?
blic.nsf/websearch/B871D05A1A4742FA86256C70006BBE00?OpenDocument>How Do I Avoid Out-of-Synch (a.k.a....
The Hot-Fix can be found:
LabVIEW Datalogging and Supervisory Control Module Version 6.1 for Windows 2000/95/98/ME/NT/XP -- Fi...
I assume you run into such a use case - maybe. It happen to me, too . And I've created a small VI which would analyze traces on back-in-time (NaN - Not a number) values. I assume the missing Data in DIAdem are those Not-a-Numbers aka Break.
If you still encounter some problems after applying the DSCEngine.ini UseServerTimestamps=false, you might contact a National Instruments Support Engineer.
Hope this helps
Roland
Attachments:
BackInTimeAnalyzer.llb ‏622 KB

Similar Messages

How do you change values after logging data to the citadel database (DSC)?

How would one go about changing values that already exist in the citadel database. So say I were logging a numeric value with a timestamp for 1 year and then wanted to change a few values for the month of June because my sensor wasn't working at that time. How could I do this? Is it possible without re-creating the whole database over?
Thanks
Matt

mattyk wrote:
Hey Ben,
Thanks for the quick response. I called NI and asked about this because this would be a big show stopper for me. Their applications engineer said that the best way to change previous data would be to read in the trace that you want to change, modify it and save it as a new trace. I am also looking into SQL querying. I was able to connect to the citadel database by setting up a ODBC data source and using the database connectivity toolkit to query the database. This worked. Now I am trying to come up with a statement in SQL to change previously saved data. I'll Keep you posted.
Thanks
Matt
That reinforces my suspicions.
I have an app running for about 8 years now that keeps a paper trail to enusr ethe cartdiges used in respirators (operated NIOSH National Institute of Occupational Safety and Health par teh CDC) that use DSC to enusre nobody can ever falsify those records. If they can be hacked I my customer and anyone using a respirator with a NIOSH sticker on the side of it, want to know.
Ben
Ben Rayner
I am currently active on.. MainStream Preppers
Rayner's Ridge is under construction

Editing tag values logged in tag history / citadel database

Hi,
I having been logging data from input/output tags and memory tags to my citadel database over the past week and due to some errors in my vi (my fault) some values are showing on my trends that are incorrect. These values are affecting my calculations (e.g. zeros, infinite numbers etc) and I would really like to delete them if possible.
Does anyone know a way of going in to the database/tag history and selecting and deleting values whilst still leaving other data in tact?
Many thanks,
Stuart

You can remove the data from a Citadel database using LabVIEW, Lookout or Measurement and Automation Explorer (MAX). This is a two step process:
First you archive the data that you want to delete. When archiving, choose to delete this data from the original database. The archiving operation will move the data from your database to new database.
The second step is to delete the new archived database. This removes the extra database that was created by the archive operation. Once the data has been archived you can remove the archived data by either deleting the database manually or by using file VIs from LabVIEW.
The document elaborates this process for MAX, LabVIEW and Lookout separately
Measurement and Automation Explorer (MAX) You must have Historical Data Viewer installed. It comes with Lookout 5.0 and LabVIEW DSC 6.1.
To remove the historical data in MAX, follow these steps:
Under the Historical Data folder in MAX, select the database that you want to remove data from.
Right-click the database and select New View»Trace. This creates a trace view with the default name of New View under the database folder.
On the Trace Attributes page, click on Add new traces. Select the traces you want to archive.
Select a starting and stopping time.
Right-click New View and select Archive.
Enter a directory path to store the archived data.
Check the Delete data from source database checkbox to delete the data from the source database.
Click Start.
Once it is complete, click Close.

How to archive citadel database. we have DSC 6.0.2

we have LabView 6i & DSC 6.0.2 . we develop a application using tags. we want to log our data to citadel database but some time the data is lost for some reason (like system hank or restart).so we want to take a backup of our citadel database.how it will possible.

Pilla,
Unfortunately, 6.0.2 does not have the capability. This was one of the new 6.1 features. Check Table 3 on page 7 in this manual. The Archive Database is what you need for programmatic archive. In 6.1 there is also a new utility called Historical Data Viewer, that allows you to do it manually.
http://www.ni.com/pdf/manuals/322955b.pdf
I remember seeing some online document about archiving Citadel database. ... oh, ya, here it is (I searched ni.com for "archiving citadel" and it was the first hit):
http://zone.ni.com/devzone/conceptd.nsf/webmain/2F24997EAD7C53A686256B6E00686D64?opendocument
Have a good weekend.
Dr.Tag

Citadel Database

Hey
I am trying to learn how to use a Citadel database, but haven't found any good pages or books to help me with the basics. I am totaly green so need help to set up the database, log some data and read the data. I have made a little program with a Sinus signal that i want to try it on.
Any help will be appriciated
Johan

jonaskj wrote:
Hey I
am trying to learn how to use a Citadel database, but haven't found any
good pages or books to help me with the basics. I am totaly green so
need help to set up the database, log some data and read the data. I
have made a little program with a Sinus signal that i want to try it on. Any help will be appriciated Johan
Citadel
database itself is not meant to be used in LabVIEW directly. It is part
of older LabVIEW DSC versions and the only way to log values to the
Citadel database is by using the LabVIEW DSC VIs. The ODBC connector
for Citadel only provides query capabilities but no way of writing to
it.
However newer versions (>= 7.0) of LabVIEW DSC do use the Microsoft
SQL server for data logging, a decision I don't fully understand but
that is another story.
Rolf Kalbermatter
Rolf Kalbermatter
CIT Engineering Netherlands
a division of Test & Measurement Solutions

Configuring Database for logging to citadel Database using shared variable engine

Hello All,
I have two systems with me here, one with LabVIEW 8.5 and one with LabVIEW 8.6, I'm using Shared Variables in my code and I am Logging to citadel Database. In a PC with LabVIEW 8.5, I am able to Log Data to citadel Database, but with the same code I am not able to Log Data from the PC with LabVIEW 8.6. Both the PC's have Database Installed and the connection with the Database Exists when I test connection with database through control panel. I would Like to know any configurations (in LabVIEW or in code or in Database or in PC) that have to be done for Logging to Database because the PC with LabVIEW 8.6 was added recently and the code was upgraded to 8.6.

It was due to a dll in LabVIEW, a dll named nitaglv.dll
Now I have a issue with Data Logging from the EXE, once the shared variables are deployed programatically from the EXE the Data Logging stops,
can I get any input on this issue..???

Slow opening archived citadel database on first read

I have archived a citadel database using the archive vi. I then copied this database to another computer. when I try to read data from this archived database, it is taking a long time to return the first set of data (5-10 minutes). New attempts to read data execute much quicker than the first read. Is there a way to improve this opening of the citadel database?? Right now I am using the OPEN DB sub vi that was sent to me to resolve an error accessing citadel problem. The database is not setup as the logging location in the tag engine. This OPEN DB vi is locked so I can't do any kind of probing to see what it is happening. Are you not supposed to work with an archived database??
Attachments:
my_citadel_viewer.llb.zip ‏108 KB

Bump,
I was unable to reproduce your problem. I archived a DB on my machine and moved it to another Win2k machine. The program works fine, whether or not I use the open and close DB VIs. How much data are you trying to retrieve? You also may have a corrupted database, and Citadel is trying to fix its corruptions when you first access the data. Have you tried to access the original (pre-archive) database with the VI? What is the result?
Regards,
Michael Shasteen
Applications Engineering
National Instruments
www.ni.com/ask
1-866-ASK-MY-NI

How to add an array of data in Citadel database

I have an array of data which I want to directly put it in the Citadel Database. Normally the write tag VI writes one value at a time. How can this be done ?

It depends on what you really want. Could you be a little more specific? Does each point have its own separate timestamp? If so, you could simply put the "Write Tag.VI" inside a FOR LOOP and insert values one after another. This would give a separate timestamp to each value. I have attached a simple VI that shows how to do this.
Or are you looking to record some type of vector, where you log a complete array of data at each interval? I don't think this is possible (I'm using DSC 6.02) unless the data-type is a bit-array, in which case you would simply use "Write Data (bit array).vi".
Another option, depending on what you are aiming for, would be to create your own "VI-based Device Server" and use the VIs on the palette submenu call
ed "DSC Server Development". With your own Device Driver you can write multiple datapoints to the input queue simultaneuosly. In this way, you can write more than one point at a time... but that doesn't mean that each time interval will contain an array of data. Assuming that the data is analog, the data will still be logged to Citadel "one after another" so that when you look at it with the HTV, you will see one curve of analog values.
Attachments:
Write_array_to_citadel.vi ‏17 KB

Write to Citadel database from LabVIEW?

Hi. I'm a newbie when it comes to Lookout and the Citadel database... so here's my question for you all:
We have various labview applications that log data from instruments. We would like to be able to take this data and store it as "tags" in a Citadel database.
What do you think is the best way to do this?
There are ODBC drivers for the Citadel database, but they appear to only allow you to query the database to retrieve values... you can't write data to the database (so its read-only).
Are they any "Lookout VIs" or Citadel VIs that can write to the Citadel DB from LabVIEW? Or a DLL or something?
Any ideas?
regards,
John Paul

Very nice question John Paul. I take it you already have Lookout and LabVIEW. In this case, with the latest versions of either, simply use the LabVIEW datasocket to publish the items on your front panel. Lookout then can subscribe to these once you register the LabVIEW computer. Once you have subscribed to them in Lookout, you are then able to set up logging. You may need to create an expression object to route the data to citadel.
If you just want to use LabVIEW and the Citadel database, you might try VI Logger. This works in our configuration utility - MAX, and also comes with a set of VIs for LabVIEW. One caveot, the Citadel database in VI logger is newer (and faster) than the Citadel database in Lookout. I believe NI is working to synchronize
the version in the future, and its hard to say when. So till then, you would not be able to merge data in the dislike citadel databases. However, you can always use and ODBC call set to extract from both and store in a more permanent SQL database such as access, oracle, ms back office, etc.
Good Luck,
Preston Johnson
Business Development Manager
Industrial Applications
National Instruments
Preston Johnson
Principal Sales Engineer
Condition Monitoring Systems
Vibration Analyst III - www.vibinst.org, www.mobiusinstitute.com
National Instruments
[email protected]
www.ni.com/mcm
www.ni.com/soundandvibration
www.ni.com/biganalogdata
512-683-5444

CITADEL DATABASE NOT CONFIGURED AS A RELATIONAL DATABASE

HI:
I have the same problem described by another member before, and I haven't found any resolution of this problem:
The problem was:
>I enabled database logging, and configured the shared variables that I wanted to log. Then I deployed the variables. When I try to read alarms and events (Alarm & Event Query.vi), I get this message:
>Error -1967386611 occurred at HIST_RunAlarmQueryCORE.vi,
Citadel: (Hex 0x8ABC100D) The given Citadel database is not currently
configured to log alarms to a relational database.
Please help me with this topic
Thanks in advance

I'm configuring the variables to be logged in the same way that appears on the file you send, but it doesn't work... I don't know what else to do.
I'm sending you the configuration image file, the error message image and a simple vi that creates the database; after, values are logged; I generate several values for the variable, values that are above the HI limit of the acceptance value (previously configured) so alarms are generated. When I push the button STOP, the system stops logging values to the database and performs a query to the alarms database, and the corresponding error is generated... (file attached)
The result: With the aid of MAXThe data is logged correctly on the DATA database (I can view the trace), but the alarm generated is not logged on the alarms database created programatically...
The same vi is used but creating another database manually with the aid of MAX and configuring the library to log the alarms to that database.... same result
I try this sabe conditions on three different PCs with the same result, and I try to reinstall LabVIEW (development and DSC) completelly (uff!) and still doesn't work... ¿what else can I do?
I'd appreciate very much your help.
Ignacio
Attachments:
error.jpg ‏56 KB
test_db.vi ‏38 KB
config.jpg ‏150 KB

Latency in accessing Citadel Database over the LAN

Hi!
I have two systems running LabVIEW 2013 with DSC 2013 and I am using one of the systems to acquire (using shared variables) and log data internally in the citadel database. I have made an application to visualize this data either locally on the same PC or over the LAN. When I am using the application on the same local machine which is logging the data, its all working fine and very fast. But when I access the same database over the LAN, the application takes a finite amount of time (anything between 2-18 secs everytime) to retrieve and show the data. Since the database size is very small (less than 5MB), logging rate is low (1Hz), number of traces are not much (14 traces), data is boolean, it should not take so much time to show the values over the LAN (the application should be as snappy as it is on the local PC). I am at my wits' end but I am not able to reduce this latency or find any logical reason for it to be present.
I specify the logging PC's name whenever I want to retrieve the trace (as shown in the attached picture), I have a feeling that if I replace that with the IP address of the system, things should become fast. But whenever I try this, LabVIEW throws up an error saying that the specified trace/database does not exist.
I am attaching a pic of the architecture and a simplified image of how I am retrieving the data from the database with this post.
Any help shall be deeply appreciated.
Attachments:
LAN Architecture.png ‏23 KB
Accessing database.png ‏14 KB

To enable external access without VPN, you must assign an external IP address and also map that external IP to the server. Additionally, you must open the associated ports (http) on the firewall (doesnt matter what firewall you use). You will have to map the extranal ip to the internal IP that you have set on the EBS server configuration. In some cases, you have to enable the proxy as well.

Citadel database missing trace

Once again we have a problem.
Client is running 6.1 (Build 27). Missing data from a few objects for 2 and a half months! Suddenly reappears. No machine down tme, no errors, just gone. Tried to rebuild database, no luck.
This is becoming a problem, as much as it would pain me to move to a new software platform we may have to at this point. Constant database corruptions cannot happen when the clients need these reports for internal and other health type agencies.
Citadel is a platform that simply is not reliable. 6.2 show the same errors as well.
Example problem. Tanks record current, high and low values that are trended. The low and high disappear but current stays. And all of the points for the pump (control, monitor and status) are gone ont he same dates. Then suddenly reappear later with no system restarts at all.
EDIT: We know the system was working, we log onto spreadsheets as well, the data is correct in there. But the report system is automated in lookout, not an excel type program.
Thanks, again,
Mike
Message Edited by Mike@DTSI on 06-19-2009 01:27 PM
Mike Crabtree - Lead Developer
Destek of Nevada, Inc. / Digital Telemetry Systems, Inc.
(866) 964-6948 / (760) 247-9512

Got some screenshots
1 - This is one week view from today (07/02-07/08). The green line (the logical connection for the Control Bit (\\.\ServerWell4\BC)) shows data as being in the database for the hypertrend.(The gold and dark red lines are other wells, which have the same issues.)
2. This is the SQL pull using the datatable object. Pulls from the Citadel database. Looking at the days 07/01-07/08 it shows 0 for everything, but as you see in the last pic, hypertrend sees the data. The SQL string:
SELECT MATH_ETM(TO_DISCRETE("TCWDSCADA/Server/Well4/BC")), MATH_starts(TO_DISCRETE("TCWDSCADA/Server/Well4/BC")), MATH_stops(TO_DISCRETE("TCWDSCADA/Server/Well4/BC")), MATH_ETM(TO_DISCRETE("TCWDSCADA/Server/Well4/BM")), MATH_starts(TO_DISCRETE("TCWDSCADA/Server/Well4/BM")), MATH_stops(TO_DISCRETE("TCWDSCADA/Server/Well4/BM")), MATH_etm(TO_DISCRETE("TCWDSCADA/Server/Well4/BS")), MATH_starts(TO_DISCRETE("TCWDSCADA/Server/Well4/BS")), MATH_stops(TO_DISCRETE("TCWDSCADA/Server/Well4/BS")) FROM IntData WHERE LocalTime BETWEEN '2009-7-2' AND '2009-8-2' and IntInterval ='1'
BC=Control, BM=Monitor, BS=Status. The reason we pull the LocalTime from 2 days into month and after the month is because the citadel database returns values from the days previous. Its wierd stuff.
July shows all 0. April has 2 days (04/01-04/02)
Well 4 in this example runs everyday. weekends, weekdays, everyday.
DBFolderT and SysfolderT have no remote connections except for the remote property is linked to a datatable field. the datatable is only loaded on system start, or manually which doesnt happen for this system.
Thanks again
Mike
Mike Crabtree - Lead Developer
Destek of Nevada, Inc. / Digital Telemetry Systems, Inc.
(866) 964-6948 / (760) 247-9512

Database very slow/hangs.

Hello,
My company server has a database of size 400GB. It has has been distributed on 2 HDD of 300GB each. Previously i had faced the problem of database performing very slow . That time the size of the database was much shorter & i just moved the REDOLOG FILE to other HDD using rename command. That worked & again the database was working great. But now the scenario is quite different than previous... as given below ...
Tablespace Name - LHSERP
No. Of. Datafiles in LHSERP Tablespace - 55 ( Which are distributed on 2 HDD )
Hard Disk Drive (HDD) in server - 2
Capacity of HDD - 300GB each.now both HDD has datafiles from LHSERP tablespace. I tried to move the redo log files to other HDD but no use. No improvement in the performance. Still the database is slow.
One more thing i need to make a point of , when i see the task manager on the server it shows some RED and GREEN color graph of CPU usage. Does it mean anything serious ???? Even the whole OS works quite slow on the server. Right from opening MY COMPUTER , to loging into the user ....to fire the query in the user ...everything is very slow. Should i try to short out this problem in some different direction ???
Can you suggest me what to do next ... to improve database performance. If you have anymore ideas please let me know.
ORACLE DATABASE 10g
Windows Server 2008 64-Bit
Thanks in advance ....

Hi,
What is your AWR snapshot keep time? I normally set it to 30 days so that in case of some problem, i can compare my AWR with my history AWRs. Now, can you take out an AWR when your database was doing good and then the latest AWR and then compare it and see what is the difference? Is redo log generation has increased? What are the top 5 wait events in the good AWR report and now in current bad AWR report. What are the top SQLs (elapsed time, CPU time) in good and bad AWR. What were the top segments in bad and good AWRs.
Doing this will give you insight of the problematic area.
Can you check ADDM report, is oracle recommending you anything t o look into?
what is the CPU usage. Even Oracle is the top consumer, can you check the CPU usage for past 24 hours, is it touching 100 %?
You should have OEM configured with the database. Form OEM, can you check host hard disks performance and check the busy percentage of your hard disks for past one month and check if there was any increase in the hard disk busy rate.
Doing all above will certainly help you identify the problematic area.
Salman

Lookout 6.2 loading buffered data from PLC into Citadel database?

We are testing a Lookout small process application <200 I/O that is designed for continuous application. This is not a critical process, therefore we don't want to commit to duplicate Lookout Systems. However, we would like to buffer 24 words(16bit) in the PLC for 5-10 minutes so we would not loose that data when the computer running Lookout does a restart for example.
Our problem is how do we get this data into LookOut Citadel database in the proper time frame.
If loading this data into Citadel is not possible with the original trace, can we load into another database file and view this data separately?
Any suggestions would be appreciated!
Solved!
Go to Solution.

First, you need to be able to get the data as well as the timestamp from the PLC.
With all the data and timestamp, you can use the Logger object to log data to Citadel. Take a look at this example. The Logger_discrete object.
http://zone.ni.com/devzone/cda/epd/p/id/3816
But the problem is that the data is not logged to the original trace. It is logged to the logger object trace.
Ryan Shi
National Instruments

Is there a way to test for integrity of the Citadel Database?

Gday,
Currently I am logging OPC data to a Citadel 5 database (Labview Version 8.2.1).
I have done some integrity testing to try and "break" the citadel database and I noticed that if you delete any of the .cdpg files no alarms are flagged or notifications given in MAX or elsewhere to show that the database has been comprimised.
The result of deleting or the corruption of any of these .cdpg files will result in valid data being 'moved' back in time as the citadel databases only log changes in value with no corressponding timestamp. For example if you delete from 9.00am to 10.00am then all data logged after 10.00am will now start at 9.00am.
So I was wondering if there is anyway to either correct or even test for corruption in a citadel 5 database.
Regards
Ben

This should not happen according to the design of Citadel. Also I tested with current Citadel, and didn't see this problem. I'm not sure if it was a bug in the old Citadel(the one with DSC 8.2), but for current Citadel, I can't reproduce it.
There are many files in database folder. The .cdpg file is the data file, containing all the logged data. Some other files, such as .cdin, .cdih, .cdib, are the index files. When you read data from a trace, Lookout will first find the index from those files, and then locate the position of the data in .cdpg files. Actually Citadel will rebuild the index files if it finds something changed or damaged. If you delete one .cdpg file, Citadel will rebuild the index files. With the new index files, you can work with the database without any problem. The only difference is that the data in the deleted file can't be viewed then, just like the data was nevered logged. So, to delete the .cdpg file won't damage the whole database. The new data will be logged to database with the real time or the user-defined time. Citadel does log the delta value, but it also records the time for each trace. The detailed architecture is too complex to describe.
Can you give some screenshots to show the problem?
Ryan Shi
National Instruments

Logging large .ctl to citadel database is slow

Similar Messages

Maybe you are looking for